Are you going to mutant pogroms? Mutant pogroms are xpiratez's terror missions, so if you don't at least show up, you'll take a significant infamy hit. Also, knocking government ships out of the sky and raiding them after tends to make your infamy tank as well, since they're the people who pay you protection money, and its not like you absolutely can't do it, just don't do it often.
Slavery and pimpcraft have nothing to do with infamy, AFAIK.
Killing/kidnapping civilians during missions is also bad, and I *think* not helping the lok'narrs also tanks your infamy, since they're your only real ally, lore wise.
Some research options can tank your infamy too, but those are rare and far between.
EDIT: There's also two other things that I know can lower your infamy, those being that one "strange signal" space mission, which gives you an infamy hit if you don't deal with it, and secret bases, which give you a monthly hit if you don't deal with them, me thinks.