Well, I can't speak for everyone, but I'd always thought that this kind of attitude arose from either one or a mixture of the following:
1. A dislike of rules that might restrict one's liberty to engage in practises that one's religion disapproves of or forbids, or that mandates/encourages things that the subject does not want to do.
2. An (often-justified) aversion to religious structures based on bad moral example, historical or current, by members of said structure. This is, of course, often mixed with an unwillingness or inability to distinguish between a religion and its structures/the people who inhabit such structures.