I think that proposition 8 has had a huge impact on how Americans view Mormonism. Some of the misinformation that was being spread by Mormons was quite hateful, and it appeared that the Mormon church was trying to get support from other right-wing Christian denominations by being so anti-gay.
As someone who comes from a familly with a non-Mormon Christian background, I know that Mormonism has a zero percent probablility of ever gaining much support from the non-Mormon Christian backgrounds. They believe that if someone worships a false prophet they are really worshipping Satan, and, not only will they go to hell, they are spreading the word of Satan. There is a lot of evidence that Joseph Smith was a false prophet, particularly on the internet, and it just doesn't matter if Mormons otherwise agree with the evangelical Christians.
Proposition 8 also offended the liberals, for obvious reasons.
Turning now to atheists, the other group mentioned. Atheists used to take the attitude that it is mean to burst the bubble of people who believed in religion. However, the Christian right has really offended younger people with its anti-gay stuff, and 911 has also made a lot of atheists feel that religion is anything but harmless. So, atheists are now aggressively trying to point out the absurdities of religions, particularly for those religions that have a strong tendency to argue against civil rights. Mormonism is one such religion.
So, that is my take on the reasons why Mormonism is unpopular among non-mormons these days.