My experience was that the bigger I got, the less attention I got overall, but the attention I did get was from hotter women. Women get intimidated too.
But regardless, if your goal is to get as big as you want regardless of what anyone thinks, that's okay, you can do that. If you want to lift for the girls, that's OK too.
Having said that I don't think the average natural lifter is in any danger of looking too freakishly big, especially if they stay lean.