I remember, a couple of generations ago, the Ravenloft setting in D&D had a social modifier for how alien your character race was (for example, Half Orks); this was, as far as I understand it, less a question of 'hey, that guy looks different' and more 'what the hell is that?'. It worked in a horror setting, especially one where even normal looking people could be monsters and followed the typical Gothic horror trait of mistrusting outsiders (especially outsiders that already looked monstrous). It wasn't specifically about race, but race did factor into it; so, basically, I think it definitely depends on the setting.
On the flip side, I'll just paraphrase a line from the Humanis Policlub contact in Shadowrun: 'Who cares about that tan looking guy standing next to you, when that thing over there has hands the size of my head?!'