The problem is that we humans are a visceral species, our gut feelings (whether we admit to it or not) rule our logic and reason, putting that split-second hesitation into our voices before we reply to someone else, makes our eyes narrow slightly. We do that because we're still primates underneath it all. We know that if we have one barrel of apples from Somalia and another one from Tasmania, and we find ten bad apples in one barrel but none in the other, well, we'll stick with the "safer" apples. Even if we later find a rotten apple or more, at the beginning, the ape in us all will pick what it considers the healthier barrel.
The problem with that is that there are a lot of perfectly good apples sitting there in the barrel and currently needing to find acceptance with us, saying "pick me! give me a chance!" and meanwhile the rest of us are slightly averting our eyes, slightly standing closer to the apples in the "good" barrel. Tensions will be elevated in this way, and the best thing we can all do is to fight the reflex, realise that it exists for an old, long-past-useful survival reason.
I don't know about you, but I'm going to work hard to meet everyone's eye, not just a select few people I consider to be "good apples..."