I lived and worked in Japan during the 1990s. I noticed that people were uncomfortable talking to me and I thought maybe they didn't want to talk to me because I am a white American.
Later, I returned to the US and spent a few more years studying Japanese. When I visited Japan again, I found that my perception changed. My Japanese was more fluent now and I did not get the same feeling of rejection when interacting with Japanese people. What I misunderstood as a racial barrier was actually a language barrier.
The Japanese place a lot of emphasis on smooth communication, and if confronted with someone they can't communicate smoothly with, they somehow feel that it is their fault for not maintaining a smooth flow of conversation, even though the problem was with my limited Japanese. So they try to avoid interacting with people like me who did not speak Japanese well.