I still speak a reasonable amount of Japanese, and am familiar with the social norms. I think I would be fine, despite being white as a picket fence.

That’s not to say I am dismissing the xenophobia - it’s real, and it’s one of the aspects of Japanese society that troubles me the most, in a general sense.