If white people want to use that word in that (or, basically, any) context, maybe they should start working towards systematically dismantling the apparatus of racism in this country and then check back in twenty years.
Not really. It's irrational, but it's also meaningful that non-blacks aren't 'allowed' to say that word. It's a reminder that things aren't equal, things aren't fair, and the way this currently affects you is mildly unfair, but also entirely symbolic.