but it's like black people calling each other the n-word,
Yeah. So if it's not racist for black people to say the n word, does that mean the n word isn't racist?

White people who use this term are too stupid to understand it's racist. They think it isn't because it doesn't apply to them, it's a political slur rather than a racist slur. But it's based on skin colour, an angry white person flushed in the face goes gammon coloured. It only applies to white people. I'm perfectly happy that this qualifies as racism. Is it as bad as racism towards black people? No I don't think it is. But that doesn't make it acceptable. If you oppose racism, you should avoid using this slur, otherwise you are a hypocrite.