If the guy next to you is yelling racist stuff, and you are not saying anything, the guy next to you is making you look bad. If you don't like it, don't show up for Jazz games. The crowd is treated as, and often acts like, a single entity, and what others do and you don't object to reflects on you.
So, you don't mind when people refer to you as a knuckle-walking low-browed self-blinding bonobo? I mean, if you object, just let me know, because I would not want to call you that if it was offensive to you. Of course, since you are not going to get all hypersensitive and PC on me, you won't bother to object, right?
Anyone can be prejudiced (and pretty much everyone is), and anyone can be bigoted. Racism involves the social power structure, and you can only be racist when supporting that power structure. So, black people are racist when they show bigotry to other black people (for example, Jesse Lee Peterson), but since the power structures favor white people the most, white people can not be the victims of racism (although they can be the victims of prejudice and bigotry).
If you were really respectful of all people, you wouldn't need to be guilted into fighting racism, you would be doing it out of you respect for all people.
I'm sure it is making things worse for you. It's making things better for me.