Sometimes yes it is. But many of the currently preferred terms are idiotic and not using preferred terms is not automatically impolite.
I agree with the literal meaning of what you said. However, based on history, I suspect you are using that literal meaning as a cover for other meanings.