This is a completely untrue conflation of two different bases for differing outcomes and two different roles that are occupied by those that benefit from the privilege of being a favored race/class.
Examples:
1830's: White person named "Ryan Johnson" is able to own a slave, treat him like property, pay him nothing for labor and enjoy protection by the laws of the state for his status.
2010's: White person named "Ryan Johnson" is more likely to get a job than equally qualified "LaShawn Perry" because of latent attitudes about race and class.
Recognizing the latter event as the product of "privilege" does not make the moral judgment that Ryan Johnson #2 is morally culpable in the same way as Ryan Johnson #1. Privilege, in many instances, is a way of describing objectively verifiable social phenomenon that occurs regularly and passively. Those who gain advantages from the privilege frequently do not seek them explicitly, and they often have no way of perceiving that the privilege is real because, for example, Ryan Johnson #2 above has no idea that LaShawn Perry even exists, much less what his qualifications are. In Rumsfeldian terms, to the person who benefitted from the privilege, the person who did not is an "unknown unknown."
Where you run into problems is when people try to deny that privilege exists. That kind of willful ignorance has the effect, if not the intention, of ensuring that privilege persists rather than erodes across generational time.
