Privilege is not power
Much discourse on the left forgets this, leading to incoherence. Privilege is a relation between groups. It exists either because of overt differences in treatment or, as is often the case with ‘able’ privilege, because of systems designed to work well for most people with little thought for exceptions.
Power, on the other hand, is a relation between individuals. I have power over you if I can, through action or inaction, affect your life and further or frustrate your goals in ways that you cannot do symmetrically to me. Power and privilege are, of course, linked. Groups (generally the Unmarked Group) whose members find themselves in positions of power often gain privilege in a feedback cycle. Other times, groups gain privilege simply by greatly outnumbering those unlike themselves. An odd case is the group of white, middle class senior citizens in much of North American and Western Europe, a group who has the welfare state tailor made to serve their interests in spite of this harming the economy as a whole and impoverishing younger generations who are just starting work and school. Particularly odd since individual seniors are seldom in positions of power and do not often enjoy what would be considered privilege relative to younger members of their demographic groups.
Privilege is, instead, having society and the world fit you so naturally that it doesn't seem remarkable. The concept of privilege has value only as a heuristic check on ones own thoughts and evaluations, an awareness that ones experience of the world is not representative, and that ones evaluation of the effects of institutions are likely inaccurate. Knowing that others have privilege is useful as a guide to what they may not understand and what experiences they may not have had when trying to educate them. It serves no other purpose. Feeling guilty about ones privilege is stupid.
Confusing privilege and power can lead to oppression of its own. Most people I come into contact with possess privilege I do not, yet I often have power over them. For me to forget this, to act as if they are somehow my oppressors (the concept of an individual oppressor as a cause of institutional oppression is, in itself, nonsensical) and treat them accordingly would be nothing but bald-faced cruelty on my part, and it would still be frank cruelty even if there were no individual difference of power. In most interactions, especially between strangers at a remove with no particular economic relation, there is no differential of power.
Privilege is not a scalar
Occasionally, members of one marginalized group will be told that members of some other group thought (by the speaker, at least) to be more marginalized do not ‘have privilege’. This is obviously wrong. A straight, black man may experience more oppression than a gay, white man. His goals may be frustrated more, his appearance may carry ‘otherness’ in a way that is only revealed in a gay, white man's behavior, and he may experience more risk of violence. Nevertheless, he has an experience of being part of the sexual norm for his community. His romantic and sexual encounters are unremarkable by the standards of those around him.
This does nothing to lessen the oppression that he is under, but privilege is not a quantitative measure of oppression. It is a description of those societal factors for which one is the exception.
Though the degree of oppression that falls on the average woman is in general much greater than that which falls on the average man, institutional gender norms do oppress men. Alongside obvious male privilege (one can express oneself forcefully without being thought shrill and domineering, basically all ones clothes have pockets, one can have a bad day without strangers assuming ones biochemistry is the cause and suddenly their business), there are examples of female privilege, like being able to show affection or close attachment to a friend without it being taken as either a sexual advance or reflective of ones sexuality and being able to cry without being judged as a failed example of ones gender.
Privilege is not very good framing
As we know, George Lakoff's attempts at reframing political discourse are kind of garbage. The familial metaphor is based on, basically, nothing, and his attempts to rechristen taxes as ‘membership fees’ and abortion as ‘development prevention’ border on farce. Framing itself is, however, important, and the discourse of ‘privilege’ feeds into the zero-sum ideology that underlies much of America's political dysfunction.
When we speak of ‘ending privilege’, it sounds as if we want to go up to people and take something beneficial away from them, when nothing could be further from the truth. Ending able privilege does not mean that now everyone gets to have the architecture of buildings and entire cities built to create second class citizenship for them. Ending white privilege does not mean that now everyone gets to be harassed by the police and denied opportunities.
On the contrary, history has taught us that releasing others from oppression is a benefit to society at large, with more of the population able to develop their full potential and contribute to the world. Apart from the obvious economic benefits that providing education and opportunities for everyone in our society and ending mass incarceration would yield, there is an argument that the homogenization and unmarkedness of ‘white’ culture is the reason why many white people find their own culture to be ‘sterile’ or ‘wonderbread’ and exoticize other cultures.
Obviously things won't be easy. If ending systematic discrimination were easy it would have been done by now. But right now we have a problem where one of the factors hurting efforts toward greater equality is that majority groups in bad economic situations view efforts to redress inequality as taking something away from them when they already feel that they are worse off than they should be.
What is the solution? I'm not entirely sure. Using able privilege as a prototype for privilege generally might help. Treating privilege as the norm and departures from it as analogous to a ‘social disability’ that needs to be redressed may capture the positive-sum essence of the thing more accurately.