I know black people who hate the "N word," no matter who uses it. Arguably, when black people use it to refer to themselves as individuals, or to each other, it perpetuates "internalized racism," the anti-black racism of black people who have come to accept a white racist view of themselves as bad or inferior or criminal, and who therefore hate themselves at some level.
OTOH, I've met other black people who will use the "N word" themselves, but would be highly offended if a white person like me used the word to describe them.
The problem with the word is that historically, it's often been used by racist white people in a hostile and derogatory way -- been used as a racist insult directed at black people.
When most black people hear a white person use the word today, I think it evokes all that racist and often violently racist history. When a white person calls a black person a "N****r," I think most black people take that as a deliberate insult, if not a threat.
Some black people use the word to refer to themselves and to other black people, however, for the same reason that the rebellious American colonists in 1776 began calling themselves "Yankee Doodle Dandy." The "Yankee Doodle" term was originally a British insult targeted at American patriots. The patriots responded by adopting the name for themselves -- as a way of defusing the insult.
In similar fashion, there are gay activists who call themselves "queers" or "fags" or "dykes," but enormously resent heterosexuals calling them these names. There are Jews I've met who sometimes will use anti-Semitic insults to refer to themselves, and I'm sure there are white southerners who call themselves "rednecks" but hate it when any outsider calls them that.
In all these cases, if you want to avoid offending the person you're talking to, I think you should call them by whatever polite, widely accepted term that they choose for themselves. Using the traditional insult that members of the group have chosen to claim for their own is risky, and it's easy to offend people and anger them without meaning any harm.
Your grandfather, of course, is probably not going to change. But YOU can change, and make sure you don't call your coworker by the "N word."
And yes, it is a double standard. But in real life, we run into double standards all the time. This one really isn't hurting you any as a white person, just asking you to think a little before using the same vocabulary that your grandpa does. Although it's a bit of a nuisance for you as a white person to observe the double standard, you can learn to do so pretty easily. So why not do it?