I once drove a friend to the hospital to undergo a relatively routine surgery. She was scared and, even knowing I am an atheist, asked if I would pray for her. My initial reaction was to laugh and say, "I don't think I'm the person you want petitioning your god on your behalf." But she was serious and asked again. I told her that I loved her and if she wanted me to, I would pray for her. And when they took her back to surgery, I did. Not because I believed there was a God to hear or to intervene on her behalf, but because she was my friend and she asked me to. It meant something to her.
I do not "refuse" to pray. It is just that in the normal course of event I fail to recognize the efficacy of requesting favors from what I believe to be a fictional and non-existent character. It is one thing to talk to empty air, it's another thing altogether to expect a response.