I do not know Perelman personally, but I have read a good bit about him.
I think the first thing to know with regards to why he refused the prize is that he is not materialistic at all... he lives alone and with relatively few material possessions. He has basically said that he has "all he needs."
This quote from wikipedia about his withdrawal from mathematics is particularly enlightening:
"As of the spring of 2003, Perelman no longer worked at the Steklov Institute. His friends are said to have stated that he currently finds mathematics a painful topic to discuss; some even say that he has abandoned mathematics entirely. According to a 2006 interview, Perelman was then unemployed, living with his mother in Saint Petersburg.
Perelman is quoted in an article in The New Yorker saying that he is disappointed with the ethical standards of the field of mathematics. The article implies that Perelman refers particularly to the efforts of Fields medalist Shing-Tung Yau to downplay Perelman's role in the proof and play up the work of Cao and Zhu. Perelman added, "I can't say I'm outraged. Other people do worse. Of course, there are many mathematicians who are more or less honest. But almost all of them are conformists. They are more or less honest, but they tolerate those who are not honest." He has also said that "It is not people who break ethical standards who are regarded as aliens. It is people like me who are isolated."
This, combined with the possibility of being awarded a Fields medal, led him to quit professional mathematics. He has said that "As long as I was not conspicuous, I had a choice. Either to make some ugly thing or, if I didn't do this kind of thing, to be treated as a pet. Now, when I become a very conspicuous person, I cannot stay a pet and say nothing. That is why I had to quit." (The New Yorker authors explained Perelman's reference to "some ugly thing" as "a fuss" on Perelman's part about the ethical breaches he perceived).
According to Interfax news, Perelman turned down the Millenium prize in June 2010 because he considers the prize unfair and Hamilton deserves as much credit as he does for the prize. A direct quote given is "To put it short, the main reason is my disagreement with the organized mathematical community... I don't like their decisions, I consider them unjust."
One important thing to consider is that he does not consider his proof of the poincare conjecture a particularly great achievement. In fact he thinks it is unfair that he should be granted such a large prize for a contribution which he thinks is very routine in the world of mathematics. I remember reading about a mathematician whose recent work Grigori Perelman thought was more important than the proof of the poincare conjecture. Grigori said it was unjust that he himself should get such a large prize while that mathematician got nothing. Unfortunately I do not remember where I read this or that mathematician's name.