I'm not medically trained so I can't tell you for sure, but it does sound like raynauds.
I have it and I always feel colder than everyone else, even in summer. I'm currently sitting here in three layers of t-shirts and two fleeces and I'm still cold.
My fingers and toes go completely white down to the knuckles and then the backs of my hands tend to start turning blueish as well. Wearing gloves doesn't stop it- it helps a bit but I still have completely white hands and it takes ages to get them warm again. I also have no feeling in them and then, when I finally start getting warm again, I have pins-and-needles through my fingers for about ten minutes.
A milder version of raynauds may just leave you with numb feeling fingers but not completely white hands, just paler (mine's pretty bad so I get the full effects).
I've been going to my doctor for a few years trying to sort it out (I've had it for about the last 20 years but only just decided to try and do something about it. Before that, I just put up with it). So far, I've tried three types of medication- two gave me a blinding headache and the third didn't do anything- and so now I'm back to putting up with it until they bring out a new treatment to try.
Check with your doctor- you never know, if it is raynauds one of the treatments may work for you.
And yes, it drives me crazy when I get people telling me "Don't be daft, it's not cold!" or deciding that just because they're warm, they can open all the windows, leaving me freezing and then complaining when I close them again.