Dont worry and ignore the wankers that think its funny- a lot of women like shy guys, I know I would prefer a virgin to someone who had slept with loads of sluts, and so would many other women.
Of course it is due to your low self esteem, but you can help yourself- I would maybe try a self help book on confidence first- as you can practice the exercises in your own home, do what the book says- no matter how small- it will all add up and build up your confidence, you should realise that you are as good as anybody and keep telling yourself in situations where you get scared "Whats the worst that can happen? It wont kill me", if you do build your confidence up a bit, then maybe try and see a hypnotherapist to help with the confidence building.
The bottom line here is that you may not ever be super confident, but enough to get you through life better than you are now.
Also when talking to women try not to see them as a potential mate- treat them as you yourself would like to be treat- ask her things, and listen, compliment her, laugh at her jokes, just be really friendly, try not to think of the bigger picture and just take one step at a time.
You would be susprised that probably most, decent women will not find you odd that you are still a virgin, esp when you explain why, trust me on that score.
At the end of the day there are no "rules" in life that you have to stick to, who wants to be like the boring majority of this world?? We dont have to stick to what everyone else is doing- not nowadays! You are you and you are absolutely fine as a man and as a person, just take one step at a time, and you will soon realise that you will meet a woman who likes you for you.
Also my last word of advice is- dont go looking for a woman, there is no race to lose your virginity, it WILL happen when you least expect it, and I bet you will find that it will go well and surprise you.
I wish you all the luck in the world :)
I am a woman over 21.