understand the grandparents postion. they have lost their daughter, and their grandson is all they had left of her. i bet they supported him, gave them a home and were always there when needed. do they have grandparent rights? YES THEY DO and a judge will award them visitation. however, UNLESS you are a drug addict, a drunk, a proven child abuser, or shown to be an endangerment to your child, they will NEVER get custody of him in any form.
this is hwere YOU take the correct initiative. put a legal proposal to them. OFFER them visitation once a month for a weekend with the child in their home PAST a certain age, when your boy is comfortable sleeping there - maybe 5-6 yrs old starting. until he's that age. OFFER them the option to babysit for you when you need someone, OFFER them a visit once or twice a month, whether in your home or theirs so they can see their grandbaby. the condition must always be that they will act in kind towards the child and not do or say anything disparaging about you to the child or in front of the child at any time, or you will retain the legal right to cease visitation.
judges and courts will look favourably on you for this and it will certainly make you a bigger and better person. remember always that your ex gf - your sons mom, and her parents and his cousins etc are always a part of his life, and do not vanish out of his life along with her tragic death.
strength to you to make the right choices. you say you don't understand why they're doign wha tthey're doing - put yourself into their shoes and see they are probably scared shyteless - they lose their daughter and their grandson at once - show them you will not be such a moron (some are) as to pretend they don't exist, and that your son will always know and have access to both sets of his legal grandparents, and they don't need to waste money in a court because they think you will cut them out of his life.
tell them they can't proive you infit in any way so this is a waste of money to pay a laywer's mortgage fighting a battle they can't win, and they're only fighting out of desperation anyway. suggest they could put this money into a college fund for their grandson and play a constructive part in your lives instead.