Don't refuse your child's father access to your child. He's the father and he has rights. It's hard for you to have had this happen and still send your child to see him and his mother, but the point is, you can hold your head up high, knowing the claim was false and now a record exists of what happened, with child services. It doesn't show either of them in a good light, and can be used against them if they ever do it falsely again. but of course, pursue child support by all means available to you.
Think of this from your child's point of view. She's the innocent in all of this. She just wants a mommy and a daddy and a grandparent too. She won't always be small and biddable. One day she's going to have views of her own, and how will it be when she says to you "Mom, why didn't you let me see my dad, and grandma?"