If what they said is an actual representation of what is going on during these visits, then no, you shouldn't force them.
After seeing your added details, I have to agree with your son; the environment there seems toxic. Of course your ex isn't going to admit he's a lousy father and could care less about the quality of the visits, that is to be expected regardless. But for him to rail against the kids like that with you, it seems he is more interested in how he comes off as the bad guy than in digesting any of the feedback his children have provided, however harsh.
When my brother and I were younger, we would spend summers with my father in Connecticut. Often times things there were boring, and we didn't have the greatest of fun. But that was because my father wasn't used to having children around; it wasn't out of disdain or lack of interest, and we were always well looked after. He never tried to garner info about our mother, never sought to put us in the middle of what ever issues they had. Even to this day, I'm not aware of them ever having argued, all discussion was done in private. Your ex needs to grow up and learn that his kids aren't annoyances to be kept at arm's length.
Anyways, forcing your kids to spend quality time with daddy when daddy isn't interested is only going to breed resentment; against him, for being a dick, and against you, for not heeding their complaints. If you've tried talking to him already, and nothing seems to change, let it go.