If this buyer became interested in the home while you were under contract with the listing agent, and if this person found the home through that agent, he is owed a commission. His job was to produce a ready, willing and able buyer, which he did.
I would definitely contact his broker if he is being a schmuck and not returning important calls. Your contract is with the broker, not the agent. The broker should be made aware that he has a moron out there dirtying up his company's name. Call the brokerage, insist on speaking to the broker of record immediately. Use phrases like - No He cannot call me back, I will hold. If he is in a meeting I will hold until I hear his voice on the line because I am mad as hell, etc.
When you speak to him, tell him your current agent is not working out, and you need another agent in his office to complete the transaction.
As far as getting out of the contract, it's possible, but he will almost certainly come after you for a commission in this case, and in all likelyhood, he will win. Pay him.