Einstein was being metaphorical, of course. He didn't believe in a personal God, rather, the God of Spinoza. Technically, that made him a pantheist, but he himself didn't like the term. He thought God was the grand sum of the universe, so far as science can reveal it.
As someone already pointed out, Einstein initially rejected Quantum mechanics. Einstein saw an innate order in the universe, and the idea of Quantum mechanics did not resonate with that belief, as Quantum mechanics is inherintly chaotic. As Quantum physicists like to put it, the more you know about Quantum Mechanics, the more mindboggingly confusing it becomes.
In Einstein's day, Quantum mechanics was an untested theory, much unlike his theory of relativity, which was supported by hard empirical data. He uttered the phrase "I am utterly convinced that God does not play dice with the universe" in a letter to one of his friends. He later used the quote in a debate with Niel Bohr, who was a staunch advocate of Quantum Mechanics. Bohr's response is oddly less famous: "Don't tell God what to do!"
After testing proved the existence of Quantum Mechanics. Einstein spent the last 30 years of his life trying to discover the grand unifying theory of physics, which would link Quantum mechanics with Newtonian mechanics and Relativity, the way the theory of evolution links all of the fields of biology together.
It was said that he worked on the morning of the day he died. According to him, he wasn't innately talented, he just had a passionate curiosity about the universe.
EDIT: It's not what we're saying, it's what Einstein said. As far as free will, it's tricky to say. There are some people who say that the presence of Quantum Mechanics negates any sort of predestination or determinism in nature, implying that humans do have free will. If you want to get a better understanding of what Einstein believed, study Baruch Spinoza, as Einstein named him as his perennially favorite philosopher.