The idea began to form (in the scientific community) around 1644.
The barometer had just been invented and it was recognized that the atmosphere has a weight, and it is this weight that gives the air pressure needed for a barometer to work. The pressure comes from the weight of the air above the barometer.
Then it was found that, as one climbed a mountain with a barometer, the air pressure goes lower and lower, showing that the weight of air remaining above the barometer was decreasing.
From this, it was concluded that air (at least, in breathable quantity) did not continue upwards forever. There would be a limit above which there would be a lot less air, and the air pressure would be insufficient for breathing.
It is difficult to establish who is the exact person who came to that conclusion. Galileo was still alive around that time; however, Galileo has written something that implied that he did not believe that air had weight. Torricelli (who is credited with the invention of the barometer) was the first scientist to publicly contradict Galileo on that idea. However, it is not clear if Torricelli thought that the air did become insufficient for breathing at some altitude, or if he simply thought that it has less and less weight as one went up.
Torricelli might have been prevented from clarifying his ideas because people who were aware of his experiment had reported his for practicing "witchcraft" (this was at the time of Galileo's trial by the Church, not an easy time for scientists in Italy).
In France, Florin Perier did a formal experiment in 1648, going up a mountain with a barometer, and noting that the air pressure did go down in a predictable way as one went up in altitude. The formula to determine the rate was worked out by Blaise Pascal.
So, if you are looking for who determined that we cannot breath in space, you could say that it was a combination of Torricelli, Perier and Pascal.
This was done by 1648, twelve years before Boyle's experiment regarding the effects of vacuum on living bodies.
Once you combine the work of the first three with that of Boyle, you could say that by the end of the 17th century, scientists were pretty confident that space was a vacuum where breathing would be impossible.