Because they were going to the deep south in the mid-twentieth century. If Dr. Shirley had ever, even to go to the gas station, been alone in the car, his life would be in danger. He would have been pulled over on suspicion of stealing the car, and there was a reasonable chance that the white guys who pulled him over in at least one of the towns they visited would have decided to execute him without a trial. They'd have done it for CERTAIN if they'd gotten a whif that he was gay as well as black. Early in the movie I think there was an indication that Tony was chosen because of his bouncer skills; he was there to protect Dr. Shirley.
Also, even IF he could have passed as the black chauffer and been safe for the whole trip, Dr. Shirley did not want to do that. His whole reason for taking his musical tour down South was to be a trailblazer and push back against the stereotypes in people's minds. That scene where he got out of the car by the side of the road and the field hands saw that a white man was driving a black man, that moment, and the opportunity to create it, was part of his entire purpose for going to the South. He also wanted to change the stereotypes in the minds of his white audience members, but he found that to be more of a losing bet.
Edit: I checked Don Shirley's Wikipedia page just now. He was not only a musical prodigy, but he'd also studied psychology and for a short time he made a career as a psychologist. He understood how significant it was for a black man to demonstrate to the white southerners that he was not only a musical genius, but that he also had the most refined manners and educated vocabulary of anybody else in the room, and to demonstrate to the blacks that by virtue of his achievements and wealth, white people were treating him with deference and respect.
I just went back and read your question again. You also asked why not all 4 men go in the same car, to save money and have more fun. They didn't need to save money. The only person on the tour who needed the money was Tony the driver. Remember how Dr. Shirley's high rise apartment was decorated? That was to demonstrate to us that he already had more money than he knew what to do with. And I think there was a scene early on where Tony asked the other 2 musicians why not keep to the North for better safety and less hassles, and the other two musicians explained that Dr. Shirley had been adamant that they tour through the South, and they were coming along because of their respect for him as a musician. They felt honored to play with him.
If Tony was going to be the driver, it wasn't acceptable in the South for whites and blacks to be passengers sitting together, although in a private car it was probably technically legal. Ever heard of the freedom riders? White and black civil rights protestors from the north chartered a bus and rode through the south seated in black/white pairs, side by side. Some white southerners were so enraged by this that they blocked the road and set one of the busses on fire while the passengers were still inside it. They got out alive, but still. This was before that. Ever heard of Rosa Parks? I don't know if you are aware, but there was enough room on the crowded bus for her to remain seated that day. There just wasn't room on the crowded bus for her to remain seated without sitting next to a white man, which was illegal even across the aisle. She was required by law to stand so that he could be the only person seated in a row that had room to seat 2 people on each side of the aisle. That was the LAW. Blacks had to ride at least one row further back than every white person on the bus.