It would be horrible - it ruined Taekwondo.
However, it would not likely happen, either. First off, Taekwondo and Karate would appear to be too similar, so, IOC committee would not likely allow it. On the other hand, Olympic Taekwondo is only sparring. So if the Karate organization were to put up a referrendum to get into the Olympics, they could choose self-defense, breaking, weapon/weaponless, or forms, then they might have a shot.
One of the problems with Taekwondo is that the Koreans keep winning. This is bad for the sport, and keeps TKD in jeopardy: IOC rules stipulate that one criteria for a sport to be in the Olympics is that the sport can't be particular to a single country. If Korea doesn't win as often, Taekwondo wouldn't be in jeopardy. It is amazing to me that Wrestling was bounced from the Olympics, and Taekwondo was kept - despite Wrestling not ever having been associated with a single country, and the fact that it is one of the more traditional sports in the Olympics. Although that has changed, it remains that Taekwondo is once again in jeopardy.