You are right that it isn't that simple; that a high mileage car is worse than a low mileage one. If you have a high mileage car who's owner is a cautious driver who does all the preventative maintenance on time, and another with low miles driven by a crazy teen, you probably want the high mileage one. So do take maintenance and owner profile into consideration. Model of car matters too, as some age better than others.
You can't just ignore mileage though. Parts wear down over time. As cars climb higher in miles, the amount of things that all begin to wear out increase. When a car is newer, a part might go bad and you replace it. Older cars, you can often see many things starting to fail all nearly the same time (just because they are all starting to reach their natural failure point). That is the risk you run; having just fixed something, and having to turn right around and fix something else, and so on.