As others have noted, the Olympus platform uses the 4/3 sensor giving all lenses a 2x crop factor. This smaller sensor produces poorer quality images. Nikon uses the larger, 1.5x cropped sensor, while Canon uses the 1.6x sensor.
Canon uses the CMOS sensor while Nikon and many others use the CCD chip. Between the two, there are no real differences. Don't have someone that owns the Canon 1Ds say Nikon has bad image quality when they are comparing it to a Nikon D70s. Comparable cameras produce comparable results. This is no truer than between Nikon or Canon.
The Canon 450d, or XSi, is a great little camera. Everything about it is good. But the build quality is, as you noted, not as good as it should have been; but still much, much better than the older models in the Canon Rebel lineup.
The Nikon D80 would be your best bet. The specs are good and the build is great. There is nothing much to fault this camera, but for you, it might be a tad big. You could just grow into the camera, that is, get use to the feel over time.
The final decision is up to you, but like I said in the previous post, you're going to have to settle on something. If it were me, I'd rather have a camera that was too big rather than too small.