Durability won't be a problem, and since you're not likely to be mashing up a steep hill with a fully-loaded bike, any lateral frame flex isn't likely to bother you if you even notice it.
What might be a problem, however, is finding a carbon frame that has the necessary bits for you to attach your racks and accessories. I haven't really looked for a touring frame in a long time, but my recollection of every carbon frame I've seen is that they're pared down with no eyelets or extras beyond a couple sets of bottle bolts. I may be wrong, though....maybe someone has a carbon tour rig out there.
As far as other frames, what marks a good touring setup is a longer wheelbase, a fork that has a little more curve at the bottom, and usually a lower bottom bracket height. All of these things help to provide some stability. They handle slower, but they do well with the odd weight distribution of a loaded bike. A dedicated touring frame will also have bunches of little extras like a 3rd or 4th set of bottle bolts, pump peg, chain holder, and proper rack mounting eyelets. You can build up any frame with parts you have...just be sure that you have sturdy wheels. Any complete new touring bike will be set up pretty well for that type of riding.
Hope this helps some. Have fun! :o)