Prior to 1996, IOC rules allowed for autonomous regions and territories to form national Olympic committees, seek IOC recognition and participate in the Games on their own. Remember that for much of the 20th century most of Africa, Asia and a large part of the Americas were territories or colonies of european countries.
There are 10 non-sovereign countries in the Olympics - Puerto Rico, Virgin Islands, Guam American Samoa, Bermuda, British Virgin Islands, Cayman Islands, Cook Islands and Hong Kong.
Current IOC rules do not allow for additional autonomous regions to join the Olympic movement - their athletes would have to represent their "parent" country. The existing ones are allowed to continue on their own so long as they remain politically autonomous.