Looking at Natural Selection favoring the fittest in any population every moment, while rejecting the unfit, any colony will have the healthiest or fittest going furthest out to mate with the fittest of another colony. The less fit keep getting rejected in the competition for survival. Mental retardation as @L. E. Gant pointed is just one of myriads of problems we all face.
However, in hard times, it is found that the rejected unfit population has a better chance of survival. A typical example is the survival of those with Sickle cell trait in Africa where malaria is endemic. Divine Nature certainly cares for both, the kailo - whole, sound, and healthy individuals, as well as the unfit, feeble, unhealthy, debilitated individuals. This ensures that nature overcomes hard times without total extinction.