They can be a little harder to grow, not just due to greater disease and pest susceptibility, but also because they are often more fussy, requiring more water, or more stable temperatures, and suchlike.
Also, the idea that heirloom varieties are always better-tasting is something of a fallacy. Which? Gardening, an independent and well-respected gardening publication, did a test, including a blind taste test, to compare heirloom varieties to modern F1 hybrids (which are not GMO, but are simply the offspring of two specific parent varieties, chosen for superior qualities), and found that modern varieties of carrots, cherry tomatoes, all brassicas (cabbages, broccoli, brussels sprouts), sweet corn, pea, green bean, cucumber, courgette (zucchini), aubergine (eggplant) and lettuce were all superior tasting, as well as better cropping.
However, heirloom varieties of beefsteak tomato, beetroot, radish, and winter squash were found to be superior. All other vegetables tested were found to be more or less equal.
There seems to be a lot of fear and distrust of modern hybrids for some reason, and it seems to be founded on misinformation.
For starters, the general public cannot buy GMO seeds for growing in their own garden. They're fairly strictly regulated. Modern hybrid varieties for amateur growers are not GMO, but either normal open-pollinated varieties, seed produced in the exact same way as those of heirloom varieties, only having undergone more years of selection to choose only the best. Or they are F1 hybrids, which are the first generation of offspring from two specific named parents, chosen to combine the best traits of those two parents. Both are perfectly natural, and definitely not GMO.
Then there is the idea that heirloom varieties taste better and are more nutritious. It is true that some modern varieties have been bred to grow quicker, and this means they have less time to absorb nutrients. But this is only certain specific types of vegetables. Most take the exact same time as the older varieties did, and thus have more or less the same nutrient content.
And as for taste, well, I've already discussed it above, but the fact is these plants have undergone decades of selective breeding to choose only the best varieties. Varieties that are sweeter, richer, tastier, not to mention lack the bitterness that certain old varieties have, crop more heavily, are more tender and less tough, are more disease resistant, and so on.
Some heirloom varieties are definitely worth growing, but others are outdated novelties at best. The idea that heirloom varieties are always, or even usually, better is just silly. That's basically saying that you think plant breeding peaked 100-150 years ago, and has only gone downhill since.
Personally, rather than fixating on heirloom varieties, I would work out what types of vegetable you want to grow, then search for recommendations for varieties of that specific type, and buy seeds for whatever seems the best.
For example, I would recommend growing winter squash Marina di Chiggia, which is an heirloom variety from Italy. I would also recommend growing snap pea Delikett, which is a modern hybrid.
It may also be worth concentrating on growing vegetables that are expensive to buy. For example, onions and cabbages are cheap to buy and taste exactly the same from your own garden or from the shop, so aren't really worth growing. Snap peas or fine green beans, on the other hand, are expensive to buy and quite easy to grow, so may be more worth growing.