Martial arts are great, but I would focus on your other reasons, rather than the self defence aspect. Avoiding real world violence is far better than a false sense of being 'able to protect' with violence.
I enjoy weapon arts, having a similar 'background' as you describe, and additionally being small and slight. (5'5" and 115lbs). Swords are fun. Swords do not require strength, or very high athleticism to perform well with, as long as a modest amount of both are available - understanding of how to move and how to generate and exploit an opening and provide a cover are far more important than raw power and speed. The manuals I work from are historical documents from the C14th to C16th, in a related family of systems of fencing - and are interesting in their own right, as well as in the information they contain, and the varied and subtly different ways of fighting that are portrayed in each (part differences in weapon design, part evolving discovery of better ways of doing or writing about doing).
Interpreting how bodies move, together with the various weapon types, and in accordance with the plausible readings of the historical treatises is fun, has something for everyone, from the loftiest scholar to the most athletic sportsman and every station in between.
Generalised anxiety is at least difficult to seem quite as tough once you have faced off against a 4ft steel longsword in the hands of someone a foot taller and with 100lbs on you... and fenced well against them.
I didn't want to do a purely striking art where my smaller stature would a bigger problem than in a weapon art, and Ringen (wrestling) is a part of the weapon and unarmed systems, so that aspect is at least partially covered, and there is a 'surprise' element when I press in against a larger fencer, which is helpful.
Do participate in something, martial arts are fun. The exercise is good for your mental well being and mood, but you should also consider whether you need to also look for other support as well. It is social, focused on a common task (best of both worlds).
I started HEMA at 44, 4 years ago, and am slightly older than the average, but far from the oldest in our school. I am the smallest and lightest in the school, but that doesn't hamper my performance, only require that I do subtly different selection of techniques and avoid those which rely on pure distance games.