Honestly it depends on your area. The township I live in most building permits other than for residences are $50. If you are building structures to support farming in any way (including horses) that fee is waived. When I built my barn my permit cost nothing.
If I lived a few miles over in a different county and a different township the permits start at $2500. It is an important question to ask when you are buying property. The low/non-existant permit costs were one of the reasons we chose that particular house.
If you have have electricity, water, & drainage (like for restrooms installed) you'll probably need inspections for that. In my area the water & drainage are inspected by the township person who issues the building permits at no extra cost, but the electrical inspection was done by the county inspector. That cost was still pretty low (I believe $150) and included in the initial bill to get the electricity installed.
Arena costs are varied. A "fabric/plastic" tarp building might be quite cheap to setup (I think I've seen small ones at $8,000-$10,000 but I don't know if that including installation), but still cold in the winter. I've ridden in a canvas arena that kept heat in really well in the winter plus the sides rolled up for a cool breeze in the summer but that arena cost around $60,000 as I recall. A pole building would probably be in-between.