A lot of energy does go into refining the metals used to make an electric engine but these are also the materials that are most easily recycled. The expenditure of energy in making an electric vehicle is essentially a capital expense, i.e.: a one time affair.
However, after that the car is at best carbon neutral if the energy comes from clean energy sources.
Batteries are very poor ways of storing energy chemically, not only are the chemicals used inefficient at accepting, storing and discharging energy but you carry the spent products of the reactants around with you till you can recharge them. The volumetric energy density of batteries is very low, maybe 250 Wh/l whereas the energy stored chemically in gasoline is about 9,000 Wh/l and the energy stored in gasoline is stable for far longer than the energy stored in batteries, with gasoline, some of the reactants are from the atmosphere and do not have to be carried by the car and the products of the reactions are just emitted to the atmosphere instead of carried around as well. In theory, you can make a battery based on oxidation of gasoline and indeed that is the holy grail of fuel cell research (fuel cells are batteries where some of the reactants are from the atmosphere and the products of the reactions are expelled to the atmosphere). The only reason why fuel cells use pure hydrogen and pure oxygen is that it's the simplest form of fuel cell and we haven't figured out how to deal with byproducts of using hydrocarbons and nitrogen oxygen air in the fuel cells. At least not yet, methanol fuel cells are now commercially available and microbial fuel cells are being deployed by the military in the deep seas.
Now everyone always thinks of fuels like gasoline and diesel as being fossil fuels and what we have now are refined from fossil reserves as that is the cheapest way of making them but it's not the only way. Gasoline and diesel can be synthesized from carbon monoxide and hydrogen gases which can in turn be obtained in commercial quantities by gasification which works with almost anything that burns including trash and dried sewage. Indeed that's how we currently get hydrogen, by gasification and steam reformation of natural gas or coal.
Synthetic gasoline and diesel can also be derived directly from CO2 and H2O as Sandia Labs has done with solar power. Of course, it doesn't have to be solar, it could be wind, hydro, biomass, nuclear or whatever else we have available (note Sandia Labs has an interest in finding niches for nuclear power). When synthetic fuels are made by gasification of biomass, not only is the entire plant used (not just the grain or fruit) or trash and dried sewage used but charcoal is produced as a byproduct which can be sequestered as biochar resulting in the removal of CO2 from the atmosphere. Synthetic gasoline can be carbon negative.
A carbon neutral vehicle will never amortize the environmental impact of it's manufacture but your gas guzzler could if we just changed how we make our fuels. Unfortunately, synthetic fuels would mean more expensive fuels such that $4 US a gallon would no longer be the record maximum price but would become the minimum price.
It isn't that electric vehicles are bad for the air, they just don't do any good either while we can achieve not only the benefits of electric and hydrogen fuel cell vehicles but also benefit the environment without changing a single vehicle, just change how we make the fuel and be prepared to pay the higher price for that fuel.