The number of elector votes is based on the number of representatives that it has in the House of Representatives (which varies by state) and the number of Senators that it has (two per state).
Article I, Section 2 of the Constitution mandated a decennial census and dictated that the number of representatives would be calculated based on "the whole number of free persons, including those bound to service for a term of years, and excluding Indians not taxed" and three-fifths of all other persons (a/k/a slaves). The Fourteenth Amendment changed this rule by eliminating the reference to "free" persons (referring only to the "whole number of persons") and the three-fifth rule.
You have to remember that at the time of the original Constitution, many states still had some form of property requirement to vote. Even by the time of the Fourteenth Amendment, women still could not vote. Even today, people under the age of eighteen can't vote. The Framers of the Constitution, however, decided not to try to calculate the number of eligible voters and opted to apportion based on people.