The original use of nut words for a human quality had little to do with madness, but with "source of pleasure, pleasurable disposition". The nut being a tasty snack, such usage, and its extension, is readily understood. The shift from a pleasurable nature to insanity occurred in the mid 1800's according to the following information.
1600's: nuts "source of pleasure"
1700's: be nutts upon "be very fond of"
1800's: be off one's nut "be insane"
1800's: be nuts / be nutty "be crazy"
1900's: nut, nutter "crazy person, crank"
from earlier be nutts upon "be very fond of" (1785), which is possibly
from nuts (n., pl.) "any source of pleasure" (1617),
from nut (q.v.).
Sense influenced probably by metaphoric application of nut to "head" (1846, e.g. to be off one's nut "be insane," 1860).
Nut "crazy person, crank" is attested from 1903,
(British form nutter first attested 1958).
Connection with the slang "testicle" sense has tended to nudge it toward taboo.
"On the N.B.C. network, it is forbidden to call any character a nut; you have to call him a screwball." ["New Yorker," Dec. 23, 1950]
"Please eliminate the expression 'nuts to you' from Egbert's speech." [Request from the Hays Office regarding the script of "The Bank Dick," 1940]
This desire for avoidance accounts for the euphemism nerts (c.1925).
Nutty "crazy" is first attested 1898. "