Typically, earth or ground faults are sensed by running both the hot an neutral wire of a circuit through the coil of a current transformer (CT). Under typical conditions, the current in the hot wire is exactly opposite the current in the neutral wire, so the net current through the CT is 0 (google Ampere's Law). If a fault occurs, the current in the neutral conductor will be less than the current in the hot wire because some current is "escaping" to ground and not returning through the neutral, so the net current through the CT will be non zero, and the secondary of the CT will output a current that trips a relay which opens the circuit.
*EDIT* GFCI breakers or outlets have the CT built in, that is why they are physically larger than their non GFI counterparts.
You can think of it like a differential current amplifier connected between the hot and neutral wires. Normally, the output is zero because the two inputs are balanced. When the two currents go off balance, the diff amp starts outputting a signal that is used to break the circuit.
On a related note, arc-fault sensing is a much more complicated matter and involves electronic monitoring of the circuit to sense non typical operating conditions (e.g. voltage or current spikes).
B.Sc. Electrical Engineering