It's always great to see answers from people who have no idea about the art and science of protective relaying...
While the specific equations of your current differential relay will be given in the manufacturer's manual, you can still determine its major characteristics without the manual.
Current differential relays use the concept of measuring the current flowing into and out of a device, and determining how much of that current is 'lost'. If this difference in current is excessive, the relay trips the breaker to de-energize the circuit(s). Usually, the discrimination is a percentage of the current flowing into the protected equipment, e.g. a 15% setting would trip for a difference or current 'loss' above that level.
Since there may be measurement uncertainty, especially at low current levels, it is common for the trip characteristic to require a definite non-zero amount of difference current before tripping. (You wouldn't want a transformer differential to trip on magnetizing branch current.) Likewise, for tremendous levels of difference current, the relay may use an altered characteristic to provide faster tripping, or more security, depending on the application.
The manufacturers usually provide very detailed descriptions of their relays. Look for ABB, Siemens, Schweitzer Engineering Labs (selinc.com), GE-Multilin.
I would start by hooking up two 3-phase sets of current sources (the test set) to the relay and shoot a little current in it. Gradually increase the differential current until it trips. Do the same for higher "through" currents, and on the other phases. With enough testing, you will determine the characteristic. The manual will help you get the testing set up correctly.
If you don't have a dual 3-phase test set, it may be possible to test the relay with a single-phase set, but only for a small number of the possible current differential relay configurations.
Voltage differential relays work on basically the same principle, but they are limited in application.
High impedance bus differential relays are a very different form of current differential relay. They rely on the saturation characteristics of matched sets of CTs to detect if a fault is within the bus protected zone (trip) or outside the zone (no-trip). These Hi-Z bus diff relays are basically over-voltage relays with a few extra features. Check with the engineer before twiddling...
Electric power engineer. Specialty in protecting relaying, including inside-the-relay and system-level design and application.