I'm confused now. Where are you making your voltage measurements? At the lights or at the battery? If the voltage at the battery is going low, then the fusible links and switch aren't likely the problem, since increased resistance will reduce current draw and should not drop battery voltage. But if you're getting bad voltage at the lights while the battery voltage is correct, then definitely you should start tracing the voltage, and see for example if the voltage into the switch is the same as that out.
If the readings are at the battery, then the lights are pulling more current out of the battery than is going in. Look for either a bad alternator, bad terminals, or bad wiring. Don't overlook the possibility that some wire, such as the main charging wire from alternator to battery, is eaten away inside even if it looks good. Measure resistance if you can.
There's also a slight possibility of a short circuit or leakage in the wiring to the lights, but it's unlikely, especially if you did already check the wiring. Since your readings are varying very reliably with the status of the headlights, you could easily check that by simply unplugging the headlights. If the voltage drop improves greatly, then you can be pretty sure that the problem is current draw from the lights, not from the wires.