Yeah, what happens in defrost (on most vehicles, not sure if the XJ is like this specifically) is that the air is passed through the AC evap coil which forces a lot of the humidity out of it, then through the heater, which heats it back up, so the hot dry air clears your windshield faster. Without the AC you'll just be blowing normal hot air across it, which works but not as fast like Jim said.
If your AC works, why kill it? If it doesn't, you will notice no difference in your defrost, rip it all out and forget about it.