You can pull the refrigerant lines, expansion valve, and dryer pretty easily if they are already discharged. The system may have leaked down, which is why it isn't working, but if it still has a charge then that needs to be evacuated before you can unbolt them. Please don't just vent them to the atmosphere. Federal law requires that the R-12 be collected and contained by a licensed technician. You should be able to find someone to do this if you let them keep the Freon, but everyone need to contribute to keeping the environment safe. Enough said about that, once empty they should just unbolt in the engine compartment. There is an electrical connection to the dryer, and one to the compressor, but they can just be unpluggeed and left there. I believe they were a standard part of all harnesses, whether you had AC or not. The only other electrical component you can get at is the relay under the black cover. You could pull it out, but it really won't hurt anything to leave it there.
Since you want to leave your compressor in place, the other main components are the condenser and evaporator coils. You can get the condenser out from by taking the header panel off above the radiator and loosening it all up and pulling it out the top. This may give you a little better airflow across the radiator, but you can leave it there until you have to go into that area and will just ride along until then.
The evaporator is located in the blower housing under the dash and access to it requires almost complete disassembly of the passangers side dash and a gppd part of the drivers side too. Removing it would serve no purpose and I have to say removal and reinstalltion of the blower housing is probably the most frustrating project I've tackled in 18 years of XJ ownership, and that includes conversion of my 2x4 to 4x4. I wouldn't wish it on anyone.
The comfort switches are no different with or without AC, so there's nothing to do there.
Just pull the easy parts off and get the rest if you ever have to work in that area.