it would help if you could indicate what year XJ you're referring to.
I can't speak to the 97+ models, but the 84-96 models have tiny little spring-steel "hooks" to hold the wiring connector onto the window/lock switch panels (they grab into the "rubber" of the connector block), they're fiddly little devils that can be easy to knock loose and hard to spot when they land on the ground under the truck. I lost a couple at one point for my driver's door (which takes six of the things), and used a couple of 4-inch zipties as a temporary substitute, which took some fiddling to get arranged. Eventually I got my hands on a used master switch panel that had a full set, giving me spares.
Here's a link to a "Standard Motor Products" version of the 85-96 style master panel - you can see the clips I'm referring to in two of the three pictures:
https://www.summitracing.com/parts/...MIpvOehI267gIVkMmUCR2L-gxsEAQYASABEgKUgfD_BwE
I've no idea if these clips can be sourced independently, but if you have a "mill supply store" around and have at least one of them still, you could see if they could match it to a known product. That, or buy a used panel on eBay and hope it comes with at least a couple clips (and then you have spare switches)...