XJFish:
It would really help if you would have said what year your XJ is, or at least update your profile to include the vehicle information.
In addition to the vacuum leaks mentioned above, if it is one of the early ones with a Saganaw column and the switch integrated with the turn signal / wiper stick, the switch and / or harness are a common failure. If you are cruising at a fixed speed and turn the cruise to on, then push the button, does it hold that speed? In other words, is it only the accelerate function that does not work? Again, assuming it is this style of switch, the place to look is at the front of the switch assembly, where the harness exits the stick and enters the steering column. The wires often break here, although the sheath will probably look OK, so you will need to wiggle it to check for broken wires inside. I had one with a break that was making intermittant connection and worked part time. A new switch fixed it. This switch is the same as one used on a year or 2 earlier GM vehicles. I found one on ebay for a 1988 Blazer IIRC. It worked just fine in my 1990 XJ. There are 2 vendors on ebay and have pictures so you can match it with yours. The switch is ~$50 at the stealer and I got it on ebay for $28.