I've found the most common problem with the starter is the brushes. Either oil has seeped in and gotten on the brushes, usually from a leaky oil filter adapter. Muddy water has gotten on the brushes or they are just worn down. When they wear down the spring tension holding the brushes down gets weaker, sometimes the wire to the brushes gets a little too short and has to be rerouted a little.
Lightly (to moderatly) tapping the side and/or end edge of the starter with a hammer sometimes seats the brushes enough for a few more starts. Usually my first test. Some people recommend against this, I've been doing it for forty years (twenty on the XJ specifically) with no problems yet.
The XJ starter needs good juice (good connections, wire and ground) and a little spin to get the solenoid and bendix to seat all the way. Most often it's the brushes (or a weak battery or a combination of the two), but can be the ground, relay or wiring. I've found dirty or worn brushes to be the most likely culprit about 4 or 5 to one over most anything else.
I'm using an 87 starter on it's third set of brushes as a spare while I service the starter on my 96 now (on it's second set of brushes). I actually have three spares, junk yard starters, cleaned, the planetary gears and Bendix re greased (synthetic), with new brushes sitting on the shelf or stored in my XJ's for insurance.