Was having some alternator issues and had to swap out the battery a couple of times (one on the charger, one in the XJ). After about the third swap, had the same symptoms you discribe. Oil/grease on the battery poles, cleaned with brake cleaner, sanded reinstalled, started right up.
When the brushes are going bad or not making good contact in the starter, a sharp rap with a hammer on the back of the starter, is usually good for a couple of more starts.
The starter relay, usually looks fairly corroded inside, never had one fail on an XJ, but have seen many act up on the Dodge (pretty much the same part).
Noticed after cleaning the grounds, where the battery ground, hooks to the dipstick bracket and the motor to the firewall. My alternator charged better and the starter turned over a little faster.
Also had a bolt back out of the starter (believe it was the bottom) starting became a sometimes, thing.