Its a common problem on Jeeps, the outer ring of the damper works free from the rubber. If you don't get a new damper it will just get worse and do lots of damage.
Luckily on mine, the rubber looked great, its just the outer ring worked free from the rubber and walked backwards toward the motor. It made a lot vibration, since I had a hunk of metal wobbling around on the end of the crank. It did scratch up timing chain cover a bit, and made noise from slipping and rubbing.
The drawback is, unless you've got a good compact puller, you'll need to pull the radiator to pull the damper. This is on the 2.5L 4cyl with a taller narrower radiator, maybe its different on the 4.0L 6cyl. If your pulling the radiator, might as well kill two birds with one stone and do a flush and full coolant replacement on the system.
A new one made a huge improvement for me in vibe and noise. I got mine from a dealer that sells parts online, just over wholesale, there are bunches of them. I think I paid $36 for a new one, can't remember, it might have been around $60, either way, I could only find it through a dealer and mail ordering over the internet at one of those dealers is usually half the cost of most dealers.