Rough Start

I have an 01 XJ that is having starting issues. when I go to start it SOMETIMES it will just keep cranking without firing for a few seconds too long and then fire with a rough idle. If I stop cranking it before it fires and try again it fires with NO rough idle. It's a little embarrassing and annoying to constantly deal with this so I'm trying to figure it out. Any help would be appreciated!

2001 XJ 4.0

Does it when stone cold--sitting overnight; or does it happen when hot--driven, shut off a short time, hot restart?