Very high idle at start-up and vibration.

Sometimes when I start my XJ it idles very high, 2,000-2500 RPM. Sometimes it will go away after a few seconds, others times after a few minutes of driving and sometimes not at all. Also, lately when I stop and start to go it vibrates kind of bad for a split second, but only in 1st gear. Same kind of vibration you would get when going too fast on a dirt road (with the evenly spaced bumps). I'm good at driving a manual and have never had problems before so I know it's not me. Could it be a Throttle Position Sensor? If so how easy would that be to replace? I have limited mechanical knowledge. I have been "priming" the fuel lines when starting it for a couple months as it takes very long to start it otherwise. I don't know if that has anything to do with the problems but I just thought I'd throw it in there.

Ideas? I need help as I know this could turn into a very serious situation (no control over fuel amount = bad)

The TPS is on the throttle body, it is at one end of the butterfly in the throttle body itself, it's held in with two torx screws and those are factory locktited with green locktite. How long since you cleaned the throttle body out ? Thats what I would do first, I just did my TJ's yesterday, took it off and brought it inside then went at it with cleaner and a brush. Idles much better now, that throttle body was filthy and to be honest thats the first time I've looked inside it since we got it about 5 years ago. The TPS can be a pain to get off, the screws face the firewall and those buggers are in there. The last two I did I ended up breaking the dam thing apart and removing it that way. Going to do my sons TJ once we get the right TPS, they sent the wrong one.
If you have a volt/ohm meter you can unplug the TPS and stick the meter on resistance then manually operate the butterfly while watching the meter [engine off of course] it should go up and down evenly with no dead spots or sudden jumps as you are moving it. All it is is a resistor, someone should have posted the values for it around here somewhere. VTVM's are cheap, $15 at radio shack, just get an analog one not a digital, the analog is easier to read.

The other thing is you might want to consider doing a tune up on it, new champion copper core plugs, spark plug wires, cap and rotor then see how it runs.

Usually a high idle is associated with a vacuum leak, or a sticky / malfunctioning IAC.