Diagnosing a bad fuel pump?

I think my fuel pump is going bad, but not sure. I need some other opinions. When I first crank, I have no trouble. As I drive for a while, the engine starts running rough above about 3000 RPM (passing gear), and gradually works its way down to lower RPM. I also read somewhere that it could be the Catalytic converter. Is there a way to diagnose? What should I be looking for?

BTW the jeep has been sitting in one spot for about two years without starting.

I am new to jeeps, and am looking for ANY advice.

1996 XJ,4.0

i would change the fuel filter, cheap and easy enough its right along the frame rail, drivers side in front of the gas tank.

Definitely change the fuel filter first, which is cheap and easy. After that, if it still misbehaves, you can check fuel pressure at the fuel injection rail and see if it's up to spec.

But before going too far, you should also make sure no mice have nested in the air intake, or in the exhaust system. When you get it revving up, check to see whether there's good exhaust flow out the tailpipe. Mice are a pretty common problem with stored vehicles.