ChicksDigWagons has it pretty much right on. How important this is to you depends on the type of off-roading you will be doing, and also on whether you will drive the Jeep long distances to the off-road venues, tow it, or only wheel close to home.
The effect of a mismatch is bump steer. At rock crawling speeds this won't even be noticed, and even if you notice it won't have any effect on anything. If you're doing the Baja 1000, bump steer at high speed would not be "a good thing." If you drive long distances to get to your wheeling sites, again bump steer is not a good thing but if it isn't too severe you can probably learn to live with it. But if I had a vehicle with bump steer, especially one I did NOT drive daily so I was accustomed to reacting to it, I would drive at or below the speed limit so I wouldn't gat caught in an emergency situation and have the vehicle zig left when I really wanted to zag right.
P.S. It hasn't happened with my Cherokees, but I did screw up the front suspension of my autocrosser many years ago and had fairly pronounced bump steer. It was NOT a fun vehicle to drive at highway speeds -- I couldn't relax for a second without finding myself in a lane other than the one I was supposed to be driving in.