Odd ball idea, but have you considered a 4-cyl XJ? Sounds like you're going for all-out handling. The only major down-side of the 4.0 is it's weight. They weigh more then most V-8s, and worse yet, because of the length of the motor, a big part of that weight is ahead of the front axle.
Build up a 2.5, bump the compression, more aggressive cam, bigger TB, injectors, a decent exhaust,... probably can't get up to the 177- to 196HP the various 4.0s put out without blowing serious coin, but 160 is probably doable, and 160 HP with a couple hundred pounds cut off the nose might be quicker on a tight course.
Whatever you decide on, look for a NP-242 select-trac transfer case. The full-time 4wd system with the center differential should handle better then the part-time system.