I am at 3.5" lift, cut out fenders, stock axles (5-speed so 3.07s), with 33x12.50s and I love it. Yeah its a bit of a dog, but its something I gotta deal with until i can find a nice set of axles. I don't think it rides to high or anything, but then again it also depends what your family is like. If your family is young, small kids, etc... maybe stick with 3.5" and 31 or 32. If you got older kids go for the bigger lift. If you do get the 4.5 you may have to deal with getting a SYE, 3.5 should be fine w/o, I never even had to use the transfer case lowering blocks they supplied me. No driveshaft vibs at all... every rig is different in that some will vibrate with 2" of lift, some with 4"-5". That all may be stuff you already know from your YJ, but anyway, What year is this XJ? If its 97 (I believe) or newer you will run into driveline vibs with less lift than you would with an older XJ from what I have heard.
Edit: Also remember, if you go 4.5" you are stuck with 4.5" springs, etc... go with 3.5" and you can always get a spacer or ACOS system for up front to get an extra inch or 2, and a shackel for the rear, make up that extra inch at a later date.
Its much easier (cheaper) to go up an extra inch than down an inch. Thats why i got the 3.5"... now that my XJ isnt my DD I plan on going to 4.5" or 5".
Overall, I would say go with the 3.5" and work from there, if you want more, its not hard to get more. Plus you can always trim like I did and fit up to 33s.