Take it off and try it. It can be scary as hell if you have to stop fast. If you don't drive much, well you may not care. I actually have mine off right now and have had it off for about 2 months because I have been too lazy to drill out my broken swaybar bracket bolts so I can reattach the bolts. Driving on the highway can be hairy at times too. Overall, it kind of makes you drive all tense when you're around other cars.
Worst case scenario...a little puppy dog runs out in front of you while running from it's 10 year old owner, you swerve, loose control due to no sway, hit the dog, go off the road and hit the 10 year old, flip over and somehow survive and then you are sued and will never own anything more than a pair of shoelaces if you're lucky. Lol...hey, you asked for worst case scenario. I'm sure some more creative people than me will chime in and give you something even more "worst case"...