If you're below around 6" lift or so, then yeah, you can take it out and you'll probably have a net gain in performance. I did this for a few years.
If you're above that, there are many more benefits to keeping it. Sidehill stability being my own reason. Take it off, try it, and if you don't like it then put it back on and pursue other options.