Along with Wheel Bearings having excess Stress put on them because of extending out the Leverage applied to them, Wheel Spacers double the amount of Lug Nuts that can come loose or need to be removed. They make twice as much that could go Wrong and the added Worry to check the Lug Nuts and the Nuts holding the Spacers on a bit more often than you normally would.
Also, you will put more Wear and Tear on your Steering Components. Any Offset (Spacers or/and Backspacing) will wear them out quicker. It's not really a big problem unless you put huge Tires on.
It's the excessive Weight of the Wheel/Tire/Spacer Combo coupled with the Offset that will wear out the Steering Components and Wheel Bearings quicker.
If it is a Rubbing problem in Turns, try putting a Washer or two onto the Steering BumpStop. IMHO