The panhard is what locates the rear. The control arms do virtually nothing for sideways (lateral) support. I looked at mine earlier and saw the same rub marks on the plastic. They're not quite as bad as yours, but my Tahoe's not quite as heavy as your XL and I may not huck it quite as hard as you do yours. I have the same tires with what looks to be the same amount of wear. I noticed what may have been some slight rubbing marks on the control arms, too.
The rear isn't exactly centered and it does shift sideways slightly as it travels up and down since the panhard swings in an arc. Here's a little video that shows it in action on a drawing:
The pivot points are opposite of how they are on our trucks (ours has the frame mount on the RH side), but it's all the same. You can see how the rear moves laterally just a hair. The axle moving inward, plus the body leaning, plus the tire sidewall rolling inward all lead to the tire contacting the inboard areas. Beefing up the control arms wouldn't do much at all for this. Beefing up the panhard is about all that could make the only real difference.