You are right, we have to replace the whole panel.I think from looking it is the plastic retainer on the door panel side for that piece, where it attaches on the posterior portion to the door panel. I have that problem on both sides, you have to replace the panel. <sigh> There's not a replaceable clip there, it's a plastic piece that's molded in - unless you want to try to fab something, which would be difficult, as you can't permanently attach it. At least I don't think so.
<edit> I bought a pair of door panels on Egay, didn't realize until I got them that they had those plastic arms that hold them to the metal doorskin broken, too. What a waste of $200. Those door panels are a PITA.