You dont have to remove the panel if your just changing the switch. But if your changing the mirrors then you do have to remove the panel because the wiring from the mirrors is held onto the inside of the door by 3-4 stubborn ass clips
+1
I just replaced my drivers mirror since I destroyed my old one backing into the garage.