I haven't done it myself but need to, from things I've read on the forum you need to take the inner panel off first.
removing door panel
Also I copied this from a post some time in the past:
Step #1 With a flat screwdriver, gently pry the speaker cover off. It helps to cover the screwdriver(s) with masking tape to prevent scratching. Remove the three #1 Phillips head screws around the speaker. These are deeply recessed, and are not the three screws that hold in the speaker.
Step #2 Pry open the small black screw cover near the chrome door handle. Pry from the front, this cover is hinged in back(I think!). Remove the black screw with a #00 Phillips.
Step #3 Gently pry the top & bottom black plastic covers off of the door pull (these pull off perpendicularly from the door, and are probably the hardest part of the job). Remove the screws underneath with a #1 Phillips. These screws have a larger head; don't mix them with the speaker screws.
Step #4 Remove the triangular piece covering the outside mirror anchors by grabbing it with your fingers near the window channel and pull it perpendicularly towards you.
Step #6 Sit down in the seat, place your feet on aluminum door bottom to brace the door. Starting at the bottom outside plastic edge, pull perpendicularly while trying to get your fingers between the plastic and the aluminum. There are six "pop-pins" that need to be released, two near the hinge, two on the bottom near the armrest and door pull, and two on the outside edge.
Step #7 When all six "pop-pins" are released raise the panel up about two inches on the rear outside edge. This is to partially release the window channel
Step #8 Gently pull the plastic assembly that holds the window switch and chrome handle from the back, towards the rear, twisting slightly. This releases the metal tab that holds this assembly in place. Because the switch and chrome handle stay with the door, it must pass through the hole in the panel in which it mounts.
Step #9 Lift the panel straight up to release it from the window channel.
Step #10 Release the speaker wire lock and remove the speaker wires.