You should be able to do that. I removed the whole carpet (it's a bitch, but you might do that just to be safe). To remove it just pull the side trim (all of it), the center console, the seats, and muscle it up.
I'd say remove it all, fix the hole(s), and then herculne the whole thing. Good time to run wires to the back if you need to as well!
Sequoia