How to remove gear shifter from shift tower?

Anybody know how to remove the gear shifter (the long lever) from the shift tower? It looks like there are spring-loaded balls possibly that once in place would require some special tool to remove.

This is from my AX-4 to AX-15 swap. The tranny is in, but the junkyard tranny I got just came with the shift tower and the square knob on top. I have the entire handle from the ax-4 that should fit on if I can figure out how to get it off. I'd rather not replace the entire shift tower at this point even though that may have been a better option before the transmission was in.

Got it off using a pickle fork and a large hammer. It had rusted on there for over a decade.