Check the hole, see if it's threaded. If it is, it will be either 3/8NPT or 1/2"NPT (I can never remember which) and that means you can buy a brass pipe \plug to put in there (for about a buck and a half.) Why brass? It won't rust, and it will last for-damn-near-ever.
I also quit using Teflon tape a while ago - if the threads don't seal on their own (which is why pipe threads are tapered) I use Teflon pipe dope instead. That way, I don't have to remember which way it winds, and I don't have it unrolling across the floor if I drop it. I also have more control over how much goes on.
Anyhow - the rubber plugs are to be avoided if you have the option. They were used not because they were reliable, but because they are cheap (the bane of mass-production.) You've got it now - make it reliable.
5-90