removing rust inside rear axle tube?

I recently picked up a used rear axle (8.25) from the scrap yard. I pulled it apart to replace the axle seals, and saw that there was a lot of surface rust on the inside of one of the axle tubes and on the corresponding driveshaft. I'm fairly sure this was from the way the axle was sitting in the yard, as only half of it was covered.
Is there a good way to remove the rust from inside the tube without disassembling too much? I imagine the rust will eventually will get into the bearings/etc. as the gear oil starts sloshing around, and I want to remove it before anything gets chewed up. Or I am overthinking it, and just don't worry about it?

Thanks!

Does the rust rub off on to your fingers or a rag when rubbed hard? If not than I think I would ignore it. Once it is coated in gear oil that will stop it. To clean it soak a rag in brake clean working it in and out with a broom handle or something like that. The axle can be sanded with 120-180 grit sand paper.
Ron

Yeah what kind of rust are we talking about

It's a fine loose rust, looks almost like rusty splinters. It rubs off easily; I cleaned off the axle with a rag and some alcohol. It seems like I need to clean out the tube, but I'm afraid of pushing it in the bearings.