For the most part bearings aren't hardened all the way through. They are surface hardened and get softer as they wear. Depends on the manufacturer and even different batches from the same manufacturer as to how durable the hardening is.
Taking the weight off the bearing and spinning it to listen/feel for roughness may or may not tell you much. It may be in the beginning stages of complete failure and not rough enough to rumble without the weight on it.
Any play in the U-joints and they are failing. I usually pick up failing U-joints by the rust dust on the rubber U-joint cap seal. The roller bearings have gone dry and total failure is imminent.
I usually pick up failing hubs by taking the weight off the tire and grabbing the tire on opposite edges and wiggling it slowly (top and bottom works best). A little play in the hub bearings turns into a lot of play at the tire edges.
Bad pinion bearings often show up as a leaky seal first.
Carrier bearings often have a totally different tone, in forward and reverse. Same with pinion bearings, but with pinion bearings you can often feel kind of a stick slip feeling as the pinion doesn't mesh well with the ring gear with excessive wear. I usually pry up on the entire carrier with a rag and big pry bar, you can feel the play in the carrier bearings.
You are likely better off swapping in new carrier bearings than swapping an axle in. I've seen carrier bearings fail with as little as 130-160,000 miles. Hard to find a used axle assembly with less miles than this anymore in a junk yard.
Don't discount the spider gears and thrust washers. I've seen thrust washers really worn down and the spider gears get really sloppy. Heat damage is always suspect, once bearings (or gears) wear through the surface hardening they tend to overheat and it kind of cascades to total failure eventually.
Sometimes a little noise from well worn parts really doesn't mean a whole lot, I've driven many thousands of miles on differential bearings that were noisy. I always do a serious inspection though, just to know exactly what I have. Whether it is normal wear and tear or some specific part in the process of meltdown. When you drain the oil save it in a clean pan and pour it through a paint filter, just to see what kind of metal bits and pieces you have in there. Depends on the differential, but many have kind of a depression under the pinion inside the carrier housing. The depression holds oil and metal debris, a good spot to look if you suspect a failing pinion bearing.
Visually inspecting bearings and races is a good way to tell what you have, but if you take it apart to inspect it, chances are you will want to replace it. Blue or black spots or even straw colored patches are an indication of overheating. After the bearings have worn through the surface hardening you can often see kind of a grain in the metal, or spalling (flakes) and usually means the bearing is in all out failure mode.