Everything you ever wanted to know about the AW4 transmission:
https://www.naxja.org/forum/showthread.php?t=1053970
Not sure it was clear earlier, but the shifting is electronically controlled by the transmission computer (aka the TCU) based on inputs from the throttle position sensor (TPS), output speed of the transmission (rarely goes bad, usual symptom would be faling to upshift out of 1st), and to some extent the sensor that tells it where the shifter is (the NSS, also does the backup lights). Poor connections, blown trans computer fuse, failing sensors, bad grounds can all confuse the computer and make it shift inappropriately.
The TCU controls the shifting by means of two solenoids, and the torque converter lockup with a third solenoid. Bad solenoids usually manifest as shifting into or starting in the wrong gear and can be intermittent based on temperature. There is also a throttle position cable that often needs adjusted (often incorrectly called a kickdown cable) which controls the pressure in the transmission and hence the shift firmness.
So boiling all my rambing down, the good news for you is that downshifting when it shouldn't is usually not a mechanical problem. The electronic problems are usually easily diagnosed and remedied. Personally, I'm betting a bad TPS sensor if it's the original with 180k miles- you pressure wash the engine recently?