You are correct to be cautious, I hosed a 2.8L that way when the the pickup plugged at 70mph on the interstate after I cleaned the sludge out of the top end.
There are a couple of ways to do it, first is with an engine flush and oil change, this works the fastest but can leave a lot of flush soaked sludge in there that will break down between then and the next oil change. Some even use a quart of ATF or Kerosene as a flush, not my cup of tea.
Second method now that you have the cover off is to take a shop vac in one hand an a plastic scraping tool in the other and use them together, as you break the crud loose the vac wil suck the pieces up.
Third method is slower and more gentle, either use a full oil change of Mobil-1 or Castrol Syntec, both are very high detergent and will break down the sludge as you use them, couple that with keeping an eye on the dipstick so that when you can no longer read the hash marks [black oil] on the dipstick you can change the filter and add a quart. Second method and the third method work well together.
Fourth method is to take it to a shop that has a flushing machine where they pump solvent in thru an oil filter adapter with the engine running. It does a good job but again leaves a residue of cleaner that will have soaked into the heavy sludge, if you do that you really need to keep an eye on the oil color and change the filter. If you don't the filter will plug up and a bypass will kick in, this bypass will cycle unfiltered engine oil until you change the filter again.
Just for laughs I dropped my Mobil-1 for it's normal oil change and refilled with some mobil 10w30 normal oil and drove it 3 miles to my friends shop, he put it on the flushing machine with a new filter and ran it, came out clean, the filter was a very very light oil colored tan. I then refilled and put my usual Mobil-1 and Mobil-1 filter on [or K&N oil filter, whichever I can get] My 98XJ has over 192,000mi on it.
I run Mobil-1, after the 5K break in when new, always have and I can see how clean it is around the filler area. My wifes 97 Olds blew a water pump and overheated, warped the motor so we had to get a new rebuilt Jasper, when our mechanic pulled the valve covers on that V6 the valve train still looked new and it had over 160,000mi on it and like my XJ had only been run on Mobil-1 after break in.