Some roughness in a cylinder head (ports/chambers) is acceptable and even beneficial because it creates turbulence which promotes better mixing of the air/fuel mixture. Unless you see major casting flaws (think every 4.0 intake manifold you've ever seen. *roll eyes*) I'd say leave it alone. Worst case scenario, I'd say get get some buffing wheels and compound and polish the chambers to a mirror like shine. I've heard of some engine builders doing this, the theory being that this directs heat away from the chamber roof, reducing carbon fouling and heat that is transmitted to the coolant. I'd still say remove the valves and clean everything really well if you plan on doing this.
Gasket matching ports is a bit more involved. To do it properly, you need to make a template out of a piece of steel or aluminum and bolt it into place, torqued to spec, with the gasket installed and mark out the areas where material needs to be removed. This is because compressing the gasket will cause it deform a bit, which can cause you to remove too little or too much material.
Since you have an aluminum head, have you considered upping the compression ratio? The easiest way to do this that I know of would be custom-made pistons, but there's plenty of companies that make them. I know that's typically a engine rebuild procedure and 4.0's run forever beforw they need a rebuild, but with an aluminum head you should be able to run around 9.5:1 on regular, ol' low-octane pump gas, safely, and with no real drawbacks other than extra power...