To improve performance think cold air in and hot air out. Removing the manifold covers increases the air temp under the hood and if you are pulling this hot air into the air intake than you will lose power. A modification of adding cold air intake helps to bypass this problem. You also see cars with thermal spacers that stop the transfer of conductive heat from the engine to the intake manifold. A hot intake manifold also heats up the incoming air and you loose power. On the exhaust side you want to keep the exhaust hot to keep the velocity up as it exits the tailpipe. As the exhaust cools, it expands and slows velocity and increases backpressure reducing power. Thus the header tape that someone mentioned earlier. The problem with header tape is that it can hold moisture against the header and cause premature corrosion to the pipes. My last car project car was a turbo and that's why intercoolers are installed to reduce the temp of the air going into the engine.
BTW having one side with a manifold cover on and the other side without is theoretically a very bad idea. Exhaust systems are designed for a venturi effect. Meaning the timing of the exhaust pulses out of the engine are timed and actually assist in pulling exhaust gases out of the cylinders as the engine fires. Uneven cooling of the exhaust gasses on both sides of the engine could upset this timing.
This is generic engine performance theory, I haven't messed with my pathfinder enough to see what (if any) difference it makes to our engines, but on some engines the difference is noticeable.
I personally have ceramic coated pacesetter headers that seem to work fine. I've had them for about 5 years so I can't really remember what it was like before with the stock manifold. I do remember that the headers did not fit properly and the installer had to customize the fit. Welding coated headers is not fun. He had to grind off the ceramic coating at the area of the welds.
One benifit that nobody has mentioned is that the aftermarket headers are significantly lighter that the stock manifolds. The lighter the vehicle the less power it takes to get you moving.