Thanks guys for all the straight up answers.
I finally got the XE, did some extensive thinking, and 90% of where I need to go will be gravel roads. So, I'm just gonna go with the common consensus and leave it be 2wd. The plan is to just do a 2" lift, a locker diff, a trans cooler, some better tires when the current ones wear out (has nearly brand new General Altimax tires), and fix up all suspension bits. And just going to use it to keep the wear off my CM Tundra down.