To get the most out of processing X-Trans files with Photoshop or Lightroom the main caveat seems to be unusual sharpening settings being required compared to other formats. Other than that I've had no problem getting satisfactory results in terms of color, sharpness, clarity, contrast, etc.
As it happens, I started photography with m4/3 equipment though I'm a mostly Panasonic shooter rather than Olympus. If you look at the album the Moraine Lake image I linked above is from [
Jasper & Banff], I actually had a Panasonic GH3 and the Fuji X-E1 for that trip. Terrible idea to bring two separate camera systems on a trip & I don't recommend it, btw. But in the end it did happen to give me a decent real world comparison between a 16MP sensor on both formats for travel and landscapes. Ultimately, either prints beautifully on my printer up to the maximum 13x19" size I can do here.
If you prefer weather sealed zooms I think m4/3 makes a logical choice if the smaller format doesn't bother you. It's small and light weight, there are optically excellent weather sealed zooms available across the entire range, and it's a very mature system. The main tradeoff is of course the sensor size and all that goes along with that if you like shallow DoF or need high ISO performance, etc.
In my opinion having fairly extensively used both, I don't believe Fuji or Olympus/Panasonic has a clear edge in the optics department. Both are very capable systems in terms of features and performance too. I suggest you make your choice based on what "feels" right to you. I've learned the hard way over a lot of bought and sold camera gear that the most important thing is always what inspires me to pick up a camera and shoot, rather than the spec sheets.