I'm in the "been there, done that" camp because I made a foray into Fuji land and really liked what I found; for the record, at one time, I owned a X-E3 with the 23mm f/2, the 27mm f/2.8, the 35mm f/1.4, the 90mm f/2 plus the initial reason I got the X-E3 as a kind of "zoomable Leica-like" camera, the 18-55mm f/2.8-4. I liked the experience as well as the images, but in the end, the system didn't offer enough over my

bodies (the E-M5 III and GX9) in terms of quality, let alone portability. Once I had moved to Nikon Z FX and found the Z 50 a really pleasant little complementary camera for EDC use, the X-E3 felt kind of redundant.
Anyhow, the Panasonic 15mm f/1.7 turned out to be easily the equal of the 23mm f/2 optically while being hardly any bigger than the 27mm f/2.8; it's also the fastest focusing of the three, and besides, I also own the 20mm f/1.7. The Panasonic 25mm f/1.4 II was a nice replacement for the 35mm f/1.4 (they have a lot in common), and both the Olympus 12-45mm f/4 and the lowly Nikon 16-50mm f/3.5-6.3 proved to be a match for or better than the 18-55mm (I've since "graduated" to the - adapted - Sigma 18-50mm f/2.8 on the Z 50; *that* lens is really fun, and very competent, too). Finally, the Z 85mm f/1.8 S and Z MC 105mm f/2.8 both optically match or surpass the 90mm f/2 which I nevertheless still consider one of the most desirable lenses in Fuji's catalogue. But: The Olympus 75mm f/1.8 would be all the reason I'd need to stick with

- even though I don't own that lens (the Z lenses again ...). The E-M5 III blows the X-E3 out of the water in terms of performance and features; the GX9, while not quite as strong a proposition, still puts one over the X-E3 by offering I.B.I.S. ...
My impression is that there's not enough between the two systems to make one of them truely superior. But the X-T5 is one heck of a camera for its size. So's the OM-1. I prefer smaller bodies, though - and would choose my E-M5 III over just about everything else in that category; that said, I enjoy shooting with the Z fc even more, but not because it's a superior camera. The APS-C body that really changed my mind was once again the Z 50 - because it's really a mini-me Z 6 minus the I.B.I.S.; this means it handles fantastically well even with bigger glass, something neither the X-E3 nor the E-M5 III were able to achieve for me, not even when using their respective additional grips. The Z 50 is considerably smaller than a G9 or OM-1 and also the X-T5, though there's surprisingly little between the latter two. All the other cameras mentioned are much more feature-complete and stronger performers than the Z 50 - but none balances size, handling and appealing results as well as the Z 50 and, by extension, the Z fc, though I have to admit that the latter's place is firmly established by its "gestalt" rather than its haptics for me. Apart from an M body, there's nothing I like pulling out of my bag more ... Anyhow, the E-M5 III remains my "marginal weather" EDC camera of choice, at least whenever I want a great zoom to go with it (the 12-45mm f/4 again).
So, YMMV. I guess the best thing would be to temporarily own both systems and use them side-by-side. Replacing one by the other could be too much of a gamble and in the end may cost you more than you think (as it was in my case - though I was able to mitigate my losses via part exchange in most cases).
M.