That closely reclects my own sentiments. Years ago I hiked with just a P&S in my pocket. IQ was adequate but not great. Then I discovered ILC cameras and bigger sensors. Not sure that was a positive development. I have at times considered selling all and getting an X100T or F as my only camera, but I'm not sure I can go backward now.
Yes, it's the 'not going backward' which the issue. Having seen what really good gear can do, but also experiencing what a pain in the bum it can be to travel with, I'm loathe to take my 5D Mark II, even though it has image quality and versatility beyond most of my other cameras. For some years, a DSLR was in my bag whenever I went out, even to the shops for groceries, but when micro four thirds was announced, it seemed like the answer to my wants. It wasn't until years later that the Olympus E-M5 that m43 seemed to fulfill its promise of high quality stills and video in a package smaller than a DSLR. Having said that, it doesn't have the IQ of the better aps-c cameras, let alone full frame. m43 is currently in my acceptable range for image quality, though.
In my gear list, I'm attempting to cover 21mm to at least 90mm so that I can shoot everything from landscapes in daylight to portraits in low light. Size is a factor, as some of my most enjoyable trips have been with just a couple of small cameras. Another option is the Leica M9 and Panasonic LX7, so I get high quality stills and an all round snapshot and video camera, but from experience, the difference in IQ would bug me. The GM1 with an ultrawide zoom like the Panasonic 7-14 or the Olympus 9-18 (thanks jssaraiva) could be a reasonable substitute for the LX7, but then we get back into larger footprint.
Constructing gear lists is a pasttime of mine, as you can tell, haha.