I'm with you, but the answer may be simpler than you imagine. Buy a S5 with the remarkably adept 20-60mm for landscape and portrait needs, and (as an example in my case) keep my GX8 with a P45-175, a PL15 and a P30 macro for their various case uses. Best of both worlds, bank not broken, etc etc. It's still a small bag to carry around.If I were going to go FF, this is almost definitely the camera I would get. But the big lenses, poor telephoto reach (vs m43), and the pain of starting a new lens mount is a real show-stopper for me (like others).
It's also not too expensive to then add a Sigma lens of whatever flavour you want to the S5 at some stage for either low light, macro or some reach. As I'd have the telephoto and macro options already in place with the GX8, I'd probably plump for the 14-24mm f/2.8 to get some UWA low-light capabilities.
These are the thoughts that I ponder on as I drive around on dark winter days, waiting for the sun to reappear. And they're all 'ifs', anyway. I'd be just as happy with some better glass for my GX8. Perhaps a PL60, a PL50-200 and an Oly60 (finally). With a TC or two.