You don't have to make a note each time you go out - it's only when you first CHANGE a battery that you need to make a mental note (of which one or ones will need charging) and since you're taking one out and replacing it with the next number up, it should be easy enough to remember your starting point. I used this method when my main camera was the RX1, where I'd occasionally go through two batteries and start on a third during a heavy day of shooting. Now I'm mainly shooting with a DSLR and I've yet to have to change a battery while out shooting with it. I wouldn't cosider this a reason to switch, but it's a very nice thing not to have to think about. I just carry my one and only spare if I'm going out for a full day (for just an hour or two I don't worry about it) on the possibility that I may eventually go through one, but I can't imagine it happening...That addresses a concern I had with this approach (how do you know which batteries are empty if you switched from the highest numbered one back to #1 in the field).
I'm not sure if I'll remember to make a (mental) note of the in-camera battery number each time I go out, but other than that this approach sounds quite feasible. I'm going to give it a try.
As for this being a first world problem, just exactly which problem/issue/question we EVER discuss on a camera forum like this is NOT a first world problem???