The viewfinder on the X-E1 is a, if not the, major plus for me. I am not a fan of rear-screen focusing and actively dislike tilty screens to the extent that the one on my X-T1 is taped - to me, like a clip-on EVF it's just something to snag on your clothing, break and go wrong.
Remember that the original price point of the X-E1 was many beans more than the A or the M - that differential bought you more than just an EVF; from personal experience the X-E1 is more substantial, with a more comfortable grip which in turn makes it more usable with longer lenses. The X-M1 gets a mode dial in place of a shutter speed dial, and also loses the exposure compensation dial.
Overall I would take an X-E1 over an M or an A every time - the only advantages - for me at least - were the lower value and reduced bulk - having the EVF is essential - for me - on an interchangeable-lens serious compact.
Hope this helps!