Gotta point out, though: the X100 sure is a lot prettier! Just sayin'....
I think that extra 1 cm in height and depth does make quite a difference (the X100 has the extra 1cm in width but it seems less important—or maybe I just truly want to return my OM-D for an X100S!) Put a 17mm or 20mm pancake on the OM-D and I think it's even more.
To cherrywood yes the X-E1 is smaller than the OM-D in two dimensions but the lenses so far are quite bigger.