The smaller the AF box, the slower the AF. If the smallest AF box of the XT20 is noticeably smaller than the EM5, that would explain that. Try a similar AF box size in both cameras.
Regarding PDAF, note that the lenses you used are your typical wide/slight-tele zooms, and slow too, so the DOF...