I own the E-P1, E-PL1s (Japan exclusive obtained via eBay), and E-P3. I used to have the E-PL1 but recently traded it. Honestly, unless you spend all day pixel peeping there is virtually zero difference between the way the current Olympus Pens render images. The E-P1 and E-P2 are "a little" softer since they have stronger AA filters on the sensors, but I really can't see much difference in image quality in practical use.
The E-P3 (and E-PL3 and E-PM1) have faster AF and faster art filter processing than the older Pens, but I rarely use the in-camera art filters and the slowest focusing Pen (the original E-P1) is just as fast as the E-P2, E-PL1, E-PL1s and E-PL2 after you update the E-P1 with the latest firmware. Lenses can make a big difference, though (the older 14-42mm kit lens is slower than the newest 14-42mm IIR kit lens).
Honestly, I tend to use the E-P3 and E-P1 more than my E-PL1s just because they handle in much the same way so I don't have to give much thought to using the cameras. If I was regularly switching between the E-PL1 and E-PM1 I might find myself stopping to remember how I change settings on one camera versus the other.
Getting back to the image quality, here are images taken with the E-P1, E-PL1, E-PL1s, and E-P3. Other than using different lenses, I can't really tell which camera is "better" in terms of how the sensor renders the image.