The GF1 is a great camera. It's very similar to the Pen cameras but has an improved AF system.
The IQ is beautiful. The X1 is also a great camera but only if you can live with a fixed Focal Length.
Most shooters will have an X1 but also have another camera with interchangeable lenses or adjustable focal lengths. The discipline of a fixed focal length camera takes
commitment and patience to find your personal vision. This usually takes it's toll on the shooter after a period of time and the lure of different focal lengths takes over.
With either a Pen or the GF1 you'll be cutting your load and still getting the images you want AND have the options of different lenses.
Good luck in your quest.
With the right lens the GF1 & E-P1&2 can match DSLR's for IQ, no problem. They have the added bonus of being small, light and portable and I much prefer to use a camera of this type rather than lug big gear around wit me.
Pentax make a small DSLR(s) - (the Kx?), and their glass is (very) good and reasonably priced - you can also use all Pentax MF glass from years back - (M screw fitting included) - cheap and good off Ebay
For use as a main camera the lack of a view finder is a big negative for me and I do find the controls on the EP-1 and to a lessor extent the G1, not (nearly) as good as on the DSLRs, (I have a Nikon and a Pentax). Trying to frame with a LCD, particularly in good daylight is not a great experience
The physical size is a big positive for M43, but as your main camera, I personally would go for a G1 or it's latest replacement rather than the EP or GF
M43 is fine with the fixed 17mm and 20mm lenses (and 45mm), but once you get into telephoto territory I have found the available lens to be less satisfying
don't get me wrong, I like my M43 stuff - but they do have drawbacks which offset their size - "they are what they are"
I would take a serious look at the X1 (as I do too). Of course you need an OVF, that increases the format.
Do you really need long distance shots? Do you really need a zoom? Is IQ not the most important?
Perhaps you should wait until the end of September (Photokina).
I also own a Nikon D40 and and EP2. I can tell you that the D40 with the 35mm 1.8 lens is a LOT lighter than the EP2 with the Zuiko 50mm lens. Of course, I don't know what lenses you are using on your D40 but just a thought!