I would buy the set right away, for the 28mm and the M mount: The 28 is a lovely lens and a very usable sensor in a very small package. Unlike the GR, the lens does not have to motor out of the body, making the camera quick to turn on (and more reliable mechanically, too). There is absolutely no issue with AF speed in good light, and in general the camera is extremely responsive when you use "snap focus" (a Ricoh specialty): If the camera does not focus quickly enough, you can full-press the shutter and the camera fires *that* instant, with the lens pre-set to the "snap focus distance". This distance is easily and quickly user-setable: Say, you are in a room and the action is roughly 15ft away – you just pre-set that (hold a button/turn a ring/takes about second with practice). If you have the time to let AF do its work, fine. If AF is too slow, you simply full-press and get a reasonably sharp image *that* instant. Given the inherent depth of field especially of the 28mm lens (remember: it's really only 19mm or thereabouts physically, so super-deep dof), this works really well. There is a good reason that Leica shooters of old used a similar technique, after all...
Also, don't forget manual focus, which is implemented really well here: There is an excellent little distance scale on the back screen, together with depth of field indication in real time, based on the chosen aperture. With even the smallest amount of practice, this setup is quicker than most AF systems in low light: You twist the lens barrel and you're done. If at any point you think you are too slow, just full-press the shutter and snap focus goes again. I've had great success with "decisive moment" type photography with this setup. I would argue it's faster than even my Pentax K3 in low light (because the camera doesn't have to measure anything at all, it's limited only by user experience/practice/technique).
The M mount is a superb piece of kit, with excellent IQ. ReidReviews continues to use the GXR M mount as a bench mark for APS-C M mount cameras. It beats the Leica T (a much newer camera, and much more expensive), when used with M mounts! I use it as a complement to a "real" Leica, and it is extremely useful: a) for macro (since you see through the lens, at working aperture), b) for tele (e.g. the crop factor means you get 200mm equivalent out of a Leica 135mm lens), and c) as a backup, should the M9 ever fail (which it never has, keeping fingers crossed). Thus, GXR M mount plus Leica is a set where each camera overcomes the other's short-comings really nicely.
The small zooms are useful if you can pack only a very small camera on a trip, but yeah, the quality is S95-like. They are good if you can live with that. The APS Zoom I once owned, but sold: To me, personally, it was too bulky, and somehow just not a very inspiring focal length/aperture/size/weight package (ymmv).
I would buy the set, sell the zooms you don't want, and be happy with one of the most intelligently designed camera systems ever to exist. It's a pleasure to use, phenomenally well thought-out, an ergonomic delight. Again, that's compared to a K3 and an M9, and I stand by that assessment with all my heart... I love the GXR!