I have dropped my GR... severely. Once, the pocket on my shorts ripped open, the GR dropped down (while I was sprinting to catch a train, with my child in my arms), and I kicked the GR five meters across the pavement. Put it back in my (broken) pocket, and sent it flying again. It broke off an entire corner of the body, and shattered the screen protection glass. The camera continued shooting, though, and remained sharp. And we did catch the train.
Another time I had a bike accident. Sailed across the asphalt and landed rather uncomfortably on the curb's edge—with the GR's lens as my first point of contact, and 75 kg of squishy human flesh to drive it home. That made a sharp dent in the lens barrel, and a bigger one in my hip. The camera continued shooting, though, and remained sharp as ever.
The GR III, I have somewhat less confidence for. I dropped it badly once, and the lens front shield popped off (where it says the focal length). To its credit, the repair was affordable (€90), quick, and the camera continued to work without the shield. I dropped it several other times without issues.
From these experiences, I believe the lens assembly is very sturdily built, and perhaps more importantly, very safely stowed when retracted.