I tend to hate lens leashes for the lens cap, but I find them invaluable for the lens hoods, since I've taken my lens hoods off and put them back on many times, they don't always stay attached when I just have the camera slung over my shoulder.
In terms of keeping lens caps, there are few products out there to address this issue:
I bought the Lens Cap Strap holder during its initial kickstarter phase. I personally don't use it, because I've gone to narrower straps (Domke gripper) that it wouldn't fit on. There is also the issue whether the sizes match the lenses you use:
The Lens Cap Strap Holder
Hufa has a clip that holds the cap on your strap. I don't have any experience with this:
hufa Home
OPtech/USA has the fast cap:
OP/TECH USA Fast Cap and filter cap:
OP/TECH USA Filter Cap. If you carry a filter, you could store the lens cap on the filter or in the filter compartment in this holder:
OP/TECH USA Filter Pack
The alternative that I tend to use, is just buy a couple of generic lens caps with a center pinch in the appropriate size, and when you lose it, just get the next one. You don't have have a lens cap that advertises what brand of lens you shoot with.
For my smaller micro 4/3rds lenses (Panasonic 20mm, Olympus 14-42mm), I have 3rd party metal lens hoods, and I just keep the cap at home.