Spend as little time as necessary for the touristy stuff.
But take time to:
1. Walk the high line early in the morning and around sunset.
2. Rent a bicycle as Manhattan has become bike friendly and this a great way to get around and see a lot.
10. B&H and Adorama are worth a visit, though visits to these gear and gadget candy stores can get pretty costly.
11. And of course, a boat ride and visits to the Central Park (great on a bicycle) are a must do when in Manhattan.
Agree with almost all of these recommendations and will add.
1. I stayed near the High Line when I spent a month there so I sort of took it for granted as just part of Chelsea and the Meatpacking area, but it really is wonderful. And stop at the Chelsea Market when you're down around 16th or 17th.
2. You can actually join the "City Bike" program on a short term basis and just pick bikes up in one part of town and drop them off in any other part of town without having to worry about looking after or returning a rental to the same place. It's an enormously successful program that's been in place for less than a year. Great for locals and tourists alike. During peak commute times you might actually have to stand in line to wait for a bike to come in - that's how successful it's been - but most times of most days you won't have any trouble finding a bike at a zillion spots all over the city.
10. B&H is a must. Even if you're not all that into cameras, it's just such an amazing organizational feat and runs so incredibly efficiently that it's worth just buying some little doo-dad and following it through the process from display to checkout. And if you spend some time with one of the sales folks looking at camera models and buy one of those, so much the better. I think I'd stop into B&H on every visit even if I didn't care about photography. You could skip Adorama - they've got a great web presence, but in person its just another of several good NYC camera shops with good inventory. Nothing terribly special about the experience. If you need something and they have it, go for it, but as a tourist experience, there's nothing all that special about it - 17th St Photo is just a few blocks away and also very nice, Photo Village is a few more blocks up, and there used to be a Calumet nearby too (not sure about it's current status given the bankruptcy). All are good shops and it's nice to have them in such a compressed area. But B&H is worth an hour or two just based on what an amazing operation they run.
11. Central Park can't be stressed enough. It's the most amazing piece of highly URBAN open space I've ever seen anywhere. On a nice day in the Spring or Autumn, Central Park is a place you can easily spend the whole day. You'll feel like you're in the city because the skyline is obvious from everywhere and there will always be people around. But it's an incredible respite from the urban intensity of the rest of the city.
One other thing I'd add is even if you're middle aged and tend to go to bed pretty early most nights, stay up REALLY late at least once or twice just to experience a city that never really shuts down. I was last there in October with some old friends and there's just something about sitting outdoors in an Eastern European restaurant of some sort (actually, I think it may have been Ukrainian) in the East Village and slurping matzoh ball soup at 2 in the morning. With TONS of people walking by the whole time. It's simply the most alive city I've ever been too. Sometimes a bit overwhelmingly, but if you pick your spots, not really.
I'm going back for a few days in May and I'm already looking forward to it with a slight ache...