This actually raises a good question - not in the least reinforced by the excellence of the J-3 you built for me: what exactly is the right frame of reference for comparisons in the digital age? Even among lenses that are already out there, it seems there is a lot of latitude that could be affecting comparisons in a way that makes one-off comparisons largely unreliable predictors of what happens when you buy the next sample of a pre-digital lens: factory variances, questionable service histories, mid-2000s re-collimations (pretty endemic with the rise of the digital Ms), factory-endorsed variances (the two major optimizations of a C-Sonnar), and the FSU units that have had the Sweeney lead-to-gold treatment.
The companion question is exactly what type of testing method would allow one to abstract from one example of a lens to all of them. I looked at the mossy-knoll test pictures for the various Sonnars, and now understanding that they were not shot with an EVF, that raises the question of whether the variation we are seeing is actually the lens or the lens' interaction with the camera's register and rangefinder (the former not apparently being identical between an M8, an M240, and an M246). They are all quite close. This is my personal feeling, but I would not make any life-altering conclusions about the subtleties of those test pictures without normalizing them by EVF focusing all of them on a high-contrast target in-frame. A closed focusing loop, I think, is critical for evaluating the optics. How you can reliably focus on a closed loop presents its own question, since with the M it still requires reliance on magnification or focus peaking.
It's certainly valid to test the system performance (focusing with the mechanical parts), provided that you have enough of a sample to establish that both the camera and lens are not fliers (being good or bad). But this, I suspect, should always be secondary to normalized optical tests, since it is possible to tweak the master focus, focal length, and RF cam of pretty much any lens in a way that substantially (if not completely) replicates the "best" performance of the glass, at least at some range of distances.
LTM adapters are also something that tend not to get normalized - I don't really having seen any test in the past 20 years in which someone factored the quality of the adapter into an evaluation of a lens. I have now measured out dozens of these, and even with a non-indexed 50mm lens, the ones on the thicker end can be capable of preventing proper infinity focus (as in not really focusing optically to infinity). Parallelism of the front and back faces also poses at least a theoretical factor to consider (I did not observe anything being off more than a few 1/10,000" around the circumference, but that is not to say that the threads are always drilled straight, either - I've now seen situations where two adapters with the same thickness behave differently).
I don't know if there is a concrete suggestion in all of this except that we have probably all been oversimplifying the fight between optical good and optical evil. I know I've been doing it.
Dante
