UPDATED: The Great 40ish Landscape Test (10 Lenses in 40MP HR Mode)

Discussion in 'Native Lenses' started by kwalsh, Nov 6, 2015.

  1. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    UPDATE: See field curvature tests in a later post in this thread

    This is a mostly ridiculous exercise in extreme pixel peeping, but I was going to test two new lenses, so why not ten? I'm cross posting this from DPR as well, I know many mu-43 users can't stomach DPR so I'm directly posting here as well ;)

    The test is 10 lenses shot around 40mm or so all at F/5.6 near infinity focus with the pixel shift high resolution mode of the E-M5II. Tested lenses are:

    • 42.5/1.2
    • 42.5/1.7
    • 45/1.8
    • 12-32
    • 12-40/2.8
    • 14-45
    • 35-100/2.8
    • 40-150/2.8
    • 45-175
    • 45-200
    Most of the lenses are shot somewhere between 40 and 45 though of course the 12-32 had to be at 32.

    In all cases the camera was moved to preserve the same framing of subject, this is to make detail comparisons easier. All shots were focused at the center using MF with a 14x live view magnification and peaking. Again, exception is 12-32 which is a pain to MF so I just used AF on that one.

    As can be seen from the center crops it does not appear there were any focus errors, all the lenses are essentially identically sharp at the center. Which is no surprise, stopped down at F/5.6 they should be very similar in the center - it is the extreme corners where we expect differences.

    All were shot at F/5.6 1/1000 at ISO 200 on the E-M5II using the high res mode. The high res ORFs which are 64MP were brought into Lightroom and the "Camera Natural" calibration was used (this roughly matches in camera JPG processing as far as color goes) and chromatic aberration correction was turned on. Crops that were 1246x1246 were taken from the 64MP RAWs, exported and then made into a contact sheet with a built in action in PS. Note that in the contact sheet the effective resolution of the crops is now lower and roughly akin to what a crop from a 40MP file would be - in other words PS has resized the crops. This is fine as the 64MP RAW images are at way too high a resolution to begin with (which is why the camera does 40MP JPEGs instead). Really the exact parameters are mostly irrelevant, what matters is all the parameters are the same for a useful comparison.

    The first contact sheet are center crops from all 10 lenses with -0.5EV exposure adjustment and camera daylight WB. These show the variation in color and transmission for all the lenses.

    The second sheet are the same crops, accept now exposure and WB have been adjusted so they all look the same. This is how one would use the lenses in practice of course and now we can compare detail without noticing color and exposure differences so much.

    The third sheet is the interesting one, these are now the extreme lower left corner with the same settings as the second sheet. Any differences in exposure are now due to vignetting. Here is where we see significant differences in performance. Again, remember these are pixel peeping crops at around 40MP - so even if a lens looks "bad" here it might be more than acceptable at 16MP in typical use and probably looks perfect at normal print sizes.

    Also I checked all four corners on the lenses and there were no corners that were significantly different from the lower left corners, and this is usually true for most "good copy" lenses at F/5.6. I know the primes show some variation in corners wide open but that is again typical even for a "good copy". All that said, of course this test is subject to "copy variation" like any other. I can say pretty confidently none of the lenses are total duds (or I would have exchanged them long ago) but it is certainly possible there are better copies around. I can only test what I own!

    Lastly, since the topic came up in another thread, realize these tests are near infinity focus. Most test sites shoot small test charts at at very close focus. This "chart" is a four story building. Modern lens designs can show surprising changes in corner performance between close and infinity focus. So don't be surprised if these results don't match numbers on test sites. Also notice these are the extreme corners and because of how most MTF software works often the "corner" targets in those test charts are further from the corner. Many lenses drop off in performance very rapidly in the corners. If you typically crop to 2x3, 5x7 or 4x5 then the worst part of these extreme corners might not even be in your final image.

    Enough talk - here are the images, and of course you'll need to click to see the originals to conclude anything at all!

    Center Crops, all with Daylight WB and -0.5EV
    i-TmsW8Nw.

    Center Crops, WB and exposure matched to look the same
    i-SWzkMmw.

    Corner Crops, same settings as second image above
    i-q3vZf44.
     
    Last edited: Nov 11, 2015
    • Informative Informative x 7
    • Appreciate Appreciate x 6
    • Winner Winner x 5
    • Like Like x 1
  2. barry13

    barry13 Super Moderator; Photon Wrangler

    Mar 7, 2014
    Southern California
    Barry
    Wow, I'm surprised the 12-40mm did better in the corners than the 45mm and 40-150mm.

    Curious - Why did you choose 46mm instead of 40mm on the 40-150mm?

    @Amin Sabet@Amin Sabet, should this be moved to the HiRes forum?

    Barry
     
  3. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    I for some reason expected the 12-40 to be a bit weak at the 40 end, but boy it sure wasn't! The 45/1.8 has in my experience not been superb for stopped down landscape, it really shines in its portrait capacity instead. Still, certainly it has always been a bit better than most of the lower priced zooms in that range and is in no way a "bad" lens for landscape - especially at 16MP. But clearly if you are carrying the 12-40 no need to bring either the 45/1.8 or 42.5/1.7 expect for shooting wide open (which you'd never do for landscape anyway).

    The 40-150 was a bit of a surprise, though I had suspected it wouldn't necessarily be amazing. When it first came out I kept harassing early reviewers to test it at infinity. The few review sample photos I could find seemed to indicate it got weaker in the corners but it was really hard to tell by how much. The 12-40 is clearly better. Or of course we should be careful and say my 12-40 is clearly better than my 40-150! Still the 40-150 result doesn't seem to contradict the little I've seen in sample photos.

    The 42.5/1.2 is just jaw dropping though, shots at F/4 are just as sharp in the corners. Clear it is *the* "reference" lens for m43 - at least at infinity focus. And it isn't just resolution, look at the microcontrast in the marble pattern. Yowza!

    Almost more of a convenience rather than a choice. I shot it from about the same position as the other long zooms which all start at 45. The 35-100 is also shot around that focal length for similar reason. For most of the zooms I was fine with just tweaking the zoom rather than moving the tripod again.

    Worth noting that of course this test is in no way meant to be "fair" to all the lenses, some of them are being shot right at the limits of their zoom range where performance is often a bit worse than nearer the middle of the range.
     
    • Like Like x 3
  4. Klorenzo

    Klorenzo Mu-43 All-Pro

    Mar 10, 2014
    Lorenzo
    Thanks for the test. In the center crop there is a strange rusty stain in the bottom left only in a few pictures with different shapes. Do you have any idea what that could be? Reflections of something? An artifact from the hi-res shot?

    A little OT: there are a few of such artifacts in the Nocticron corner crop, on the flower/grass. Also the 14-45 has some "vibrations" in the center crop. Did you spot many of these when looking at the pictures?
     
  5. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    Reflections - of a tree I think. Depending on the exact focal length being used the camera was set at a slightly different distance from the target to preserve the same framing. So the angles for reflections changed between some shots. Most of the telephoto zooms and the 45/1.8 were shot from the same spot which happens to be where you see the largest reflection. The 12-40 and 12-32 were shot closer and there was no reflection from that location.

    Yes, these little diagonal artifacts are from high res. The most obvious are usually in the corner crops around the plants which occasionally moved in a gentle breeze. I do, however, often see them along other high contrast edges particularly in longer focal length shots outdoors. In these shots you do see them in the center crops on occasion along the edges of the marble blocks or even along the edges of the blinds in the windows. Obviously camera motion due to wind blowing the camera and tripod is a possible cause - but in this case there was very little wind and a pretty darn stable tripod. Plus the artifacts don't seem to be "global" as you'd expect from camera motion - instead they seem very patchy and local.

    I suspect what is happening is an effect normally seen more obviously at much longer focal lengths - scintillation from atmospheric mixing of different temperature air. This is the waviness you see over a road on a hot day. The ground gets hot because of solar radiation, it then warms the air close to it and this air rises slowly into the colder air above. The index of refraction of air changes with temperature and while these pockets and bubbles of warmer air rise slowly mixing with the cold air they distort the optical path just like little lenses. Once the warm air and cold air mix together to a uniform temperature the effect disappears - so usually the higher up from the ground you are the less of a problem it is and the closer to the ground the bigger a problem (put your head down on the sunny pavement and look along it to see how much more severe it is than when viewer from a standing position).

    I'm used to it causing softness in 16MP images shot at say 100 to 150mm when shooting over ground on a hot day. Often you can see the waviness in a 14x zoomed in live view when using long focal lengths. These shots were over a lawn taken near mid-day, not the case where you'd expect a significant problem at 45mm but we are looking at high res shots here so in many ways our 45mm shots are more like 90mm at 16MP when it comes to resolution. Usually these scintillation effects just turn into slight softness of the image - but high res mode turns minor "motion" blur into nasty diagonal artifacts. In the case of scintillation neither the subject or the camera moved - but the changing optical path created an apparent motion of the subject. I've discovered when testing longer lenses in high res that the high res mode is an amazing "scintillation detector".

    This would also potentially explain why all the center shots look almost the same as far as sharpness goes. It is possible some of the lenses would have looked sharper than others but if scintillation is effectively slightly blurring everything we wouldn't be able to see the difference.

    The scintillation problem is a real hassle with telephoto landscape. It is surprisingly difficult to predict in what situations it will be really bad or more manageable. Counter intuitively having it just a little bit windy (10mph or 15km/h) actually reduces the effect by causing more rapid turbulent mixing of the air while still air can make it much worse. I've found it essentially impossible to reliably test lenses over 150mm on targets just 1km away in an urban setting even when doing everything I could to minimize the effect (e.g. shooting out of a 5 story parking garage from the north side to ensure there is no solar load on anything and my optical path is 60 ft above the ground). This of course also means that as far as practical field use goes for distant telephoto landscape photography this is little point in carrying around a very high IQ long telephoto - just use a cheap one or crop a shorter focal length. Obviously for wildlife, sports and portraiture the shooting distance is much closer and so more reach is still practically useful. But in my experience shooting an even modestly distant landscape you'll see no advantage of a 400mm lens over a cropped 200mm.

    Here is a very extreme example in which the effective focal length is really long and the shot was intentionally setup to have about the worst imaginable scintillation. Of course in my test the scintillation was likely more than an order of magnitude smaller - but of course we are looking at it with nearly an order of magnitude higher magnification so we should not discount the effects of the atmosphere in outdoor tests of very high angular resolution optics.

    i-sZfCZ8F-L.
     
    • Informative Informative x 4
    • Like Like x 1
  6. SojiOkita

    SojiOkita Mu-43 Top Veteran

    619
    Feb 23, 2014
    France
    Thanks for sharing.

    The 12-40 results are impressive.
    The Nocticron is very impressive.
    The corners of the P42.5/1.7 are very good too and seem to be superior to the O45/1.8 (not a very big deal though for portraits shots).

    I'm quite surprised by the results of the 12-32, but I think 32 mm is not the best focal length for this zoom.

    I wonder how the P12-35 would compare to the 12-40.
    (there are 10 lenses compared, and yet somebody complains for a missing one ;) )
     
  7. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    Me too. I tested the 12-32 at closer focus last year at 16MP and it did show a bit of field curvature that would lead to softer corners when shot at 32mm - but this test looks worse than I expected. Maybe difference between closer and further focus along with the extreme pixel peeping of 40MP?

    I'd like to go back and test a few of these lenses again focusing in the corner to see how many of them are just showing field curvature.
     
  8. drd1135

    drd1135 Zen Snapshooter

    Mar 17, 2011
    Southwest Virginia
    Steve
    Rather pretty for artifacts.
     
    • Agree Agree x 1
  9. SojiOkita

    SojiOkita Mu-43 Top Veteran

    619
    Feb 23, 2014
    France
    I always found the 12-32 quite soft in the corners (partly due to the distorsion correction), but very sharp in the center.
    On your test I find the center crop not so sharp too.
    However, I always tested it at 12 mm and only with 16 Mpix. Maybe the limits of this zoom are reached at 40 Mpix.
     
  10. drd1135

    drd1135 Zen Snapshooter

    Mar 17, 2011
    Southwest Virginia
    Steve
    My compliments on some nice work.
     
  11. johnvanatta

    johnvanatta Mu-43 Regular

    181
    Aug 5, 2014
    Oakland, CA
    i appreciate the rigorous test, I feel we don't get enough of those here.

    My 40-150 doesn't wow me at the short end either.
     
  12. Benzy

    Benzy Mu-43 Regular

    156
    Mar 18, 2014
    Very nice post. The Nocticron is outstanding.

    This makes me wonder more about DXO. Their lens sharpness evaluation depends on sensor resolution, so comparing m43 lenses to Sony FE lenses (Or FE lenses on a a7 vs a6000 for that matter) are essentially pointless.
     
    • Agree Agree x 1
  13. Vandy

    Vandy Mu-43 Regular

    53
    Nov 8, 2015
    VA
    Marc
    Very good information, thanks a lot. Will have to look into the Nocticron, just bought the mzuiko 12-40, and have the 45 1.8. Wowsers, a little outta my budget. Nice results though. . .
     
  14. pdk42

    pdk42 One of the "Eh?" team

    Jan 11, 2013
    Leamington Spa, UK
    Thats a fantastic test - thanks for doing it. Would you be prepared to make the raws available so that we can do our own pixel peeking? I'm really interested to see how well the HiRes mode stacks up - esp since I'm looking to picking up an A7r!
     
  15. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    UPDATE: Field curvature and repeatability tests

    In the original tests there was the natural question "what if we focus in the corners?" We know of a few lenses that have field curvature and the corners get sharper if we focus on the corners rather than the center. And in some cases a nice happy medium can be found that makes the corners sharper without degrading the center.

    Same test setup before, but on a different day. Only five lenses shot this time, the ones I was curious about field curvature on along with the 12-40/2.8 which showed no obvious field curvature as a "reference" lens. Each lens was shot twice, once with focus on the center and once with focus in the corner. On the E-M5II you can only put the magnifier so far into the corner so in these tests the focus was roughly on the word "FOR".

    This also allows this testing to provide some insight into how repeatable the results are. For this test all the "center focus" shots should look the same as the earlier tests. In fact they do, though with a lot of staring I think I could claim some very subtle differences between some.

    One last thing to note in general. I watched the 14x magnified liveview for awhile on the corner through the 12-40/2.8 (the part of the image closest to the ground and thus with an optical path that follows the ground the longest) and I could definitely see very minor small motions likely because of scintillation as described in an earlier post. Nothing severe, but certainly at these resolutions could be the source of the small high-res artifacts we seen in some of the crops and also might explain why the lenses look so similar in the center - the atmosphere is limiting in that case.

    My observations:

    • 42.5/1.7 shows almost no perceivable difference when focused on the corner. BUT, when focusing in liveview at 14x it is clear that astigmatism is at play. Keep in mind focusing occurs at F/1.7 where such things are much more obvious, but the details in the image corner came into sharpest focus at slightly different positions depending on the orientation of the detail. I chose a "best" focus that was between those two points. We are talking very small focus movements and probably by F/5.6 the effect is gone.
    • 12-32 can't be manually focused well. Here we see no difference between the shots, but I honestly wonder if my AF selection was done properly. In other words there is a chance the "corner focus" sample was actually still "center focused".
    • 12-40/2.8 looks the same and tests as wonderfully as the earlier test. Yes, I refocused in the corner but it sure didn't look like it needed it in liveview - but I went ahead and moved the focus ring and refocused anyway. It is a bit reassuring to have shot this lens on two different days and focused three times and gotten essentially the same results every time. One thing I wonder about is the reliability of focusing at these resolutions in the field. The combination of 14x magnified liveview and peaking seems to make it reliable.
    • 40-150/2.8 also looks the same as the last test. Here focusing on the corner appears to have had a modest effect in improving the corner - but note the small focus change does appear to have made the center slightly worse. Hard to say if the marginal corner improvement was worth it. Presumably there are multiple aberrations at play in the corner and not just field curvature.
    • 45/1.8 is the most interesting. Again the center focus results seem to make the previous test. Now the corner was never that bad really, but look at the result when we focus on the corner - beautifully sharp! And better still the center really hasn't degraded noticeably either.
    Image corner for both center and corner focus
    i-3LkqVCZ.

    Image center for center and corner focus
    i-dPmSD58.
     
    • Appreciate Appreciate x 2
    • Funny Funny x 1
  16. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    I'd be happy to do that, but even compressed in a ZIP file the 10 shots are 500MB in total. My free Dropbox account is full enough it doesn't have space for that. If you've got some free and easy place I can stick them for you let me know and I'll put the file up.
     
  17. pdk42

    pdk42 One of the "Eh?" team

    Jan 11, 2013
    Leamington Spa, UK
    Yes, I can see the problem. Let me see if I can arrange something.
     
  18. barry13

    barry13 Super Moderator; Photon Wrangler

    Mar 7, 2014
    Southern California
    Barry
    @kwalsh@kwalsh, thanks for continuing to test...

    If you have a Google account, you can upload & share files on Google Drive; several GB are free.

    Otherwise, I could share them from my account for awhile, or I have a web server I could host them on for a month or two.

    Barry
     
  19. datagov

    datagov Mu-43 All-Pro

    Dec 2, 2012
    New York
    While I appreciate the effort involved in doing this comparison, a set of shots focused on windows and text on stone don't really tell us much about the capabilities of these lenses. A diverse landscape scene with depth of field, color, contrast, and lighting challenges would be much more interesting.
     
  20. kwalsh

    kwalsh Mu-43 Top Veteran

    775
    Mar 3, 2012
    Baltimore, MD
    Well, interesting in one sense I agree. Unfortunately such scenes are typically not good for checking for field flatness, de-centering and the like in a single shot. And re-framing for slightly different focal lengths is challenging as well on most landscape scenes unless you happen to take a picture of a cliff with a flat treeless foreground, which in the end would be nearly as featureless and "boring" as this building. Shooting in changing light is nearly impossible when doing 10 lenses so you are left with mid-day shooting to have any sort of sensible comparison.

    And that's the key point here - it is a comparison. I'm pretty confident that anyone who has shot landscapes much can infer the differences between these lenses easily with this target and extrapolate that to a given landscape in the field. I'd say it is glaringly obvious which lenses do better in the corners from this test.

    So while I agree I'd rather do a comparison on a pretty landscape the constraints of making comparison shots - especially comparisons of the four corners of the same lens to verify optics symmetry - make it nearly impossible. So that leaves you with having to at least start with a test like this to verify none of the optics have an issue to begin with and understand which lenses might need special techniques in the field (e.g. off center focusing).

    Now, having done this test I now know how the lenses behave and that none are obviously decentered. So I might have time at some point to try a subset of them in the field on an appropriate landscape scene. I sometimes have time to kill in mid-day on photography trips (last trip I shot nearly every lens I own on an IR body to kill some time, boy do aberrations become nasty when you shoot at wavelengths the lens wasn't designed for). Right now that'd probably be the 12-40/2.8, 35-100/2.8, 42.5/1.2, 42.5/1.7 and 45/1.8. The other five are "also-rans" in my book and not worth taking the time in the field to play with.

    So I completely agree, I like to "test like you shoot" and so since the point is to shoot landscapes that'd be an ideal target. Problem is in my experience it is a very poor first target to shoot because it isn't appropriate for initial evaluation of lenses - it is easy to miss things or draw false conclusions from a complicated landscape scene when testing a lens that isn't characterized ahead of time.
     
    Last edited: Nov 11, 2015
    • Agree Agree x 3