How DxO mark tricks you into being stupid

Discussion in 'Open Discussion' started by Lisandra, Nov 9, 2014.

  1. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    havent written anything polemic in a while...



    well...no one can actually trick you into being stupid, but you can certainly be tricked into saying something stupid. Case in point:



    "Zeiss 16-35mm FE gets DxOmarked: Best f/4.0 wide angle zoom on market."



    that was on my rss feed headline a couple of days back, and being that i wasn't too impressed with the native 24-70 f4, i took a look see with hopes of finding some native glass redeemeness (yes its a real word, you dont have to look it up). This is what i found



    Sony_FE_16-35__Canon_16-35__Nikkor_16-35
    Sony_FE_16-35__Canon_16-35__Nikkor_16-35.





    Ive always been a numbers gal, numbers tend not to lie and all. So that said it took around 15 minutes of absolute puzzlement to finally realize where he got his conclusion; the DxO overall score. It took me another good 10 minutes in trying to figure out how the sharper, more transmissive, less distorted, less vignetting, and lower aberrated nikon got a point less than the sony despite using the same pixel count. I gave up, it doesn't make sense.



    Speaking of which, isn't it terribly unfair to compare 36mp to 22? doesn't make sense, its borderline silly. At least not if your really trying to assess which lens is best. and before you go into the "does it matter" mode, yes it does. drop a pin here, we'll pick it up later.

    So i went into trying to make things more even by changing the cameras and here's what happened


    15750971692_8b5367a25d_b.



    my how things have changed haven't they?

    switching to similarly pixeled (but still higher) cameras on the sony and the nikon reveal that both those lenses fall behind the canon. in sonys case it churns out half the resolution despite de 2 million more extra pixels, and look at all that yummy aberration!! 3 times as much!!!!!

    So at this point youre probably yelling at your screen that "the A6000 is APS C!!!!! that's unfair you raging ball of hypocrisy!!!!!!!" well i hear ya, lets change that as well


    15563558519_ffd1c3837b_b.


    well that didn't do nikon any favors did it? and that's without forgetting that canon is still a full 4 million pixels behind both the sony and the nikon, and the nikon doesn't have a AA filter. And more to it personally I think canon sensors are mad cow crap. But this is as fair as it gets, and in that state the zeiss is far and away from being " the best f4 wide angle zoom on the market". Its actually the suckiest by a good margin. Sure an even fairer comparo could be made using the old 16 and 18mp sensors, but dxo didn't test the zeiss with the 16, and I doubt things would change. And I could get deeper into analysis using their charts and whatnot, but the sony only suffers more for it.



    Remember the pin I told you to set down earlier? go pick it up, ill wait...

    does it matter? well if you only using native then no, but why in the hell are you comparing other brands if youre only shooting native?? bragging rights??? internet warrior weaponry???? you're being a D bag, stop it.

    No, if you're looking into that its because maybe you're into the lovely world of mirrorless, and comparing brands is actually a good idea, and in that case yes, yes it matters. So if you believe DxO mark"s overall score or the post that was made about the zeiss on that "other" site, you might go "hey!!! my canon 16-35 is amazing!! are you telling me the zeiss is even better??" and poof before you know it your L is on ebay and you've charged your aching plastic fantastic card with the zeiss. Then, when it finally comes, you make a trip to that amazing place you're so fond of, with dreams of getting even more detail out of that amazing landscape, you shoot it oh so perfectly and when you get home...

    A MUCH softer image (yes, much) with more than twice the color aberrations that you had before!!!!

    but but! its the best f4 wide angle zoom on the market!!!!

    Its my copy bad????



    Nope, you've been DxO'd
     
    • Like Like x 17
  2. usayit

    usayit Mu-43 Hall of Famer

    LoL

    Every forum needs a DXO rant thread...
     
    • Like Like x 2
  3. OzRay

    OzRay Mu-43 Legend

    Jan 29, 2010
    South Gippsland, Australia
    Ray, not Oz
    And don't forget Imaging Resourced! :biggrin:
     
  4. pellicle

    pellicle Mu-43 All-Pro

    Feb 10, 2010
    Southport, OzTrailEYa
    pellicle
    DPeeved Review?

    Seriously all metric systems require you to grasp what they are doing and visualise that for it to make sense. At least DxO aren't spinning subjective yarns like some others are (yet)

    One thing that is not obvious in DxO is their not factoring in lens curvature of field. I suspect that this shows up on some lenses as reduced edge sharpness. They shoot a fixed target which is flat. Mainly only copy lenses are made to perform under such situations.

    I noticed that my Sigma 30mm f2.8 focused in steps which prevented me from getting exactly where I was focusing in focus. Such a point will reduce the PMpix values unless the camera is subtly moved to compensate (as do macro photographers).

    In real life flatness of field is mostly not critical nor is such accuracy of focus.

    Thus one needs to take the figures as being a bit 'representative' - sus the data across a variety of camera bodies and see if its a good lens for you.

    Unfortunately this is not like pulling out a micrometer and measuring a thickness. Its more complex :)
     
  5. fortwodriver

    fortwodriver Mu-43 Top Veteran

    959
    Nov 15, 2013
    Toronto, Ontario, Canada
    Frank
    Hey, it makes for great toilet reading when you can't get out and take photos... ;-)

    Was this one of those "I can't find my pills" kinda rants?
     
    • Like Like x 1
  6. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    indeed it does, and with good reason!!
     
  7. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    True. in all seriousness I dont hate dxo, or any other major site, its just that in the process of trying to over simplify things for the less knowledgeable and get more page hits, they end up giving people with heads up their butts the tools to say stupid sh*t. Sometimes said people have pages with thousands of viewers a day, and when they say "best wide angle ever" they do more damage than they think. Lens testing is a complex, difficult to understand thing, ive done it, its overwhelming. Dont try to simplify it, force people to educate themselves on the subject, especially if they wanna use it to argue on forums and such. Saying "the zeiss gets 26, and the nikon 25" is just irresponsible.
     
  8. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    hahaha it was actually a "ive just read 86 stupid posts on how the zeiss is amazing" kinda rant
     
  9. fortwodriver

    fortwodriver Mu-43 Top Veteran

    959
    Nov 15, 2013
    Toronto, Ontario, Canada
    Frank
    The Cynic in me says all of the content DxO has on their website does one thing: make click-through money... It's "information" and "advertising" and masses of arm-chair photographers who never take their cameras out of their houses sit around and debate lenses and resolution they'll never use. On the other hand, lots of "baked" pros use gear they're told to use. In the old days, courses mandated Hasselblad or Mamiya because the instructors told people to use those cameras. Today, instructors mostly use Nikon and Canon. You can't argue with sheeple.

    At my wedding, the photographer used Canon full-frame gear. She shot at relatively small apertures to ensure everyone was in focus for group shots and then manually photoshopped in blur where it worked. Works for me!

    Those of us who take photos, print photos, and mount them have learned that websites like DxO really don't mean anything for practical photography.

    But, like I've seen before, you seem to get awfully butt-hurt about these sites. They're just businesses - lots of people make money spreading data and statistics around. That's why we have statistics courses in university and college. I don't think you're really the people they're preaching too.

    I am a part-time professional musician... a bass player... which means I almost always have work in music.
    I use Yamaha basses... Lots of people like to crack that same joke that I play basses made by a motorcycle company...
    That's what they see "online" and that's what they believe. Then they come back and ask how I'm getting such great sound from these "basses made by a motorcycle company"... meh... idiots - but they're entitled to live too. ;-)
     
    • Like Like x 4
  10. pellicle

    pellicle Mu-43 All-Pro

    Feb 10, 2010
    Southport, OzTrailEYa
    pellicle
    Hi

    LOL ... indeed ... I have a small rant on my blog about exactly this. I call it "King Wang"

    coincidentially I had my own little rant about this (and involving DxO) just lastnight

    fudging figures

    anyway, the King is dead ... Long Live the King
    King Wang , King Wang, King Wang King ...
     
  11. fortwodriver

    fortwodriver Mu-43 Top Veteran

    959
    Nov 15, 2013
    Toronto, Ontario, Canada
    Frank

    Lol! Well, yes, and I do in fact have a Jena Zeiss lens which is absolute garbage.
    Again, arm-chair photographers who never actually use their gear often have no idea...
     
  12. Why is that? Because they don't score well on DXOMark, lol!
     
  13. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    hey man that fudging figures is a good read, and further proves the point!
     
  14. Lisandra

    Lisandra Mu-43 Veteran

    234
    Nov 16, 2010
    hahahahaha aaaaa

    seriously if canon ups the sensor game they would rule all!!! I cant help but feel disappointment with the 5d3 and the 7d2 (so many years for that!!!). Whne the 5D2 came out it was like BOOM!! greatest thing ever!! but the 5D3? meh at best. Its amazing to me that the best sensor out there currently is from samsung of all people...
     
  15. MAubrey

    MAubrey Photographer

    Jul 9, 2012
    Bellingham, WA
    Mike Aubrey
    Lisandra, you're 100% right.

    But you're also missing something. Look at the width of the blue bar that the DXO total score appears inside. It's much narrower for the Sony than it is for the Nikon. And that means that the Sony is more consistent across its zoom range than the Nikon is in the various metrics. The other numbers averages all apertures and focal lengths. And for zoom lenses, consistency is worth something. The Nikon is certainly better in many ways. But it isn't universally better.
     
  16. mattia

    mattia Mu-43 Hall of Famer

    May 3, 2012
    The Netherlands
    My take home from this is that all three lenses will perform quite admirably on their respective bodies. The samples shots I've seen from the Zeiss convince me it's sharp enough and performs well enough to be my widangle solution for the A7r, and likely replace the flaremongering 7-14 (which is great for landscapes with no sun in frame, but really quite terrible for places like churches).
     
  17. drd1135

    drd1135 Zen Snapshooter

    Mar 17, 2011
    Southwest Virginia
    Steve
    I don't need anyone to trick me into being stupid.:suspect2: I usually give Dxo scores +/- 5% error bars. Within that and its' usually a wash for sensors. My favorite lens site is photozone.

    Edit: Actually, my favorite lens site is mu43.com, although it takes more time than accepting a single number.
     
  18. Ulfric M Douglas

    Ulfric M Douglas Mu-43 Hall of Famer

    Mar 6, 2010
    Northumberland
    Lisandra did you get your redemption?
     
  19. Promit

    Promit Mu-43 All-Pro

    Jun 6, 2011
    Baltimore, MD
    Promit Roy
    Some comments on DXO scores: http://www.dslrbodies.com/newsviews/do-you-believe-in-dxomark.html

    It's been obvious for some time that DXO "overall" scores are total garbage. I don't think DXO is wrong in their subscores and measurements, but one needs to understand exactly what they measure, how they measure, what the error bars are, and what the measurements actually represent in real images. Unfortunately DXO doesn't do a great job of disclosing all of these things, and relatively few people dig into the measurements tab and actually understand what's going on.
     
    • Like Like x 1
  20. wjiang

    wjiang Mu-43 Hall of Famer

    I actually find the DXO detailed measurements to be quite informative. Agreed that the summary (and dependence on sensor) is rather silly though.