Are camera comparisons fair?

Discussion in 'This or That?' started by Speedliner, Sep 9, 2015.

  1. Speedliner

    Speedliner Mu-43 All-Pro

    Mar 2, 2015
    Southern NJ, USA
    Real Name:
    No wisdom to disclose here. Just thinking that sometimes comparisons and reviews seem heavily biased against m43.

    I was just reading about the Nikon 7200 and the review was positive, but within the details were things like C-AF missed on occasion. Max FPS was less than 6. Buffer full in 14 images.

    Then you read in an e-m1 review that C-AF works best in the "slow" L mode at "just" 6fps and that the buffer fills in "only" 22 images.

    Then there is the fact that many well regarded enthusiast and entry level DSLRs don't teach e-m1 performance but are cast in the "superior" DSLR category.

    I know, it's just a review, always take with a grain of salt, but I remember making my buying/platform choice it took a LOT more research than it should have just trying to interpret the inconsistency between facts and opinions.

    I guess it doesn't matter, bug it bugs be sometimes.
    • Agree Agree x 1
  2. Ulfric M Douglas

    Ulfric M Douglas Mu-43 Hall of Famer

    Mar 6, 2010
    Your defensive feelings are natural if you feel a connection to (for example) your E-M1.
    Just remember it's not a person, not your dog.
    Reviewers carry bias and subconcious quirks and for the most part can be forgiven for the stuff that's squeezed into their 'conclusions' footnote.
    What bugs me is when they skip over or leave out fundamental features ... the latest was when some reviews completely failed to mention the E-M10MkII's focus-bracketing. Doh!
    • Agree Agree x 2
  3. InlawBiker

    InlawBiker Mu-43 Veteran

    Feb 1, 2012
    Seattle, WA
    Real Name:
    I think what's happening is, since just about any camera you can buy these days is pretty awesome the reviews are forced to discuss the minute differences that, in reality, don't matter very much.

    I've seen reviews test say, focus on a moving subject and compare the keeper rate. They're all getting a keeper in almost every frame, but the only thing left to do is point out that one of them got 98% while the other got 88%. The reality is we delete most of those photos and keep just one of them anyway.
  4. Carbonman

    Carbonman Mu-43 All-Pro

    Jul 10, 2014
    Vancouver BC
    Real Name:
    Reviewers aren't often as well schooled in photography and the craft of using a camera as in the manual focus, film camera era. There were a lot fewer reviewers then because there were only photo magazines for information and ideas on photography. The magazine publishing schedule gave reviewers months to conduct their reviews and hone their wording.
    Now, anyone that's built up an audience or plugs their page full of keywords for Google bots to catch can have a camera or lens review up and presented as the truth in hours. A reviewer that waits 2 or 3 weeks to publish online is behind the marketing curve. Expertise has become an uncertain benchmark.
    • Agree Agree x 6
  5. dwig

    dwig Mu-43 Top Veteran

    Jun 26, 2010
    Key West FL
    An perhaps more importantly, the magazines were well edited. Editorial review is totally lacking in all but a very very few modern web reviews. These days the vast majority of reviewers publish their own reviews without any oversight.
    • Agree Agree x 3
  6. nstelemark

    nstelemark Mu-43 Hall of Famer

    May 28, 2013
    Halifax, Nova Scotia Canada
    Real Name:
    I have always found this interesting, the 7200 vs the E-M1 comparison. A friend of mine shoots the 7100 and the indoor images are noisy and the buffer is small, but it is a Nikon! :biggrin:
    • Like Like x 1
  7. Lcrunyon

    Lcrunyon Mu-43 Top Veteran

    Jun 4, 2014
    Real Name:
    It bothers me too. There is a lot of that.

    Bias is inherent in the way people think, so it's kind of unavoidable to a point. There may be some reviewers who are purposefully skewing the record, but I would prefer to think the vast majority are well intentioned. The former will resist anything we say, as fanboys; but the latter can overcome their bias with a little eye-opening education.

    Just yesterday I read an article from a lifelong Nikon shooter describing why she liked the LX100 so much. She said she tried all the other mirrorless options, including m4/3, and was disappointed, but every benefit she cited from the LX100 was actually better on the E-M1, and many of the other OM-D models as well. At least she was praising something in the family.

    All we can do is reply, and hopefully they will respond well to our points, I guess...
  8. gr6825

    gr6825 Mu-43 Veteran

    Oct 10, 2012
    Fairness is in the eye of the beholder. Here we have a thread about unfairness, and it did not take long for an unfair anti-DSLR comment to pop up.
  9. pellicle

    pellicle Mu-43 All-Pro

    Feb 10, 2010
    Southport, OzTrailEYa
    Real Name:
    The only thing I pay attention to in reviews are the facts. So I still use dpreview where I can examine data like images, side by side. Specs I can interpret myself and I don't need to be spoon fed koolaid on what's awesome.

    as @Carbonman@Carbonman said above often they have limited expertise and knowledge. For instance I got sick of a reviewer at dpreview saying that Olympus were first to market with micro43 when Panasonic was. There was as I saw it a lot of pro Olympus bias in their writings. I regularly pipped him about it but by 2010 I had lost interest in reading their words anyway.

    Read the facts and make your own decisions , rely on your own judgement.
    • Agree Agree x 3
  10. gryphon1911

    gryphon1911 Mu-43 All-Pro

    Mar 13, 2014
    Central Ohio, USA
    Real Name:
    Almost every review of anything comes with a bias. Its actually quite natural. If someone doesn't like something about camera X, they may out right pan it and may not give it the just due it deserves.

    They can also over glorify camera X, if they do like it, often overlooking the faults of the camera because they find other parts of the design invaluable.

    End of the day, as the reader, we need to be able to read between the lines and comprehend what is a fact about the comparison and what is an opinion. What matters to me in a camera may not matter to someone else. What is a deal breaker to me might be a thing that someone else doesn't care about or use. What matters most is that you get the information about what does matter to you, in the context that you need it.

    Are they fair - probably not, but not much in life is truly fair.
    • Agree Agree x 1
  11. Speedliner

    Speedliner Mu-43 All-Pro

    Mar 2, 2015
    Southern NJ, USA
    Real Name:
    I expect some bias and understand it. It's when camera X shoots Yfps with Z keeper rate and camera A does same but the reviewers judge one great and one awful that annoys me.
  12. bassman

    bassman Mu-43 Top Veteran

    Apr 22, 2013
    New Jersey
    Real Name:
    The only way to approach a review is to know the reviewer and their biases in advance. At the risk of being flamed, I read KRs reviews and usually get something from them - but I know what to ignore and how to interpret what he says. The same goes for Thom Hogan, DPR, and all the others.
    • Agree Agree x 1
  13. Gary5

    Gary5 Mu-43 Veteran

    Jan 15, 2014
    I think the answer to different reviewers having different biases, priorities or frames of reference is to read a lot of reviews. Most reviews don't even mention some of my highest priorities, and some base their conclusions on things I don't care about. So I read more reviews until I feel confident enough to buy or decide not to buy.

    ...adding: A different kind of unfair comparison, I read a review this morning that compared E-M1 to A7R2. Unsurprisingly, the reviewer preferred the camera that's two years newer and currently costs three times as much. I thought most of it sounded reasonable, but I disagreed strongly with his predictions on future viability of the formats, and as usual, the reviewer didn't even mention some things that are my top priorities. But it was interesting and mostly cool to read.
    Last edited: Sep 10, 2015
  14. Clint

    Clint Mu-43 All-Pro

    Apr 22, 2013
    San Diego area, CA
    Real Name:
    I have found a lot in Rockwell's and Hogan's reviews - they have been doing it for while and have a large frame of reference, and actually pretty vocal about biases. @Carbonman@Carbonman and @dwig@dwig are right on target. And lthough I read DP Review I've seen too many problems with their comparison photos to now even give them any weight at all.

    When Olympus stopped the 4/3s line I was reviewing Nikon and Canon to see which way to go. What I found was so many that lacked capabilities to use the equipment in the best fashion yet they were writing reviews. Many don't even do their own test but parrot what they've heard. The other thing is many seem to have to say that one is better than another.

    However in photography, everything has a cost or trade off - weight, price, size, ease of use, etc. More often than not direct comparisons should not be made and many things are not necessarily better on one than the other. While Model A may be have better IQ (by a slight bit) than Model B, the size and weight of 'A' could easily offset the positive if it made the camera not as likely to be taken out to shoot by the user. My specific example was comparing D7000 images with a Nikon 80-200mm lens compared to a Olympus E-30 with the 50-200mm lens. While the Nikon had the edge on ISO, almost everything else was not better or worse, just different.

    And then there is the overblown hype, both positive and negative, on forums including this one. After the E-M5 had been out for a while I really wanted to move back to something smaller (I was shooting a Nikon 800 and D7100 with Pro lenses) and I still had the 12-60 and 50-200mm lenses. But trying to decipher what was tremendously bad or not about the E-M5 was hard to do. It took me a couple of months to make up my mind in big part because of this site. The focusing ability of the 4/3s lenses were misrepresented (which I did not find out about until I had one in hand) and items people had problems with, seemed to be large issues - which they probably were to a comparatively very small group of people. BTW - this explains some of post I've made on here that you may have read.

    All said and done, reviews are only worth the value the reader wants to place on them.
  15. Jonathan F/2

    Jonathan F/2 Mu-43 Hall of Famer

    Aug 10, 2011
    Los Angeles, CA
    I think the most hyped cameras are from Sony. It's as if every camera they release will make all other cameras obsolete. I find it funny when someone decides to ditch DSLRs, moves to mirrorless, gets caught in the Sony hype, gets an A7x mk.xx and then ends up with something ridiculously large like their 24-240mm lens. It's like you were better off sticking with DSLRs in the first place!
  16. piggsy

    piggsy Mu-43 Top Veteran

    Jun 2, 2014
    Brisbane, Australia
    For me the most telling thing about it is how from the A7 on everything they've made has simultaneously been the best thing since sliced bread, and also the solution to all the horrible problems nobody could possibly have ever put up with for a moment that were in their last model. After this cycles around a few times you start to think - y'know - there doesn't actually seem to be any ideal time to buy one of these.
  17. AcridSaint

    AcridSaint Mu-43 Regular

    Feb 25, 2014
    North Carolina
    Well, there's no apples to apples here, but I think that if you were trying to make a fair comparison, then it shouldn't be the D7200. The cameras are the same price, yes, but not intended to be in the same class. The E-M1 is the Olympus flagship camera for m43. If you were to compare it to the flagship Nikon camera, 6FPS might seem a little slow.
  18. TransientEye

    TransientEye Mu-43 Regular

    Sep 18, 2013
    Real Name:
    I think this is an understatement. I just blogged a 10k word comparative review of Capture One, based on my migration from Lightroom. It was hard work. The post itself is seeing fairly normal view rates - except for one short comparison made with the A7rII, which has generated more than a month's normal traffic in two days.

    It is pretty clear that if you seek advertising revenue, writing long considered pieces is not sensible. Write the shortest most sensationalist thing possible on any currently hot topic. Think Steve Huff's so-called reviews (he alway reminds me of those old 6am US TV advertisements for exercise equipment - buy this it will make you look amazing!).

    The financial incentives this creates are severely distorting all forms of traditional publishing.

    It is also a bit sad that people get so worked up about gear, yet no one seems to get anywhere as excited about photography.
    • Like Like x 1
    • Agree Agree x 1
  19. pellicle

    pellicle Mu-43 All-Pro

    Feb 10, 2010
    Southport, OzTrailEYa
    Real Name:
    agreed ... one needs to inspect the EXIF data and wonder if there is some other issue (like deliberate focus errors)

    they've also made it non-trivial to compare (say) a GF1 with a GX7
  20. Geoff3DMN

    Geoff3DMN Mu-43 Veteran

    Feb 29, 2012
    Castlemaine, Victoria, Australia
    Real Name:
    I respectfully disagree, I strongly suspect that the average shooter decides upon a budget then looks at the options within that budget. It would be uncommon IMO to look at options 1/3 of the budget.

    I speak as someone who does significant purchasing (and deciding of same) for my job so purchasing isn't something I'm unfamiliar with.

    So yes I think it's totally fair to compare the EM-1 with the D7200 (in fact I cross shopped the D7100 against my EM-1 as possible upgrades from my D7000 kit).

    YMMV :)
    • Like Like x 1
    • Agree Agree x 1