Ouch...

Discussion in 'Micro 4/3 News and Rumors' started by alex g, Jul 15, 2017.

  1. alex g

    alex g Mu-43 All-Pro Subscribing Member

    Mar 30, 2016
    New York / Bath
    An unfortunate Panasonic QC failure undermines a new Lenstip review of the PL 12/1.4. :doh::shakehead:

    I guess at least they're being honest and not picking out good copies to send to reviewers...
     
  2. wjiang

    wjiang Mu-43 Hall of Famer Subscribing Member

    That's kind of terrible from Lenstip... Publishing summaries, conclusions and recommendations based on a clearly flawed copy doesn't really help their readers... yes we want to know that there may be QC issues but that happens with every lens... we want to know what a premium lens is capable of when we get a properly made copy, not some copy that somehow failed QC or got misaligned during transport...
     
    • Agree Agree x 5
  3. alex g

    alex g Mu-43 All-Pro Subscribing Member

    Mar 30, 2016
    New York / Bath
    I agree with you in principle, and I would like to see a review of a non-flawed copy. But I do feel that considering that flawed copy could potentially have gone to anyone, not just a reviewer, Panasonic do to some extent deserve to get a poor review, if only to encourage them (and other companies — they're not unique in this respect, obviously) to improve their quality control standards. It's all very well saying that it doesn't really count because it's a flawed copy, but there shouldn't be such badly flawed copies in circulation in the first place, certainly not at that kind of price. Personally, I think it's appropriate for a review site such as lenstip to apply the same rules to each item that comes through their door, and if the supplier sends them a dud, then they can only blame themselves if they get a stinky review.

    I hear what you're saying, but how would a reviewer decide whether a copy is flawed or not in your scheme? I think it would put the reviewer in a difficult position because they would then have to be making value judgements based on "reasonable expectations of performance" or something. In this particular case, it's fairly obvious that there's something amiss, but in others it might not be so clear cut.

    Just my personal view... :)
     
    • Agree Agree x 5
  4. VSTR61

    VSTR61 Mu-43 Regular

    100
    May 15, 2017
    Perth, Western Australia
    Vic
    I'm kind of glad they got a bad copy. Nobody would care if it had been me!

    As a buyer of an expensive lens, I would like to know all the facts and would likely read many reviews before buying it. If their QC can be dodgy, I would like to know this too. Everyone from Leica down is paying a lot to have that name on the front of the lens and I wouldn't be swayed by just this one review. However, if this happened to other reviewers...
     
    • Agree Agree x 1
  5. johnvanatta

    johnvanatta Mu-43 Veteran Subscribing Member

    220
    Aug 5, 2014
    Oakland, CA
    I'd wager most buyers receiving the lens they reviewed wouldn't have even noticed they got a bad copy. The equipment far exceeds the output for so many now...How many of the sample shots would an average buyer flag if they didn't know what to look for?

    I wouldn't be at all surprised if the lens wasn't an outlier that failed QC or was damaged. If I scrutinize enough, I can see a lot of lens issues: modern sensors are pretty demanding. I've identified flaws in about half the lenses I've bought new recently.
     
  6. wjiang

    wjiang Mu-43 Hall of Famer Subscribing Member

    Upon getting decentering, I would publish that I had a bad copy but contact the manufacturer to get an explanation. Finish the review based on a properly QC'd sample, but still mention in the intro and conclusion to beware of QC issues.
     
    • Agree Agree x 1
  7. saladin

    saladin Mu-43 Top Veteran

    814
    May 29, 2015
    jason
    Lenstip are a less than effusive review site at the best of times. A decentred version was never going to end well.

    But It is of some concern just how many reports of decentering are emerging for the 12PL and 8-18PL. Are we as consumers just looking for it more these days? Either way, Panny have a problem as these are flagship lenses and i dont imagine Leica will be happy.
     
  8. pdk42

    pdk42 One of the "Eh?" team

    Jan 11, 2013
    Leamington Spa, UK
    This isn't an isolated report of poor QA with the recent Panasonic Leica high-end lenses. The 8-18, 12/1.4 and 12-60/2.8-4 all seem to carry the "copy variation " stigma now. This isn't something the Oly 12-100, 300/4 or 25/1.2 (to name recent releases) seem to have acquired so I'd say that Panasonic has a manufacturing problem. I'm sort of interested in the 8-18, but right now I'm hesitating because of reviews like this and similar experiences from early buyers.
     
    Last edited: Jul 16, 2017
    • Agree Agree x 4
    • Like Like x 1
  9. christofp

    christofp Mu-43 Regular

    160
    Jul 21, 2012
    While I don't like QC issues, I have to say that LensTip has a little history on finishing tests in a questionable way.

    I remember the PL100-400mm test, it was severely flawed by EPM1's shutter shock but they finished the test and blamed the lens for the bad resolution without even mentioning the words "shutter shock".

    A better idea would have been to either mention the issue or to restart the test using a more recent camera with a shutter better suiting a 400mm tele lens. Imaging ressource was smarter with their Oly300/f4 test, they mentioned SS and presented a little comparison with both results (flawed and flawless).
     
    • Agree Agree x 2
  10. kwalsh

    kwalsh Mu-43 Top Veteran Subscribing Member

    812
    Mar 3, 2012
    Baltimore, MD
    It was worse than that. It was pointed out they had shutter shock and then instead of taking one picture that would have proven the point one way or the other they waved they waved their hands and made BS arguments as to why SS couldn't be a problem.

    And then someone else proved they had been impacted by SS and their response was total silence. Their completely invalid and bogus 100-400 review still sits on their site unedited.
     
    • Agree Agree x 1
    • Wow Wow x 1
  11. christofp

    christofp Mu-43 Regular

    160
    Jul 21, 2012
    I have to agree, unfortunately.

    I did not hesitate to buy a used 12-100 and in fact it was flawless.

    At the same time, caused by negative rumours, I would not buy the PL8-18 without a proper return policy.

    Anyhow, I was positively surprised to find a flawless 8-18 at my brick and mortar store where I could compare the lens against my Oly 7-14 (I bought the PL8-18 immediately ...).
     
    • Like Like x 2
  12. tkbslc

    tkbslc Mu-43 Hall of Famer

    That's why you have to read a handful of reviews. If you have multiple sites with similar results, then obviously there is a problem. If only one site has an issue, then it is likely an error or bad copy.

    I really like it when Roger at LensRentals does reviews, because he has 10+ copies of every lens for his rental biz. Everyone else seems to test just one copy.
     
    • Like Like x 1
  13. Giiba

    Giiba Something to someone somewhere

    714
    Aug 19, 2016
    New Westminster, BC
    Where is this proof? None of their images display any kind of motion blur, just very soft... they may well have gotten a poor copy but that just brings us back to the issue of the thread.

    Lenstip only review a single copy of any lens so sample variation isn't controlled for. The copy of the 12/1.4 they got had a clear decentering problem, and while this may not be all lenses it does point to a quality control problem somewhere in their manufacturing chain.

    Consider, the lenses are provided by Panasonic to them for testing. We can at least say Panasonic doesn't screen the lenses they send for review.
     
  14. alex g

    alex g Mu-43 All-Pro Subscribing Member

    Mar 30, 2016
    New York / Bath
    Yes, I'd be interested in seeing this proof too. Do you have a link?
     
  15. kwalsh

    kwalsh Mu-43 Top Veteran Subscribing Member

    812
    Mar 3, 2012
    Baltimore, MD
    Re: Read well: Micro Four Thirds Talk Forum: Digital Photography Review

    That's the most concise post showing the clear evidence, but there is other data spread through a few different DPR threads.

    SS isn't always motion blur, slightly soft is exactly how it looks in most cases.

    They could have gotten a poor copy, but the reality is that Cris was able to reproduce their exact shutter speed range they test at and show clearly there are dramatic effects of shutter shock on that particular lens that also perfectly overlap with the focal range they claimed was soft.

    LensTip steadfastly refused to do the one test that would show one way or another whether the issue was SS or not for their copy - take a single resolution chart shot with a strobe. That would have answered it, instead they just spouted non-sense which was trivially demonstrated to be false. Had they cared at all about their audience rather than their egos they could have spent five minutes testing to put the issue to rest and then updated their review with a very helpful warning to their users that the lens is extremely sensitive to SS at the long end.

    Now as you point out there is the whole issue of single copy testing - which seems to be the 12/1.4 issue for sure. But if you have demonstrated over confidence in your test methods and fail to take an investigative approach to your testing you'll end up with garbage in-garbage out results just like they did with the 100-400.

    As someone else mentioned - this is why you can really never depend on just one test site. Besides the individual copies problem you have the fact that people just make mistakes and not all of them are mature enough to be capable of recognizing their errors when clearly demonstrated and then taking the next step to correct them.
     
  16. DanS

    DanS Mu-43 Top Veteran

    822
    Mar 8, 2016
    Central IL
    I agree! He also seems to approach his testing in a much more methodical and analytical way.
     
    • Like Like x 1
    • Agree Agree x 1
  17. alex g

    alex g Mu-43 All-Pro Subscribing Member

    Mar 30, 2016
    New York / Bath
    Hrm, for me the jury's out on this one. I have absolutely no axe to grind with regard to Lenstip — I wouldn't claim that their testing procedure is necessarily perfect, and I agree with you that an investigative approach to such matters is always the most productive one — but I'm not sure whether the counter-evidence I've seen is what I'd call "conclusive proof" that their results were totally bogus. Personally, I'm a little doubtful about how accurately one can make deductions about precise test conditions by extrapolating from evidence acquired elsewhere — I'm not discounting the results you linked to, but I find the conclusions slightly tenuous. I think that Lenstip's point that the 300/4 performed significantly better than the 100-400 under what they claim to have been identical test conditions is a reasonably valid one, especially when one considers that the 300/4 is itself sensitive to shutter-shock (as reported by Imaging Resource), as indeed any lightweight 600mm+ equivalent system is likely to be.

    I agree that Lenstip does sometimes project a slightly dogmatic and entrenched attitude, which is unfortunate, but not altogether surprising from a company with over ten years of experience. As I mentioned earlier, I'm not trying to defend their stance, but given their generally methodical and even-handed approach to testing coupled with their breadth of experience, I find it hard to believe that they would be both unfamiliar with the various aspects and implications of shutter-shock in general and equally unwilling to acknowledge its relevance to this case in particular if they considered it to have been a factor.

    I think that the most valuable observation is that it's unwise to put too much faith in any single review, regardless of the source, and that as in most things, it makes much more sense to seek out as many opinions as possible.
     
    • Agree Agree x 1
  18. kwalsh

    kwalsh Mu-43 Top Veteran Subscribing Member

    812
    Mar 3, 2012
    Baltimore, MD
    I wouldn't disagree with your summary at all. I think it is better stated than mine. To clarify a bit further by "proof" I meant specifically their claim that they shot at shutter speeds in which SS is not an issue. That was conclusively proven false by Cris's tests which show the range of shutter speeds significantly affected by SS overlaps with the ranges they were shooting their resolution charts at. That itself doesn't mean their copy wasn't also affected by some defect - but it does show their claim that SS doesn't play a role isn't right. Further if you follow the data a bit further you can see other evidence in the review that points to SS playing a role as their diffraction limited center resolution curves don't make sense either for longer focal lengths.

    So we are left with their hand waving against pretty clear evidence that there is a better hypothesis explaining their anomalous results. "Bad copy" is one way to explain that. Test data showing clear impact of SS on the lens is another perhaps stronger explanation.

    The larger issue is when asked questions they provided no evidence - just assertions of what they claimed to do and "trust us". When evidence was presented that some of their assertions weren't correct they discontinued the conversation. Not confidence inspiring.

    And for the record, I really like LensTip reviews in general and typically rank their results pretty high in my weighting of reviews. Which makes it all the more disappointing when they don't pursue if not completely obvious errors than very suspicious results. They have constraints though - it is quite probable that by the time people raised questions they no longer had the lens on loan to do further testing.
     
    • Agree Agree x 1
  19. alex g

    alex g Mu-43 All-Pro Subscribing Member

    Mar 30, 2016
    New York / Bath
    Incidentally, on the general topic of Lenstip — in case anyone else has wondered if they conduct their flare tests with or without a hood fitted to the lens, the answer is "without". I emailed them to find out, because it's a relevant factor which doesn't seem to be stated anywhere on their website.
     
    • Like Like x 1
    • Informative Informative x 1
  20. saladin

    saladin Mu-43 Top Veteran

    814
    May 29, 2015
    jason
    I tend to trust results and reports of faults from this site more than the lens testers sites. Mainly because you get real-world use and continually evolving feedback.

    Eg- I'm more bothered by reports of decentered lens elements in the latest PL lenses on this website than I am by a single incident on Lenstip or the like.

    Which i guess is really saying what others have noted - get info from as many varied sources as you can.
     
    • Agree Agree x 2