Why are there no standardized low light tests??? :(

Discussion in 'This or That?' started by bee, Apr 30, 2012.

  1. bee

    bee Mu-43 Rookie

    Mar 27, 2012
    Why isn't this a standard test? I see flawed ISO performance tests where are the real ones? I am new to this so bear with me.

    I think most of us care how a camera performs in low light situations where we need to minimize camera shake or subject movement. We are looking for the least amount of noise in these situations.

    Example (from my real life)
    I am shooting a non moving object hand held on my GF1. My shutter speed is 1/30th with a 20mm 1.7 Panasonic lens wide open at 1.7, no flash allowed. I have maxed out the light that can enter the camera. The ISO is adjusted to properly expose the image and I press the shutter. I pull it up on the computer, and say holy crap that's noisy I can't use this......Let me see if the new GX1 would have been able to pull that shot off. I go to dpreview.com and find a completely useless comparison tool.

    So here are the variables:
    Available light (Can be measured and standardized)
    shutter speed (Standardized in seconds)
    aperture (Standardized in F stops)
    ISO (NOT Standardized different from camera to camera)
    Noise (Subjective, there are different kinds of noise and personal opinion involved)

    My test procedure for comparing cameras:
    Set cameras with same lens to a specific shutter speed and aperture, properly expose image by adjusting ISO. View resulting images at a standardized size. This seems like a good real world test.

    DPreview test procedure:
    Set camera with specific ISO and aperture, then adjust exposure with shutter speed. View results at 100% crop.
    FLAWS: they use non standardized ISO number. And then you view the pictures at 100% crop which isn't realistic between sensors with different megapixels.

    I was comparing the GX1 to the GF1 and DP used different shutter speeds between the cameras at 800 ISO. So the test is meaningless as far as I can tell. I downloaded the raw pictures and the GF1 was 400th of a sec and the GX1 was 200th of a second. To me that means that the GX1 was working with more light and shockingly there was less noise.
    Here is the link: Panasonic Lumix DMC-GX1 Review: Digital Photography Review

    So are there flaws with my logic? Are there any reviews that use a comparison system that is better then DPReview?
  2. pcake

    pcake Mu-43 Regular

    May 3, 2010
    i changed the ISO on that page so all the camera's crops were ISO 200, then downloaded the full sized jpgs for the GX1 and OMD. in the exif info, both show ISO 200, 1/50 of a second, f/6.3. when i downloaded the full sized jpg for the GF1, i found the shutter speed was 1/80 rather than 1/50 like the other two, but that could be because dpreview has changed their standard studio shutter speed.

    i haven't found a better one myself, although some people and review sites swear by dxo - here's their side by side comparison of the GX1 and the GF1.
    DxOMark - Compare cameras side by side
  3. dhazeghi

    dhazeghi Mu-43 Hall of Famer

    Aug 6, 2010
    San Jose, CA
    Real Name:
    To be fair, they do try to keep things constant, but it's difficult and apparently they changed their studio setup some time back. Most of the newer samples are self-consistent, but older ones like those from the GF1 are not.

    DXO does seem to be more scientific in their approach. The downside is that they provide analysis, but no images, so it's impossible to tell exactly how a given numeric score translates into actual output.

  4. Hikari

    Hikari Mu-43 All-Pro

    Nov 26, 2010
    I think the DPreview comparisons are very good. What you want is to see the effect of the noise without changing conditions. Which is what DPreview does. Unlike your test, DPreview shows the entire ISO range of there camera.

    I think you don't understand the DPreview target and can't use it. Why can't you pull up the two cameras and see the difference. If your camera is no different from the other camera you are thinking about, then you know it would not have been better. And you can download the files from DPreview so you are not limited to 100% crops. And the Dpreview test target gives shadow and highlight information and well as chroma/color. It is really good. The Image Resource provides a similar set of images.
  5. songs2001

    songs2001 Mu-43 Top Veteran

    Jul 8, 2011
    Because ISO noise is different at different EV levels. The GX1 was presumably shot at 1 EV brighter scene than the GF1.

    What the OP wants is a test that shows low light, and low shutter speed, so he'll know if the noise, DR etc... are better at the same ISO under the conditions he uses them.

    I have to agree, I would like the tests to show different EV levels and different lighting conditions. But this probably won't happen.
  6. bee

    bee Mu-43 Rookie

    Mar 27, 2012
    The fundamental problem with DPReview is that they are considering the iso settings in the camera to be the same from model to model. If you look at the chart from DXO DxOMark - Compare cameras side by side
    You will see that the iso settings in camera are not standardized. This chart from DXO matches up with the shutter speed difference I mentioned on the DPreview site. (Or perhaps the DPreview site just happened to have slightly different lighting conditions as song suggested. With DP testing method I doubt lighting would be considered a critical parameter.)

    You can easily use my test throughout the iso range by simply using a dimmer to control the lights in the studio. You leave shutter speed and aperture alone, then turn down the lights to specific metered points and raise the iso till you reach proper exposure. By doing this I am actually testing low light performance not just increasing shutter speed AND I have now removed the non-standardized variable.

    DXO standardizes the ISO in there testing which is great, but without pictures its really only half the story.

    Edit* I was thinking about it and using a dimmer would be a bad way to control the brightness because it will probably have an effect on white balance/color tone. My system would probably use some kind of ND filter to decrease light.
    • Like Like x 1