1. Reminder: Please use our affiliate links for holiday shopping!

Why RAW?

Discussion in 'Image Processing' started by oldracer, Mar 17, 2016.

  1. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    I am a confirmed RAW/JPG shooter though I almost never use the JGPs except to browse folders in Windows Explorer.

    But why do we have all these incompatible file formats? Today Adobe has announced yet another RAW converter that will accept yet more RAW formats. Good for them, they can use RAW compatibility to force upgrades. Not good for us.

    Lossless image storage formats are hardly a mystery. That's all a RAW file is AFIK; an image stored in a lossless format together with image-related parameters and metadata like exposure information, etc.

    So what's with every camera model needing to have its very own RAW format? For a given manufacturer, say Panasonic, it's guaranteed that successive RAW generations are created with basically the same software and have basically the same format. No one is going to start from scratch writing a RAW generator for every new camera.

    Has anybody looked at the RAW formats? What are the obstacles to standardizing a file format for RAW images?
     
  2. Bruce McL

    Bruce McL Mu-43 Veteran

    I think what you want is possible. I would like to see it as well. DNG as specified now would make a good starting point, but it's not perfect.

    In creating a TIFF or JPEG from RAW, these two things happen (along with other things).

    1) A tone curve is assigned. Sometimes there is no guidance in the RAW EXIF data for what tone curve should be applied. When that happens, each software package has to make up their own curve, or use "Linear" interpretation, which looks terrible and leaves the ultimate tone curve decision to the user. If I was in charge, I would require tone curve info in my universal RAW format.

    2) A white balance is applied. This involves throwing away a lot of color info. To me this is one of the big benefits of RAW, being able to decide on white balance when processing instead of when taking the photo. There is usually a lot of info in the RAW EXIF data about WB.

    Neither of these things prevent the development of a universal RAW format though. A few years ago Panasonic added some extra lens correction data to RAW that other manufacturers were not doing, but that was just another line in the EXIF data. A RAW standard could allow for things like that.
     
    • Agree Agree x 1
  3. bikerhiker

    bikerhiker Mu-43 All-Pro

    Dec 24, 2013
    Canada
    David
    To be clear, RAW is not an image. It is a binary file that needs a RAW converter to convert into an image based on its own subjective interpretation. It is this subjective interpretation that makes RAW conversion so unique on its own. If RAW is standardized, then there will not be any improvement is image conversion of older cameras and existing cameras. Take the Olympus E-1 for example. If it wasn't for the improvement of the RAW converter of the latest lightroom, you wouldn't get the benefits of the latest RAW conversion algorithms which actually make my 10 year old E-1 still usable. Too bad DXO Optics Pro doesn't support the E-1, but with the E-5 DXO Optics Pro has extended its lifespan by a few more years I take it just because of its PRIME noise engine and its Clearview help improve the E-5's image quality compared to OV3.
     
    • Agree Agree x 1
  4. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    True enough. JPG is also not an image. It is a binary file that needs a rendering engine ... Same-o for TIFF, PNG, GIF, ESRI Shapefiles, etc. I don't think that is very relevant.

    I am suggesting standardization of the RAW file format, not standardization of RAW converters. The number of things that one needs to represent in a [binary] file describing an image is finite and actually not very big. From the content of the standard RAW, people can play as much as they like with conversion tricks and algorithms to give you benefits that are backward-compatible.
     
  5. tkbslc

    tkbslc Mu-43 Hall of Famer

    You could get most of what RAW delivers if the cameras all just offered 16-bit TIFF format.
     
  6. ean10775

    ean10775 Mu-43 All-Pro

    Jan 31, 2011
    Cleveland, Ohio
    Eric
    But wouldn't that result in larger file sizes by default?
     
  7. Turbofrog

    Turbofrog Mu-43 Hall of Famer

    Mar 21, 2014
    And our 16MP files would only be 80MB apiece...

    Really, love it or hate it, DNG is the closest thing we have to a standard RAW format. Manufacturers recognize it, too. Both Pentax and Leica offer DNG as their RAW files, and most any other device (i.e. smartphones) that now offers RAW will do so in DNG format.

    Adobe offers its standalone RAW/DNG converter for free, so it's a total stretch to say that they use RAW compatibility to force upgrades. They are not the bad guys in this situation.
     
    • Agree Agree x 1
  8. tkbslc

    tkbslc Mu-43 Hall of Famer

    Most definitely. I was going to edit to add that before you replied.

    When I typed that, I was thinking that It would be nice if a new standard would emerge that offered a nice balance of the size advantages of JPEG with the quality and lossless editing of TIFF.
     
  9. tkbslc

    tkbslc Mu-43 Hall of Famer

    Well it is not Adobe's fault that new cameras with new RAW files come out all the time, I agree. And it is nice they offer at least some path to usability with older software for those camera. But it is definitely a calculated move not to provide ACR updates for older software to strongly encourage (not force) people to upgrade. DNG conversion with their standalone tool is a fairly obnoxious extra step in the process that makes you want to buy the new version rather quickly. I mean the day that LR6 came out, LR5 gets nothing in terms of camera updates. That's a deliberate marketing choice.
     
  10. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    We're starting to mix apples and oranges here. A standard RAW file would require lossless representation of the image. One lossless representation is to simply store all the pixel values individually. I think that's what TIFF does. There are also lossless compression techniques the do reduce image size in most cases and I would expect that our hypothetical standard RAW file would at least permit of not require use of one of these. So I don't think the hypothetical standard RAW would necessarily be significantly larger than current RAW files.

    Re wishing for a new standard, what you're really asking for is a new and better lossless compression algorithm. People been working on this kind of thing for a long time, but certainly it's possible that some sharp PhD candidate somewhere is working on your project. There's no magic, though. The size advantages of JPGs come from lossy compression which, IMO, would be unacceptable in a new standard RAW format.
     
  11. DaveEP

    DaveEP Mu-43 Top Veteran

    684
    Sep 20, 2014
    York, UK
    That's exactly what Adobe wanted to do with the DNG. As I recall my Leica M8 produced DNG files. However, there's clearly something else going on because no camera makers seem to want to embrace and/or endorse it. Maybe it's politics, maybe it's loss of control, maybe Adobe is too big and has too much control already. As mere users we're unlikely to find out the truth.
     
  12. tkbslc

    tkbslc Mu-43 Hall of Famer

    I know. But that also means that RAW is not magic. We should be able to come up with a true image format that is the size of RAW files and contains the same quality.

    Leica generally bundles Lightroom as their "free" RAW editor with their cameras. So I think they have a bit more of a relationship with Adobe than most.

    I think Pentax still offers DNG as well, but you can also select their proprietary PEF format instead. Would be nice to have a choice in camera and shouldn't be hard to implement in firmware
     
  13. SVQuant

    SVQuant Mu-43 Top Veteran

    855
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    I think one of the main drivers behind all the raw formats is the desire of camera companies to maintain control on their RAW files and also to maintain market differentiation (and maybe retain market share?). In my personal experience, Nikon and Sony have in the past encrypted parts of their RAW files. I am sure that other manufacturers do so as well.

    A free, open RAW standard does not exist to my knowledge. DNG is open, but not free. I do not know what Adobe's licensing terms are, but maybe they do financially impact margins on camera sales.

    A similar situation exists in music files. With most people using mp3 (think jpeg) and while a free open lossless format exists (FLAC), Apple chose to implement Apple Lossless instead and Microsoft went with WMA Lossless. Though DRM was part of the reason behind the decisions, control and creating closed ecosystems was a huge part of it as well.
     
  14. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    Yes, possibly. I have seen this type of thing happen over and over:
    1. Every player in the marketplace develops their own interface.
    2. Users complain, asking for standardization.
    3. Major players resist standardization, attempting to deflect competitors who might benefit from it.
    4. Large users form an alliance, usually via standards organizations like ANSI, JEDEC, etc.
    5. Major players cave.
    One particularly amusing time I was in a standards committee meeting when the Exxon guy told Foxboro and Honeywell that there would be interface standards for process control systems. Another, from the dim mists of time, was when the Justice Department explained to IBM that they would be releasing the technical specifications of their disk drive interfaces.

    This case is a bit of a puzzle to me, though, because I don't see what market advantage the camera OEMs get from proprietary RAW files. It's easy to see why they wouldn't adopt a format controlled by another company, i.e. DNG, but they have happily contributed to and adopted consensus standards like TIFF, MPx, and JPEG. Why not "URAW" ? (Universal RAW :biggrin:)
     
  15. dwig

    dwig Mu-43 Top Veteran

    622
    Jun 26, 2010
    Key West FL
    This by no means "all a RAW file is".

    Not true.

    RAW files contain the raw data from the sensor. They do not contain R, G, and B data for every photosite since sensors, with one rare exception, only record information for one color per photosite. RAW file vary due to differences in sensors. Change the sensor and the RAW file has to change. DNG is Adobe's attempt at creating a containerized RAW file that embeds extra information needed to demosaic the RAW data and to rearrange the data sections into a standard arrangement. it's not perfect, but it works for the most part.

    It takes a "RAW converter" to process the data in a RAW file, even a DNG, and interpolate a full set of RGB data for what are in the output file "pixels". An RGB file with the same number of pixels as its source RAW file will be 3x the size of the RAW, assuming the same type of compression is used and discounting the trivial difference that might exist in the file header data and metadata.
     
    • Informative Informative x 3
    • Agree Agree x 1
  16. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    All that's fine, dwig, but I don't see that any of it precludes development of a URAW file format. Your comment about DNG "working for the most part" more or less proves the point. Down the road, as experience is gained and sensors change, there might be a need for a "URAW II" format but that is a common occurrence in the world of standards so that still isn't an argument that precludes starting with a URAW.
     
  17. SVQuant

    SVQuant Mu-43 Top Veteran

    855
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    I agree with you there. I don't think there is any marketing advantage, but there is differentiation. There are claims out there some RAW files have richer information than others. Also, it may just be that the RAW shooting world is already segmented by manufacturer and there are not enough users demanding a common format. Given that anyone who cares can convert to DNG anyways, it is possible that camera manufacturers don't want to take the trouble to move to a common standard.

    On a side note, there was an openRAW push at some time and it seems to have died (or at least gone dormant). Another sign that there is just not enough demand for it?
     
  18. oldracer

    oldracer Mu-43 All-Pro

    Oct 1, 2010
    USA
    Well, or a sign that users are small and weak. When Exxon told the process control world that there were going to be standards, it was from a position of probably being the biggest process control buyer in the US if not the world. So Exxon's procurement department could simply edict that all equipment from a certain point in time would have ISA standard interfaces. We don't have anything like this in the photography world, so we are probably stuck until/if OEMs decide it is in their best interest to adopt a standard.
     
  19. bikerhiker

    bikerhiker Mu-43 All-Pro

    Dec 24, 2013
    Canada
    David
    You do have a point here with JPEG, but don't we have already a new RAW standard that can be backward compatible? JPEG 9.1 which has 12bit color, Lossless and HDR support. When camera makers adopt this new standard, it would be very easy to manipulate images from apps to applications seamlessly.
     
  20. SVQuant

    SVQuant Mu-43 Top Veteran

    855
    Sep 20, 2015
    SF Bay Area, California, USA
    Sameer
    Fair enough.

    For my purposes, I think I will be very happy if and when JPEG 9.1 shows up in cameras. But more fundamentally, a JPEG is already a positive (decoded, demosaiced, WB and NR applied, etc) while RAW files are really negatives (sensor data + sensor information), so I think that JPEG 9.1 may not have enough in it to be a universal RAW format.