RAW Conversion

[0000]

I'm quite surprised that commercial software that use LensFun database don't even participate in making it better. They would also benefit from it.
Having no automatic distorsion correction for my panny lenses on my oly body is not something I consider acceptable for a paying software.

Purely speculation, but maybe these vendors just see Lensfun as an easy bolt-on solution so they can check off a box for what they consider a marginal feature without putting resources into it. The thing is, with more and more lenses (and fixed-lens cameras) being produced with the assumption that software correction will be applied to the RAW files, it really isn't a marginal feature anymore... although they may still see m4/3 users as a relatively small part of their customer base, for all I know.

 

[archaeopteryx]

it really isn't a marginal feature anymore...

Right! I had a bit of time and, being motivated by this thread, had a look into correction behaviours. As of their current builds (Adobe DNG converter 10.1.0.864, G7 firmware 2.2, RawTherapee 5.3) DNG, SOOC jpeg, uncorrected RW2 from RawTherapee, and RawTherapee+Lensfun RW2 all tend to produce somewhat different results on the Panasonic 12-60 3.5-5.6 and 100-300 II images I selected to try. Me being a nature photographer, it's unclear which are best corrected. So my main conclusion is I should find a brick wall and redo the test.

However, there don't appear to be geometric or vignetting difference between DNG and SOOC, only expected tone curve differences and---in certain images---the field of view in the SOOC follows its slight crop relative to raw (4592x3448 versus 4600x3456) rather than exactly matching. The uncorrected and corrected RW2 are comparatively outliers, so I'm tempted to conclude lens correction is performed as a part of DNG conversion when embedded profiles are available (and supported). If so, that'd be convenient since the .dngs are a few MB smaller than the .rw2s and have a native Windows 10 codec (rather than requiring one be added like .rw2).

I think this aspect often goes overlooked.

In fairness, it is a bit technical and obscure and many folks aren't familiar with contributing to open projects. It's also my sense m43 users are more likely to apply money (DxO, C1, LR, ...) than time (most any freeware) to such things. And it is rather nice when things just work rather than needing fairly low level configuration since there's rarely enough time in life to futz around with sorting out innards of software packages. Unfortunately, though, raw processing hasn't worked out in way where much of anything is simple.

anybody who can use a text editor should be able to grok what they need to do just by looking at one of the files

Yep. Many folks understandably aren't familiar with the distinctions between text and other editors, though. Finding the files is also not entirely simple. For example, with RawTherapee on Windows it's

• C:\Program Files\RawTherapee\5.3\share\lensfun\mil-olympus.xml
  
• C:\Program Files\RawTherapee\5.3\share\lensfun\mil-panasonic.xml

Many users wouldn't want to be making edits under Program Files and not everyone knows how to take admin permissions to make such changes. (There's also a nontrivial proportion of environments where users don't have admin to make such edits, though Lensfun probably isn't present in many of them.)

 

[0000]

As of their current builds (Adobe DNG converter 10.1.0.864, G7 firmware 2.2, RawTherapee 5.3) DNG, SOOC jpeg, uncorrected RW2 from RawTherapee, and RawTherapee+Lensfun RW2 all tend to produce somewhat different results on the Panasonic 12-60 3.5-5.6 and 100-300 II images I selected to try. Me being a nature photographer, it's unclear which are best corrected.

Neat... any observations about transverse CA correction? I have a hunch that this may be a weak spot in some of the Lensfun profiles, based on what I've seen of the hit-or-miss behavior of the tca_correct tool (from Hugin) commonsly used. Half the time I end up deriving these corrections manually when making a profile.

And it is rather nice when things just work rather than needing fairly low level configuration since there's rarely enough time in life to futz around with sorting out innards of software packages.

True, but it's also nice for any reasonably curious grease-monkey to know where to poke around when the magic isn't happening. I guess my point is that the Lensfun database should perhaps not be seen so much as the innards of a software package as some useful information about lenses that happens to be formatted in a specific, but human-friendly way. I just get the feeling that this is one of those cases where people who are more than capable enough might prejudicially decide that it's too scary without actually looking into it. Honestly, any page of real-world HTML nowadays would be orders of magnitude more intimidating.

The nice thing about getting comfortable with this is that it can enable you to create your own profiles for all your weird adapted lens mutations that you can't possibly expect somebody else to provide for you.

Many folks understandably aren't familiar with the distinctions between text and other editors, though.

Hmmm... I'm deluding myself that if somebody opened an XML file in, say, Word® or whatever default thing, that said thing would by now do something sane. In the likely event that I'm completely wrong about this, please don't tell me.

Many users understandably wouldn't want to be making edits under Program Files and not everyone knows how to take admin permissions to make such changes. (There's also a nontrivial proportion of environments where users don't have admin to make such edits, though Lensfun probably isn't present in many of them.)

The thing about that is, from what I've heard recently, at least some of these vendors direct their users to update the database by downloading a tarball and dumping it into the database directory. I'm guessing that on Windows there's a userland location that gets searched (and prioritized) in addition to the installed system copies, equivalent to ~/.local/share/lensfun used for the Unix-like environments. Possibly others provide other means, or just rely on users updating the software frequently? I'd be interested to hear more about this aspect of various software that uses the database.

 

[archaeopteryx]

any observations about transverse CA correction?

Both the Panasonic 12-60 3.5-5.6 and 100-300 II often measure less than half a pixel of CA at 16-20MP. I'm not set up to test to that level of detail. The third lens I have, the Panasonic-Leica 45, is even lower and has negligible distortion with not much falloff. Other than around 12, the zooms are fairly well behaved so distortion adjustments are modest. There is some difference of opinion between Panasonic and Lensfun over how much vignetting to correct but this is not difficult to control for.

I'm guessing that on Windows there's a userland location that gets searched (and prioritized) in addition to the installed system copies

I think you'd need to look in the code as RTFM indicates the Lensfun folks don't know either. Not sure this qualifies as human friendly.

 

[coffeecat]

Regarding the thing about camera/lens combinations: I noticed with my first paid version of Windows Affinity (1.6.something) it was not correcting distortion for my PL25/1.4 on my E-PM2, which could look pretty ugly with straight line features near picture edges sometimes. However, after an update (current version is 1.6.1.93) it now seems to work OK. But I've no idea if that was due to a LensFun update, or Affinity using a later version of LensFun, or Affinity fixing something in their software, or what. It's all very obscure.

 

[BigTam]

Having no automatic distorsion correction for my panny lenses on my oly body is not something I consider acceptable for a paying software.

That's the advantage of DxO: they correct for each body/lens combination separately.

 

[SojiOkita]

That's the advantage of DxO: they correct for each body/lens combination separately.

All the software that are able to correctly read the exif data can do the same (mostly Lightroom and Capture One, but they may be others)
(and on an oly body, what you see in the viewfinder includes the distorsion correction even for panny lenses because it's part of the standard).

The only difference is that DxO uses its own corrections instead of the included ones, so for example when a lens has a 5% distorsion, the included profiles corrects it up something acceptable (approx. 1%), and DxO tries to corrects it perfectly.

And that if you buy a very recent lens, you'd need to wait a little until dxo supports it.
When using the included profiles, if you have the lens, you have the profiles.

 

[archaeopteryx]

Me being a nature photographer, it's unclear which are best corrected. So my main conclusion is I should find a brick wall and redo the test,

After an exciting session with a tripod, levels, grids, a brick wall, and a hex editor, some results:

• The DNG contains a WarpRectilinear in OpcodeList3 whose parameters match those applied to the SOOC jpeg. Most Windows apps therefore display the DNG and SOOC with identical geometry. Since RawTherapee doesn't support WarpRectilinear (#2383) its default rendering of the DNG and RW2 is the same. The distortion correction in from Lensfun is not necessarily comparable to that provided by Panasonic, which may or may not be important depending on the purpose. For example, Panasonic retains a bit over 1% barrel on the 12-60 at 12 whereas Lensfun provides somewhat more correction.
• None of the DNGs contain a FixVignetteRadial opcode even when Panasonic applies vignette correction in the SOOCs (e.g. Rec -> Shading Comp = On, which appears in the RW2 and SOOC Panasonic makernotes). It's not easy to check if the RW2 includes a vignette correction; could be either the G7 doesn't emit this or that it's dropped in DNG conversion. Lensfun lacks vignette corrections for my lenses so I suppose I might have a diffuser project at some point. A quicker approach would be a white wall and the vignette controls in GIMP, RawTherapee, and so on.

 

[0000]

I think you'd need to look in the code as RTFM indicates the Lensfun folks don't know either. Not sure this qualifies as human friendly.

The "not knowing" part is actually in reference to the Windows equivalent for a /var/lib/lensfun-updates/ directory, which I've never actually heard of being used. In practice, there's the system /usr/share/lensfun/ files installed with the software, and the location in the user's home directory. It seems that there's increasing interest in the project by Windows users lately, so I'm sure I'll be learning more about that shortly, one way or another...

Lensfun lacks vignette corrections for my lenses so I suppose I might have a diffuser project at some point. A quicker approach would be a white wall and the vignette controls in GIMP, RawTherapee, and so on.

Should you decide to actually take this project on, there's a rant collection of notes on some of the messy details here, with particular emphasis on the value of profiling at more than just infinity focus... it can be kind of a tricky business... but don't let that deter you!

 

[SojiOkita]

Vignetting is quite complicated to handle, I think.
For example, sometimes with my 45-150, at the same focal length on two consecutive shots, the vigneting level is not the same.
I never understood why.
As that is not something that I really care about, I never really tried to understand it.

I generally deactivate the vignetting correction when I can - I prefer to deal with it on the few pictures where it's a problem for me (and that means very few pictures).

 

[0000]

This can happen due to IBIS moving the sensor around relative to the image circle, particularly with longer lenses. Which means that IBIS needs to be disabled when taking calibration shots for vignetting - thanks for the reminder about that!

This is also why, for any variances that you can't somehow track when applying correction, you want to correct for the least case, and avoid over-correction at all costs.

 

[speedy]

I reckon vignetting is the easiest thing in the world to correct for. I like adding a bit in at times for effect.
SilkyPix lets you move the center point of correction, as well as the angle, and I "think" the graduation, all around the frame. Very handy at times

 

[0000]

I reckon vignetting is the easiest thing in the world to correct for. I like adding a bit in at times for effect.
SilkyPix lets you move the center point of correction, as well as the angle, and I "think" the graduation, all around the frame. Very handy at times

For the purposes you're describing (adjusting an individual image for subjectively pleasing results) this is all good and true... and I disable automatic corrections in many cases myself. The situation is a little different when producing a lens profile, where the goal is correctness, such that, for example, the images could be stitched into a panorama without causing banding in the overlapping areas. Whether this full correction is the look that any given person want for less critical applications is a whole 'nother story. It astonishes me that some software apparently gives no option to disable it.

 

[speedy]

For the purposes you're describing (adjusting an individual image for subjectively pleasing results) this is all good and true...

Well, to me, that's the whole point of photography. To make an individual, pleasing image. I'm not interested in turning it into an automated production line, feeding in raw materials, pressing a button, and waiting for a clinically perfect result to pop out. It's a hobby.
Even if the vignetting is automated, the software I've used allows you to increase the vignette, even if you can't disable it. Same result in my book

 

[0000]

Well, to me, that's the whole point of photography. To make an individual, pleasing image. I'm not interested in turning it into an automated production line, feeding in raw materials, pressing a button, and waiting for a clinically perfect result to pop out. It's a hobby.

Totally agree, but consider that just to produce the files that you start with in this endeavor involves a whole bunch of automated process before they get to you... doubly so if you shoot JPEG. Point being, there are uses for accurate lens profiling, and there are perfectly good reasons not to apply it blindly, too.

Even if the vignetting is automated, the software I've used allows you to increase the vignette, even if you can't disable it. Same result in my book

Yeah, I kinda figure this is how they look at it. Depending on how it's implemented, though, it doesn't always give the same results. For example, bright clipped hilights in corners might be greatly reduced in brightness when applying a vignetting "effect", where they wouldn't have been by actual vignetting. This flattening of contrast is fairly characteristic of many simple "instagram" type effects.

 

[archaeopteryx]

This can happen due to IBIS moving the sensor around relative to the image circle, particularly with longer lenses. Which means that IBIS needs to be disabled when taking calibration shots for vignetting - thanks for the reminder about that!

Mmm, wouldn't the calibration ideally cover the whole IBIS accessible portion of the image circle with correction then determining the sensor position to apply the appropriate bit? It's a measurement which requires a larger sensor (or some remarkable firmware hacks and post processing) but the error from measuring only the centre of the image circle and extrapolating might be acceptable.

My lenses are all OIS and the G7 lacks IBIS. So it's not something I personally need to address immediately. I'd caution, though, that the 45-150 mentioned above is also an OIS lens and it hasn't been indicated in this thread that it's operating dual IS or IBIS priority. Perhaps elsewhere?

SilkyPix lets you move the center point of correction, as well as the angle, and I "think" the graduation

Offhand, the image editors I can think of which offer vignette correction have similar flexibility. Likely that's distinct from vignette effects for darkening corners. It's my impression those are usually more limited but I'm hardly a connoisseur.

details here

I was thinking of maybe trying to be a little more structured by applying LED diffusion film to a cheap clear filter. Or knocking the glass out and inserting a diffusion disc. My working thought is either approach acts against side light leaks, might offer somewhat improved results, and---if it works out---it'd be easy to on other lenses.

Where I am polystyrene and polypropylene sheets of ~1-3 mm which should serve for discs are inexpensive and easily had. Opal frosted acrylic also looks interesting but I haven't yet figured out if it's only made in the UK or if it just goes by other names elsewhere. Once trimmed, some of those materials should work well for folks with slide in filter mounts. Particularly if they've the gaskets for ND filters.

 

[0000]

Mmm, wouldn't the calibration ideally cover the whole IBIS accessible portion of the image circle with correction then determining the sensor position to apply the appropriate bit?

I have no idea to what degree IBIS systems are able to report that kind of information... from what little I understand, these are crazy, delicate little realtime gyroscope systems, not CNC machines. But if it's feasible, then yeah, somebody in some twisty little dungeon is probably working on it...

I was thinking of maybe trying to be a little more structured by applying LED diffusion film to a cheap clear filter. Or knocking the glass out and inserting a diffusion disc. My working thought is either approach acts against side light leaks, might offer somewhat improved results, and---if it works out---it'd be easy to on other lenses.

Based on my experience, I would be hesitant to introduce anything that might contribute even the slightest bit more vignetting. Even with a diffuser flat against the face of the lens tube with a wide margin of overlap, I've still ended up with calibration images that vignette more than real-world images, and I still don't know exactly why. My primitive monkey instincts tell me that maybe at that tiny distance the curvature of the front element becomes significant in some way, but I'm still waiting for somebody with a good grasp of optics to sort it out for me.

I think I've used 0.010" white styrene in at least one of my attempts, the only issue being that at that low transmission rate the shots at f/22 can be a little looooong... since I've never had a stack of lenses to do this with, my efforts always end up being ad hoc affairs a couple of times a year. There's also the possibility of using Hugin to stitch a full 360° panorama - it will crunch the numbers and produce a nice analysis of the lens vignetting, but the idea of doing this for multiple apertures and focal lengths and focus distances, in an environment that provided good feature-matching in all cases, borders on laughable.
But anyhow, this is all getting pretty far off the topic of how various RAW processors work...

 

[archaeopteryx]

I have no idea to what degree IBIS systems are able to report that kind of information...

Well, you could make the GPS tags in the EXIF really precise. Less facetiously, sensor position has to be instrumented for the IBIS control algorithm to use feedback. Listing the track followed by the sensor in the makernote is therefore a basic diagnostic of how the system's functioning, though it might not be turned on in firmware released externally. Regardless, careful vignette correction would deal with the sensor's integration of the portion of image circle intercepted by the sensor over the duration of the exposure, which becomes increasingly complex with increasing exposure length. I would expect similar difficulties to occur with OIS since the lens performs pitch and yaw (and potentially x and y) correction by steering the image circle over the sensor. The main distinction might be the image circle isn't constant since the optical formula is changing slightly.

Profiling seems underdesigned for either consideration. Though, without measurements of uncorrected images and residuals in corrected images, it's unclear how much of a problem this might be. As a class, I'd expect smaller and lighter lenses to have smaller image circles. Since small and light are in vogue for the time being it follows the importance of this aspect of lens correction might be increasing.

But anyhow, this is all getting pretty far off the topic of how various RAW processors work...

True, but I'm reasonably as to the source of least part of the trouble. I'll PM later.

 

[0000]

Profiling seems underdesigned for either consideration.

Yep, playing with PanoTools back around the turn-of-the-century, you just had to worry about your scans being off-center with respect to the image circle.

True, but I'm reasonably as to the source of least part of the trouble. I'll PM later.

Created a thread for discussing this stuff: Lensfun correction profile discussion

 

[archaeopteryx]

Happy to help with specific questions about deconvolution.