Thomas Chamberlin asks an interesting question in the Flickr D3 user’s group:
“With the D3 and D300 we are now offered lossless compression, no compression, or compression with loss for RAW files. Nikon says there is no effect on image quality using lossless compression. The file is compressed 20-40%. Write times are faster than with no compression. How can they compress the file 40% and not affect the image quality? What is lost and then regained?
The great thing about getting a new camera is that I get another chance to put “on paper” words that lost to the ether.
North Beach, San Francisco, California
Panasonic DMC-LX1, Ultrapod
2 exposures at .8sec @ f/3.6, iso 80, 11.6mm (52mm)
From right to left. The Leica M8 shoots RAW into the Adobe DNG format (not sure of the compression), the Nikon D70 which introduced lossy NEF compression, the Nikon D200 which shoots lossy and uncompressed NEF RAWs, the Nikon D3 which shoots lossy, lossless, and uncompressed NEFs in both 12 and 14-bits. The photo was taken with a Panasonic LX-1 which shoots a Panasonic/Leica RAW file format that is uncompressed and 13x larger than the JPEG sidecar file! Ironically, I processed this image, by accident, from the sidecar files instead of the RAWs, even though it was shot at ISO 80.
[Weighing in on lossy RAW after the jump.]
The Nikon line has had this stuff for a while now. The D200 and D2X offered both lossy compression and uncompressed, and all mid-range Nikon camera RAWs since the D70 were lossy compressed (while the D50 had uncompressed RAWs only—go figure). The difference in filesize was about 50% which explained why the pre-firmware updated D70 and D200 would have the RAW image counter off by a factor of two!
Now back then they called it “visually lossless compression.” The cynic and engineer in me must say:
Any time they have to qualify an absolute with an adjective that isn’t a superlative, then it’s marketing-speak for the opposite.
In other words:
| “lossless” | =lossless | =not lossy |
| “visually lossless” | … | =lossy |
Go figure.
So I guess that I can thank Nikon for finally fessing up this time around and calling it what it is: lossy compression.
Lossy, Lossless, and Uncompressed
Actually the new D300 and D3 have three modes: Uncompressed, Lossless, and Lossy. Nikon lossy and lossless compression are both losslessly compressed using an LZW-style lossless compression.
So let’s get that out of the way first. LZW algorithms will typically save you 15-30% in the file size. This means that you’d pretty much be an idiot to use “uncompressed” as your RAW setting in the D3 or D300.
We must next ask “What is lost and regained?” we mean to compare lossy compression to lossless compression.
Aside: The math is strong with this one
Some of you may be wondering, if uncompressed NEFs are 12-bit (or 14-bit), how come they’re only twice as big as lossy-compressed JPEG-fines and much smaller than TIFF?
The reason there has to do with the fact that the NEF’s are un-demosaic’d, which means the NEF data is actually monochrome.
Trust me, the math works out. 🙂
Onward.
Lossy vs. Lossless
Luckily I don’t have to delve into the math because there is already a great discussion of the mathematics of lossy RAWs written by Fazal Majid: the upshot of it is that the lossy RAWs are reduced from 12-bit to 9.4 bit before being losslessly compressed. This means:
Nikon lossy compressed 12-bit RAWs can be seen as 9.4 bit lossly-compressed RAWs.
(I don’t know what happens in the D3 when you use 14-bit RAWs and lossy compression. Does anyone know yet?)
But, while I largely agree with the article, I do differ in one key aspect: the speculation in the conclusion.
You could argue it does not [really matter], as most color spaces have gamma correction anyway, but highlights are precisely where digital sensors are weakest, and losing resolution there means less headroom for dynamic range compression in high-contrast scenes. Thom’s argument is that RAW mode may not be able to salvage clipped highlights, but truly lossless RAW could allow recovering detail from marginal highlights. I am not sure how practicable this would be as increasing contrast in the highlights will almost certainly yield noise and posterization. But then again, there are also emotional aspects to the lossless vs. lossy debate…
Whoa, there bessy! That isn’t true at all!
Highlights are where digital sensors are the strongest except when clipped which has nothing to do with lossy vs. lossless compression.
Clipping is not done, Noise isn’t introduced, posterization, however, is!
I’ll not disagree that there is an “emotional aspect” to the debate here, but let’s not dismiss the factual aspects so quickly. I’ll justify this below.
Understanding highlights
You can go read Jeffrey Friedl’s excellent article on a practical analysis of the lossy compression. It has the added benefit of being much better written than anything I can do as well as not requiring knowledge of math or physics.
The conclusion is firm, incontrovertible, and factual:
In the end, the only reason to use compression in the first place is for convenience (smaller files, faster write times, and more images on a card), and that convenience is paid for by irrevocable loss of image detail, albeit a very small amount of image detail.
In other words, if you want smaller files and faster write times (more images on your card and disk and less delay between bursts), then you should use lossy compressed RAW, but it will result in a factual loss of image detail in the highlights which is so small as to be invisible in the processed shot.
To me, the interesting part is to ask why. Why is the loss in the highlights? Why is it not visible? When is it going to be visible?
Understanding our collection devices
No, I don’t mean the camera. I mean our eyes and ears. You cannot study cell biology, neuroscience and neural computing and not appreciate the engineering marvel that is ourselves. And it really is a marvel.
Very telling in the Majid article is this comment he makes in passing:
[Requantizing to discard bits] is a fairly common technique—digital telephony encodes 12 bits’ worth of dynamic range in 8 bits using the so-called A-law and mu-law codecs.
Why do they discard bits in telephony? Why does Nikon discard bits in photography? And, in both cases, why are the bits in the high end?
Because that’s how our eyes and ears work.
When we hear we talk about decibels, when we take a photograph we talk in “stops of light” (eV). Both of these are logarithmic. Our ears and eyes are logarithmic data collection devices, but our digital recorders are linear ones.
As I mentioned before, the last stop of highlights in the image (visually, as interpreted by our eye) takes up half the data! Therefore, you can remove data by quantization in that area and still not lose a single thing visually.
That’s why the data is lost on the highlights and preserved in the shadows. And I’m surprised more people don’t mention this fact when they engage in the emotional appeals of “lossy” vs. “lossless.”
When and what do I lose?
It will not recover clipped highlights. As I mentioned before, that shit fell off the edge. Welcome to the world of digital. So stop talking about that.
When you say “dynamic range is preserved,” you must mean it. Stop being an indian-giver in your discussions.
It will not allow you to denoise better because, as I mentioned before, you are in the part of the poisson distribution where the noise relative to value is minimized.
When you say “expose to the right” digitally, you must mean it. Stop obsessing over highlights if you can’t understand the physics of perception.
FAIL.
When lossy does fail is when you are postprocessing heavily in the highlights. At that point you will definitely posterize with a lossy NEF before you posterize with a lossless NEF.
An practical example is if you are taking a bracketed shot for work in HDR or contrast blending. The most noise free data will be in the highlights of your shots and therefore those are the bits that will be used in procressing. It makes sense that throwing away the most useful data during collection in such a heavily processed image would lack foresight. Besides, if you are taking such a photograph. Do you really care about your buffer or about the file size? 😉
Another nit pick
As I mentioned before, I love to nitpick Thom Hogan. Now this isn’t because he’s wrong more than he is right like Ken Rockwell is, but it is because he is right so often than an error stands out.
In all of his PDF guides, he has a discussion on NEF compression in the section under “Compressed NEFs” and that discussion is good. If you are interested in it, go buy one of his digital SLR ebooks. But I will pick two nits.
“That’s party becuse our eyes work in a non-linear fashion with brightness (sensors are linear—the NEF compression scheme mimics our eye’s non-linearity), but also because our eyes generally are thought to distinguish tonal changes only about equivalent to those produced by 8-bit RGB data.”
—Thom Hogan, Thom Hogan’s Complete Guide to the Nikon D200, p 150
That’s so close to being totally true. The two nitpicks here are tiny.
The first is the caveat that by using the argot of “tonal” he is specifically referring to the luminance, not the chroma: i.e. 8-bit per channel RGB or 24-bits total.
But more importantly is the missing qualifier: “at once.” See, our eyes have this thing called a pupil, other eye muscles, and shit like that. This means that when taking in a scene in the real world, our eyes can focus in on different parts of it, our pupils can dilate, and the total dynamic range of the scene we can perceive may be 14-bits of “tonality” or higher!
Just because our eye can only distinguish about 10 million colors doesn’t mean that our eye only has just shy of 24 bits of total dynamic range!
Perspective
It really is an amazing piece of engineering, our eye. And it is an amazing instrument to see, our camera.
Let us never forget in our discussion, what this camera is: an instrument that teaches another to see.
Compressed or uncompressed. RAW or JPEG. Teach me by taking a photo today. 🙂
When I say sensors are weakest in the highlights, I was precisely referring to the fact digital sensors clip harshly, unlike the more gradual shoulder of film. Film is more forgiving of overexposure (well, negative film), whereas digital sensors give better shadow detail than film.
As for the visual system, it is even more amazing in how it uses saccades, short but extremely rapid eye movement to build a high-resolution picture from an essentially 6MP sensor, with some very fancy processing in the visual cortex to mask the fact our eyes are shifting over ten times per second. They also have the original idea for the SuperCCD HR with our rods and cones…
It’s funny – we both live in San Francisco, both have a M8 and a D3. I haven’t experimented with the lossy NEF compression in my D3, but each NEF file actually has the quantization curve built-in, so this is conceivably something that could be changed from one firmware release to another.
You might be interested in this article from astrophotographer Christian Buil, which concludes much of the D3’s high-ISO performance is actually from preprocessing in the camera itself and that the intrinsic sensor capabilities are similar to Canon’s, not a breakthrough. It would not surprise me too much as sensors are close to the physical limits (the M8 is not a particularly good camera for high ISO, but its detective quantum efficiency in the green channel is already 40%), I am not sure whether I agree entirely with his conclusions, but it is clear even the lossless or uncompressed NEFs are actually cooked.
You are not a real photographer, look at your shots. Stick to being a dork and don’t post this nonsense. Your camera doesn’t help your images.
@Fazal: We should meet up sometime. 🙂
Yeah, I was a little harsh on you. It was clear you knew what you were talking about, but I just had to pick a single nit you mentioned haphazardly in the conclusion (nobody called you out on the difference between clipping/dynamic range and luminance resolution). Besides, you wrote that article a long time ago so it’s understandable (though it bugged me that nobody pointed it out).
As for the part that is unrelated to this post…
I was planning on writing an article on the Active D-lighting (this is the “preprocessing” Christian Buil is talking about) since I’ve had to explain it so many times to people. In my opinion it actually is a “breakthrough” because the preprocessing is done on the sensor in analog space while the sensor is being detected, instead of the in-camera RAW file digital postprocessing. The D200 also did that with color space (which got overshadowed by the whole “white balance encryption” controversy). It is not “cooked” in any way because you can turn the burner off (simply turn off Active D-lighting), did he remember to do that? Did he acknowledge that processing in analog space must be done in all cameras so there is no way not to cook a RAW? Hmm…
(I do have one complaint though, the way it works is that the sensor gains up as the data is being collected when that pixel is dark meaning the image is slightly underexposed. That seems a little backward. Ideally you’d want to attenuate the photosite as it fills up allowing for slight overexposure.)
In any case, the proof is “in the pudding.” It is incontrovertable that no matter what Christian and other Canonophiles say that the D3 can take photographs that the 5D (which is basically the same quality sensor) can’t take no matter how much postprocessing is done.
When Canon introduced image stabilization and piezoelectric autofocus motors, a lot of Nikonites were saying why it’s crap and how it can’t replace a good tripod or freeze the subject, blah blah. And I called them out on that. The criteria I use is simply, “If my camera had that feature, would I be adverse to using it” the answer in the case of of IS and USM is “yes” which is why we have VR and SWM today.
Are we going to honestly stipulate that Canon photogs wouldn’t use Active D-lighting and nanocrystal coatings? I don’t see any of their L lenses with fluorite crystal anymore. Hmm…
I forgot to put the link I mentioned:
http://www.astrosurf.com/buil/nikon_test/test.htm
The 70-200mm f/4L IS introduced 20 months ago has a fluorite element (like its excellent and very affordable non-IS predecessor). Fluorite is very soft and fragile, so it requires extra hard coatings to endure real-world use, or be used in inner lens elements nt directly exposed.
Only Zeiss and Canon master the technology for photographic lenses, and a few specialty houses like Takahashi for very expensive but wonderfully corrected fluorite apochromatic refractor telescopes.
Fazal,
Are you 100% sure of the statement that the 70-200mm f/4L IS uses fluorite crystal in one of their elements? Let me explain.
Here is the Canon marketing literature on it. The summary is I mentioned that fluorite elements in photgraphic lenses is a marketing claim by Canon that does not stand the smell test because 1) Nikon ED class was available before Canon introduced fluorite elements, and 2) ED elements back then can combine to create the same apochromatic, 3) newer ED glass have almost identical dispersion characteristics as fluorite without the manufacturing cost and fragility it entails.
In other words the only reason Canon and Zeiss bothered to copy this technology from the astronomy world was because they were excluded themselves from more economical high-index specialty glass (by patents or by choice).
Read the literature closely, where does it say the 70-200mm f/4L or the 70-200 f/2.8L IS for that matter use fluorite (they do use UD glass, which is what Canon switched to when ED patents expired), especially since Canon no longer has glass manufacturing capability (photographic glass is still made by Cosina, Hoya (Tokina/Pentax), and Nikon.)
If I’m wrong, please correct me, otherwise I posit that this is more myths of the white lens.
By the way the very first line of the Astronomy article you reference is absolute bullshit. First the “filtering” above 1 second is entirely optional (it is a custom menu) and the second is that the moron doesn’t even know why this occurs which leads me to believe the guy must have failed his astronomy course! All astrophotography does digital subtraction of black point exposure, the difference here is Nikon offeres it “in camera” and the Canon is superior to the Nikon D80, etc. because it is a CMOS sensor instead of a CCD. CCD sensors are lower noise (even in long exposure astrophotography) than CMOS, but the CCD requires an image processing sensor which operates on the order of a volt (as to millivolts for the same processing to be done on the CMOS chip itself). In high end astronomy work, the processing chip and the CCD are separated and the CCD is often liquid cooled to eliminate the photodetectors extreme sensitivity to heat. This is impractical for a consumer digital camera resulting in a CMOS chip being better for night photography.
I mean the guy is fucking lying (and he must know it or he’s an incompetent astronomer) in his very first sentence! (Note, I’m not disagreeing with the conclusion that a Canon 20Da is superior to say an 80D for astrophotography, but remember the D300 and D3 both have CMOS so are going to be the same and all late model Nikons and all Canons other than the 20Da have a ultra strong cut filter that needs to be removed for serious astrophotography).
Why hasn’t anyone called him out on this? Is it because he’s French? Geez, I hope he didn’t go to Ecole Normale or Ecole Polytechnic because those uni’s went down in my book if so (if he didn’t go to those schools, then how the hell is he qualified to even talk about this stuff).
The guy needs to crack open his f—king manual and look up “Long Exposure NR = off” (which addresses all his previous and future “dark current” bullshit) and “Active D-lighting = off.” (which is why his “display vs. actual” ISO measurements are off. The manual clearly states with Active D-lighting on, the image is underexposed). He is the science equivalent of a criminal.
The Canon product page for the 70-200mm f/4 clearly states it has 1 fluorite element and 2 UD elements (the little icons at the bottom).:
http://www.usa.canon.com/consumer/controller?act=ModelInfoAct&fcategoryid=150&modelid=7345#ModelDetailAct
Canon’s EF Lens Work III book says the same. In their diagram the fluorite element is the third from the front of the lens.
The product page for the f/4L IS does as well:
http://www.usa.canon.com/consumer/controller?act=ModelInfoAct&fcategoryid=150&modelid=14260
Now you may accuse them of fraud, but the burden of proof is upon you to substantiate your claims. As for the relative merits of ED/Super ED vs. Fluorite, I am not qualified to discuss them, and I suspect neither are you. Nikon has used fluorite in the past as in their 300mm medical imagery lens. Even Nikon only claims robustness, lower cost and reduced sensitivity to temperature as advantages of ED technology (which is basically a blend of glass and fluorite, and thus extremely hard to produce consistently from one melt to another).
As for Zeiss using fluorite due to lack of access to specialty glass, I’m sorry but that statement is laughable. Zeiss is a sister company of Schott, probably the highest end glass maker in the world. Both are owned by the Carl Zeiss Stiftung charitable trust, established by Ernst Abbe, the University of Jena optics professor who also happens to be the pioneer in the use of fluorite to correct for chromatic aberration.
Canon definitely makes glass and crystals through its subsidiaries Ohara and Optron. The complex structure of japanese keiretsus can be confusing, e.g. Nikon gets its AF-S motors from Mitsubishi but they are part of the same extended family.
As for Christian Buil, I have no affiliation with the guy, and have no idea where he studied. It is possible but unlikely he does not understand Nikon settings given the lengths he has gone to in order to avoid automatic dark frame substraction in his other article on the D70. That said, profanity-laced ad hominem attacks do not do much for your credibility. From some cursory Google searches, his credentials are quite impressive, from checking his list of publications, at least one book that was translated in English (http://www.willbell.com/ccd/ccd4.htm), his IRIS astrophotography software, and the fact he is an optics expert for the CNES and the Pic du Midi observatory and generally working with CCDs well before either of us even graduated from high school.
Please don’t bring my alma mater into this. It’s spelled École Polytechnique, by the way, and the other school is the École Normale Supérieure Ulm-Sèvres (the presumably inferior Écoles Normales only train elementary school teachers). There are other fine French schools that specialize in optics and astronomy, e.g. Sup Optique, the alma mater of Angénieux, where Alain Aspect has his lab. Perhaps you have heard of him?. As for French-bashing, I guess I was mistaken in thinking it was the exclusive preserve of red-state philistines.
I like Nikon, they make fine optics and cameras, but Nikon fanboyism is just as tiresome as that of Canon or Leica.
@Fazal:
I never responded to your comment (it was in an article I never published). I apologize.
Nikon ED glass has flourite particles suspended in silica to reach ultra-low dispersion (it’s glass, but with Ca-F2 crystals in it). That’s what the glass patent that Nikon had was for, and it’s why Canon lenses used to use Flourite crystal until they got access to same glass patents as Nikon (Japanese companies being intertwined in a strange, opaque way). I didn’t realize, Canon still used Ca-F2 crysta especially given how fragile the latter is. But since they have icons for both UD and Ca-F2, I guess they still do. I apologize.
Second, my rant on Christian Buil was way out of line and due to a misreading of his post (it came from hearsay on the forums). I reread the article a couple times just after I posted the comment and realized it. He is still wrong, but he isn’t bullshitting—he is just unaware of a natural consequence of Active D-lighting. He insinuates this is a deception (like the Nikon D70 "NR off" setting fiasco), which it is not.
Because of Active D-lighting, Nikon photosites in cameras that have the Active-D feature are connected electrically. Therefore there is leak charge between photosites that exhibits the directional behavior in the dark current he is seeing (even when Active D is disabled). In other words, it’s a feature, not a bug. His problem comes from using a consumer camera for something it was never designed to do and then bitching about it.
He is free to rant, but really, he’s standing athwart of history here. I expect other camera companies will follow suit with the same technology as soon as they can get the Japanese government to okay the transfer (in fact, as of this writing, Canon already has with the 1D mk IV). The benefits of Active-D for consumer and professional photography are to great.