/usr/share/rawtherapee/camconst.json

/*

   DO NOT EDIT THIS FILE!

   All changes made here will be lost on software update.
   If you want to add custom values or changes existing ones,
   create a "camconst.json" file next to your personal "options" file.
   Its values will then override and/or complete the ones of this file.

   If you add values for your own camera and are okay to share them with
   RawTherapee's community, please drop a link on the user's forum

   IMPORTANT:
   ----------
   
   1. If you set the dcraw matrix in your user file for an already existing entry
      in RT's file (same camera, same model), your values will replace RT's ones.
   2. If you set the Black level(s) values in your user file for an already existing
      entry in RT's file, your values will replace RT's ones, even if RT's ones are
      more complete and/or detailed. You might want to copy/paste RT's levels first
      (if provided) to your user's file and complete/modify it.

      Same for the White level(s), independently from the Black level(s).


----------------------------------------------------------------------------------


This file is in JSON format and contains camera constants which RawTherapee uses
when parsing raw files.

Raw files themselves unfortunately do not contain all information needed for making
a raw conversion, typically color response information and black/white levels are
missing. That's why this file is needed.

It's read once during startup, so if the file is updated you need to restart
RawTherapee in order to take effect. The file is not intended for modification by
the casual user, but advanced users can add missing camera information to this file.
If you do so please report at http://code.google.com/p/rawtherapee/issues so we can
extend the distributed version of this file so your provided camera information
becomes available to all.

RawTherapee uses DCRAW as the raw format parser. DCRAW contains hard-coded camera
constants, but not for all cameras and not always accurate information. For example
DCRAW only support one white level, while some cameras have different white levels
per channel and per ISO. If a camera is not listed in this file the constants from
DCRAW will be used, if listed here this information will override any constants in
DCRAW (if any).

Some cameras may only have partial information here, for example if the raw file
itself contains a color matrix it's not entered here. A camera whose black level
is measured on special pixels in the raw file should only have white levels here
etc.

Examples:

    {
        // make and model separated with single space, must match make
        // and model as provided by dcraw (case-insensitive).
        "make_model": "ManufacturerA ModelB",
        // ColorMatrix with D65 Calibration Illuminant, in dcraw format
        "dcraw_matrix": [ 7530, -1942, -255, -4318, 11390, 3362, -926, 1694, 7649 ],
        // black and white level same for all colors at all ISOs
        "ranges": { "black": 10, "white": 1000 },
	// crop away masked sensor borders, 10 pixels left, 20 pixels top, resulting image width/height 4000x3000
	// instead of width/height you can write a negative number which specifies how much of right/bottom border
	// that should be removed
	"raw_crop": [ 10, 20, 4000, 3000 ],
	// same as MaskedAreas DNG tag, used for black level measuring here two areas defined
	"masked_areas": [ 51, 2, 3804, 156, 51, 5794, 3804, 5792 ],
    },

    {
        "make_model": "ManufacturerA ModelB",
        "dcraw_matrix": [ 7530,-1942,-255,-4318,11390,3362,-926,1694,7649 ],
        // black and white levels per ISO per channel
        // this example only two ISOs, normally the list should be more populated.
        // When RawTherapee asks for black/white levels for a specific ISO the closest
        // match is picked.
        "ranges": {
            "black": [
                { "iso": 100,  "levels": 10 }, // here only one level, same level for all channels
                { "iso": 3200, "levels": [ 50, 60, 50 ] } // 3 levels, G2 same as G1
            ],
            "white": [
                { "iso": 100,  "levels": [ 10000, 11000, 10000, 12000 ] }, // 4 levels, G1 and G2 different
                { "iso": 3200, "levels": [ 11000, 11000, 10000, 12000 ] }
            ]
        }
    }

How to measure white levels:
----------------------------

Dcraw which provides the default values to RawTherapee often provides too high
white levels, and only provides a single value regardless of color channel, ISO
or aperture. If you open an image with a large clipped area and that is
rendered in a pink/magenta color rather than white it usually means that the
white level constant is too high. You can fix this by adjusting the
"Raw White Point" in the raw tab inside RawTherapee, or permanently fix it by
measuring and providing a more exact white level in camconst.json so
RawTherapee gets to know from start where the camera actually clips.

Providing a complete and detailed white-level profile can be a quite large
and complicated effort. As an alternative you can provide a simpler profile.
We suggest one of the following alternatives in rising difficulty (and
generally diminishing return):

A) Provide a single white-level value measured on the native ISO (base ISO).
   For many cameras this will actually be complete information, those that
   don't vary on channel, ISO or aperture.
B) Check through all ISOs and if there are differences in white level provide
   an array with white level per ISO.
C) In addition to ISO, check for aperture scaling and add that.
D) In addition to ISO and aperture scaling check for color channel
   differences and add that.

Doing A) is often better than nothing, as dcraw's default is often too high.
B) can also be worthwhile for some cameras (or else you'll get pink highlights
for some ISOs), while C) and D) can generally be seen as fine-tuning.

Here follows a guide how to measure white levels (clipping levels):

Shoot with your camera into a bright light source, such as a lamp, and make
sure the shutter speed is long enough to get overexposure (we want
clipping!). Preferably overexpose lightly, say 1 or 2 stops if you can. The
reason for this is that some cameras with fuzzy white levels may look less
fuzzy than they actually are if over-exposure is heavy.

Use f/5.6 or smaller aperture (=larger f-number) to avoid any raw scaling
the camera might have for large apertures.

Open the file in a raw analyzer such as Rawdigger and check the pixel values
for the clipped areas (if you are using Rawdigger, make sure you have disabled
"subtract black" in preferences or else sample values can be wrong). In this
stage we always look at white level before black level subtraction! White
levels can be different on color channel (R, G1, B, G2, note the two greens,
most often both green channels have the same white level though) and vary
depending on ISO setting, so if you want to provide a complete profile make
one shoot for each ISO (even 1/3 steps, so yes it can be quite a lot of
pictures to shoot and check).

In addition, many cameras scale the raw values for large apertures. It's
generally not that important to cover this, but if you want to extract most
out of the camera you should cover this too. Then you need to shoot with a
wide aperture lens (ideally the widest available from the manufacturer) and
test each aperture (1/3 steps) from the widest (say f/1.2) until the camera
stops scaling the raw values (usually f/2.8 or f/4.0). If the camera also
have ISO scaling you need to shoot at these different ISOs to detect any
differences in scaling, there can be a bit of variation. If you don't have
access to the widest lens available for the system (say only an f/1.8 lens
instead of an f/1.2) it can still be valuable to have the values down to
what you can provide. Brands known to have models that have aperture scaling
of white levels include Canon and Nikon. Note that if white levels are not
scaled the camera may have raw scaling anyway (Sony for example), but as
such scaling will not affect raw decoding we don't need to care about that.

PROVIDE CONSERVATIVE VALUES. Most cameras have a little noise at the white
level, and some can have a lot. In your raw analyzer, move around and look at
the values in the clipped areas to get a sense of the variation, and/or look
at the histogram. While it's common to with very little variation, say only
+/-2 units, some can have +/-500 or more (some may have different variation
depending on ISO). There can also be camera-to-camera variation.

If the white level is set too high RawTherapee will not think the pixels are
clipped and you can get discoloured highlights (usually pink), this is what
we want to avoid. If white level is set too low RawTherapee will clip early, ie
you lose a little highlight detail, but the color is rendered correctly and
highlight reconstruction can work properly, so this is not as bad. This is why
we want conservative values.

By conservative values we mean that if you see a white level of most often
15760 and occassionally 15759 (ie very small variation of white level which
is a common case), you set the white level 10 - 20 units below, say at 15750 in
this example, this way we get a little margin from noise and camera variation.
Since sensor raw values are linear you lose in this example log2(1-10/15760) =
-0.001 stop of detail, ie irrelevant. Thus it's better to provide RawTherapee
with knowledge where the image clips rather than keeping that last 0.001 stop
of highlight information and risking that clipping will not be detected
properly.

If you have a fuzzy white level look at the linear histogram; you will probably
see a normal/gaussian distribution (bell shape) noise peak at clipping and
probably also a peak at a hard raw data clip level usually at or close to a
power of two - 1, such as 4095 or 16383. Then you pick a value just before the
bell shape rises, ie to the left of the bell meaning that you cut away the
whole fuzzy noise peak. If a little of the starting edge of the noise will be
included it's not harmful, but 99% of it should be above.

If you have used Adobe's DNG Converter and analyzed it's output you may have
noticed that it's very conservative regarding white levels, ie it cuts away
quite a lot from the top. While we also recommend to be conservative, you can
generally be a little bit less so than Adobe's DNG Converter. RawTherapee is
meant to max out what you can get from your camera, and the white levels should
mirror that, within reason.

The aperture scaling feature is meant to raise the white level to not miss out
on highlight detail when the camera has scaled the raw values (and thus
raised white levels). Many cameras do this, but not all, and can only do it
for lenses that report aperture to the camera (ie you see it in the EXIF
data). Providing proper aperture scaling values is a bit more advanced task,
so if you are unsure we recommend to skip that part.

Beware that the raw format may have a ceiling so that it clips scaled values,
for example the Canon 5D mark II maxes out at 16383 which happens at f/1.8
for ISOs with the white level at 15750, but for ISO160 when the white level
is 12800 it does not max out. If there is such a raw limit it must also be
provided ("ranges":"white_max"). Usually you will not need a margin on
white_max as it clips there as a result of an in-camera math operation.

Note that aperture scaling can be quite small, for the 5D mark II it's only
0.1 stop down to f/1.4 and then it can be discussed if it's worthwhile to care.
The "worst" cameras scale about 0.6 stops though, and then it's more
valuable to compensate. If you skip aperture scaling RawTherapee will clip the
files a little bit too early and you miss that last fraction of highlight
detail, but you get no processing problems. Setting unconservative scale
factors can on the other hand cause a too high whitelevel and break highlight
processing, so be careful.

Scaling can vary sligthly depending on ISO (if white levels vary) so make
sure to provide conservative scalings so regardless of ISO you don't get a
too high white level. We recommend to keep a wider margin here than on the
white levels, ie 0.5-1% lower or so. For example if base (conservative!)
white level is 15750 and the scaled is 16221 we have a scaling factor of
16221/15750=1.0299 ie +2.9% we set the factor to 1.02 or +2% to keep a
margin.

The scale factor you provide here is applied on the white level before black
level subtraction (if any), ie directly on the white level value you provide in
the camconst.json file. Black level (if provided) is not scaled. Please report
to us if you come across a camera which scales black levels, then we can add
that as an option. Usually the camera applies an offset to shift back the
black level to the standard level after scaling.

If RawTherapee doesn't find an entry for the aperture used in the image, it
will pick the closest above. Ie if the apertures 1.0 and 2.0 is in the table
and the image has aperture 1.2, it will pick scaling for 2.0, even if 1.0 is
the closer aperture. The reason for always checking the closest above is that
we rather get a bit too low white level than too high, as discussed before.

Some cameras have different white levels on different color channels. When
this is the case the difference is often so small so you can just provide a
single value instead, ie a conservative value based on the lowest clipping.

What we know at the time of writing about different brands/models (not
complete info):

 - Canon CR2: typically same clipping per channel, but significant variations
   on ISO and aperture. Maxes out at 16383, black level measured on masked
   black pixels, ie don't provide that.
 - Nikon NEF: sometimes different clipping per color (most often negligible
   though). Will do aperture and ISO scaling, but often to a lesser extent
   than Canon files, ie not as much to gain.
 - Sony ARW2: no scaling. Generally black level around 512, and white level
   16350.

Note that some raw formats may go through a certain amount of pre-processing
based on meta data, such as curve and levels adjustments and various
calibrations. The Phase One IIQ is one example, and this means that if you
look at the data in a raw analyzer such as RawDigger it may perform a
different type of preprocessing than RawTherapee's loader does, and you may
end up providing incompatible black/white levels.

You can use RawTherapee for analysis too, it's safer as you are using it's
own raw decoder but it's not as user-friendly: enable verbose mode in options
so you get output on the console (you need to start RT from a console to see
the output). When you load a file you will see a message of current black
and white levels and if they came from dcraw or camconst.json. If you're
adjusting an existing camconst.json value you can just read what it is in
the file and not need to enable verbose output.

Reset exposure sliders to neutral, and zoom in on a large clipped highlight.
Move around the mouse pointer within, it should show stable 100% on R G B. If
so, the white level is not too high, it could however be too low. To test that,
go to the raw tab and adjust the "whitepoint linear correction factor", reduce
it until one of the channels is no longer 100%, and then increase in steps of
0.01 until all are 100 again. Usually you play around in the range 0.90 to
0.99, ie a very small adjustment. When you've found this factor you should
apply it on the old white level to find a new larger one. As RT's "whitepoint
linear correction factor" work after blacklevel subtraction and camconst.json
want values without it we need to do some math:

BL = black level (typically something near 0, 256, 512 or 1024, find it in the
     verbose output or if available in camconst.json)
F =  whitepoint linear correction factor you just found out (typically in the
     range 0.90 to 0.99 if you need to increase white level, 1.01 to 1.10 if
     decrease)
oldWL = old white level, found in verbose output or in camconst.json if
        available.

new white level = BL + (oldWL - BL) / F

Note that if black level is 0 which it is for many cameras, the formula
simplifies to: new white level = oldWL / F.

Here's an example from a Canon 1000D: black level is 256, old white level is
3651, whitepoint correction factor becomes 0.90, then new white level is
256 + (3651 - 256) / 0.9 = 4028.

If your camera have different black levels per channel use the one which
yields the smallest white level (can be the largest or smallest, test!).

This new white level you then enter in your camconst.json file. The same
procedure can be used if the white level is too high, ie if you see pink
highlights, then increase the correction factor above 1.0 until you just start
seeing stable 100% on all channels, you use the same formula to calculate the
new smaller white level.

About black levels:
-------------------

Unlike for white levels it's much more common that black levels can be
derived from the format. Either it's simply 0 (typical for Nikon cameras), or
it can be derived from masked pixels (typical for Canon cameras) or otherwise
be extracted from some tag. Some formats are have built-in subtraction
information and are pre-processed by DCRaw to end up at a black level of zero
(Phase One's IIQ). In all, you typically should not care about the black
level in camconst.json: any information that can be derived from the raw file
itself should not be specified in camconst.json! Sony's ARW2 is one of the
few exceptions (which has a single black level around 512), but DCraw
generally has good constants for these already.

Currently we have chosen not to provide any guide how to measure black levels
as we don't think it will be a common task (it's also more difficult to do
than measure white levels). If you experience a black level issue it's more
likely due to a format parsing bug which should be fixed in DCRaw and/or
RawTherapee's raw format parser.

How does a black level issue look? If the image has a color cast and is
possibly duller than normal it's likely that black levels are off. The color
cast is typically stronger in darker colors but it can be hard to see, it's
more often experienced as a cast over the whole image.

*/
{"camera_constants": [

/*

When adding camera constants please set a quality level so we know the status for future updates

Quality A: complete information, no need to add more, to the best of our knowledge
Quality B: not complete, but very little to gain from adding more
Quality C: complementing with additional information would provide significant gain
Quality X: unknown, ie we knowing to little about the camera properties to know if
           we have enough info or not.

*/


    { // quality A (only aperture scaling for f/1.2 missing)
        "make_model": "Canon EOS 5D Mark II",
        "dcraw_matrix": [ 4716,603,-830,-7798,15474,2480,-1496,1937,6651 ],
        "ranges": {
            // black levels are read from raw masked pixels
	    // white levels are same for all colors, but vary on ISO
            "white": [
                { "iso": 50,  "levels": 15750 }, // typical: 15760
                { "iso": 100, "levels": 15750 },
                { "iso": 125, "levels": 15750 },
                { "iso": 160, "levels": 12800 },
                { "iso": 200, "levels": 15750 },
                { "iso": 250, "levels": 15750 },
                { "iso": 320, "levels": 12800 }, // typical: 12810
                { "iso": 400, "levels": 15750 },
                { "iso": 500, "levels": 15750 },
                { "iso": 640, "levels": 12800 },
                { "iso": 800, "levels": 15750 },
                { "iso": 1000, "levels": 15750 },
                { "iso": 1250, "levels": 12800 },
                { "iso": 1600, "levels": 15750 },
                { "iso": 2000, "levels": 15750 },
                { "iso": 2500, "levels": 15750 },
                { "iso": 3200, "levels": 15750 },
                { "iso": 4000, "levels": 15750 },
                { "iso": 5000, "levels": 15750 },
                { "iso": 6400, "levels": 16370 }, // typical: 16383
                { "iso": 12800, "levels": 16370 },
                { "iso": 25600, "levels": 16370 }
            ],
	    "white_max": 16383,
            "aperture_scaling": [
		/* note: no scale factors known for f/1.2 and f/1.0 (had no lenses to test with), but the
		   typical 15750 white level maxes out at "white_max" for f/1.8 and below anyway. */
		{ "aperture": 1.4, "scale_factor": 1.077 },
		{ "aperture": 1.6, "scale_factor": 1.054 },
		{ "aperture": 1.8, "scale_factor": 1.039 },
		{ "aperture": 2.0, "scale_factor": 1.031 },
		{ "aperture": 2.2, "scale_factor": 1.021 },
		{ "aperture": 2.5, "scale_factor": 1.016 },
		{ "aperture": 2.8, "scale_factor": 1.010 },
		{ "aperture": 3.2, "scale_factor": 1.0046 },
		{ "aperture": 3.5, "scale_factor": 1.0031 }
            ]
        }
    },
    { // quality B, more aperture scalings desired
        "make_model": "Canon EOS 5D Mark III",
        "dcraw_matrix": [ 6722,-635,-963,-4287,12460,2028,-908,2162,5668 ],
        "ranges": {
            // black levels are read from raw masked pixels
	    // white levels are same for all colors, but vary on ISO
            "white": [
                { "iso": 50,  "levels": 15270 }, // typical: 15282
                { "iso": 100, "levels": 15270 },
                { "iso": 125, "levels": 15270 },
                { "iso": 160, "levels": 13300 }, // typical: 13306
                { "iso": 200, "levels": 15270 },
                { "iso": 250, "levels": 15270 },
                { "iso": 320, "levels": 13300 },
                { "iso": 400, "levels": 15270 },
                { "iso": 500, "levels": 15270 },
                { "iso": 640, "levels": 13300 },
                { "iso": 800, "levels": 15270 },
                { "iso": 1000, "levels": 15270 },
                { "iso": 1250, "levels": 13300 },
                { "iso": 1600, "levels": 15270 },
                { "iso": 2000, "levels": 15270 },
                { "iso": 2500, "levels": 13300 },
                { "iso": 3200, "levels": 15270 },
                { "iso": 4000, "levels": 15270 },
                { "iso": 5000, "levels": 13300 },
                { "iso": 6400, "levels": 15270 },
                { "iso": 8000, "levels": 15270 },
                { "iso": 10000, "levels": 13300 }, // no data, guessed
                { "iso": 12800, "levels": 15270 },
                { "iso": 16000, "levels": 15270 },
                { "iso": 20000, "levels": 15270 },
                { "iso": 25600, "levels": 16300 },
                { "iso": 32000, "levels": 16300 },
                { "iso": 40000, "levels": 16300 },
                { "iso": 51200, "levels": 16300 },
                { "iso": 102400, "levels": 16300 }
            ],
	    "white_max": 16383,
            "aperture_scaling": [
	        // limited scaling info available, please provide more
		{ "aperture": 2.0, "scale_factor": 1.024 },
		{ "aperture": 2.8, "scale_factor": 1.010 }
            ]
        }
    },

    { // quality B, lacks aperture and ISO scaling, known to exist, but little to gain as the levels are so close to white_max
        "make_model": "Nikon D7000",
        "dcraw_matrix": [ 7530,-1942,-255,-4318,11390,3362,-926,1694,7649 ], // matrix provided by Tanveer(tsk1979)
        "ranges": {
	    // measured at ISO 100. ISO differences not measured, but known to exist
	    "white": [ 16370, // 16383 typical
	               15760, // 15778 typical
		       16370  // 16383 typical
		     ],
	    "white_max": 16383
	    // aperture scaling not measured, but known to exist, at f/1.8 the G channels hits white_max
	}
    },

    { // quality A
        "make_model": "Sony SLT-A77",
        "dcraw_matrix": [ 5126,-830,-261,-4788,12196,2934,-948,1602,7068 ], // matrix provided by Colin Walker
	// note: Sony ARW2 file format gets 14 bit values, not 12 bit like in dcraw 9.19
        "ranges": { "black": 512, "white": 16350 }
    },
    { // quality A
        "make_model": [ "Sony SLT-A99", "Sony SLT-A99V" ],
        "dcraw_matrix": [ 6344,-1612,-462,-4863,12477,2681,-865,1786,6899 ],
        "ranges": { "black": 512, "white": 16350 }
    },

    /* Phase One: color matrices borrowed from Adobe DNG Converter, black/white levels tested on actual raw files.
       Note: the dcraw decoder makes black level subtraction and various corrections to the raw values based on
       metadata embedded in the IIQ format, so what we see here is the result after that, ie black level is 0,
       and white level is typically at or close to 65535. However sensors vary a bit and can be noisy around clip
       so we cut away 0.05 stop from top down to 63300 to be a bit conservative. */
    { // quality A
        "make_model": [ "Phase One P40+", "Phase One IQ140", "Phase One P65+", "Phase One IQ160" ],
        "dcraw_matrix": [ 8035,435,-962,-6001,13872,2320,-1159,3065,5434 ],
        "ranges": { "black": 0, "white": 63300 }
    },
    { // quality A
        "make_model": [ "Phase One IQ180", "Phase One IQ280" ],
        "dcraw_matrix": [ 6294,686,-712,-5435,13417,2211,-1006,2435,5042 ],
        "ranges": { "black": 0, "white": 63300 }
    },
    { // quality A
        "make_model": [ "Phase One P20", "Phase One P20+", "Phase One P25", "Phase One P25+" ],
        "dcraw_matrix": [ 2905,732,-237,-8135,16626,1476,-3038,4253,7517 ],
        "ranges": { "black": 0, "white": 63300 }
    },
    { // quality A
        "make_model": [ "Phase One P21", "Phase One P21+" ],
        "dcraw_matrix": [ 6516,-2050,-507,-8217,16703,1479,-3492,4741,8489 ],
        "ranges": { "black": 0, "white": 63300 }
    },
    { // quality A
        "make_model": [ "Phase One P30", "Phase One P30+"],
        "dcraw_matrix": [ 4516,-244,-36,-7020,14976,2174,-3206,4670,7087 ],
        "ranges": { "black": 0, "white": 63300 }
    },
    { // quality A
        "make_model": [ "Phase One P45", "Phase One P45+" ],
        "dcraw_matrix": [ 5053,-24,-117,-5685,14077,1703,-2619,4491,5850 ],
        "ranges": { "black": 0, "white": 63300 }
    },

    // dummy test entry to test the parser and show the format with all entries active
    {
        "make_model": "DummyMake DummyModel",
        "dcraw_matrix": [ 7530,-1942,-255,-4318,11390,3362,-926,1694,7649 ],
	"raw_crop": [ 10, 20, 4000, 3000 ],
	"masked_areas": [ 51, 2, 3804, 156, 51, 5794, 3804, 5792 ],
        "ranges": {
	    "aperture_scaling": [
	        { "aperture": 1.2, "scale_factor": 1.1 },
	        { "aperture": 1.4, "scale_factor": 1.08 }
	    ],
            "black": [
                { "iso": 100 , "levels": [ 10, 20, 10, 20 ] },
                { "iso": [100, 200] , "levels": [ 30, 40, 30 ] },
                { "iso": 3200, "levels": [ 50, 60, 50, 60 ] }
            ],
            "white": [
                { "iso": 100 , "levels": [ 10000, 11000, 10000, 11000 ] },
                { "iso": 3200, "levels": [ 11000, 11000, 10000, 11000 ] }
            ],
	    "white_max": 16383
        }
    }
]}
rawtherapee-data 4.0.12+dfsg-2 / usr / share / rawtherapee / camconst.json
