Processing RAW files in Linux

[ Sunday, 15 June 2008, michalrz ]


In this article I’m going to show the benefits of storing the photos in RAW “format”. I’ll be also decribing GNU/Linux tools that can help you with processing such images.

Dragonia Magazine

I framed the word “format” with quotation marks on purpose, since it’s not a particular way of saving an image; it is rather an umbrella term for many formats, implemented by digital camera manufacturers in their products. Nikon makes use of the nef format with partially hidden info about correction curves, Fuji has its raf format, etc.

The term RAW, used in the field of digital photography, comes from English and has the same meaning as “raw”. It is common for manufacturers to supply customers with software for decoding their proprietary software.

Decoding is not an easy thing to do, because the light-sensitive points on an array (CMOS or CCD) do not sense colors, only light intensity (the light is filtered separately for each primary color), which means creating a “mosaic” image in the camera. The camera then has to process the image in such manner, as to provide each pixel with full information about the color (based on the neighboring monochromatic points). It is also necessary to correct the white balance, and some cameras also adjust the contrast and saturation. The electronic hardware in the camera has thus a plenty of complex operations to accomplish after the shutter button is pressed.

Because the computational power of even the most expensive cameras is limited, and an important feature affecting the choice of a camera is the number of photos per second and the overall operation speed. Manufacturers often take the shortcut and implement such processing and compression algorithms (to JPEG or TIFF), which do not optimize the picture’s quality. They even use tricks such as ignoring data from pixels on the edge, since they are computationally “difficult”. Meanwhile we could perform all these actions ourselves using much more powerful processors (present in a typical desktop computer), while preserving better image quality.

Advantages of RAW in a nutshell

  • Ability to use a better algorithm for converting colored dots into pixels while processing the pictures using a computer.
  • Better control over white balance (while correcting the balance of a JPEG we are working on a picture already “spoiled” by the camera).
  • JPEG is limited to 8 bits for each color channel (256 levels), while the camera usually records 12 bits (4096 levels). This data is accessible in RAW.
  • Better exposition correction than simply darkening or brightening of the picture, or gamma correction.

The basic and the most obvious drawback is the size of the resulting file (e.g. to a 2 MB JPEG there may correspond 15 MB of losslessly compressed RAW) and the need to convert the image before previewing in popular image viewers.

The problem for many camera owners is that not all manufacturers support RAW photography; sometimes they even give up TIFF in favor of JPEG in several quality variants.

In the following part of this article we will focus on a couple of GPL-licensed applications capable of handling these digital negatives.

The dcraw tool

This command line program authored by Dave Coffin and constantly enriched with support for new formats is a basis for many other GNU/Linux applications. The documentation is available multiple languages and the program itself allows you to use several interpolating algorithms (changing the “mosaic” of points into pixels), reconstructing details in overbrightened areas, noise reduction, choosing color space (sRGB, Adobe RGB, XYZ…), etc.

One serious drawback is the lack of preview, which means converting to ppm or tiff each time you want to view changes made (and this takes a couple of seconds).

The upper photo in figure 1 shows the result of running the command:

dcraw -n 150 -H 5 -w -q 3 -b 2.5 photo.raw

and saving the result as JPEG using GIMP, while the lower image has been cut out of a lossily compressed photo created by the camera. The option -n sets the level of noise reduction, -H – the way of interpreting very bright areas (0 – reduction to whiteness, 5 – reconstruction), -w forces the use of white balance info supplied by the camera.

Figure 1 – Upper photo – conversion from RAW using dcraw, lower – JPEG from a camera (Panasonic DMC-FZ30). A section (1/5 of the original) framed from the middle.

The interpolation algorithm is chosen by the -q switch (where 0 – low quality, 3 – AHD algorithm). It may be necessary to set the brightness level by trial and error (adding a number after -b)

Adding a -v switch will make the program verbose. For the sake of comparison the upper photo has been saturated with color (to make them similar). The colors of the photo created by the camera are rich, but even with the highest quality set, there are typical JPEG artifacts visible. Image created with dcraw out of RAW has more details in the bright areas, the colors are more balanced but also more realistic (lacking the spots visible on JPEGs, especially on tree trunks).

Figure 2 presents one of dcraw functions – restoration of overbrightened areas. Restoring areas from too dark or too bright will be much more successful using RAW and curve correction, than the similar correction done in GIMP on a lossy JPEG image.

Figure 3 shows restoring details in RAW (upper) and JPEG. In both cases a similar flattening correction curve was used.

The upper image appears to have more different colors, but with less saturation. Green in the lower photo is more juicy, but practically homogenous. Moreover, the reddish clouds on the left have much less details in the lower photo. The lower, in turn, has sharper edges and more saturated colors (but less hues). The original has been marked in figure 2 by a red stripe.

Rawstudio graphics editor

For much easier processing of the RAW pictures we can use a program called Rawstudio. For opening images it utilizes dcraw behind the curtains, thus granting support for a broad spectrum of popular digital cameras. However it is equipped with a very friendly, dark (facilitating working with graphics) interface made in GTK and the ability of opening the entire directory with photos simultaneously. In the upper part of the interface there is a toolbar displayed with thumbnails allowing you to switch between the pictures. Corrections may be made to any of the photos and immediately after that one may proceed with another (The program records changes on the run in the XML files placed in the picture directory). Controls altering the images are arranged in three tabs, enabling quick comparison between different configurations. Additionally, there is a split-screen mode, juxtaposing two versions of the same photo.

Among other things one may adjust correction curves and color saturation; exposition and modifications are made in a 16-bit color depth. Images can be queued to launch automatic modifications according to a predefined pattern (including a pattern for file naming).

Parameters can be copy-pasted between photos. A bit annoying detail is the fact that while using the correction curve some adjustments may yield a total blackening of the picture. But, that aside, Rawstudio is a very potent and convenient application, undergoing constant development. What may seem strange is that it lacks noise reduction, despite it being supported by dcraw.

Figure 2 – Uppermost photo – conversion from RAW using dcraw with -H9 option (emphasis on bright bright areas), lower – attempts at achieving similar results using gamma correction and JPEG exposition; red frame marks the original.

Figure 3 – Upper – curve correction in Rawstudio (working on a RAW file), lower – similar curve on JPEG in GIMP.

There may also be problems opening pictures taken by some Fuji cameras, since an unorthodox layout of the array points results in the picture being rotated. Such pictures may be properly opened using dcraw, although they get rotated counterclockwise, but that may be handled easily (-t option in dcraw). Rawstudio, in turn, shows the photo in a very weird position, which (as for version 1.0) I haven’t been able to handle properly.

Figure 4 – Rawstudio interface, split-screen view.

UFRaw

This is yet another free solution, operating either as a plugin for GIMP, or a stand-alone application.

Launched, the program welcomes the user with a picture choice window, which, even after opening an image for editing, remains present (but inactive) under the main window, allowing to open another file when needed. It lacks the photo directory browser known from Rawstudio.

The program supports (among others) Polish language interface, but it still requires many fixes. Nonetheless one quickly gets accustomed to the manifold of different options. These options are categorized under particular concepts like white balance, curve, color management, framing and rotating.

The program displays a lot of useful information like dimensions, EXIF data, histogram. It also lets the user choose details such as interpolation algorithm (just like dcraw).

In general it is much more mature than Rawstudio. The interface lacks directory preview and it’s bright, but instead stuffed with more options.

Figure 5 – UFRaw.

All these programs allow compressing raw data into a lossy format (usually JPEG), and certainly grant the user much more control over retouching the photo, offering more details to work with. What’s more, it can often be done much better (closer to our expectations), than by the camera.

Certainly, not every picture requires such elaborate processing, and hard drives tend to fill quickly if we decide to keep photos exclusively in RAW. Some cameras (like Panasonic FZ30 used for our tests), even when set to RAW, make an additional JPEG file to allow for later choice which one to keep for further modifications.

Links:

Subscribe to RSS feed for this article!

15 Comments

fold this thread oled  Tuesday, 24 June 2008 o godz. 1:28 pm #  Add karma Subtract karma  +0

Take a look at Raw Therapee, it’s alot more mature than any of the above.

(Comments wont nest below this level)
 
fold this thread Lóránd Somogyi  Friday, 11 July 2008 o godz. 12:18 pm #  Add karma Subtract karma  +0

Really nice article, with lots of useful general information about raw formats. And fine links too. Thx.

I have a short article on RAW processing too, including few non-OSS too.

(Comments wont nest below this level)
 
fold this thread kaktus  Friday, 11 July 2008 o godz. 12:55 pm #  Add karma Subtract karma  +0

The biggest advantage of using RAW images are possibilities of controling colours – using RAW it is more effective when you try to change anything, because when camera change picture format to bitmap, camera includes gamma-correction* and because of that all post-processing operations are nonlinear (because gamma-correction is nonlinear)

*gamma-correction is colours correcion which is used to give better results on CRT screens, because of different wave-lenghts of different colours, some colours are changed (plasmas and LCDs do invert gamma-correction, because they are prepared for typical signal – it is a left-over after first colour TVs)

btw. sorry for my english:)

(Comments wont nest below this level)
 
fold this thread Area Code 819  Friday, 27 April 2012 o godz. 9:50 pm #  Add karma Subtract karma  +0

It is perfect time to make some plans for the future and
it is time to be happy. I’ve read this post and if I could I want to suggest you few interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read even more things about it!

(Comments wont nest below this level)
 
fold this thread Gabriel597  Monday, 1 October 2012 o godz. 10:20 pm #  Add karma Subtract karma  +0

Its really a nice and helpful piece of information.
Im glad that you shared this useful information with us.
Please keep us informed like this. Thanks for sharing.

(Comments wont nest below this level)
 
fold this thread Elissa  Sunday, 16 December 2012 o godz. 3:53 am #  Add karma Subtract karma  +0

Hello, i feel that i noticed you visited my web site thus i came to return the desire?
.I’m trying to find things to enhance my web site!I suppose its adequate to use a few of your ideas!!

(Comments wont nest below this level)
 
fold this thread Kathleen  Friday, 11 January 2013 o godz. 2:50 pm #  Add karma Subtract karma  +0

I do consider all the ideas you’ve introduced in your post. They’re really
convincing and will certainly work. Still, the posts are very quick for newbies.

May you please prolong them a bit from subsequent time?
Thank you for the post.

(Comments wont nest below this level)
 
fold this thread Bettye  Friday, 26 April 2013 o godz. 5:13 am #  Add karma Subtract karma  +0

In most cases they are, but when it comes to updates, patches, etc to the mmorpgs.
If you don’t have much skill in programming, but still think that creating and developing games is the most suitable for you, there are special courses that will give you training in graphic designing, game development, and computer animation.
It’s fast to build and play and will challenge your memory.

(Comments wont nest below this level)
 
fold this thread Anonymous  Thursday, 2 May 2013 o godz. 7:54 pm #  Add karma Subtract karma  +0

99, while the Sansa Clip+ has an entry price of $39. I know a lot more music now than before and have changed my tastes.
We are literally not the same person we were a minute ago,
let alone a day, a month or a year ago.

(Comments wont nest below this level)
 
fold this thread gratis Spiele  Sunday, 5 May 2013 o godz. 10:31 pm #  Add karma Subtract karma  +0

While some are great fans of Motorbike games, others love to pay games
where a lot of strategy comes into play. With some game clients are
growing, web games will ensure players enjoy the same quality, and eventually catch up with
the client game in the image. Everyone with plenty of cash to commit could be a winner at Small Hand Texas Holdem
Poker soon after obtaining familiarized with a handful of primary dos and don’ts.

(Comments wont nest below this level)
 
fold this thread Gclub Casino  Friday, 14 June 2013 o godz. 4:05 pm #  Add karma Subtract karma  +0

Hmm is anyone else having problems with the images on this blog loading?
I’m trying to determine if its a problem on my end or if it’s the blog.
Any responses would be greatly appreciated.

(Comments wont nest below this level)
 
fold this thread Hatteras Fishing Guides  Tuesday, 22 July 2014 o godz. 9:32 am #  Add karma Subtract karma  +0

Hey there! I realize this is kind of off-topic but
I had to ask. Does running a well-established website such as yours
take a lot of work? I am completely new to blogging however I do write in my journal
daily. I’d like to start a blog so I will be able to share my personal experience and
views online. Please let me know if you have any kind of
recommendations or tips for brand new aspiring blog owners.
Thankyou!

(Comments wont nest below this level)
 
fold this thread Tree Removal Fort Lauderdale  Friday, 1 August 2014 o godz. 6:52 am #  Add karma Subtract karma  +0

Great post! We are linking to this particularly great post on our site.

Keep up the great writing.

(Comments wont nest below this level)
 
fold this thread Pompano Beach Subs  Sunday, 3 August 2014 o godz. 5:06 pm #  Add karma Subtract karma  +0

Wow that was odd. I just wrote an very long comment but after I clicked submit my comment didn’t appear.

Grrrr… well I’m not writing all that over again. Anyways, just wanted to say superb blog!

(Comments wont nest below this level)
 
fold this thread Pompano Beach computer repair  Wednesday, 27 August 2014 o godz. 9:41 pm #  Add karma Subtract karma  +0

Hello, after reading this amazing article i am as
well glad to share my experience here with friends.

(Comments wont nest below this level)
 
Name (required)
E-mail (required - never shown publicly)
URI

Adjust field size: shrink | enlarge)


You can use simple HTML in your comments. Some examples are as follows:
  • A hyperlink: <a href="polishlinux.org">GNU/Linux for everyone!</a>,
  • Strong text: <strong>Strong text</strong>,
  • Italic text: <em>italic text</em>,
  • Strike: <strike>strike</strike>,
  • Code: <code>printf("hello world");</code>,
  • Block quote: <blockquote>Block quote</blockquote>

About the Author

Michał Rzepka

Dragonia Magazine editor, proofreader, translator. KateOS user.

New AdTaily ads!

Are you a film buff?

film buffs community, movie recommendations and reviews

RSS: Comments

You can follow the comments to this article through a special channel RSS 2.0 .

Related articles: Graphics

 more »

PolishLinux Top Content


Become our fan on Facebook!

PolishLinux.org on Facebook

Follow PolishLinux on Twitter!

Follow polishlinux on Twitter

Google Ads