RAW files, ImageMagick and WoodWing Enterprise

Recently I had a partner that I was working with who asked the following question:

“How can Enterprise accept RAW files and build large usable previews?”

Good question.


This set of explanations and instructions assume that you have a certain level of skills that allow you to compile some server tools, modify Enterprise config and Server Plugin files.

In addition, since we are modifying the stock Enterprise installation you NEED to have a plan in place that will allow you to propagate the changes you are making to newer versions of Enterprise as updates and upgrades are released.

What are RAW files?

The best explanation that I can give is that a RAW file is exactly what the image sensor recorded at the time the image was taken UNAFFECTED by any processing. I think of a RAW file as a digital negative. (Don’t confuse that term with Adobe’s DNG stuff…that’s for another post!)

When I first heard people talk about ‘RAW’ files I thought that implied that there was one file type. Not true. Each camera manufacturer creates a RAW file type that then has a file extension defined by the camera manufacturer. A tool like Photoshop understands how to interpret the file but in the end will want to write out to something like the .psd or jpeg formats.

See more here: http://en.wikipedia.org/wiki/Raw_image_format

In my case the sample files include extensions like .NEF, .CR2 and .IIQ

Image files and WoodWing Enterprise

WoodWing Enterprise uses several different mechanisms to deal with image formats.

The GD module that can be compiled with PHP is the first candidate to take a look at.

However, I know that GD does not support ‘RAW’ files. GD supports JPEG, GIF, PNG, SWF, TIFF and JPEG2000 files. If you want to see what your installation is set up to support take a look at how by reviewing this page: http://www.php.net/manual/en/function.gd-info.php

A second option is to use SIPS (Scriptable Image Processing Server). Using SIPS with Enterprise requires that:

The Enterprise server be running on Mac OS X.
The SIPS Enterprise Server plugin is installed (available on the Community site and free).
SIPS was designed by Apple to provide tools for performing basic image alterations and support for various image formats. It is a OS level tool and support includes these image file types:

jpeg, tiff, png, gif, jp2, pict, bmp, qtif, psd, sqi, tga.

As you can see it doesn’t look like RAW files are supported here either.

That leaves the last ‘standard’ choice: ImageMagick

A quick review of this page clues me that all of the sample file formats (.NEF, .CR2 and .IIQ) are supported by ImageMagick

ImageMagick and RAW files

How are ‘RAW’ files supported by ImageMagick?

They are supported by using ‘ufraw’ (Unidentified Flying Raw) which is an open-source tool that allows for working with RAW files such as those from Canon, Nikon, Pentax, Samsung, Sony, Fuji and even Adobe. Here’s the web site: http://ufraw.sourceforge.net/index.html

Most installations of ImageMagick will probably NOT have ufraw so you will have to install it so that ImageMagick can work with it.

Installing ufraw…or in my case re-installing ImageMagick and then installing ufraw

In other parts of this web site you will find entries that refer to my Mac using MacPorts. I have installed PHP5/ImageMagick using this tool and find it easy to use. Keep in mind that you have to have ImageMagick installed if you are going to use ufraw with Enterprise.

To install ‘ufraw’ using MacPorts start ‘Terminal’ and enter:

sudo port install ufraw

Hit return and enter your password.

That should do the trick. Unless, of course, you are me. I had to do a variety of messing around with MacPorts that included uninstalling a series of unused port versions before I could finally get this installed.

Once MacPorts has worked its wonders you will be left with an installation that has ‘ufraw’ installed.

Enterprise and MIME Types

Out of the box Enterprise handles a LOT of file types. This is a case, however, where we have to add a MIME type to the system so that it can handle the files.

There are a variety of file types that are considered RAW files but for this example we are going to deal with three: .nef, .cr2 and .pef

Assuming you know how to work with Enterprise at the level where you can edit the configuration files take the following steps…assuming YOU HAVE MADE A BACKUP!

Open the configserver.php file and navigate to the section labeled ‘extensionmap’. Add the following line:

‘.nef’ => array( ‘image/x-nikon-nef’, ‘Image’),
‘.cr2’ => array( ‘image/x-canon-cr2’, ‘Image’),
‘.pef’ => array( ‘image/x-pef’, ‘Image’),

Be careful here and make sure that you follow the conventions laid out in the file. I.E. if you screw up the array your users will know very quickly.

Making these changes tells Enterprise that it can handle the indicated file extension, that the MIME type for the file extension is ‘image/xxx’ and that the file type will be a ‘Image’ when uploaded to Enterprise.

The Enterprise ImageMagick Server Plugin (which ships free with Enterprise)

Right now, with the MIME type added to configserver.php you can upload the RAW files to Enterprise and they will upload and be usable. The problem is that there will still not be a preview associated with the file.

We have to make a few changes in the plugin. We have to first change a file so that plugin recognizes that it can deal with our new MIME types. Then we have to change the code a bit so that a preview can be extracted from RAW files and uploaded to Enterprise.

Changing the ImageMagick_Preview.class.php file

To modify the plugin you will need to go your Enterprise installation to /server/plugins.

Open the directory labeled ‘ImageMagick’. Inside of that directory find the file labeled ‘Imagemagick_Preview.class.php’. Open this file with a text editor.

Find the area labeled ‘canHandleFormat’. This is a function that tells the plugin how to handle a variety of formats. Add the following:

case ‘image/x-nikon-nef’:
case ‘image/x-canon-cr2’:
case ‘image/x-pef’:

Save and close the file.

What would our results be at this point?

Right now, with the configuration that has been done up to this point and the installation of ufraw Enterprise will give you a 160 x 120 preview of uploaded images. Generally that’s not going to work for most uses as the preview is to small.

How can we get a larger preview? That is going to take a bit more work. This will include modifying the Enterprise ImageMagick server plugin a bit more.

Changing the ImageMagick.class.php file

The changes here are a bit…tricky.

What we have modify this code to look for a specific set of MIME types. If the MIME type is found we then have to process the file, extract the preview from the RAW file and write it to disc so that Enterprise can upload it.

It’s the extraction part that is a bit tricky. Enterprise/ImageMagick cannot extract the preview without the help of a program called exiv2.

The exiv2 tool is pretty cool in that it has a command that will pull the jpeg preview out of the RAW file and write it to disc. In general RAW files will contain up to 3 previews. One preview will be small, one medium and one large.

In the case of my sample file I have the following:

Preview 1: image/tiff, 160×120 pixels, 57600 bytes
Preview 2: image/jpeg, 570×375 pixels, 113163 bytes
Preview 3: image/jpeg, 4288×2848 pixels, 813532 bytes

Note: you can get the above output by entering the following into ‘Terminal’
exiv2 -pp /filename.extension
Once we have asked exiv2 to extract the preview we’ll need replace the preview that Enterprise extracted with preview exiv2 extracted.

Instead of laying out the code to do this here I will post it in WoodWing’s community forum

Testing preview generation

Make sure the ImageMagick Server plugin that you have modified is loaded by going to the Enterprise Admin area. Navigate to the ‘Server Plugins’ area and make sure that the ‘ImageMagick’ plugin is loaded.

Start up Content Station. Find a Dossier. Drag a .nef file (or other RAW file depending on your configuration) to the Dossier so that you get a upload dialog. Set the meta data and press ‘OK’.

The file will upload and a preview will be created.

Memory and large files
One thing about RAW files is that they can be quite large. Make sure that the setting PHP’s php.ini file are set correctly to allow Enterprise to upload files of this size. This should have been part of the initial configuration of Enterprise but check it anyway.