Aligning Multiple Exposures in Hugin

article by Ben Lincoln

Once you have a set of multispectral exposures (probably using the method from Taking Multispectral Pictures, you can proceed to process them on a computer. The first step is to align them, and the method I use is based around Hugin, which is a front-end for the Panorama Tools command-line utilities. There is a longer explanation of Hugin in the Other Tools article. These instructions assume the use of Hugin 0.7.0, because at the time of this writing that was the most recent version for which a precompiled Windows version was available, and I don't expect most photographers (even among the subset who are unusual enough to want to use this process) to be interested in compiling their own binaries. Hugin is also available for Linux and MacOS, and I imagine these instructions are more or less valid for those versions.

The high-level reason for this work is twofold: first, to correct for any jostling of the tripod in-between your exposures. Second, to correct for the slightly different effective focal lengths that the exposures will have due to the longer (in the case of near infrared) and/or shorter (in the case of ultraviolet-A) wavelengths of light. If you are not a human, but an advanced robot or cyborg with motion-control software that guarantees the exact same shot for multiple exposures, and you are using an expensive multispectral lens whose design renders the focal lengths identical, you can skip this lengthy process altogether.

Remember that Hugin's intended purpose is for panorama stitching, not aligning multispectral exposures, so some additional steps are required for this repurposing.

RAW Conversion and Pre-Processing

Before Hugin even comes into play, it's necessary to turn the RAW files from your camera into something it can use. There are two variations on this section depending on whether or not your image-editing software supports 16-bit-per-channel images^[1]. If they are supported, you should most definitely use them because they provide an enormously greater dynamic range and fidelity than older 8-bit-per-channel formats. In the case of these steps specifically, much of the work can be performed using non-destructive editing (effects layers, etc.).

If your image-editing software supports 16-bit-per-channel images, use these steps:

Open the RAW files in the RAW converter.
If you used a grey card while shooting, use the grey card shots to automatically set the white balance of the real exposures. If performed correctly, this should cause any near infrared exposures to appear as greyscale ("black-and-white"), and reduce the purple tint of any ultraviolet-A shots. If you did not use a grey card, adjust the white balance manually to the best of your ability. For near infrared and ultraviolet-A exposures, it is generally possible to approximate a grey card by clicking on an area of the image of medium brightness. For the human-visible-light exposure, if there are any true grey objects in the shot (a concrete sidewalk, etc.), clicking on them with the white balance tool will serve the same purpose.
Configure the RAW conversion software to convert the near infrared and ultraviolet-A exposures to greyscale. This will remove any lingering tinting artifacts, and in better software will perform an initial optimization of the dynamic range.
Disable any other processing. For example, Adobe RAW defaults to boosting brightness, contrast, etc. Make sure all of these options are set to the equivalent of 0 (no change from the RAW file).
Make sure the output format is 16-bit-per-channel.
If the RAW converter is integrated with the image editor (for example, Photoshop/Adobe RAW), open the files. If you are using a separate RAW converter, save the images as 16-bit-per-channel TIFFs or another lossless format that supports 16-bit-per-channel files (PNG, etc.), then open those files in your image-editing software.
In the image-editing software, create a Levels effects layer for each of the exposures and use it to normalize the dynamic range by adjusting the left and/or right sliders to "trim off" the section of histogram that doesn't contain any data. This step is performed using an effects layer so that if you change your mind later, you can come back and adjust the levels in a different way.
Save all of the exposures as separate files in the native format of the image-editing software.

If your image-editing software does not support 16-bit-per-channel images, use these steps:

Open the RAW files in the RAW converter.
If you used a grey card while shooting, use the grey card shots to automatically set the white balance of the real exposures. If performed correctly, this should cause any near infrared exposures to appear as greyscale ("black-and-white"), and reduce the purple tint of any ultraviolet-A shots. If you did not use a grey card, adjust the white balance manually to the best of your ability. For near infrared and ultraviolet-A exposures, it is generally possible to approximate a grey card by clicking on an area of the image of medium brightness. For the human-visible-light exposure, if there are any true grey objects in the shot (a concrete sidewalk, etc.), clicking on them with the white balance tool will serve the same purpose.
Configure the RAW conversion software to convert the near infrared and ultraviolet-A exposures to greyscale. This will remove any lingering tinting artifacts, and in better software will perform an initial optimization of the dynamic range.
Use the RAW converter's processing options to normalize the dynamic range of all of the exposures (for example, by adjusting the response curves). The histogram of the image should generally extend from the left side to the right, but not be "cut off" at either end. Near infrared and ultraviolet-A shots will often start out with a histogram that appears as a narrow spike in the middle of the graph; the goal here is to spread that spike out into a nice curve. Performing this step in the RAW converter is necessary for this variation because after downconversion to 8-bit-per-channel, over 90% of the image fidelity (depending on the specific RAW format) is permanently lost.^[2]
If the RAW converter is integrated with the image editor (for example, Photoshop/Adobe RAW), open the files. If you are using a separate RAW converter, save the images as 8-bit-per-channel TIFFs or another lossless format (PNG, etc.), then open those files in your image-editing software.
Save all of the exposures as separate files in the native format of the image-editing software.

White Balance of Non-Human-Visible Exposures

Uncorrected

Corrected

The first of these images shows one of the ultraviolet-A exposures from "Stovetop Bokeh" (see Thermal versus Near Infrared) as it appeared uncorrected in RAW format. The purple tint is an artifact introduced by the white balance settings. After adjusting the white balance as much as is possible for UVA exposures (which - unlike near infrared - cannot be fully compensated for by white balance alone) and selecting the Convert to Grayscale option in the RAW converter, the exposure appears in more accurate form as depicted in the second image.

Normalizing Dynamic Range Using Histograms and Levels

An image with a poor histogram

An image with a better histogram

How to turn the former into the latter using levels

The two versions of this image differ only in the the second has had a Levels adjustment layer applied in Photoshop® in order to normalize its dynamic range. In the original version, you can see that most of the histogram is empty, with a narrow spike in the middle. By "cropping" the dynamic range and expanding what's left to fill the entire histogram (which is what this use of Levels does), the clarity is greatly improved. This type of processing is especially critical for exposures that capture non-human-visible light. Obviously you should endeavour to shoot the photos in a way that results in as well-balanced a histogram as possible, but this software enhancement is usually necessary even with a very well-composed shot when using lenses not specifically designed for multispectral use.

Image Cleanup

Now that the images are open in the editing software, use whichever means you are comfortable/familiar with to remove any dust or other minor defects. If there are any near infrared or ultraviolet hotspots, do not attempt to correct them at this stage, because doing so convincingly requires working with a composite image in a later step.

While checking for dust, I recommend using a very close eye, because dust spots that are barely noticeably in the RGB rendition of an image may become glaringly obvious in a false colour composite. For example, if a shot contains a blue sky, switch to a view of just the red colour channel and you are likely to notice a number of dark dust spots that are next to invisible otherwise. If that uncorrected red channel is shifted to the green channel (for example, for an NIR-R-G variation), those dust specks will stand out like a sore thumb. It can also help to temporarily add a brightness/contrast adjustment layer to make any defects easier to find.

Hidden Dust Specks

R-G-B

Red channel only

This human-visible image looks fine normally, but when the red channel is isolated, several dust specks (highlighted) are easily visible. If they are not removed, they will become glaringly obvious when the red channel is shifted into the green channel for false colour compositing.

Once you are satisfied that the images are good enough and have saved them (again, in the native format of the image-editing software in order to preserve any adjustment layers, etc.), save copies as TIFFs (TIFF is the best format that Hugin supports). For the near infrared and/or ultraviolet-A exposures, be sure to convert them to RGB colour before saving them as TIFFs. This additional step is necessary because while the Hugin front-end supports greyscale TIFFs, some of its underlying components (Autopano-SIFT, etc.) only support RGB formats.

Aligning the Images in Hugin

The remainder of the alignment process takes place in Hugin.

Why Registration and Effective-Focal-Length Compensation is Important

Comparison

Here's a comparison between two versions of the NIR-R-G false colour variation of a photo that I took in Bodie State Historic Park. On the left is the result of working directly with the original exposures. On the right is the result after the exposures were processed through Hugin to correct the registration and compensate for the different effective focal length of the near infrared shot.

Launch Hugin.
Select the Images tab.
Click the Add individual images... button.
Navigate to the location where you stored the final TIFFs in the Image Cleanup section, above. Select all of them, then click Open.
In the Camera and Lens data dialogue, enter the focal length of the lens you used in the Focal length: field, and the crop factor of your camera's sensor in the Focal length multiplier: field. This will automatically populate the HFOV (v): field. If you don't know the crop factor for your camera's sensor, you will need to look it up online. For example, Nikon "DX" bodies (the D70, D80, etc.) have sensors with crop factors of 1.5, and "FX" bodies (The D3, D700, etc.) have a crop factor of 1. For single-frame shots, this information is not critical, but it is very important to get it right if you are constructing a multi-shot panorama (discussed below). If your TIFFs contain EXIF data for the lens focal length and crop factor, this dialogue will not be displayed.
Verify that the Lens type: drop-down is set to Normal (rectilinear) and click OK. Again, if your TIFFs contain EXIF data for the lens focal length and crop factor, this dialogue will not be displayed.
In the list of images, select the human-visible-light version. Click the Anchor this image for position and Anchor this image for exposure buttons near the bottom of the main Hugin window.
If you have not already done so previously, go to the File menu and choose Preferences. Select the Control Points Editor tab. These values are the ones I've found most useful so far: Patch width: 21, Search area width: 5, Local search area width: 14, Correlation Threshold: 0.7, Peak Curvature Threshold: 0, Enable rotation search: checked, Start angle: -10, Stop angle: 10, Steps: 12. Select the Autopano tab. In the Select Autopano drop-down, choose Autopano-SIFT. In the Arguments: field, enter --ransac off --maxmatches 256 --absolute-pathnames on --refine %o %i (turning RANSAC area filtration on can provide better results in some situations and will make the search run faster. The corresponding set of arguments is --ransac on --maxmatches 256 --absolute-pathnames on --refine %o %i). Finally, click OK. You should only need to perform this step once - Hugin will retain the settings unless you change them again^[3].
In the main window, select the human-visible-light image and one of the others (not both at once! This will make the search take much longer and usually not provide good results). Click the Create control points button. A new window should appear. Depending on the specifics of your images and the speed of your computer, it can take a few minutes or over an hour for this step to finish. If you immediately receive a non-specific error, verify that you saved all of the TIFFs as RGB and not greyscale. Autopano-SIFT will fail instantly if one of the images is greyscale. If all of the images are RGB, you may need to manually specify the control points, which is discussed below. You can also try temporarily modifying the near infrared and/or ultraviolet-A images to make them easier for Autopano-SIFT to detect matches in with the human-visible-light shot (discussed below).
If you have multiple non-human-visible exposures, repeat the previous step for each of them (performing a match between them and the human-visible shot).
Select the Camera and Lens tab. If there is a near infrared exposure listed, select it. Near the bottom of the window, uncheck the two Link boxes in the Image Center Shift section (Horizontal and Vertical). Click the New Lens button.
Repeat the previous step for any other non-human-visible exposures (ultraviolet-A, etc.), although the Link boxes will already be unchecked. It is critical that you assign each exposure a separate lens, because the different spectral bands mean that they effectively have a slightly longer or shorter focal length (usually about 0.1mm) than the human-visible version. If you skip this step, the registration of the near infrared and/or ultraviolet-A shots will become more and more inaccurate the further away from the center of the image.
Select the Control Points tab.
In the left half of the window, use the drop-down to select the human-visible exposure. In the right half of the window, select one of the other exposures.
If there are already a large number of control points displayed, and they appear more or less accurate, you can skip to the next step. Otherwise you will need to manually specify control points. Remember that the more accurate control points are present, the better the final result, so I try to end up with at least 10-20 (and preferably over 100) between each image. Control points are specified by clicking on one of the two images. If Hugin already has at least one control point between the pair, it will attempt to automatically position the point in the other image. Otherwise, you will need to click in roughly the same place in the other image. Verify that the two control points represent the same location in both images - not just sort of the same location, but the exact same location. Choose high-contrast, fixed points for this - geographic features, buildings, tree trunks, etc. are good choices. Leaves which are blowing in the wind, shadows, and clouds are poor choices. Repeat this process until you believe you have a sufficient number of control points between the images.
If there are more than two exposures in your set (for example, human-visible, near infrared, and ultraviolet-A), leave the human-visible shot selected in the left half of the window, select one of the additional exposures, and repeat the previous step. Note that it is not important (and sometimes actually detrimental) to attempt matches between e.g. the near infrared and ultraviolet-A shots^[4].
Click the Optimiser tab. In the Optimise drop-down, select Positions and View. Then click the drop-down again and select the Custom parameters below. This will cause most of the correct custom options to be selected, with only a minor correction necessary: in the Lens Parameters section, uncheck the 0 checkbox in the view (v) subsection. Leave all of the other checkboxes alone^[5]. Experienced Hugin users will realize that the goal here is to use one of the images (in this case, the human-visible exposure) as an unchanging reference for both image orientation and field of view. Leaving all three view (v) boxes checked will usually cause bizarre results due to the lack of a solid reference point.
Click the Optimise now! button. After a few seconds to a few minutes (depending on the number of images and control points), a results dialogue will be displayed. You can make a note of the values here, or just click OK.
Open the Control Point Table (either by clicking the button in the toolbar which is second from the right, by pressing the F3 key on your keyboard, or by selecting Control Point Table from the View menu.
In the Control Points window, click on the Distance column header to sort by that column. Scroll to the beginning or end, depending on which of them contains the largest numbers.
Highlight any entries in which the Distance is greater than or equal to 1.00 and delete them. These control points are spurious and will cause alignment problems. If this means deleting all of your control points, then yes, you will generally need to start over, although usually this only results if you try to optimize parameters other than the orientiation and view. If you have a great bounty of control points whose distance is less than 1, you can nitpick further and e.g. delete any that are greater than 0.70 (or some other value).
In the main window, click the Optimize now! button again. Verify that in the Control Points window, the values are still all below 1. If they are not, delete the values greater than 1 and consider adding more control points.
Once you are satisfied with the optimized control point list, select the Stitcher tab.
In the projection (f): drop-down, select Rectilinear. This will result in an image of the same projection as with a standard camera lens. You can of course experiment with other projections, but for single-frame shots I find that all the other options are either gimmicky or lack the character of a real camera lens.
Click the Calculate Field of View button. This should populate the two adjacent fields with values dependent on the focal length and crop factor you specified when opening the images. For example, a 200mm lens with a crop factor of 1.5 will result in values of 6 and 4, whereas a 28mm lens with the same crop factor will result in values of 46 and 32.
This step is optional, but I like to have more control over the cropping - increase the two Field of View values by 1. This will result in a black border around the results, which I will trim away to my own liking in the image-editing software.
Click the Calculate Optimal Size button. This should cause the two Panorama Canvas Size values to adjust to numbers that are reasonably close to the resolution of the TIFFs you imported (plus a bit of headroom if you chose to follow the previous step). Verify that the Output section has only one box checked - under Normal, the Blended panorama option. Verify that the Normal Output: dropdown is set to TIFF, and that the Blender: option is set to enblend^[6].
Open the Preview window (either by clicking the third button from the right in the toolbar, pressing CTRL-P on your keyboard, or selecting Preview window in the View menu.
Verify that the result looks more or less as expected - you should see the first exposure in the list on top, possibly with one or more of the others visible behind it depending on how well they were aligned. If the appearance is radically different, you will need to go back and verify the previous steps. Close the preview window.
Save the panorama (I call the file "Panorama.pto" and put it in the same directory as the TIFFs).
Without making any changes to the project, save one copy of the panorama for each of the exposures (for example, I end up saving "Panorama-UVA.pto", "Panorama-NIR.pto", and "Panorama-RGB.pto", in that order).
At this point, you should have the last of these files still open ("Panorama-RGB.pto" - the human-visible variation if you're using my convention). Select the Images tab. Remove all of the exposures except the one to which this file corresponds. If your exposures were shot with a "non-CPU" lens (that is, there is no EXIF exposure information), do not make any other changes!
If your exposures were shot with a lens that provides EXIF exposure information, open the Preview window. Click the "curved arrow" button which is just to the right of the EV: label. Click the Update button at the top of the window. Close the Preview window. This step corrects for oddities introduced by Hugin's assumption that the exposure value of all exposures should be equalized (which is generally true for a traditional panorama, but not for multispectral exposures).
Save the file again.
Select the Stitcher tab. Click the Stitch now! button.
Choose a name for the output file. I use the same as the name of the panorama file, but with a .tif extension (for example, "Panorama-RGB.tif").
One the export is complete, open the next of the copies you saved (for example, "Panorama-NIR.pto") and repeat the previous five steps. Keep repeating this process until you've generated new TIFFs of each of the exposures.
Close Hugin.

Now that you have a set of aligned versions of the exposures, you can proceed on to the next article, Postprocessing, although there is additional information below that may be of use.

Temporary Modifications That Make Matches Easier (In Some Cases)

As previously mentioned, Hugin was not designed to align multispectral images. Its "similarity-detection" algorithms are heavily geared towards images of human-visible-light, or at least images that are all from the same spectral band. Frequently Hugin and/or Autopano-SIFT will fail to detect similarities that are obvious to us, such as areas on the boundary between earth and sky in a standard image (in which the ground is usually darker than the sky) and a near infrared version of the same photo (in which the sky is usually darker than the ground).

If automatic control point detection fails (and/or worse, manual control point creation results in errors due to low similarity), you can sometimes "help out" the software by temporarily modifying the near infrared version. I have only once ever had to perform this type of step for an ultraviolet-A photo (the "DIY punk spectrum" from A Detailed Introduction), but it does occassionally come in handy there.

The easiest approach is to invert one (but not both) of the images. This will make e.g. the sky/ground boundary mentioned above match between them.

You can also perform other processing, such as edge detection or modifying the brightness/contrast. Remember to keep an unmodified version of the image to revert to once the control points are created!

Inverting One Exposure To "Help Hugin Out"

RGB vs NIR (Normal)

RGB versus NIR (Inverted)

In the first image, you can see that most of the high contrast keypoints between the two exposures (from my photos of Yellowstone National Park have their brightness reversed. That is, in the RGB exposure the features may be dark against a bright background, but in the NIR exposure they are bright against a dark background, preventing Hugin from recognizing them. The second image illustrates the technique of inverting one of the exposures to allow Hugin to detect the similarity, which is how these keypoints were generated. Remember to revert to the original version before stitching and compositing!

Multispectral Panoramas

It is only a small logical step to use a modified version of this process to create a multispectral panorama, but the amount of work required is often far greater, and in my experience the results are only rarely worth it (although there are some notable exceptions). Assuming you captured the requisite exposures, the above processes can be modified in this way:

When importing the exposures into Hugin, import all of the panorama components.
When using Autopano-SIFT to generate control points, I've obtained the best results by first generating matches between all of the human-visible exposures, then generating matches between each of those exposures and the other exposures which correspond to the same shot. That is, I do not generate matches between the various near infrared shots, only between each of them individually and the corresponding RGB image. I use the human-visible exposures as the "chain" or "spine" that connects everything together. For particularly problematic panoramas, I will create control points that tie the RGB exposures together with the adjacent near infrared and/or ultraviolet-A shots, but I will still not attempt to link any near infrared to any other near infrared exposure directly.
When using the New Lens button, select all of e.g. the near infrared exposures at once, so that they all end up with the same lens number. The same is true for any other spectral band (other than human-visible, which as the reference point remains unchanged). IE for the sort of three-exposure work I do, all of the RGB shots will have a lens number of 0, all of the NIR versions will have a lens number of 1, and all of the UVA images will have a lens number of 2.
When optimizing the control points of a multispectral panorama, you must lower your expectations. You will rarely obtain values lower than 3-5, and virtually never see many that are below 1. This does mean that the alignment will not be as good, but it is essentially unavoidable for multispectral panoramas.
A rectilinear projection will not work for anything beyond a 2-shot-wide panorama, so you'll need to pick one of the other options (I typically select Equirectangular).
Keep in mind that the resulting composite will almost certainly exhibit registration errors between spectral bands. This is because the stitching software (enblend, etc.) attempts to perform its work along boundaries it determines are the least-obtrusive. Due to the differences between spectral bands, these locations will often be different, and so the result will be that some elements of the panorama are offset between the versions.

"Cross-Stitching" Multispectral Exposures

Cross-Stitching Example 1

Cross-Stitching Example 2

As noted above, I've obtained the best results with multispectral panoramas by avoiding keypoints that link adjacent exposures of the same type (other than human-visible light, of course). These two images (from Vasquez Rocks - see Drive 2007 - Day 06) illustrate my "cross-stitching" technique. In the first, the second NIR exposure is associated with the first RGB exposure. In the second, the first UVA exposure is associated with the second RGB exposure. This is in addition to keypoints that associate the first NIR and UVA exposures with the first RGB exposure, the second NIR and UVA exposures with the second RGB exposure, the first RGB exposure with the second RGB exposure, etc.

Footnotes

1.	Not to be confused with the far-more-common 16-bit-per-pixel image format. See the image-editing software section of Other Tools.
2.	Most Nikon DSLRs, for example, support at least 12-bit-per-channel RAW files. Newer/higher-end models support 16-bit-per-channel. 8-bit-per-channel images provide 256 distinct possible brightness values per channel. For RGB images, this gives a total space of 256^3 or about 16.8 million colours, which sounds like a lot until you do the same math for 12- and 16-bit-per-channel images and discover that the former provides about 68.7 billion and the latter provides over 281 trillion distinct colours. However, a more important concern is with regards to greyscale, because that's how the near infrared and ultraviolet exposures are interpreted. 256 shades of grey is very poor fidelity compared to the 4096 offered by 12-bit-per-channel or the 65536 provided by 16-bit-per-channel formats.
3.	Like all of the information in these articles, the Autopano-SIFT arguments listed are the ones that have worked best for me, not some mathematically-ideal super-values. In particular, I cap the maximum number of matches at 256 because higher values tended to cause Autopano-SIFT to crash.
4.	As with virtually all rules, there are exceptions to this one. If you were unable to obtain any matches (even manually) between the human-visible exposure and one of the others, you can attempt a match between two of the other exposures. This usually fails as well (meaning you will either have to abandon the inclusion of the problematic image, or fully manually specify control points with no automatic fine-tuning), but I have had success this way on a very few occassions.
5.	If you have an interest in experimenting with the other optimization parameters (for example, to correct various types of lens distortion), don't let me stop you. I've just found that I prefer the results when minimal interference is involved. Hugin is also a potentially very complicated piece of software, and I don't have time to write an exhaustive guide to all of the tweaks you can perform.
6.	Enblend is the component which attempts to create smooth transitions between elements of a panorama. It does not serve a useful function in the single-frame process, but if you attempt to generate the output without it, the images will lose their full X/Y alignment relative to each other. Earlier versions of Hugin allowed the output of the image with the full black border, but this was removed by the time 0.7.0 was released.