TMSB Tutorial 4D: Klaus D. Schmitt Custom Colour-Processing Matrix

article by Ben Lincoln

This tutorial describes the process for creating a custom input and processing configuration that can be easily modified to provide exactly the output you're looking for, instead of the vast number of permutations that The Mirror's Surface Breaks will normally generate. It is based on a request by Dr. Klaus D. Schmitt, who wanted effectively a reconfigurable multiplication matrix to map input images to false-colour output.

Before getting into how to build this configuration, it's important to take a detour and discuss the model I recommend for coming up with input and processing configurations in TMSB, because this configuration breaks those rules, and it's important to understand why those rules are recommended before doing that.

TMSB provides an abstraction layer between the input images and the processing configuration in the form of "special band" definitions: near infrared, red, ultraviolet-A, and so on. This is because different input configurations may contain the same spectral bands, but in different files and/or channels, but if the processing configuration is defined in terms of the special bands, those different input configurations can all benefit from the same processing configuration.

For example, the various "Vegetation Index" processing configurations are based around methods like NDVI, in which a mathematical operation is performed using the near infrared and red imagery to generate a map of where vegetation is likely to be, as well as give an idea of its general health (see Calculated Greyscale Images for a discussion of this topic). Because those configuration use e.g. the "near infrared" and "red" special bands instead of e.g. "IR1.TIF, channel 1" and "Red.TIF, channel 1", they work unchanged with all of the input types that TMSB supports (multispectral photography, OnEarth satellite imagery, and so on).

TMSB Custom Image Theoretical Model

The recommended method, and the discouraged method

Why the recommended method is preferred

The recommended framework for use when considering input and processing configurations.

The idea here is that if a particular image compositing method is useful for one type of multispectral imaging, it's probably useful (or at least interesting) for others that capture the same spectral bands.

To phrase this another way, if you have a multispectrally-converted Nikon DSLR and the common set of filters and are processing the resulting images with TMSB, I encourage you to think of the image data as being e.g. "near infrared", "red", "green", "blue", "ultraviolet-A from about 400nm to about 380nm", and "ultraviolet-A from about 380nm to about 360nm", instead of "an infrared image, a visible-light image, and an ultraviolet image, each of which contain three channels". That is, think of the images in terms of which spectral bands they represent, not the form that they happen to occupy on disk at any given time.

Now that we've covered the recommended approach, we'll discuss the configuration that uses the non-recommended approach. In this case, Dr. Schmitt is an expert scientist who knows exactly what he has to work with, and how he wants to transform those images. For Star Trek fans, this is the equivalent of Mr. Scott or Geordi LaForge using warp engine settings other than the ones that the designers at Utopia Planitia put in the warp engine manual: it's OK for them to do it because they know what they're doing, but if Ensign Greenhorn Butterfingers starts tweaking around the intermix settings, everyone had better run for the nearest escape pod.

These instructions assume that the reader has gone through all of the previous tutorials in this section.

Like all of the tutorial configuration files, complete (and tested) versions of the results below are included in the Tutorial subdirectory of TMSB_Config wherever you unpacked the software to (TMSB version 1.2 or later). These can be used to refer to if you run into trouble, or you can just copy them into your TMSB_Config directory if you would rather just use them instead of know how to create custom configurations.

Input Configuration

The standard input configurations assume that the near infrared and ultraviolet source images are greyscale, so the first phase of this example involves building a custom input configuration to make all three channels of colour near infrared and ultraviolet images available for use by the custom processing configuration which will be defined in the second phase.

In the TMSB GUI, switch to the Input Configuration tab.
In the drop-down menu, select NIR-R-G-B-UVA (TIFF - 3 Files).
Click the Export XML button.
In the File name field, enter Input-NIR_RGB_UVA-TIFF-3F-9C.xml, then click Save.
Launch your text editor, and open the file you just saved.
In the <Metadata> section:
1. In the <Name> section, change the text to read NIR-R-G-B-UVA (TIFF - 3 Files, 9 Channels).
2. In the <Description> section, change the text to read This input configuration - for advanced users who have read all of the previous tutorials and studied the recommended theoretical model enough to understand why this sort of thing is generally discouraged - makes all three channels of three source images available for use in very specialized processing configurations. If you think something else is appropriate instead, go back and re-read the first part of this article until you realize why you were wrong in that belief.
3. If you like, enter the date/time, in XML format in the <Created> and/or <LastModified> sections.
Make two extra copies of the entire <SpectralBand> block that defines the NIR spectral band, so that there are three copies total.
In the first copy:
1. Change the <BandName> value from NIR to NIR(1).
2. Remove all three lines of the <SpecialBandName> block.
3. Change the <UseAsLuminanceChannel> value from true to false.
In the second copy:
1. Change the <BandName> value from NIR to NIR(2).
2. Change both of the <Plane> values from 1 to 2.
3. Leave the <SpecialBandName> block alone - the NearInfraredNonspecific special band name can only be assigned to one band in the input configuration, and the green channel is usually the best one to use.
In the third copy:
1. Change the <BandName> value from NIR to NIR(3).
2. Change both of the <Plane> values from 1 to 3.
3. Remove all three lines of the <SpecialBandName> block.
4. Change the <UseAsLuminanceChannel> value from true to false.
Make two extra copies of the entire <SpectralBand> block that defines the UVA spectral band, so that there are three copies total.
In the first copy:
1. Change the <BandName> value from UVA to UVA(1).
2. Remove all three lines of the <SpecialBandName> block.
3. Change the <UseAsLuminanceChannel> value from true to false.
In the second copy:
1. Change the <BandName> value from UVA to UVA(2).
2. Change both of the <Plane> values from 1 to 2.
3. Remove all three lines of the <SpecialBandName> block.
4. Change the <UseAsLuminanceChannel> value from true to false.
In the third copy:
1. Change the <BandName> value from UVA to UVA(3).
2. Change both of the <Plane> values from 1 to 3.
3. Leave the <SpecialBandName> block alone - the choice of the blue channel as the preferred channel here (and as the luminance channel) is debateable, but my experience is that the blue channel is usually the "safe" bet, whereas the red channel is often blown out, and the green channel usually doesn't register much at all.
Save the file and exit.

Processing Configuration

Making effective use of the custom input configuration requires a custom processing configuration. If you attempt to use the input configuration defined above in the standard "generate a bunch of permutations" mode of TMSB, you will end up with a huge number of images, most of which are identical to a lot of the others. This is a situation where (like Dr. Schmitt) you need to define more or less exactly what you want each output image to consist of.

In the TMSB GUI, switch to the Processing Configuration tab.
Make sure something is selected in the drop-down menu (it doesn't matter what), then click the Export XML button.
In the File name field, enter Process-Custom-KDS_Matrix.xml, then click Save.
Launch your text editor and open the file you just saved.
In the <Metadata> section:
1. In the <Name> section, change the text to read Custom: Klaus D. Schmitt Processing Matrix.
2. In the <Description> section, change the text to read A processing configuration for expert users who really know what they're doing, and have read all of the TMSB documentation multiple times, because otherwise they are going to cause a warp core breach and destroy the Enterprise.
3. If you like, enter the date/time, in XML format in the <Created> and/or <LastModified> sections.
Delete everything between the <ProcessingConfig> and </ProcessingConfig> tags, but leave those two tags themselves in place.
Copy the following XML code, and paste it in-between those two tags:

<StatisticalGreyscaleCubes>

</StatisticalGreyscaleCubes>

<CustomGreyscaleChannels>

</CustomGreyscaleChannels>

<CombinedBandChannels>

</CombinedBandChannels>

<CalculatedChannelSets>

</CalculatedChannelSets>

<CustomThreeChannelImages>

</CustomThreeChannelImages>

<DCSVariations>

</DCSVariations>

<TintMapping>

</TintMapping>

<GradientMapping>

</GradientMapping>

<GenerateStatisticalChannels>

false

</GenerateStatisticalChannels>

<GenerateCalculatedChannels>

false

</GenerateCalculatedChannels>

<GenerateCustomChannels>

false

</GenerateCustomChannels>

<GenerateTintedImages>

false

</GenerateTintedImages>

<GenerateGradientImages>

false

</GenerateGradientImages>

<GenerateThreeChannelImages>

false

</GenerateThreeChannelImages>

<GenerateStandardThreeChannelImages>

false

</GenerateStandardThreeChannelImages>

<GenerateCustomThreeChannelImages>

false

</GenerateCustomThreeChannelImages>

<GenerateCalculatedThreeChannelImages>

false

</GenerateCalculatedThreeChannelImages>

<GenerateLumaColourImages>

false

</GenerateLumaColourImages>

<GenerateDCSImages>

false

</GenerateDCSImages>

<GenerateStatisticalImageThreeChannelPermutations>

false

</GenerateStatisticalImageThreeChannelPermutations>

<GenerateStatisticalImageThreeChannelPermutationsUnrestricted>

false

</GenerateStatisticalImageThreeChannelPermutationsUnrestricted>

<GenerateLumaColourVariationsFromIndividualBands>

false

</GenerateLumaColourVariationsFromIndividualBands>

<GenerateLumaColourVariationsFromCombinedChannels>

false

</GenerateLumaColourVariationsFromCombinedChannels>

<GenerateLumaColourVariationsFromStatisticalChannels>

false

</GenerateLumaColourVariationsFromStatisticalChannels>

<GenerateLumaColourVariationsFromThreeChannelImages>

false

</GenerateLumaColourVariationsFromThreeChannelImages>

<GenerateLumaColourVariationsFromDCSImages>

false

</GenerateLumaColourVariationsFromDCSImages>

<GenerateDCSImagesFromIndividualChannels>

false

</GenerateDCSImagesFromIndividualChannels>

<GenerateDCSImagesFromCombinedChannels>

false

</GenerateDCSImagesFromCombinedChannels>

<GenerateDCSImagesFromCustomThreeChannelImages>

false

</GenerateDCSImagesFromCustomThreeChannelImages>

<GenerateDCSEnhancedContrast>

false

</GenerateDCSEnhancedContrast>
In the XML block you just pasted in, change the values for the following from false to true:
1. <GenerateCustomChannels>
2. <GenerateThreeChannelImages>
3. <GenerateCustomThreeChannelImages>
Scroll back up to the <CustomGreyscaleChannels> tag. This block is where most of the content will be defined, as this configuration is built up from several layers of discrete greyscale images.
In-between the <CustomGreyscaleChannels> and </CustomGreyscaleChannels> tags, paste in the following XML code, which defines the red, green, and blue channels of the output image in terms of a 3x3 matrix made from the entries defined above in the custom input configuration.

Note that in each channel, the coefficients are placeholders - the given values will create an image in which the red channel is plane 1 from NIR.TIF, the green channel is plane 2 from RGB.TIF, and the blue channel is plane 3 from UVA.TIF. The idea here is to use different values, combining data from multiple images, e.g. = (%ChannelByName:NIR(1)% * 0.8) + (%SpecialBand:Red% * 0.2) + (%ChannelByName:UVA(1)% * 0.0) to create something that's vaguely like the red channel of an image shot with a Hoya R72 filter.

<CustomGreyscaleChannel>

<ChannelName>

Custom_Mapping-Red

</ChannelName>

<Type>

CustomScriptCode

</Type>

<CustomScriptCode>

= (%ChannelByName:NIR(1)% * 1.0) + (%SpecialBand:Red% * 0.0) + (%ChannelByName:UVA(1)% * 0.0)

</CustomScriptCode>

<UseAsLuminanceChannel>

false

</UseAsLuminanceChannel>

<UseAsThreeChannelChannel>

false

</UseAsThreeChannelChannel>

</CustomGreyscaleChannel>

<CustomGreyscaleChannel>

<ChannelName>

Custom_Mapping-Green

</ChannelName>

<Type>

CustomScriptCode

</Type>

<CustomScriptCode>

= (%ChannelByName:NIR(2)% * 0.0) + (%SpecialBand:Green% * 1.0) + (%ChannelByName:UVA(2)% * 0.0)

</CustomScriptCode>

<UseAsLuminanceChannel>

false

</UseAsLuminanceChannel>

<UseAsThreeChannelChannel>

false

</UseAsThreeChannelChannel>

</CustomGreyscaleChannel>

<CustomGreyscaleChannel>

<ChannelName>

Custom_Mapping-Blue

</ChannelName>

<Type>

CustomScriptCode

</Type>

<CustomScriptCode>

= (%ChannelByName:NIR(3)% * 0.0) + (%SpecialBand:Blue% * 0.0) + (%ChannelByName:UVA(3)% * 1.0)

</CustomScriptCode>

<UseAsLuminanceChannel>

false

</UseAsLuminanceChannel>

<UseAsThreeChannelChannel>

false

</UseAsThreeChannelChannel>

</CustomGreyscaleChannel>
In the configuration file, locate the <CustomThreeChannelImages> and </CustomThreeChannelImages> tags, which should have nothing between them at this time.
In-between those two tags, paste in the following content, which defines the custom colour image that results from combining the three custom greyscale channels defined above.

<CustomThreeChannelImage>

<UserDefinedName>

FC-3C-Custom_Mapping

</UserDefinedName>

<RedChannelName>

Custom_Mapping-Red

</RedChannelName>

<GreenChannelName>

Custom_Mapping-Green

</GreenChannelName>

<BlueChannelName>

Custom_Mapping-Blue

</BlueChannelName>

<IncludeInSecondOrderImages>

true

</IncludeInSecondOrderImages>

<OutputTransformation>

Normalize: Standard

</OutputTransformation>

</CustomThreeChannelImage>
Save the file and exit.

Output Configuration (Optional)

By default, TMSB will not overwrite existing files in the output directory. This becomes an inconvenience when using this type of reconfigurable processing configuration, because without changing that behaviour, you'll need to manually clean out the output directory every time you change the coefficients in the processing matrix.

To create a custom output configuration that forces overwriting, you can follow these steps. Be extremely careful, because it will introduce an enormous performance penalty when using TMSB with other types of processing configurations.

In the TMSB GUI, switch to the Output Configuration tab.
Select your favourite output configuration in the drop-down menu, then click the Export XML button.
In the File name field, put something like Overwrite_Existing_Files somewhere in the filename, then click Save.
Launch your text editor and open the file you just saved.
In the <Metadata> section:
1. In the <Name> section, add something like !!!DANGER: OVERWRITES EXISTING FILES!!!.
2. In the <Description> section, add something like !!!DANGER: OVERWRITES EXISTING FILES!!!EVEN IF IT TURNS OUT THAT I DIDN'T ACTUALLY WANT THEM OVERWRITTEN!!!.
3. If there is already a <Warnings> section, replace its text with !!!DANGER: OVERWRITES EXISTING FILES!!!EVEN IF IT TURNS OUT THAT I DIDN'T ACTUALLY WANT THEM OVERWRITTEN!!! Otherwise, add this block after the <Description> block:
  
  <Warnings>
  
  !!!DANGER: OVERWRITES EXISTING FILES!!!EVEN IF IT TURNS OUT THAT I DIDN'T ACTUALLY WANT THEM OVERWRITTEN!!!
  
  </Warnings>
4. If you like, enter the date/time, in XML format in the <Created> and/or <LastModified> sections.
Locate the <RegenerateExistingFiles> block, and change the value in it from false to true.
Save the file and exit.