Home > Software > The Mirror's Surface Breaks

The Mirror's Surface Breaks

article and software by Ben Lincoln

The Mirror's Surface Breaks is an application that I wrote to automate the process of turning greyscale multispectral image sets into colour final products. It is capable of producing a wide variety of output, from fairly typical false colour:

...to very colourful, "far out" imagery:

...to truly alien interpretations that have very little in common with how the same subject matter appears to a human observer:

TMSB can be used to process a variety of different types of source material, such as multispectral satellite imagery and images taken by the Mars rovers:

TMSB is free and open source (GPLv3), and can be downloaded at the bottom of this page. It runs on Windows and can also run on Unix-like operating systems with some additional work. I've tested it under openSUSE 10.3, Fedora 16, and Kubuntu 10.10 Linux successfully.

It acts as a (multiprocessing-capable) front-end to DaVinci's Shadow, a minor fork of DaVinci, which is a command-line data-processing tool written by Arizona State University for NASA's Jet Propulsion Laboratory that is typically used for processing images obtained by the Mars rovers and orbiters as well as other spacecraft.

It is a .NET 2.0 application with both GUI and command-line interfaces, and uses XML configuration files to determine how it will process a given type of data. It includes prewritten files for handling the sort of NIR/R/G/B/UV-A photos that I take with my modified DSLR, satellite data from JPL's OnEarth website, and both pancams on the Mars Exploration Rovers. Any of these can be processed using a variety of options, and additional configurations can be created by anyone who understands XML.

For more information on the various types of image that can be generated, as well as many more sample images, see the following articles:

• Basic Greyscale Images
• Calculated Greyscale Images
• Statistical Greyscale Images
• Three-Channel False Colour
• Gradient-Mapped False Colour Images
• Luminance/Colour Images
• Decorrelation Stretch Images
• False Colour From Filters (and Simulated Filters)

The Name

"The Mirror's Surface Breaks" is a reference to "Waser", a song written by Nimpf in the late 1990s^[4]. I thought it made a great name for this particular piece of software, for two main reasons:

• A mirror reflects a single image of the world, but a broken mirror reflects many slightly different versions of that same image.
• Breaking a mirror involves the use of brute force, and this software is effectively a brute-force image permutation generator.

Unlike the mirror in the song, using TMSB is unlikely to result in "absolving [of your] mistakes". In fact, it is very likely that any defects (dust on the image sensor, compression artifacts, and so on) will be magnified many times in at least some of the variations that it generates.

Background

TMSB is very different than typical photo-processing software, because such software (Photoshop®, etc.) is typically designed based on a number of assumptions:

• The image is made up of a fixed number of channels (red/green/blue, cyan/magenta/yellow/black, etc.) which represent the scene as it appeared to the photographer, and which also represent (more or less) how the final version will appear to viewers. There are no additional channels which represent aspects that are not visible to human eyes.
• The desired end result is a single image, and therefore the progression from input to output is a linear one.
• The user has some idea of what they want to get out of the image.
• The main method of use is interactive - the user is generally manipulating the images by hand, with some excursions into simple automation.

For example, with conventional image-processing software, if I have taken a picture of a red rose, my goal with the software is typically to make that photo look as visually pleasing as possible, but in the end it will still appear on screen or in print as a red rose, and if someone were to see the end result and the real flower, they would immediately grasp the relationship between them. Sometimes the scene may be modified (removing unwanted elements, changing the colour of an object, and so forth), but it is still an attempt to convey a single image to the viewer.

In contrast, these were my assumptions when writing TMSB:

• The image may be made up of some subset of any number of base channels (spectral bands, etc.). Some or all of these may appear very different in comparison to how a human would perceive the scene.
• It is very likely that not all of the input data can be "fit" into a single output image. That is, there is so much raw information that it must be winnowed down in one or more ways.
• The end result may be many different images, each representing different aspects of the raw data.
• The end user most likely does not know what to expect from the output images - the process is one of discovery.
• The main method of use is automated. The user points the software to the input files and then stands aside while the processing takes place.

When I take a set of multispectral photos of a flower, I don't know what to expect, because my eyes can't see all of the spectral bands that are being captured. The end result may look similar to - or very different from - what I saw when I was photographing it. There may be many variations on the same image, each of which reveals something else about the same flower. There are so many possible ways to interpret the same data that it's much more efficient to have the software do it for me, then sift through the results for the ones that I like best^[5].

Tutorials

Because TMSB works so differently compared to standard image-processing software, I've written a number of tutorials on different aspects of its use:

• TMSB Tutorial 1: Basic Use
• TMSB Tutorial 2: OnEarth Satellite Imagery
• TMSB Tutorial 3: Mars Rover PanCam Images
• TMSB Tutorial 4: XML File Customization
• TMSB Tutorial 5: Custom Scripts

XML Configuration File Documentation

While TMSB allows different pre-existing configurations to be selected via the GUI, modifying those configurations or creating new ones requires editing XML files in the current version. Features common to all of the XML files are covered in the first article.

• TMSB XML Schema Part 1: Overview
• TMSB XML Schema Part 2: Input Configuration
• TMSB XML Schema Part 3: Output Configuration
• TMSB XML Schema Part 4: Processing Configuration
• TMSB XML Schema Part 5: Colour and Gradients
• TMSB XML Schema Part 6: Transformation Profiles

Why Is It Free?

TMSB is "free software" in both senses of the word.

It is "free" in the sense of not having a purchase price because I hope to encourage other people to work with multispectral imaging. There isn't really any good, general-purpose software in this field (at least that I'm aware of), and manually performing the same work is time-consuming and tedious.

I decided to make it "free" in the sense of releasing it under version 3 of the GPL because I felt like the makers of other free software (DaVinci and Hugin in particular) had set a good example by giving som ething useful to the community and not asking for anything in exchange, other than that improvements made to the software would be available in a similar manner. In the specific case of DaVinci, being able to build TMSB on top of that existing framework saved me a lot of time and added capabilities it wouldn't have otherwise, and while I could have done that and still kept TMSB closed-source, I would have felt like kind of a jerk.^[7]

I certainly wouldn't turn down a donation of the spare digital X-ray system, thermal imager, box full of camera lenses in excellent optical condition, or other interesting hardware that someone has sitting unused in their basement, but the main thing that I hope to gain is seeing the uses the people put this software to. So if you do use TMSB to process images in some way, I'd love to see the results, or at least know about the uses you've found for it if you can't share the images. Please also let me know if I can mention them here. If you'd like to share custom configurations or scripts, please send those my way too.

Future Features

TMSB has a lot of capabilities in its current version, but there are numerous features I plan to add in future releases:

• A GUI for editing the input, processing, and output configurations. Currently this requires working with the XML configuration files. Because of the complexity involved, this will be a fairly major effort.
• Internationalization/translation into languages other than English.
• The ability to create a "whitelist" of output images, so that only those permutations which are present on the whitelist (or required for the creation of one or more images on the whitelist) will be created.
• More filter simulations. This will be introduced much sooner than the other features on this list, as I just need to make time to take measurements and create the appropriate configuration file(s).
• An add-on component/second application which will group similar output images and rank them by quality (using various user-customizable means), so that automation can not only generate the images, but perform the first pass of finding the best results.
• The ability to output scaled images as well as full-size (e.g. to create thumbnails and web-sized images automatically in addition to the regular images).
• The ability to define more types of custom mapping (for example, for luminance/colour and DCS images).
• More ways for advanced users to incorporate custom DaVinci script code (for example, in transformation profiles).

Known Issues

This initial release of TMSB is certainly not free of problems. Here are the ones that I'm aware of:

• You will need to use the 64-bit DaVinci's Shadow script processor to process images more than about 13MP in size using the advanced processing functionality. See TMSB Troubleshooting and Optimization for more details on specific image resolutions. This requires running everything on a 64-bit OS.
• TMSB uses a lot of disk space. This is by design. Most working directories for my 6MP images end up being about 10-30GB in size.
• Images which have been saved in a lossy format at any point in their existence will yield poor results in many cases, even though on their own they appear to be of good quality. This is particularly noticeable in DCS variations. This is a limitation of the lossy formats (JPEG, etc.). Use lossless formats (TIFF compressed with zip/deflate or LZW, etc.) instead.
• The ability to estimate the remaining time until completion does not work in the GUI. It functions as expected in the command-line version, but for some reason, the results in the GUI were so comically wrong that I disabled their display entirely.
• Statistical calculations and gradient-mapping are noticeably slower than other operations. See the DaVinci's Shadow page for more information.
• In the GUI, there is no way to view the filename of an image that is being displayed as a thumbnail on the sliding display other than right-clicking on it to bring up the context menu. Originally, I had ToolTips which would display the filename when the mouse was hovered over thumbnails, but I discovered that there is a bug in .NET 2.0 in which ToolTips leak User or GDI objects (I forget which of the two), and so after displaying about 5000 thumbnails, TMSB would crash. Unfortunately, while there is a workaround for this in older versions of the .NET framework (explicitly calling Dispose() on the ToolTips), this does not prevent the leak in .NET 2.0. It may be fixed in newer releases of the Framework (3.0, 4.0, etc.), but using newer versions of .NET would make the application more problematic to support under Mono. An alternate approach would be to display the filename underneath the image, but that would use up valuable screen space. I am considering making ToolTip-style objects (without the bug, of course) from scratch and using them in a future release, and other potential workarounds.

Frequently-Asked (and Anticipated) Questions

• Can I use it to process regular photos, as opposed to multispectral images?
  Yes, but in most cases the results won't be very interesting. In particular, there need to be at least four distinct spectral bands for most of the processing to take place. If you're interested in trying it out, but don't have multispectral photos of your own, try using the example sets that are also available here.
• What fidelity/bit-depth is used for processing?
  Internally, TMSB converts all data to 32-bit floating point values, with the exception of gradient-mapped images (which use a 16-bit integer value pointing to a palette containing 65536 colours, each of which is represented as 32-bit-per-channel floating point). This is the highest precision that DaVinci currently supports (it has a double data type, but this data is truncated to 32-bit in some/all sections of code). The bit depth is further constrained by the primary output format selected by the user, which is currently recommended to be 16-bit-per-channel except for expert users with enormous amounts of disk space.
• Are you going to make a realtime/interactive version?
  Probably not, unless some sort of miracle occurs and I become able to work on this sort of thing professionally. I would love to see such a tool, but even this batch-mode software took a considerable amount of time to develop.
  My thinking is that a tool like TMSB was necessary before anyone could make effective use of a realtime utility, and even if one is ever developed, a batch/permutation-generator like TMSB is still the best way to determine what output of the realtime version is likely to be the most interesting. This is simply due to the vast number of ways that the raw data can be interpreted.
  If you are familiar with business intelligence/data warehousing concepts, you can think of TMSB as being the data-mining stage, in which exploratory work is done to determine where and what the valuable data and relationships are. Only then can the most effective work be done by the people who develop the end-user-facing, repeatable-process interfaces to that data.
   
  In the amazing science-fiction future-world of my imagination, here is how things work:
  1. Engineers use TMSB (or a similar tool) to determine the top n best ways of interpreting a given type of multispectral image data for a given purpose (art, forensics, mineral analysis, et cetera).
  2. Those methods are used as presets in a realtime/LiveView camera system (or augmented-reality headset - which you can think of as an incredibly advanced and versatile grandchild of today's "night vision" goggles), so that the user can flip through them in quick sequence, or the system itself can automatically determine the best one to display using some criteria like highest-contrast/best historam, and so on. Here I am imagining something like the "Predator POV" shots in Predator 2, where the Predator switches through several modes in his/her/its helmet until it finds one in which the humans' ultraviolet flashlights show up brightly.
  3. The realtime system still captures the raw data for each band.
  4. Once the user of the realtime system imports the captured data into a workstation of some kind (whether that's a traditional desktop, mobile/tablet-type device, or some other model is not really relevant), they can either run it through something like TMSB, or use a more interactive tool.
  5. At the end of the day, the engineers and the users both travel home using personal antigravity flight packs and watch true holographic entertainment, with all of the equipment being powered by fusion reactors that provide clean energy at a price so low, it's hardly worth metering.^[6]
  
   
  The interactive tool I'm envisioning is something like the visual equivalent of a software modular synthesizer (StudioFactory, Max/MSP, the editor for the Nord Modular, et cetera). That is, there are graphic objects which represent the raw bands, and then manipulation/combination objects which are linked to them in the GUI.
  For example:
  1. The user draws lines from three greyscale channel objects to three inputs on a "three-channel false colour" combination object. This object has one output which is the result of using the first greyscale channel as red, the second as green, and the third as blue.
  2. The user draws lines from all of the greyscale spectral band objects to the inputs of a statistical image combination object. This object has outputs for Median, Standard Deviation, Kurtosis, and so forth.
  3. The user draws a line from the output of the three-channel combination object to one input of a second (luminance/colour) combination object, and then draws a line from the Variance output of the statistical image combination object to the second input of the luminance/colour combination object. Now, the output of that luminance/colour object is a colour image in which the colour of the three-channel image created in the first step is applied to the luminance created by calculating the variance across all bands of the data.
  4. Finally, the user drags a line from the output of the luminance/colour object to a decorrelation-stretch processing object. This object has only one input and output, but has virtual knobs/sliders/switches to control parameters like variance and whether or not to perform a "running DCS". The output of this final stage is the one the user actually treats as the end product.
  5. Like TMSB's configuration files (and modular synthesizers) this model constructed by the user is then available for use as a template to process any input that matches the pattern of the one used to create the original model.
  
  However, that dream-world is far, far in the future, if it exists at all. For now, I leave its creation as an exercise for the reader.
• Will it run on a Mac?
  It should be possible, but I don't have a Mac to test it on and work out any issues. For now, the closest you are likely to get to running it "on a Mac" is to run a virtual Windows machine (in Parallels, or a similar product) and run TMSB on that. If you are very adventurous and technical and get it working natively, please let me know. If someone wants to donate a PowerBook or other MacOS laptop (it doesn't have to be high-end or brand new, but it does need to have an Intel processor, unless you want me to try to make it work on the older Power-based hardware), I'll see what I can do. I don't have room for a MacOS desktop, even if it's free.
• Why didn't you write it in Java, or another language that has a more robust cross-platform presence?
  I used to do Java development in the late 90s. The main issue I have with Java (as a language/platform) that would have caused me issues when creating TMSB is that it has a very poor method of building graphical interfaces. Basically, I didn't want to end up with something that looked like I'd stolen it from a Unix machine running the X Window System circa 1989.
  Also, now that Oracle owns Java, I figure they will use their legendary anti-Midas touch to ruin everything even remotely good about it, and I don't want to invest in a platform doomed to that fate.
  Finally, I have a lot of experience with .NET. I don't learn programming languages unless I have a need that isn't met by one that I already know - I have enough other projects on my list already. This is why - for example - I didn't use C or C++: there is nothing those languages can do that I want to make use of that I can't already do in C#, and so I have no motivation to learn them well enough to build an equivalent application in them.
• Is there a public version-control repository?
  No. I maintain my own, private repository. If you see a public repository that includes TMSB, then someone else created and maintains it. This website is the only official source for TMSB.
  If you have source-level contributions that you think others would benefit from, please get in touch with me using the Contact page.

Update History

In reverse chronological order:

Version 1.3 - released 2012-01-15

• Updated the DaVinci's Shadow script processor to take advantage of the version 2.09 code from the original DaVinci project and some additional changes of my own (see the information on version 2.9.2 in the DaVinci's Shadow article for details).
• Added a 64-bit version of the DaVinci's Shadow script processor to the Windows® distribution, for working with high-resolution images.
• The builds of DaVinci's Shadow included with the Windows® distribution (both 64- and 32-bit) are now cross-compiled using MinGW-w64 on Linux instead of being compiled using MinGW/MSYS on Windows®
• TMSB can now be used on Linux!
• ...and BSD!
• Memory use is now greatly reduced for decorrelation-stretch images.
• Added a uname.bat file to the Windows® distribution so that timestamps are recorded correctly in the logfiles - testing of that functionality in 1.2 had been done on a Windows® machine with MSYS and MinGW installed, and one of those packages includes a uname.exe that is not present on most Windows® systems. This will also correct other DaVinci/DaVinci's Shadow scripts which use OS detection, as that depends on the uname command.
• Improved error handling in several areas of the code.

Version 1.2.2 - released 2011-12-20

• Memory use is now greatly reduced for luminance/colour image processing and statistical image cube generation.

Version 1.2.1 - released 2011-06-04

• Improved luminance/colour blending, so that it now matches the results in Photoshop® - see Secrets of Photoshop's Colour Blend Mode Revealed (Sort Of)

Version 1.2 - released 2011-05-01

• Added support for custom script variables in the processing configuration.
• Modified the pre-XML-parser XML handling to better support Mono and its wacky UTF-16 handling that is different for MemoryStreams than for FileStreams. Yes, even after the Mono team closed the bug I opened about this.
• Added tutorial configuration files.
• Added an "absolute difference" variation to the Focused: Calculated Images and Focused: Calculated Images (Greyscale Only) processing configurations.
• Added a limit (currently hardcoded to 12) on the number of source spectral bands that may be present before TMSB will not attempt to calculate the number of output variations when the processing configuration is changed in the GUI. This is to prevent extremely long wait times (e.g. five minutes) for unusual configurations. This limit will be user-configurable in a future release.
• Fixed a bug in which existing files would never be overwritten even if the RegenerateExistingFiles option was enabled.

Version 1.1 - released 2011-02-22

• Fixed an issue related to internationalization (specifically, countries where the comma is used instead of a decimal point).

Version 1.0 - released 2011-02-21

• This was the initial public release of TMSB.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Patch)	73 KiB	1.3.1	2013-05-05	Ben Lincoln
The files to update TMSB to version 1.3.1. Apply these onto either the Windows or Unix/Linux/MacOS version.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (for Windows®)	73 MiB	1.3	2012-01-15	Ben Lincoln
Version 1.3 of TMSB - includes 64-bit and 32-bit versions of DaVinci's Shadow compiled for Windows®. If you are upgrading from a previous version, you will need to update your script processor path in the Execution tab! See TMSB Tutorial 1: Basic Use!

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (for Unix-like systems)	2 MiB	1.3	2012-01-15	Ben Lincoln
Version 1.3 of TMSB - DaVinci's Shadow is not included in this package, which is intended for users of Unix-like systems who will be compiling it themselves. You must download and compile the separate DaVinci's Shadow 2.9.2 package in order to use TMSB!

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Source Code)	1 MiB	1.3	2012-01-15	Ben Lincoln
This is the source code for version 1.3. You only need this file if you want to build your own version of TMSB in Visual Studio or another C# development environment.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Patch)	24 KiB	1.2.2	2011-12-20	Ben Lincoln
A file to update TMSB to version 1.2.2, which includes optimizations for memory use in statistical and luminance/colour images.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Patch)	23 KiB	1.2.1	2011-06-04	Ben Lincoln
A file to update TMSB to version 1.2.1, which includes improved luminance/colour blending.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks	37 MiB	1.2	2011-05-01	Ben Lincoln
Version 1.2 of TMSB.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Source Code)	1 MiB	1.2	2011-05-01	Ben Lincoln
This is the source code for version 1.2. You only need this file if you want to build your own version of TMSB in Visual Studio or another C# development environment.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks	38 MiB	1.1	2011-02-22	Ben Lincoln
This is a bug-fix release that corrects an issue which caused TMSB to crash when run in an environment that used characters other than a period for the decimal separator.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Source Code)	2 MiB	1.1	2011-02-22	Ben Lincoln
This is the source code for version 1.1. You only need this file if you want to build your own version of TMSB in Visual Studio or another C# development environment.

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks	38 MiB	1.0	2011-02-21	Ben Lincoln

Download
File	Size	Version	Release Date	Author
The Mirror's Surface Breaks (Source Code)	2 MiB	1.0	2011-02-21	Ben Lincoln
This is the source code for version 1.0. You only need this file if you want to build your own version of TMSB in Visual Studio or another C# development environment.

Related Articles:
DaVinci's Shadow
TMSB on Unix-like Operating Systems
TMSB Tutorial 1: Basic Use
TMSB Tutorial 2: OnEarth Satellite Imagery
TMSB Tutorial 3: Mars Rover PanCam Images
TMSB Tutorial 4: XML File Customization
TMSB Tutorial 5: Custom Scripts
TMSB Troubleshooting and Optimization
TMSB XML Schema Part 1: Overview
TMSB XML Schema Part 2: Input Configuration
TMSB XML Schema Part 3: Output Configuration
TMSB XML Schema Part 4: Processing Configuration
TMSB XML Schema Part 5: Colour and Gradients
TMSB XML Schema Part 6: Transformation Profiles
TMSB XML Schema Part 7: Execution Options
A Detailed Introduction
Basic Greyscale Images
Calculated Greyscale Images
Statistical Greyscale Images
Three-Channel False Colour
Gradient-Mapped False Colour Images
Luminance/Colour Images
Decorrelation Stretch Images
False Colour From Filters (and Simulated Filters)

Last updated: 05 May 2013

You have Javascript disabled in your browser. This website will look a bit nicer if you turn it back on.

Copyright 2009-2013 Ben Lincoln, except where explicitly noted.

Copyright Information - Privacy Policy - Download Policy - Advertising Policy