Lepton 3.5 Thermal Imaging Camera

I finally had some time, after the holidays, to build a Prusa i3 MK3 3D printer and I've been having fun using it.  It's a fine machine.  I also took a little time to design an enclosure for tCam-Mini.   The basic enclosure is comprised of two pieces, a base and a bezel, held together with 4 4-40x5/8" metal screws with the tCam-Mini board captured between.  There are two different designs for the base.  One for the existing Rev 2 PCB and one for the upcoming Rev 4 design.

There is also a three piece GoPro™ compatible mount that clips onto the enclosure.  It's light and designed to be the thing that breaks to support tCam-Mini on a drone.

The design is in OpenSCAD and both the design and STL files ready for slicing may be found on my website or in the github repository.

It's a mobile age.   My vision for the tCam devices were to access them through an application running on my laptop but there are a lot of scenarios - as many users have pointed out - where it makes more sense to view through a mobile device.  For example when flying a tCam-Mini on a drone.

The Xojo development environment I use to write non-firmware software has long supported iOS as an output and they are now in beta test of an Android output as well so I figured it was time to finally dig in to app development.  Although I personally use Android (despite being an Apple desktop user) I started with iOS because of its maturity (and the fact that Xojo Android beta doesn't support sockets yet).  A few weeks later I'm proud to announce the first version of an iOS app available in the App Store.  It runs on iPhone and iPad.  The first version lacks some features compared to the Desktop app but I hope to improve it over time.

It provides for control of the camera's AGC, Gain and Emissivity.  It stores the radiometric images in the same json file format as used by the Desktop application (.tjsn files) and exports its private documents folder so these files can be downloaded from a computer. And it can export jpg images with or without metadata to any of the normal iOS destinations.

The libraries and rendering code I had written for the Desktop app ported right over.  Most of the time was spent dealing with the very different paradigms involved in creating mobile apps (types of controls, screen management, dynamic aspect ratios) and preparing for the App Store.

I will port this code into a xojo Android project just as soon as they support sockets (required to communicate with the camera).  I want this app running on my phone too.

There are always bugs...

A user set a password on his tCam-Mini and found the unit bricked!  Fortunately he emailed me asking what was going on.  Turns out he had set a password with less than 8 characters but the code configures the wifi stack to use WPA security (which requires >= 8 characters).  The camera had stored the password and then every time it booted and tried to start the wifi stack, the Espressif wifi library threw an error because of the bad password and my startup code froze blinking an error message.  Unfortunately I didn't sample the button when in the boot error condition so the camera was effectively bricked.  Fortunately a reload of the firmware using the Windows utility fixed the user's immediate problem.

Time for code changes.  I fixed this bug in two places.  The camera's firmware which will now ignore wifi updates with an illegal password (or ssid) and also allows a wifi reset even in the boot error condition.  I also modified the desktop application to display a pop-up error message for an illegal password instead of just blindly sending it.

There are some other issues in the Desktop application fixed.  I use a bilinear interpolation algorithm to try to generate a smooth image when it is magnified from the Lepton's native 160x120 pixel resolution to whatever higher resolution the application is showing.  Turns out the code is compute intensive enough that the application's controls could become unresponsive when streaming at the full 8.7 fps to a large window.  I didn't really see this on my OS X machine but a couple of people complained about the application freezing on Windows and Linux machines (probably some difference in the way xojo handles the event loop).  I fixed this problem two ways.  I increased xojo's optimization levels and I added a new preference to disable bilinear interpolation.  There is still some smoothing from the built-in image magnification routines.  Now even on a Raspberry Pi 4 (the slowest computer I test on) you can get full frame rate displays in large windows.

Both the firmware and new app versions are on my website and in the github repository.  If you've already loaded FW version 2.0 on your camera then you can use the desktop application to update the FW to version 2.1 (with just the "tCamMini.bin" file) instead of having to run the Espressif Windows utility.

Contributed Enclosure

I also finally got a new 3D printer (a Prusa i3 MKS+ that I highly recommend) and could print a copy of the enclosure that Zeke Gustafson designed for tCam-Mini.  He sent it to me with permission to share.  This is now in the repository and on my website.  Thank you Zeke!

New Group Gets Campaign

Group Gets recently introduced  the inexpensive Lepton FS, which is really a Lepton 3.5 that failed some tests.  It still works as a thermal imaging camera but its accuracy as a radiometric camera isn't guaranteed.  I played with one and my unit was pretty accurate at room temperature (perhaps its internal compensating temperature sensor wasn't working).  The modules identify as a Lepton 3.5 so work with the tCam-Mini FW just fine.

Group Gets is having a campaign pairing a Lepton FS with a tCam-Mini for $129. My commission and their profit on these units is pretty much zero but the campaign is a good way to get an inexpensive thermal imaging camera. 

https://groupgets.com/campaigns/978-tcam-mini-with-lepton-fs

New Firmware

Over the summer and into the fall I've been working on and testing new firmware for tCam-Mini and have posted it to the github and my website.  This version 2.0 firmware has support for the new hardware Slave IF necessary for tCam-Mini to be the sensor for tCam and several other good new  features.

• Improved performance over Wifi - The camera can stream up to the full 8.7 FPS the Lepton is capable of putting out.
• Support for Lepton 3.0 - The non-radiometric 160x120 pixel camera module is now recognized by the FW and handled properly.
• Support for over-the-air firmware updates - After loading this new version via the serial interface, future updates can be sent from the Desktop application running on any platform.

One note is that the new firmware will only run on tCam-Mini's with revision 3 ESP32 chips and at least 8 MB of Flash memory (e.g. the PCB version).  The increased memory is required to hold the multiple OTA firmware slots used by the ESP32.  Cameras based on the TTGO development board can't be upgraded beyond FW 1.3 but will remain compatible with future versions of all my supporting software.

In addition to the firmware source now on github, there are precompiled binary files, instructions and the necessary Espressif Windows utility in github and on my website.

https://danjuliodesigns.com/products/tcam_mini.html

I will document the new hardware Slave IF soon.  In addition to use with the tCam firmware running on the gCore hardware (also soon to be documented), I think it would be useful for other controllers like the Raspberry Pi which have traditionally struggled to keep up with the VoSPI interface. 

Updated Desktop Application

In addition, I've release version 2.0.0 of the Desktop application.  The primary feature is support for OTA FW updates but it also has several other changes.

• Support for Lepton 3.0 - since the Lepton 3.0 doesn't support radiometric output the application suppresses all temperature displays and controls.
• Added the ability to resize the main window.
• Added three new color palettes - Banded, Black Hot and Sepia
• Modified the FPS display to use timestamp information from the camera or video file and changed the averaging algorithm for a more accurate display over time.
• Support the new Apple Arm Macs with a universal build - still have a separate x86 build for older versions of OS X.
• Some internal restructuring and minor bug fixes.

Built for 64-bit x86 Linux, 64-bit x86 Windows, 64-bit x86/Arm Mac OS X, and 32-bit Arm Linux (Raspberry Pi).  Available in the github repository or my website.

I am so happy to make this log entry.  It's been a long haul - probably much more time than I should have spent - but my final vision for tCam is coming together. 

The hardware is based on a new version of gCore, my ESP32 dev board, which replaces the FireCam prototype I have been developing firmware on.  Aside from form-factor the main feature the new gCore brings is a much higher performance display which allows the full 8.7 fps frame rate.  It accomplishes this by using the ESP32's SPI interface running at its fastest rate of 80 MHz feeding a serial-to-parallel converter driving the ILI9488-based display through an 8-bit parallel interface.

Adjusting task priority and SPI bus speeds, along with a larger checksum, has made the communication between tCam-Mini and the tCam firmware reliable.  The shutter control has become a physical button (not shown in the pictures here) which will be mounted in a yet-to-be-designed 3D printed enclosure along with the other electronics.

About five days of coding and debug of the tCam and tCam-Mini firmware has yielded the first attempt at a higher performance version of tCam with the Lepton VoSPI pipeline and json image encoding being performed by the ESP32 on a tCam-Mini and the rest of tCam operation running on the ESP32 in the tCam prototype.  With mixed results.

The two boards are connected using a serial interface (at 230400 baud) and SPI interface (at around 6 MHz).  Communication between the two boards is via the same json command protocol that is used over the network.  All packets, except image packets, are sent back and forth via the serial interface.  This allowed easy re-use of the command processing code.  The image packet (at about 52 kB) is too large for the serial interface and is read by the tCam ESP32 from the tCam-Mini ESP32 with a SPI interface after tCam receives a new "image_ready" packet via the serial interface.

My main goal in splitting the processing between two ESP32 chips (four cores) was to improve performance and that seems to have been achieved.  I can now get over 7 fps on the tCam LCD display (up from 3-4.5 fps before).  The GUI update process is now the longest running activity per frame and limited, I think, by the maximum 26.67 MHz SPI clock rate of the HX8357 LCD controller.   The update process should be shortened, and fps increased further, by using a controller like the ILI9488 with a 40 MHz SPI interface.

Unfortunately there is occasional data corruption in the image data transferred via SPI between the two ESP32s.  SPI Signal quality looks good (clean edges and plenty of setup and hold time) so I suspect that the tCam-Mini ESP32 SPI Slave can't always keep up (or there is some data corruption by the tCam SPI Master).  The ESP32 SPI Slave is a pretty simple device and comes with a lot of caveats.  For my scenario, where it only supplies data, I setup a buffer full of data, trigger the driver to start operating when it sees chip select asserted and wait for a callback from it indicating it's ready.  Then I send a short json "image_ready" packet via the serial interface which triggers the tCam SPI master to initiate the transaction.  The transaction is pretty long, typically around 52 kB.  It might be possible to try smaller transactions except each one will need a handshake via the serial interface which, at some point, will probably make it impossible to keep up.  A kludge might be to include a checksum and simply discard corrupt images but that seems like an ugly fix and will impact performance. 

So unfortunately my initial experiment is of mixed success and there is more digging to do...

Trawling eBay netted a reasonably inexpensive Accuthermo FC100D TEC (Peltier thermo-electric cooler) controller mounted in an enclosure from Wells-CTI with a 12VDC 8A power supply and TEC H-bridge driver.  The FC100D uses PWM and bi-directional control of the TEC along with a NTC temperature sensor to provide precise temperature control (heating and cooling).  I used a 12V TEC sandwiched between an aluminum block (with a hole drilled in it for the NTC temperature sensor) and a CPU cooler heatsink with fan.  This setup can be controlled between 10-60°C.  The front surface of the aluminum block was coated with Black 3.0 paint to form a high-emissivity black body radiator.  Unlike the IVAC9000 this radiator can be adjusted to specific temperatures.

Temperature Readout Script

Previous testing (two log entries ago) revealed how the Lepton’s radiometric output varied for the first approximately 30 seconds after a flat-field correction.  Using the python library I wrote a simple script that triggers a FFC, waits 45 seconds and then averages approx once/second spot-meter readings over the next 90 seconds to create an average temperature reading for use in determining a Lepton’s accuracy and linearity over a temperature range.

I also removed the potentiometer from the IVAC 9000’s high-side because it caused too much temperature drift.  The IVAC9000 is now my temperature reference at 26°C and 38°C.  Using the new script produced the following results using two cameras with a distance of 15 cm between the camera and the radiator.  I made several runs with each camera at each temperature to check the repeatability of measurements after a FFC.  Camera tCam-Mini-B132's emissivity = 1.0, tCam-Mini-CB49 emissivity = 0.97.

The longer average values show fairly low deviation between runs (each run takes over two minutes) and the average accuracy of the Lepton is pretty high (operating at room temperature).

Comparing new radiator with IVAC9000

Next I adjusted the new black body radiator to 26°C and used the same cameras to measure its output using the new script so I could get an idea of the accuracy of the FC100D and its temperature sensor.   Several runs were done again.

Averaging all runs showed that the new radiator runs less than 0.5°C lower than the IVAC9000 unit at 26°C.

Measuring Lepton accuracy across a series of temperatures

Finally I ran a series of readings with two cameras at 10, 20, 30, 40, 50 and 60°C and plotted both the individual averages to look at variability and then an average of averages to look at a highly averaged accuracy.

Plotted data

I think the results are good news.  The Lepton appears, at least over this temperature range, to be linear device amenable to a two-point calibration (gain/offset).  Up next is to decide if a calibration is best applied to radiometric data once it’s been read from the Lepton or if it can be applied to the Lepton’s internal calculations using its RBFO parameters.

Working on tCam required changes to the Desktop Application to support file transfer.  In addition there are some new commands and responses that I just added to tCam-Mini as well.

The github repository has been updated and the files are also available for download from my website for people who don't want the entire github repository.

tCam-Mini gets the following new commands and responses

• run_ffc - Triggers a Flat Field Correction in the Lepton.  I plan to use this for data collection while reading values for calibration but it can be used anytime a FFC is desired.
• get_lep_cci/set_lep_cci - a generic mechanism to access the Lepton's CCI registers via I2C.  I think this might be useful for anyone who wants to explore the Lepton's Command Interface (within reason - it's easy to crash the Lepton or camera, for example by disabling the VSYNC signal).   These commands generate a new cci_reg response.
• cam_info - a generic response message that is used to carry things like acknowledges for other set commands or can be used by the camera to indicate a problem.

tCam-Mini also got a small performance boost by moving a buffer out of PSRAM into the faster ESP32 local memory.  I can sometimes get over 8 fps on my unit with an external antenna now.

The application got a bunch of new features.

• Ability to download files from tCam and a preference for a default folder for all image files (whether downloaded from tCam or created by the application).
• A FFC button on the main window.
• A new log window that can be displayed to watch the json packets going back and forth between the application and camera.  It's useful for me while debugging and I figure it might be useful for people who want to create their own software to communicate with the camera as they can see the contents of the json commands.
• A new CCI Register access window to allow reading and writing Lepton CCI registers (and executing run commands).
• A couple of bug fixes.

The Lepton closes its shutter, performing an action called a Flat Field Correction (FFC), occasionally as part of its non-uniformity correction (NUC) to generate a uniform output compensating for temperature changes, pixel variation and lens effects.  Closing the shutter exposes all pixels to a uniform temperature so the NUC algorithm can generate calibration offsets for each pixel.  A FFC can be triggered manually but is usually performed automatically by the Lepton based on several inputs including ambient temperature changes since the last FFC and a 3 minute timer.   Without a periodic FFC, drifts in the output values of each pixel cause increasing temperature error measurements and degradation of visual images over time.

All very good except that there is a noticeable change in the absolute radiometric temperature data after a FFC as shown below.   The graph shows the output of the spotmeter measured once per second over a 10 minute interval while imaging the 26°C IVAC black body radiator.  The FFC operation, occurring every 3 minutes, is clear.

The change is less than the specified accuracy of the Lepton but still significant if one wishes to improve on that accuracy.  I found two interesting pieces of data in this experiment.

1. It takes about a minute for the Lepton to settle down after a FFC so this time constant should be used when collecting data for calibration.
2. Outside of the period after a FFC the Lepton appears to hold about a 0.5°C precision. 

The second item has me considering a modification to the tCam firmware to show something like a green dot during the period of stability so users know when the radiometric readings are more stable and likely more accurate.

Following are expanded (time axis) portions of the above graph showing behavior during the FFC and during steady-state.

I have not given up on the goal of a full feature handheld thermal imaging camera with the features I want.  For the past several weeks I have had a chance to put significant effort into the original tCam prototype, now running on Firecam hardware because it has a 4-bit SD Card interface. 

Most of the work has been adding filesystem support and the ability to save and playback both images and videos with file formats that are compatible with the desktop application.  In addition I added commands to allow the application to offload image files from the camera and remotely control it.  While there are still performance issues - especially when attempting to stream or record and display images - the camera is feature complete - at least with respect to the features I envisioned when I started this project.  I plan to do some experimentation with buffering in the tCam-Mini code to improve performance (and also backfit some of the new command features).  I’ll see if those help tCam but I suspect it will still be necessary to offload Lepton VoSPI capture to another processor for best performance.

I am still contemplating the best hardware design for the final tCam camera but I plan to use this code with only driver changes.  The tCam codebase is about 24k total lines of C code, runs in seven FreeRTOS tasks and makes heavy use of the external PSRAM in the ESP32 WROVER module.  I did experience one of the ESP32 Revision 1 silicon bugs related to PSRAM access.  Very occasional data corruption (but repeatable, fortunately).  Replacing the strncpy library function with memcpy fixed the bug for me - probably because they have different instruction sequences.

Here are the various GUI screens showing operation of the camera from control to file management and image/video playback.