Interactive Color LED grid with IR touch sensing

After a ling delay, this project is being resumed. As of yet, I have not been able to get a working diffuser for the front that is transparent to IR for the touch sensing and semi-opaque or translucent white for the colors. Since the LEDs work well enough and I already have a frame for the assembly, I'm moving on to just making it a programmable art display.

I installed a 5mm thick light diffusion panel in the front, and finally drilled holes to mount the board in the frame with small screws. The resulting gap is about 40mm between the front of the LEDs and the back of the diffuser.

I also picked up a Raspberry Pi Zero W to use as the main CPU. The Pi will run Processing on Rasbian, and will

be connected to the Teensy CPU as before for serial communication. While reading about the methods for getting processing running on a Pi, I saw tey mention a "Upload to Pi" tool that solves a few problems that I was seeing coming up. I had worked out a whole scheme for having the Pi read files from my NAS to determine which sketch to execute, then would have had to figure out enough linux to get it loading properly automatically and when files changed for development. The Upload tool appears to cover ALL of this for me, so this post is laregely going to be setting up this tool and getting it working on a fresh laptop and a fresh Raspian. This list will also be for my reference in the future.

• Use Etcher to flash the downloaded Rasbian+Processing image
• Boot the pi and set up the following
  • Configure the hostname, etc using the system preferences
  • Configure wifi using the gui interface
  • (optionally) sudo apt-get update and upgrade all the things
• On my pc, download processing 3.x (3.3.6 currently)
• Install Upload to Pi using instructions on the GitHub page
  
  • set the hostname of the pi in the processing preferences (in my case ledgrid.fios-router.home, because i don't have mDNS set up.)

Honestly, that was about it. As I test I used the DepthSort example that comes with Processing, and I uploaded it to the Pi while I had a screen attached, and it. just. worked. I even rebooted the pi, and the sketch started again automatically!

Next was to to get it working with the test sketches I had written in the past to talk to the Teensy. I updated the code on the teensy to take a sixth byte per pixel (whether to interpret the bytes as HSV or RGB) and to take a brightness command via a magic pixel (with an x index of 100).

I made a minor tweak to the test Processing sketch to send along the sixth byte and to set the brightness. After figuring out which serial port index to use, it also just worked!

Pictures to come.

The new brackets are printed and work fine for keeping the board centered in the frame.

I failed to mention in the last post that one of the reasons I made the brackets was that the assembly got damaged, and all the connections between the boards saw heavy stresses. Quite a few pads got ripped off of the edges of the boards.

I hit each joint with the soldering iron to figure out what pads were gone, and then scraped the solder mask off each bad ground joint to make a new connection. The power connections have vias nearby, so I could use a short piece of resistor leg to make the jump. There is some redundancy due to the grid layout, but I still don't want too many joints to go bad. The brackets keep all the relative motion out of the assembly.

I also took a moment to re-evaluate the power requirements. I had originally planned for each board to be 9x9 LEDs, so a total string of 729 LEDs. At ~60mA each at full brightness, we were talking about over 40 Amps at full brightness! I decided that the we'd just not be able to run them at full power and bought a 5V 20A supply, which is still huge.

I needed to fill an item for free shipping on Amazon this week so took some time to recalculate the power supply needs.

Adafruit estimates 20mA per LED for normal use, which is about 4.5 Amps for my display. The WS2812 LEDs are by far the largest current consumer, but the IR LEDs take probably 50 or so mA more per board. I also measured the current with all LEDs at full brightness as just a hair over 2A. That means each LED is only drawing 8mA, which seems a little light.

In any case, I found a 5V 4A supply so I still have plenty of headroom, and it is a simple wall wart, so looks much better than the industrial supply. Mine does not have a power LED as shown in this image.

(Image from ebay)

I also upgraded the board to use a barrel jack that I can mount to the frame.

I have a 3d printer that hadn't been used in over a year. After a brief round of troubleshooting a bad motor driver, I was actually able to design and print some brackets for the LED board without much trouble. My design tool of choice is Designspark Mechanical (it's a stripped down but free version of Spaceclaim, which is amazing.)

I started with a basic bracket to join the four sets of inner corners. The cutout is to get around an LED that is a tad close to the corner mounting holes.

I then made models of the frame itself (a shadow box that was on sale) to help with the edge brackets. (2nd rev shown)

The brackets bolted up nicely with one adjustment to the hole diameters

But unfortunately the brackets did not make up the gap to the edge of the frame. (Gap on right here)

The prototype partition grid fits well, but still isn't even enough for the final product.

And finally, the rev 2 of the edge brackets, which should be easier to mount and should take up the remaining gap.

2 of these need to have the cutout on the long edge, and two on the short edge. The remaining 4 don't need a cutout, but I'll use ones with a cutout anyway to reduce the versions needed.

I finished routing the DAC voltage to all 9 boards, and decreased the resistance of the pull-up on the LM339 used as a voltage follower to 1K. I've increased the sensing sweep to 16 steps (but the first step is always 0V). I also updated the serial interface on both ends to ship out the touch level data, which involved removing all the unpacking code. I've captured the analog sweep on the scope with and without the serial connection active.

Without serial data: 53Hz internal refresh rate:

And with serial activity and a 49 Hz refresh rate.

So as long as the processing sketch is set to less than 45 as a framerate, it should work just fine.

I'm still not sure why it just worked, but I didn't have the problems this time around with the serial data from the Teensy to processing being slow or otherwise annoying. I did another quick video showing the full touch sensing with the depth capability.

The hardware reports the state of the touch system to the processing sketch, which decides what color to set each cell to, and sends that back to the Teensy. In this case I did a simple linear scaling of the touch distance to grayscale. In the video, it's easier to see the gradients in the processing status window.

What I'd really like to do is the full dynamic depth sensing. The idea is simple enough - vary the DAC voltage that is used as the threshold sent to all boards, and then capture what the voltage level was when the touch was first seen. I think this means sweeping from near to far. Since the IR sensors read closer to 0V when near, this should be an increasing voltage sweep. For this to be effective, I would want to get as many measurements as possible between frames of display.

The starting point is to measure what the current refresh time is, and then see how fast I can make it. The WS2812 output is fairly fixed, as I need to send out 225*24 bits at the ~800kHz data rate. A quick measure on the scope shows that the LED output takes about 7ms, and that I start sending a new frame every 9.5ms. This implies that the IR only takes 2.5ms to read, but was also measured with no serial data going.

In the touch interface, I have a hard coded constant #defined that sets the delay used in us between each bang of the bit banging, as well as some extra delay between each set of 32 bits to help me count bits on the scope. I reduced all of these to a single microsecond each, shaving 750 us off the reading cycle.

I tried to measure the time needed for serial communication, but found it hard to capture on the scope. It seemed to be only a few ms, so let's call it 6ms. If I want a ~30 Hz refresh rate, I get 33ms for each frame. With 7ms for display, and 6ms for serial, i have 20ms remaining for touch sensing shenanigans. At 2ms each, I should be able to get 10 readings in. 8 is a nice binary number, and I may need a little more headroom for serial, since I now will need to be returning 1 byte for every 1-2 cells vs the older 1 byte for every 5 cells. I think I'll use 0 to indicate "not touched" and 7 to indicate "close range". I will start with a hand tuned array of sense threshold voltages to try to get a linear response.

The #define TESTING flag allows the Teensy full control over the display instead of asking for serial input. I made a quick Green (far) to Red (near) gradient using this Gradient Maker. The resulting 7 step gradient (I did HSV through yellow) was applied to the 7 distances. I arbitrarily chose steps of 200 analog counts, which are steps of 160mV. Here's a video of the new feature in action:

Keep in mind that the first three boards are the only ones hooked to the DAC so far. I will need to increase the strength of the pull up resistor on the output of the LM339 to have enough current to supply all 9 boards.

In the future, I could consider having a calibrated distance offset for each cell to help reduce variation. The center cell of each board tends to sense sooner than the rest as you approach.

(Unlike earlier project logs, this is happening in real-time, and not a post about a previously occurring event)

In earlier discussions with my wife, when the project was in a "moving along quickly" phase, we had the idea that maybe the board would need to be smart enough to auto tune the IR sensitivity. The current scheme has a single 10K pot on each of the 9 boards, run in parallel to all 25 comparators on that board. Each 10K pot is a single turn, and is hooked to the 5V rail, so they can be pretty touchy...

Now that there is a Teensy 3.2 on board, I have an actual analog output to work with. I was considering just tying the DAC output to all 225 comparators (probably through a voltage follower to act as a buffer), but I really like the ability to tweak each board individually. So instead, I think i'll hook the DAC output to the top of all 9 10K pots. This would be the equivalent of a 1K load on the DAC, so it would draw ~3mA at 3.3V. A brief search suggests that the DAC can really only put out 1mA, so the voltage follower is still on the table as an option. This should give me global software control of the voltage, with local tweaks for any irregularities.

A blue sky version of this might actually sweep the voltage and use that to build a 3D distance map, but that seems excessive...

Since the output of the 10K pots is usually in the 0.5 to 0.7 V range, I'll target 1.5V as a starting value with all the pots around halfway to see if this improves the sensing, especially as I go to add the diffuser and such, which will mess with the IR signal.

Step 1 - cut the trace from the adjustment pot to the 5V rail:

Step 2 - set the DAC output to a constant voltage to test.

Step 3 - construct a voltage follower on a spare LM339. The blue wire comes from the DAC.

Step 4 - daisy chain the output voltage across the first three adjustment pots and test.

This actually all seemed to work. I was able to adjust the pots up to .62V which gives a reasonable IR sensing range, and the actual adjustment was far less touchy. In the next log, I'll continue the daisy chain to the rest of the pots, and then start thinking about a calibration routine (and how to trigger it!).

(June 2016)

I finally decided that the sparkfun level convertor was to blame for the bad clock bits. Yes, I know the clock wasn't even going through the level convertor, but the data was. I removed the PCB with prejudice. As I mentioned before, the shift registers are fine with 3.3V control signals, so I just needed to change the remaining two signals to the right voltage. The serial feedback from the shift registers was taken care of by a simple 5V->3.3V voltage divider. The speeds I'm talking about are not going to mind this at all.

The WS2812 remained as the last signal that needed a boost. As you can find many places online, there are standard ways as well as tricks for boosting the voltage up, ranging from a transistor to using a WS2812 powered off of a slightly lower supply rail to act as a buffer. I considered using a LM339 that was spare on my PCB anyway, but knew there must be something simpler. I found a comment to that last linked post that mentioned an unusual method that sounded just wrong, but was actually really clever. When you look at the schematic (falstad simulator), it feels wrong because it would seem that the data is flowing backwards through the diode. In reality, it is, or more accurately, the electrons that aren't the data are being pulled into the Teensy through the diode. The square wave generator in the link represents the Teensy output, and the 100k resistor represents the high impedance input of a WS2812. I used a 1N4148 and a 1k resistor from stock.

With the new Teensy to board interface done (only requiring 3 passives and a diode!), the test code worked! All that remains is to get the serial interface working on both ends in both directions.

The 12Mbps connection works seamlessly, as advertised. Processing, however, seemed to have issues keeping up with large amounts of data coming in the serial port. Originally, I had a scheme where the 6 byte packet sending color data to each color cell would cause the hardware to generate a 4 byte packet of return data (X, Y, touched, [255]), resulting in 900 return bytes per frame. After that, I had it set up such that any LED write to the last pixel would generate a packet of 225 bytes of data back to Processing, plus a [255] stop byte (226 bytes per frame). To minimize the packet size of touch data coming back over the serial port, I decided to pack the 225 bits of data into bytes, such that each segment of 5 cells was stored in a byte, so each of the 9 boards would be represented in 5 bytes. The resulting 46 byte per frame packet is still not as small as it could be, but is small enough to do the job. In contrast, the LED writing data is 1350 bytes per frame.

In debugging this mode, I set up a fixed target (my logic analyzer, now free from debugging the clock glitches) with the teensy only test code, and then switched to the serial enabled code to get the serial packets for debugging. This picture was taken so I could verify that the bits I was unpacking matched with the pixel positions I thought they did.

once the order of packing and unpacking the bits was settled, I was able to uncomment the processing code that was set to highlight any pixel being touched in green, and any untouched in blue.

The screen to the right shows processing's internal representation of the hardware, which works even if the hardware is not present so that Erica can write code without dragging around the assembly. I was able to get the processing refresh rate well above 50 Hz, so the motion and responsiveness is very smooth. I currently have the sensors set to pick up objects in the 0-5cm range. I also just noticed that you can see the IR transmitters glowing in that image because of the IR sensitivity of my cell phone camera.

The next step is getting a diffuser in front, and having the touch sensors still work. I did experiments earlier with a polypropylene cutting board that showed promise, and some poking online shows that white polycarbonate should still be transparent to IR, but can act as the diffuser in the visible range. I also experimented with using clear PVC walls in between the cells to give a little more definition to each cell, and to reduce cross-talk in the IR sensing department, which looked promising.

(March 2016)

When the Grid was running on the Arduino, the very first pixel of the last board in the chain seemed to always want to trigger differently than the rest. Otherwise, it seemed to work just fine (low bitrate not withstanding).

When I upgraded to the Teensy, I had to do somthing about the 3.3V to 5V logic level conversion. The LED board and all it's chips are 5V, and the Teensy 3.0 is 3.3V. I had one of these sparkfun level translator boards on hand, so I started there. I didn't actually need the bidirectional feature on any pin, but the 4 channels and ease of use looked great. Neither of the libraries I used on the Arduino were exactly the right fit for the Teensy, so i had to also write a bunch of new code.I replaced the NeoMatrix library (and underlying NeoPixel library) with FastLED by PJRC. It just works. I did keep the NeoMatrix lookup for converting X,Y coordinates to string index, but removed all the options for other matrix configurations.

On the shift register side, I just wrote a bit-banging routine using the Teensy's DigitalFastWrite commands to reduce the Teensyduino interpretation delays. This is where the bits went to die. For a while I thought I was just clocking things too fast, and I went through various hoops to prove that it wasn't my software, and it wasn't the PCBs. The symptom was that each board thought the last few LEDs were being touched, no matter what, but randomly. It also got progressively worse the further down the chain I went. I didn't capture the scope traces, but I was able to see both with my scope and my logic analyzer that the bits were disappearing. Since each PCB in the chain has 25 bits of real data followed by 7 bits of padding, the padding bits were getting shifted into the data area. Actually touching the IR areas of the board made it worse.

After weeks of intermittent troubleshooting, I determined that the data bits were somehow being coupled into the clock line, causing it to randomly clock extra times. I happened to have used pretty high speed shift registers, so they had no problem at all keeping up with a glitched clock. I tried just about everything I could think of, from buffering the signals through extra channels of the LM339s, to even moving the clock and chip select lines to direct I/O. The 3.3V output of the Teensy is well above the input spec for the CMOS shift registers, so didn't even need conversion. Somehow, even with this, the data is being coupled back into the clock.

Just to test everything, I went back to the Arduino and everything still worked. I also removed a row from the assembly and tested sets of 1, 2, and 3 boards with the Arduino and the Teensy. The Teensy was Just Not Working.

(January 2016)

Progress! Each board was tested, debugged, repaired, and marked. I put together a new piece of test code that can handle multiple panels at a time, and started connecting the boards together.

I designed them to be joined together with solid wire along 3 bridges on each edge. These are also connected to the power planes, so acts as power distribution. Early in the design process, I thought I'd be clever and come up with a series or solder jumpers and traces going to all four edges to make all the signals be passed along. Certain combinations of settings in the Adafruit NeoMatrix library would have lent to solving some of the routing problems, but in the end it was just making work. I used some old ribbon cable strips to join the boards together instead.

I plan to 3D print connector pieces to hold them together at the corners, but for now, I'm using standoffs and being careful not to flex the assembly at the solder joints.

At this point, I have basic functionality using an Arduino. Erica also gave me a Processing script that generates some test patterns for the array, so I could start on getting the serial interface working (on both sides). I started with the easier part, which is getting color data from processing to the Arduino, which then feeds it into the NeoMatrix library. I settled with a relatively simple format of a 6 byte message (X, Y, R, G, B, [255]). I ensure on the processing side that any color that wants to use [255] actually sends [254] instead, so that [255] can be my end of message byte.

It became pretty clear that 115200 baud was not going to cut it long term. Assuming 10 bits per byte (which includes time for the stop bits, etc), that works out to 1920 pixel packets per second . At 225 pixels per frame, the max refresh rate is only 8.5fps. 230kbaud is also not out of the question, but 17fps is still pretty pokey. I did also consider a format where the colors are just streamed without coordinates, and using the stop byte of [255] to return to (0,0), which is another effective doubling of the data rate (3 bytes per pixel instead of 6). I felt keeping the coordinate in the message would be useful in the long term, and left open the possibility of creating a command structure.

The longer term solution is to increase the bandwidth, and a Teensy 3.0 that I had on hand seemed like it would fit the bill. Not only does it have more CPU speed, it also had a nifty side feature that the "serial port" it opens will always run at 12Mbps even if you only ask for 9600bps. That's 100x faster than the 115200baud on the Arduino.

(November 2015)

Or not.

I wanted to get each panel tested individually before ganging them together. For this, I used an Arduino Leonardo because I had one handy. I wrote the basic functionality without any of the serial interface, and a simple test mode that just lights up any LED with touch data. This leaves the serial port free for debugging, and I can focus on the hardware debugging.

Since it's just for a single panel, I used Adafruit's NeoPixel library for the WS2812Bs and the Arduino's native ShiftIn for the shift registers. The hardware setup is not dissimilar to this sparkfun tutorial, though that was not the code I followed.

After some testing, I discovered that the shift registers appeared to be working (I could short high some of the spare bits at the end of the chain), but all the touch sensors always returned 0, even though the comparators should all have been putting out a 1... To the datasheet! Now look at the schematic and guess what i missed.

[screenshot is on my other computer]

Yes, the LM339 is open collector. No, I did not pull up the outputs in any way. No, I don't have enough parts (or project budget) to order a third rev of PCBs.

The only remaining answer is a great british/australian word for wedging more components into a design: bodging! The space between two LM339s just happens to be exactly the length of two 0603 resistors end to end, and the pin adjacent on one side also happens to be Vcc. The resistors on edge also happen to be the same height as the TSSOP chips, and the thickness is just a hair under the TSSOP pitch.

225 bodge resistors later and strands pulled from the thinnest stranded wire I had means the hardware is complete yet again. This time, the code all worked, and the boards could be individually debugged and fully tested.