-
Artist's talk: showing the installation and describing its ideas and meanings
02/12/2021 at 19:45 • 0 comments -
How do the machines communicate?
02/03/2021 at 18:28 • 0 commentsOne of the most fun parts of building Maelstrom has been getting ~35 machines to speak to each other. In a setting with solid infrastructure, perhaps I would use WiFi. Art galleries aren't always happy about this idea. I also wasn't thrilled about the machines possibly getting Internet access; in my perfect world, they would be completely isolated from wider networks.
I have worked with Nordic NRF24L01+ radios in other projects and decided to use them in Maelstrom. These radios accept 32 bytes of data and let you, as a developer, "press send and walk away" while they handle the mechanics of retransmission, error checking, etc.
They have other, interesting properties, like a five-byte "pipe" identification scheme that (in radio firmware) lets you ignore messages on the same channel that are not intended for a given client. And it's possible to set up a multicast network. I chose this approach for Maelstrom, using a common "message bus" with target device identifiers baked into the data payload. So all the radios are reading and writing from a common channel, at a rate of a message every few seconds (with some exceptions).
There are additional complexities with this, particularly when sourcing NRF24L01+ radios on a budget. Some of the modules sold today are using mostly-compatible clone parts. One, in particular, clones the datasheet very well--one signal described by Nordic in their own datasheet is flipped in practice in actual Nordic devices, and the clone devices cloned the datasheet instead of the real device behavior… making them incompatible in practice with real devices.
In my perfect world, I'd be able to keep each radio listening both on a group channel and on an individual address. I wasn't able to get this working across my mix of radio modules. I ended up using Pipe 1, as recommended by the RF24 documentation, to listen to just the multicast channel, and encoding an individual address into the message types that need it.
What data is shared over the radio?
I wanted to have visitor data literally being shared among the machines. A given datum is observed by one particular Maelstrom node and it shares it to two others. Each of them, if they successfully receive it, can choose to share it to two others, themselves. A given node's output generally shows the most recent datum it received. In this way, a given rumor (datum) can quickly spread among the Maelstrom network.
It exhibits an "echo chamber" effect in real time, with some facts growing in prominence and other facts being suppressed over time. After a few minutes all of the machines might be repeating the same thing back to each other.
It's very important to me that visitor data not be retained for longer than 15 minutes. So each internal radio transmission includes an expiration time, along with the data, the visitor ID (and color--it's a 6-digit hex code, as used in HTML/CSS). If a node receives a radio transmission with an implausible expiration date, it throws it out.
There are also some "housekeeping" transmissions sent among the nodes--to wit:
How do the nodes know which other two nodes to send their data towards?
This is the fun part. So, if all the machines always functioned perfectly, I could set this per machine in advance and call it a day. This would lead to a boring structure where a given machine always shared its data with the same two machines, and so on, but it would work fine. More like a rushing river than a truly chaotic environment.
The problem is that in the real world, these computers and radios don't function perfectly. Some radios in particular seem to drift out of frequency or something, causing them to stop "hearing" after many hours or days. I can reset them every so often, but I can easily imagine a scenario where a few particular nodes fail and the entire installation becomes unresponsive.
That would not be good! So, I built a system to dynamically build a directed graph of all responsive Maelstrom nodes and renegotiate it as needed as nodes appear or disappear.
It works something like this:
- Each node emits a "hello!" every five seconds or so.
- Each node keeps track of how many other nodes it can hear, and tracks this over time, computing a "signal strength."
- The "hello!" I mentioned also includes a mention of how many nodes it can hear well. So each node is really saying, "hello! and I can hear 5 nodes well."
- After powering up, each node waits to hear if a "leader" has been elected. It waits a staggered amount of time.
- If not, it calls an election, sending a special message to the multicast channel.
- Each node votes for whichever node it thinks is best, scoring by that node's signal strength. So it votes for whoever can hear the most nodes, essentially. (there is a special case of "over-listener" where a radio receives lots of garbled transmissions and thinks it hears hundreds of ghosts; I exclude this case a special way.)
- The votes are sent back to the multicast channel, and everybody listens.
- Each node decides for itself who won, sorting by node number in case of a tie.
- If a node decides that it won, it emits a "Leader!" declaration.
- If a second node thinks that it won and hears about a leadership dispute, it calls a second election; this nearly always resolves quickly due to other nodes' previous determinations.
- The leader node listens for as many other nodes as possible and computes a directed graph, using networkx.
- It attempts to connect each node to two downstream nodes and two upstream nodes. It makes a second pass and overloads some nodes to ensure a minimum 2 up/down for each node.
- Once a graph is computed, a graph version number is assigned. The graph is transmitted in a series of multicast messages.
- A given node listens only for itself and expects two nodes. If it doesn't receive them within a timeout window, it requests the graph shard for itself with a request sequence number.
- It retransmits this request as necessary, incrementing the request sequence number.
- If the request sequence number gets above a certain threshold, peer nodes assume that the leader node cannot hear the request.
- Peer nodes retransmit the request.
- If a node senses that its downstream nodes are not in range, it declares them a "bad peer" and emits a special message.
- The leader node listens to this, decimates the graph (by which I mean removes 10% of connections) and removes the connection between Peer X and Badpeer, and reweaves the missing connections to generate a new graph.
- If this approach fails it will rebuild the graph from scratch as needed.
The hard parts have been working around these failure cases:
- A node can transmit but not receive
- A node can receive but not transmit
- Node X can't hear Node Y
- Node X can't hear Leader Node but can hear other nodes
- Leader Node can't hear Node X but can hear other nodes
- A node is receiving lots of garbled messages
- A node is sending lots of garbled messages
One interesting aspect to all of this is that as someone trying to replicate rumormongering in device format, I don't necessary want perfect message transmission. I like when the machines are playing "Telephone," sharing a typo-ridden version of a fact instead of the real thing.
To help with this, I don't apply any extra CRC or error correction in my Maelstrom software. I do leave hardware CRC enabled at 16 bits, and throw away messages that are obviously badly structured--but if a given message turns "Christopher" into "Chrisjrote," to me, that's just part of the fun.
As part of building and troubleshooting all of this, I have an internal diagnostic tool that outputs the graph in visual form. Sometimes it makes fun shapes. As a bonus, here are some of the weirder graphs it's formed.
-
What kind of data is collected, and how?
02/03/2021 at 17:33 • 0 commentsWith "Maelstrom," one of the hardest parts has been striking the right tone. I want visitors to understand that their data can be gathered and reused in ways difficult to predict, but I also want their visit to be a rewarding experience--and not too scary.
The installation passively listens for these kinds of data:
- WiFi connection probe requests. Your phone sends these transmissions in an effort to see whether your favorite WiFi networks are nearby. With a special device, a Maelstrom machine listens for these requests. The phone's unique WiFI network identifier (MAC address) includes a code that indicates the device's manufacturer. Maelstrom's software looks for this.
- Nearby Bluetooth low-energy (BT LE) devices occasionally emit a peep letting nearby devices know that they are present. A Maelstrom node uses its Raspberry Pi's Bluetooth radio to listen for these BTLE devices and their names, like "Charge 2." It also looks at their MAC addresses and attempts to determine the device's manufacturer, like "Fitbit."
If you interact with the WiFi network, Maelstrom_PUBLIC, provided by the installation, it can further sense some attributes about your device:
- When signing in to the "captive portal" provided by Maelstrom's software, details about your browser are revealed:
- user agent
- browser version
- operating system
- screen size
- whether it is a touch screen
- These attributes, and others, can be used as a fingerprint to identify your device model, and perhaps to uniquely identify you.
- The Python library user_agents also attempts to discern whether your device is a phone, tablet, or desktop computer; your device's overall family, such as "iPhone"; and whether you are a human, or an automated "spider bot."
The captive Maelstrom_PUBLIC network also asks you to provide these details to "sign in to the network":
- first name
- then, in random order:
- last name
- middle name
- favorite color
- mother's maiden name
- street address
- childhood phone number
- phone number
- ZIP code
- age
- nickname
- first pet's name
- There's a Skip button, which doesn't really skip the question, they just end up again later. Easter egg/commentary on my part.
- There's no real network to join at the end. It just thanks you for providing all your data and shows you what you provided.
You can use a card-scanning station to detect attributes from a credit card or ID card (with a visual "2D barcode" on the back):
- shortened card number, last 4 digits
- expiration date
- name
- any other attributes exposed by your motor vehicle department or card issuer, such as your height and eye color
- it will also attempt to scrape any useful text (using an algorithm similar to the *nix "strings" command) from other types of cards, like gift cards or membership cards
And several machines use attached cameras and the face_recognition Python library to monitor for faces. I didn't go too far with this yet, since it doesn't work well with masks. These are not currently associated with visitor dossiers, but are shown on dedicated screens (the End-User License Agreement shows them in a "Thank you for agreeing!" box; a Maelstrom FR-Class machine shows them as tiles).
All of these data are only retained for 15 minutes, and populated into text written by me, using either Chevron/Mustache (for HTML output methods) or a simplistic templating language similar to Mustache (for text/audio outputs).
There are a few machines which show the full dossier data that the machine knows for each visitor, but most of the displays will only show or speak whitelisted attributes that I included in the text templates.
-
What kind of computers to use?
02/03/2021 at 16:47 • 0 commentsWell, as much as "Raspberry Pi" isn't a standard--it is a de facto standard. I ended up using a controlled version of the Raspberry Pi pinout as my host controller's connector.
After scoping everything out and tossing out some overly expansive ideas, the host controller board needed to include the following hardware interfaces:
- WS2812 for glow
- NRF24L01+ radios
- A few GPIOs: LEDs for blinkenlights, button for last-ditch software repair
- I2S bus for amplified audio
- I2C bus for attached displays and other outputs
I wanted to be able to source and replace the actual computers with ease. But as an independent artist self-financing this project, I didn't have the resources to be able to throw, say, a Compute Module at each of the ~40 machines. Thus, my key design goals for the computers were:
- Long-term support and good availability of supply chain
- Supports the above hardware interfaces
- Minimal cost
You might be thinking, "sounds like a Raspberry Pi Zero to me!" And you're not wrong--I did end up using many Zero Ws, slowly collected over 18 months, as part of this installation. But as a pandemic slowly crept across the globe in early 2020, and supply chains were torn asunder, I renewed my focus on making Maelstrom a multi-platform project through the design of "controller boards" that adapt non-Pi platforms to the Maelstrom/Pi pinout.
I ended up supporting these three platforms in Maelstrom:
- Raspberry Pi (Zero W, 3A, 3B, 4)
- ESP32
- Orange Pi Zero LTS
I was able to develop host controller boards for ESP32 and Orange Pi Zero that support the above hardware interfaces. My software detects the platform and adjusts the pinout and software modules to suit the host machine.
Despite my best efforts, there are some per-platform strangenesses.
ESP32:
- This is a very minimal C port of the Maelstrom Cython codebase that only supports the radio and WS281x for now. It is possible to add more device support down the road but this let me build ~12 of the machines with $4 host controller (vs ~$20 for Zero W, $18 Orange Pi Zero, $40-60 for Pi 3/Pi 4. The lack of SD cards helps bring the cost down.)
- This platform needs 3v3, which I initially planned to provide via 3V3 pins but later had to move to a regulator on the controller board.
Orange Pi Zero LTS:
- I wasn't able to locate Python or C libraries for the WS281X addressable LEDs. I added a microcontroller to this board's I2C bus
- There aren't enough GPIOs, so I added a PCF8574 I2C GPIO expander on the controller board to handle the extras.
Raspberry Pi:
- On some Pi boards, if you externally supply 3V3 via the GPIO header, it keeps the board from booting, perhaps due to power sequencing complexities.
By adapting the ESP32 and Orange Pi Zero LTS to my limited subset of the Raspberry Pi pinout, I ended up with a "Maelstrom pinout." This took a lot of staring at datasheets, library docs, and spreadsheets, and I still got it wrong several times--especially on ESP32, where some of the "GPIOs" are just "GPIs."
There was one big idea I wanted to build out that was totally incompatible with the main Maelstrom pinout, however: "HUB75" RGB LED matrix panels.
I ended up addressing this by making a separate, matrix pinout, only compatible with Raspberry Pi, and adding an extra pinout adjustment in the Maelstrom software. So a Raspberry Pi node will start up, read the EEPROM data from the attached host board--sort of like a HAT--and use that to decide whether it loads the "Maelstrom pinout" or the "Maelstrom matrix pinout."
This introduced a few new weirdnesses to work around in the Maelstrom matrix pinout:
- The matrix pinout, like Orange Pi Zero LTS, was short on GPIOs. I added a PCF8574 I2C GPIO expander here too. This GPIO expander needs to work alongside my directly-attached GPIOs, which use RPi.GPIO.
- I built out a shell script that attaches the PCF8574 as sysfs GPIOs, and launch a secondary instance of my software's GPIO module that uses sysfs GPIO.
- The existing sysfs GPIO Python module, named "gpio," had some performance problems, so I built out a simple shim version that functions like RPi.GPIO. It doesn't have edge detection or pull-up/pull-downs, since sysfs GPIO doesn't support these.
- Raspberry Pi 3 includes a GPIO expander that instantiates at the same sysfs GPIO number (/sys/class/gpio/gpio504) as a PCF8574 on a Pi Zero. If you write to this expander, you can power off the machine or make it unresponsive. (this took … a long time… to figure out.)
- The matrix library, rpi-rgb-led-matrix, is fantastically performant and has some particular pinout needs. Because of these, I needed to move the SPI0 CE pin that the NRF24L01+ radio uses. I built a device tree overlay and a Python DTO loader that remap CE0. (This is the point at which I realized that a second EEPROM on the i2c ID bus would have been handy for HAT autoloading!)
All of these hoops were helpful to jump through. I now have a multiplatform system that works reasonably well across the boundaries, and a given host controller board can be replaced with a completely different architecture in most cases--making it easier to address future electronics failures.
-
A few machines, or a lot of machines?
02/03/2021 at 16:39 • 0 commentsWith Maelstrom, I want to give visitors the experience of seeing their own data "go viral" in the safe confines of a creepy exhibition--without seriously scaring them. You can see how one sentence in, this project is full of contradictions: is it fun or creepy? is their data at risk, or not?
I felt it was important that the installation itself felt slightly overwhelming or stifling--like a "server room of the damned." This guided many aesthetic decisions I made, and also one key architectural decision: it needed to feel like a lot of individual machines all around you, each commenting on you at a slightly-too-fast pace (mimicking users on a social media network).
I ended up using one computer per display. You might be wondering…
Why not use fewer computers?
One idea I considered early on was to have just a few computers, each with lots of outputs attached. It would be possible to put a lot of displays on one machine with a bus like I2C or diff-I2C, or even USB-attached microcontrollers. I explored this option a bit--perhaps it could be 5 or 10 machines driving 40 displays, instead of my goal of 40 machines. The downsides with this approach are additional complexity for the software/overall behavior, power management, and installation.
Software and overall behavior:
Each machine would need to stagger the display of data among its attached displays--which is absolutely doable, but more complicated. The "phone tree" of radios would also become less of a factor in the installation's behavior (read more about the radios).
Also, let's say a given display fails and needs to be replaced. How does a given host machine know what devices are attached? It's possible to look for them on an I2C or USB bus, to some extent, but it is hard to distinguish between some similar-looking but very different devices. (HD44780 via PCF8574T: is it 16x2? 20x4? 40x2? is there a backlight attached?)
I would need to store configuration either inside the display devices--by adding an adapter board with EEPROM memory between the machine and the display: added complexity--or in a centralized way, in software/local storage or EEPROM memory on the host board: possible to get "out of sync" with the physically attached devices.
Some of the devices I wanted to use would have the same I2C address by default, so I'd need to make each unique--another installation failure point. And some I2C devices' addresses aren't configurable at all, so there's a max of one per machine.
Let's say a given display device fails--perhaps it runs short on power, perhaps a cosmic ray hits it. On an I2C bus this can freeze the entire bus. The more output devices per machine, the higher the likelihood of having all of its output devices fail.
Power management:
Many of the vintage displays I'm using require their own odd voltages, so a given host machine with four displays might need to provide 3v3, 5v, 6v, 9v, and 12v to its attached host displays. I could pass 5V around and use local boost converters inside each display; this leads to a large current rush at startup. I could add power sequencing to fix this. More complicated.
All these displays would be pulling a much larger current through the Maelstrom host board, leading to secondary problems with current ratings of connectors, larger PTC self-resetting "fuses,"
Some of the outputs need clean power, like speakers/amplifiers and analog meters (DACs). Each additional device, especially with a boost converter or switching regulator, makes the machine's power supplythat much noisier. It is possible to fight against the noise with local power filtering inside each device, but I hoped to stick with commercial modules and would need to buffer them with my own power filtration boards. This problem scaled quickly in practice.
Installation complexity:
The last problem with the "multiple displays per machine" architecture was how to make it humane to "use" as an installer--with all this additional complexity to track through a chaotic few days of drilling, attaching, labeling, spackling, and so on.
If I have a single machine with an octopus of screens attached to it, I need to disconnect all this stuff, number and label it, perhaps assign it certain ports on the host machine, and preconfigure the machine to know what displays are attached.
Why I chose self-enclosed machines instead
Keeping each Maelstrom node completely self-contained* makes the boring parts of keeping track of the individual nodes much easier. I just need to find the machine, attach it to the wall, and power it up.
*as much as possible; a few are "exploded views" of bare components attached to the wall.
I can store all of the configuration details for a machine's attached outputs in EEPROM memory on the node's single host board. The output devices themselves are installed inside the case. There is no external wiring to snag or catch when unpacking or installing the machines.
Each machine can deal with its own inrush current from boost converter powerup. In a more tightly controlled environment, it's easier to measure and make adjustments to the power filtration on my host board.
The software can read everything it needs to know about the attached peripherals, configure critical system parameters, and compile the necessary Cython modules dynamically on startup. I can easily replace a host controller in case of a failure without worrying about any per-machine configuration stored in the controller's storage.
Getting this big decision made let me move on to deciding what kind of computers to use.
-
What it's like to walk through "Maelstrom"
02/03/2021 at 16:34 • 0 commentsA given visitor is intended to approach the installation and have the machines begin responding to their presence within a few moments, detecting, say, an Apple iPhone device trying to connect to "My Home Wifi Name." They may see this show on a screen or two, and perhaps hear a mechanistic voice utter something cryptic that sounds a bit like it's mentioning an iPhone or "My Home Wifi Name."
At the entrance to the installation is a welcome screen, greeting them as they enter the "Maelstrom Networks Corporate Museum." It mentions, in a fourth-wall violation, that the installation uses data about them, contains flashing lights, and will not retain any data longer than 15 minutes after they depart the installation.
The welcome screen encourages them to use their phone to join a "Maelstrom_PUBLIC" wireless network, provided by one of the machines, which prompts them to "sign in to the network" by answering just a few questions, starting with their first name. If they do, it designates them with a color, which begins to propagate on the lights throughout the exhibition. The sign-in screen asks another question (what's your favorite color, say) and then another. If they answer these questions, the machines begin to discuss the datapoints entered. ("I heard that their first name is Chris.") The "written voice" of the machines is that of an amateur sleuth on social media, attempting to form connections between disparate facts.
If they answer all of the questions, a "dossier" of the data they provided--and the data sensed from their device, such as their operating system, device manufacturer, etc.--is shown to them on their phone.
Nearby is the "End-User License Agreement" console, which informs them that by appearing in the installation, they consented to the use of their image and data by the (fictional) Maelstron Networks, and as they read this, a camera attempts to recognizes their face and display it in a list of those who recently consented ("Thank you!").
As they walk through the ~35 machines with screens and speakers, set in several different vintages, the colors, displays, and speakers reflect the information gathered about them. Some of these machines are endearing in appearance: old-timey radios, clean white machines, a chunky Game-Boy-esque device.
Another data entry station lets them scan a magnetic stripe card or ID card (with a "2D barcode" on the back) to "autocomplete their profile" in the "Maelstrom-D data management platform," the signature product of the fictional Maelstrom Networks, Inc.
It is my hope as an artist that visitors will be more thoughtful about how their data spreads after experiencing "Maelstrom," and advocate for a better, brighter future for data management in the US.