Project Logs

Collapse

• Artist's talk: showing the installation and describing its ideas and meanings

  Chris Combs • 02/12/2021 at 19:45 • 0 comments

• How do the machines communicate?

  Chris Combs • 02/03/2021 at 18:28 • 0 comments

  One of the most fun parts of building Maelstrom has been getting ~35 machines to speak to each other. In a setting with solid infrastructure, perhaps I would use WiFi. Art galleries aren't always happy about this idea. I also wasn't thrilled about the machines possibly getting Internet access; in my perfect world, they would be completely isolated from wider networks. 

  I have worked with Nordic NRF24L01+ radios in other projects and decided to use them in Maelstrom. These radios accept 32 bytes of data and let you, as a developer, "press send and walk away" while they handle the mechanics of retransmission, error checking, etc.

  They have other, interesting properties, like a five-byte "pipe" identification scheme that (in radio firmware) lets you ignore messages on the same channel that are not intended for a given client. And it's possible to set up a multicast network. I chose this approach for Maelstrom, using a common "message bus" with target device identifiers baked into the data payload. So all the radios are reading and writing from a common channel, at a rate of a message every few seconds (with some exceptions).

  There are additional complexities with this, particularly when sourcing NRF24L01+ radios on a budget. Some of the modules sold today are using mostly-compatible clone parts. One, in particular, clones the datasheet very well--one signal described by Nordic in their own datasheet is flipped in practice in actual Nordic devices, and the clone devices cloned the datasheet instead of the real device behavior… making them incompatible in practice with real devices. 

  In my perfect world, I'd be able to keep each radio listening both on a group channel and on an individual address. I wasn't able to get this working across my mix of radio modules. I ended up using Pipe 1, as recommended by the RF24 documentation, to listen to just the multicast channel, and encoding an individual address into the message types that need it.

  What data is shared over the radio?

  I wanted to have visitor data literally being shared among the machines. A given datum is observed by one particular Maelstrom node and it shares it to two others. Each of them, if they successfully receive it, can choose to share it to two others, themselves. A given node's output generally shows the most recent datum it received. In this way, a given rumor (datum) can quickly spread among the Maelstrom network. 

  It exhibits an "echo chamber" effect in real time, with some facts growing in prominence and other facts being suppressed over time. After a few minutes all of the machines might be repeating the same thing back to each other.

  It's very important to me that visitor data not be retained for longer than 15 minutes. So each internal radio transmission includes an expiration time, along with the data, the visitor ID (and color--it's a 6-digit hex code, as used in HTML/CSS). If a node receives a radio transmission with an implausible expiration date, it throws it out. 

  There are also some "housekeeping" transmissions sent among the nodes--to wit:

  How do the nodes know which other two nodes to send their data towards?

  This is the fun part. So, if all the machines always functioned perfectly, I could set this per machine in advance and call it a day. This would lead to a boring structure where a given machine always shared its data with the same two machines, and so on, but it would work fine. More like a rushing river than a truly chaotic environment.

  The problem is that in the real world, these computers and radios don't function perfectly. Some radios in particular seem to drift out of frequency or something, causing them to stop "hearing" after many hours or days. I can reset them every so often, but I can easily imagine a scenario where a few particular nodes fail and the entire installation becomes unresponsive. 

  That would not be good! So, I built a system to dynamically build a directed graph of all responsive...

  Read more »

• What kind of data is collected, and how?

  Chris Combs • 02/03/2021 at 17:33 • 0 comments

  With "Maelstrom," one of the hardest parts has been striking the right tone. I want visitors to understand that their data can be gathered and reused in ways difficult to predict, but I also want their visit to be a rewarding experience--and not too scary.

  The installation passively listens for these kinds of data:

  • WiFi connection probe requests. Your phone sends these transmissions in an effort to see whether your favorite WiFi networks are nearby. With a special device, a Maelstrom machine listens for these requests. The phone's unique WiFI network identifier (MAC address) includes a code that indicates the device's manufacturer. Maelstrom's software looks for this.
  • Nearby Bluetooth low-energy (BT LE) devices occasionally emit a peep letting nearby devices know that they are present. A Maelstrom node uses its Raspberry Pi's Bluetooth radio to listen for these BTLE devices and their names, like "Charge 2." It also looks at their MAC addresses and attempts to determine the device's manufacturer, like "Fitbit."

  If you interact with the WiFi network, Maelstrom_PUBLIC, provided by the installation, it can further sense some attributes about your device:

  • When signing in to the "captive portal" provided by Maelstrom's software, details about your browser are revealed:
    • user agent
    • browser version
    • operating system
    • screen size
    • whether it is a touch screen
  • These attributes, and others, can be used as a fingerprint to identify your device model, and perhaps to uniquely identify you.
  • The Python library user_agents also attempts to discern whether your device is a phone, tablet, or desktop computer; your device's overall family, such as "iPhone"; and whether you are a human, or an automated "spider bot."

  The captive Maelstrom_PUBLIC network also asks you to provide these details to "sign in to the network":

  • first name
  • then, in random order:
    • last name
    • middle name
    • favorite color
    • mother's maiden name
    • street address
    • childhood phone number
    • phone number
    • ZIP code
    • age
    • nickname
    • first pet's name
  • There's a Skip button, which doesn't really skip the question, they just end up again later. Easter egg/commentary on my part.
  • There's no real network to join at the end. It just thanks you for providing all your data and shows you what you provided. 

  You can use a card-scanning station to detect attributes from a credit card or ID card (with a visual "2D barcode" on the back):

  • shortened card number, last 4 digits
  • expiration date
  • name
  • any other attributes exposed by your motor vehicle department or card issuer, such as your height and eye color
  • it will also attempt to scrape any useful text (using an algorithm similar to the *nix "strings" command) from other types of cards, like gift cards or membership cards

  And several machines use attached cameras and the face_recognition Python library to monitor for faces. I didn't go too far with this yet, since it doesn't work well with masks. These are not currently associated with visitor dossiers, but are shown on dedicated screens (the End-User License Agreement shows them in a "Thank you for agreeing!" box; a Maelstrom FR-Class machine shows them as tiles).

  All of these data are only retained for 15 minutes, and populated into text written by me, using either Chevron/Mustache (for HTML output methods) or a simplistic templating language similar to Mustache (for text/audio outputs). 

  There are a few machines which show the full dossier data that the machine knows for each visitor, but most of the displays will only show or speak whitelisted attributes that I included in the text templates. 
  

• What kind of computers to use?

  Chris Combs • 02/03/2021 at 16:47 • 0 comments

  Well, as much as "Raspberry Pi" isn't a standard--it is a de facto standard. I ended up using a controlled version of the Raspberry Pi pinout as my host controller's connector.

  After scoping everything out and tossing out some overly expansive ideas, the host controller board needed to include the following hardware interfaces:

  • WS2812 for glow
  • NRF24L01+ radios
  • A few GPIOs: LEDs for blinkenlights, button for last-ditch software repair
  • I2S bus for amplified audio
  • I2C bus for attached displays and other outputs

  I wanted to be able to source and replace the actual computers with ease. But as an independent artist self-financing this project, I didn't have the resources to be able to throw, say, a Compute Module at each of the ~40 machines. Thus, my key design goals for the computers were:

  • Long-term support and good availability of supply chain
  • Supports the above hardware interfaces 
  • Minimal cost

  You might be thinking, "sounds like a Raspberry Pi Zero to me!" And you're not wrong--I did end up using many Zero Ws, slowly collected over 18 months, as part of this installation. But as a pandemic slowly crept across the globe in early 2020, and supply chains were torn asunder, I renewed my focus on making Maelstrom a multi-platform project through the design of "controller boards" that adapt non-Pi platforms to the Maelstrom/Pi pinout. 

  I ended up supporting these three platforms in Maelstrom:

  • Raspberry Pi (Zero W, 3A, 3B, 4)
  • ESP32
  • Orange Pi Zero LTS

  I was able to develop host controller boards for ESP32 and Orange Pi Zero that support the above hardware interfaces. My software detects the platform and adjusts the pinout and software modules to suit the host machine.

  Despite my best efforts, there are some per-platform strangenesses.

  ESP32:

  • This is a very minimal C port of the Maelstrom Cython codebase that only supports the radio and WS281x for now. It is possible to add more device support down the road but this let me build ~12 of the machines with $4 host controller (vs ~$20 for Zero W, $18 Orange Pi Zero, $40-60 for Pi 3/Pi 4. The lack of SD cards helps bring the cost down.)
  • This platform needs 3v3, which I initially planned to provide via 3V3 pins but later had to move to a regulator on the controller board.

  Orange Pi Zero LTS:

  • I wasn't able to locate Python or C libraries for the WS281X addressable LEDs. I added a microcontroller to this board's I2C bus
  • There aren't enough GPIOs, so I added a PCF8574 I2C GPIO expander on the controller board to handle the extras.

  Raspberry Pi:
  

  • On some Pi boards, if you externally supply 3V3 via the GPIO header, it keeps the board from booting, perhaps due to power sequencing complexities. 

  By adapting the ESP32 and Orange Pi Zero LTS to my limited subset of the Raspberry Pi pinout, I ended up with a "Maelstrom pinout." This took a lot of staring at datasheets, library docs, and spreadsheets, and I still got it wrong several times--especially on ESP32, where some of the "GPIOs" are just "GPIs."

  There was one big idea I wanted to build out that was totally incompatible with the main Maelstrom pinout, however: "HUB75" RGB LED matrix panels. 

  I ended up addressing this by making a separate, matrix pinout, only compatible with Raspberry Pi, and adding an extra pinout adjustment in the Maelstrom software. So a Raspberry Pi node will start up, read the EEPROM data from the attached host board--sort of like a HAT--and use that to decide whether it loads the "Maelstrom pinout" or the "Maelstrom matrix pinout."

  This introduced a few new weirdnesses to work around in the Maelstrom matrix pinout:

  • The matrix pinout, like Orange Pi Zero LTS, was short on GPIOs. I added a PCF8574 I2C GPIO expander here too. This GPIO expander needs to work alongside my directly-attached GPIOs, which use RPi.GPIO.
  • I built out a shell script that attaches the PCF8574 as sysfs GPIOs, and launch a secondary instance of my software's GPIO module that uses sysfs GPIO.
  • The existing sysfs GPIO Python...

  Read more »

• A few machines, or a lot of machines?

  Chris Combs • 02/03/2021 at 16:39 • 0 comments

  With Maelstrom, I want to give visitors the experience of seeing their own data "go viral" in the safe confines of a creepy exhibition--without seriously scaring them. You can see how one sentence in, this project is full of contradictions: is it fun or creepy? is their data at risk, or not? 

  I felt it was important that the installation itself felt slightly overwhelming or stifling--like a "server room of the damned." This guided many aesthetic decisions I made, and also one key architectural decision: it needed to feel like a lot of individual machines all around you, each commenting on you at a slightly-too-fast pace (mimicking users on a social media network).

  I ended up using one computer per display. You might be wondering…

  Why not use fewer computers?

  One idea I considered early on was to have just a few computers, each with lots of outputs attached. It would be possible to put a lot of displays on one machine with a bus like I2C or diff-I2C, or even USB-attached microcontrollers. I explored this option a bit--perhaps it could be 5 or 10 machines driving 40 displays, instead of my goal of 40 machines. The downsides with this approach are additional complexity for the software/overall behavior, power management, and installation. 

  Software and overall behavior: 

  Each machine would need to stagger the display of data among its attached displays--which is absolutely doable, but more complicated. The "phone tree" of radios would also become less of a factor in the installation's behavior (read more about the radios).

  Also, let's say a given display fails and needs to be replaced. How does a given host machine know what devices are attached? It's possible to look for them on an I2C or USB bus, to some extent, but it is hard to distinguish between some similar-looking but very different devices. (HD44780 via PCF8574T: is it 16x2? 20x4? 40x2? is there a backlight attached?) 

  I would need to store configuration either inside the display devices--by adding an adapter board with EEPROM memory between the machine and the display: added complexity--or in a centralized way, in software/local storage or EEPROM memory on the host board: possible to get "out of sync" with the physically attached devices. 

  Some of the devices I wanted to use would have the same I2C address by default, so I'd need to make each unique--another installation failure point. And some I2C devices' addresses aren't configurable at all, so there's a max of one per machine.

  Let's say a given display device fails--perhaps it runs short on power, perhaps a cosmic ray hits it. On an I2C bus this can freeze the entire bus. The more output devices per machine, the higher the likelihood of having all of its output devices fail.

  Power management:

  Many of the vintage displays I'm using require their own odd voltages, so a given host machine with four displays might need to provide 3v3, 5v, 6v, 9v, and 12v to its attached host displays. I could pass 5V around and use local boost converters inside each display; this leads to a large current rush at startup. I could add power sequencing to fix this. More complicated. 

  All these displays would be pulling a much larger current through the Maelstrom host board, leading to secondary problems with current ratings of connectors, larger PTC self-resetting "fuses,"

  Some of the outputs need clean power, like speakers/amplifiers and analog meters (DACs). Each additional device, especially with a boost converter or switching regulator, makes the machine's power supplythat much noisier. It is possible to fight against the noise with local power filtering inside each device, but I hoped to stick with commercial modules and would need to buffer them with my own power filtration boards. This problem scaled quickly in practice.

  Installation complexity:

  The last problem with the "multiple displays per machine" architecture was how to make it humane to "use" as an installer--with all...

  Read more »

• What it's like to walk through "Maelstrom"

  Chris Combs • 02/03/2021 at 16:34 • 0 comments

  A given visitor is intended to approach the installation and have the machines begin responding to their presence within a few moments, detecting, say, an Apple iPhone device trying to connect to "My Home Wifi Name." They may see this show on a screen or two, and perhaps hear a mechanistic voice utter something cryptic that sounds a bit like it's mentioning an iPhone or "My Home Wifi Name." 

  At the entrance to the installation is a welcome screen, greeting them as they enter the "Maelstrom Networks Corporate Museum." It mentions, in a fourth-wall violation, that the installation uses data about them, contains flashing lights, and will not retain any data longer than 15 minutes after they depart the installation. 

  The welcome screen encourages them to use their phone to join a "Maelstrom_PUBLIC" wireless network, provided by one of the machines, which prompts them to "sign in to the network" by answering just a few questions, starting with their first name. If they do, it designates them with a color, which begins to propagate on the lights throughout the exhibition. The sign-in screen asks another question (what's your favorite color, say) and then another. If they answer these questions, the machines begin to discuss the datapoints entered. ("I heard that their first name is Chris.") The "written voice" of the machines is that of an amateur sleuth on social media, attempting to form connections between disparate facts. 

  If they answer all of the questions, a "dossier" of the data they provided--and the data sensed from their device, such as their operating system, device manufacturer, etc.--is shown to them on their phone.

  Nearby is the "End-User License Agreement" console, which informs them that by appearing in the installation, they consented to the use of their image and data by the (fictional) Maelstron Networks, and as they read this, a camera attempts to recognizes their face and display it in a list of those who recently consented ("Thank you!"). 

  As they walk through the ~35 machines with screens and speakers, set in several different vintages, the colors, displays, and speakers reflect the information gathered about them. Some of these machines are endearing in appearance: old-timey radios, clean white machines, a chunky Game-Boy-esque device.

  Another data entry station lets them scan a magnetic stripe card or ID card (with a "2D barcode" on the back) to "autocomplete their profile" in the "Maelstrom-D data management platform," the signature product of the fictional Maelstrom Networks, Inc.

  
  It is my hope as an artist that visitors will be more thoughtful about how their data spreads after experiencing "Maelstrom," and advocate for a better, brighter future for data management in the US.
  

View all 6 project logs