Real World Fluxly, Physics Looper

Now that the Fluxum controller and the Yak Shaver are talking to each other, the next step is to read the gyro sensor and send some actual data. I started with Kris Winer's MPU9250 example and added the Wifi and UDP broadcast code, this time calculating the angular velocity in degrees per second and sending that as four bytes. The gyro code hasn't been made into an Arduino library so the setup code is a little long to post here (it's in the files section). Here's the loop() code snippet:

void loop() {  
  
    // If intPin goes high, all data registers have new data
    if (readByte(MPU9250_ADDRESS, INT_STATUS) & 0x01) {  // On interrupt, check if data ready interrupt
        readGyroData(gyroCount);  // Read the x/y/z adc values
        getGres();
 
        // Calculate the gyro value into actual degrees per second
        gx = (float)gyroCount[0]*gRes;  // get actual gyro value, this depends on scale being set
        gy = (float)gyroCount[1]*gRes;  
        gz = (float)gyroCount[2]*gRes;   
  
    }
  
    Now = micros();
    deltat = ((Now - lastUpdate)/1000000.0f); // set integration time by time elapsed since last filter update
    lastUpdate = Now;

    sum += deltat; // sum for averaging filter update rate
    sumCount++;
  
    if (!AHRS) {
        delt_t = millis() - count;
        if(delt_t > 50) {
            count = millis();
        }
    }

    byte val1 = 0;
    byte val2 = 0;
    byte val3 = 0;
    byte val4 = 0;
    int v = abs(gz);
    
    int thisVal = v;

    if (!((thisVal < (prevVal + 2)) && (thisVal > (prevVal-2)) || (thisVal < 5)) && connected) {
      //Send a packet
      udp.beginPacket(udpAddress,udpPort);
      packetBuffer[0] = command;
      packetBuffer[1] = dir;
      if (thisVal < 5) val1 = 0;
      packetBuffer[2] = val1;
      packetBuffer[3] = val2;
      packetBuffer[4] = val3;
      packetBuffer[5] = val4;
     
      udp.write(packetBuffer, 6);
      udp.endPacket();
  } 
  
  prevVal = thisVal;
  delay(20);
}

For the first iteration of this project I am only sending the absolute value of the z axis angular velocity. The next iteration will use the command field to indicate the direction of rotation so the sample can play forward or backward.

Here's the first working iteration of the Fluxum controller:

(Text from Richard Brautigan's All Watched Over By Machines of Loving Grace, copyleft 1967 by author):

...I like to think
of a cybernetic forest
filled with pines
and electronics
where deer roam peacefully
past computers
as if they were flowers
with spinning blossoms

Next steps:

• Play samples forward and backward
• Make custom PCB with 3 evenly spaced AA batteries for more of a flywheel effect
• RGB LEDs to backlight Fluxum character
• Use ESP32 streaming audio example to play sound on the actual device instead of the server

The first step in making the Fluxly controller was to make a jig that would allow the microcontroller to spin. I laser cut a couple of acryliccircles with a cutout and a masonite base to hold the ESP32 module and a battery. A skate bearing is press fit in the center.

The sensor is a MPU9250 gyro/accelerometer. It has to be centered on the circle so it sits on top of the ESP32 module. I'm using a module that I made called the Fluxamasynth-32; it has an additional sound-making chip on board that I'm not using here. 

Unfortunately the EAGLE part I was using for the ESP32 module had the I2C lines marked incorrectly (I may have drawn it!) so I pulled the wrong pins out to the header. Instead I used prototyping wire soldered directly to the pins on the module.

Here's what it looks like spinning:

Once I had the microcontroller spinning, the next step was to get the ESP module talking to the Pure Data patch running on the computer. I decided to use Wifi rather than Bluetooth, since Pure Data is well equipped to handle messages over the network. I set up a router running OpenWRT as an access point, which each Fluxum will connect to. The laptop running the patch will also connect to the router.

The idea is to broadcast UDP packets over wifi, using a simple 6 byte protocol: fluxum_id, command, data1, data2, data3, data4. I started with the Arduino example code that comes with the Wifi library and modified to send 6 bytes with random data, just to get them talking. 

 #include <WiFi.h>
#include <WiFiUdp.h>

// WiFi network name and password:
const char * networkName = "dd-wrt";
const char * networkPswd = "**********";

byte packetBuffer[ 6];

byte command = 1;
byte id = 2;

//IP address to send UDP data to:
// either use the ip address of the server or 
// a network broadcast address
const char * udpAddress = "192.168.1.255";    // broadcast
const int udpPort = 3001;

//Are we currently connected?
boolean connected = false;

//The udp library class
WiFiUDP udp;

void setup(){
  // Initilize hardware serial:
  Serial.begin(115200);
  
  //Connect to the WiFi network
  connectToWiFi(networkName, networkPswd);
}

void loop(){
  if (connected){
    //Send a packet
    udp.beginPacket(udpAddress,udpPort);
    packetBuffer[0] = id;
    packetBuffer[1] = command;
    packetBuffer[2] = random(255);
    packetBuffer[3] = random(255);
    packetBuffer[4] = random(255);
    packetBuffer[5] = random(255);
    udp.write(packetBuffer, 6);
    udp.endPacket();
  }
  delay(30);
}

void connectToWiFi(const char * ssid, const char * pwd){
  Serial.println("Connecting to WiFi network: " + String(ssid));

  WiFi.disconnect(true);
  WiFi.onEvent(WiFiEvent);
  WiFi.begin(ssid, pwd);
  Serial.println("Waiting for WIFI connection...");
}

void WiFiEvent(WiFiEvent_t event){
    switch(event) {
      case SYSTEM_EVENT_STA_GOT_IP:
          Serial.print("WiFi connected! IP address: ");
          Serial.println(WiFi.localIP());  
          udp.begin(WiFi.localIP(),udpPort);
          connected = true;
          break;
      case SYSTEM_EVENT_STA_DISCONNECTED:
          Serial.println("WiFi lost connection");
          connected = false;
          break;
    }
}

On the Pure Data side, the netreceive object listens for data on a particular port, 3001 in this case. It receives 5 bytes and the list object splits them into individual outlets. The id byte routes the data to a particular input, command is as yet unused, and the 4 data bytes get added back together. Here's the netreceive patch added to the YakShaver engine:

There's a packet counter at the lower left that calculates how many packets per second are coming in. The goal is to reach 60 packets per second (currently around 30). 

Next step is to add the gyro sensor.

Fluxly is written in C++ and uses the openFrameworks toolkit for creative coding. openFrameworks is aimed at making coding easier for artists and designers and provides convenient libraries or addons that provide an embarassment of riches for someone willing to take it on. Fluxly uses the built-in OpenGL capabilities of openFrameworks, and the OfxBox2d port of Box2D and danomatika's libPd addon.

All of the hard work of the sound synthesis is done in the embedded Pure Data patch, the Yak Shaver Engine called Yakshaver.pd, which you can download and open using the freely available Pure Data.

There are a few parts to the patch: an input stage, the Yak Shaver engine, and an output stage, which handles panning and reverb. There are also the "scopes" that are arrays that hold the data of the sample so they can be rendered in the app. A block diagram looks like this ("Ofx" below refers to the openFrameworks app that the patch is talking to):

The main interface to the app has the four "subunits" at the top, an interface for testing the on/off and tempo controls, and a section at the bottom for loading the default values. Clicking on the messages to the right will load different samples into the 8 yak shaver channels: 

The Yak Shaver engine makes extensive use of send and receive blocks (the "s" an "r" blocks) as well as throw~ and catch~ blocks, which are the audio signal equivalent. The "r" or receive blocks here will catch messages sent from within the patch or from the openFrameworks app.

Each of the Yak Channels (there's 8 of them) loads a WAV file into a buffer and plays it back as long as the on/off toggle is on. The tempo (marked "multiple" here) controls how fast the file is played back. The openFrameworks app sends a message to this inlet every time the angular velocity of one of the Fluxum circles changes. If the app sends a message to the "filename" inlet a new file is read into the buffer.

Each of the voices is panned in the final output depending on the X coordinate of the Fluxum looper. This subunit of the output stage handles the panning of each channel:

Each of the panned outputs of the 8 channels are added together and run through the Pure Data "Freeverb" processor. The inputs are sent as a message from the app depending on the rotation of the white Fluxum in each scene. The output is toggled on or off when the sample selector is chosen.

Recording is handled by the input stage, which uses Pure Data's analog to digital converter to read in microphone audio and save it to a buffer. When the stop button is pushed a message is sent and the buffer is trimmed to size and saved to the app's documents directory. Right now the limit is about 10 seconds (400000 samples at 44.1kHz, as seen in the "resize" command upper right), but that could easily be changed if there's interest in longer samples.

The audio output is also sent to the "scopes" subunit, a group of arrays that are sent back periodically to the main app and used to draw the waveform on the screen.

One of the benefits of using openFrameworks is that it provides a simplified framework for writing your app. Anyone familiar with the "sketching" approach of Processing or Arduino wil recognize an openFrameworks app: it consists of a setup() function called once, an update() function called repeatedly, and a draw() function called at a certain framerate. The draw() function should be as lean as possible, focused just on things that need to be done to draw a frame of the screen play. The update() function is where any number crunching should be done. There are also convenient callbacks for handling user interface events like touches.

Fluxly uses two openFrameworks addons: ofxPd as a wrapper around Pure Data and the Yak Shaver patch, and ofxBox2d which provides the physics model. Here's an illustration of how they interact:

Fluxly began in 2008 as a wizard duel game where you had to make your own game controller in hardware in order to play it. The first demo controllers were cigar boxes with pennies, an Arduino and a USB connection. It was originally written in Flash using the Box2d physics engine (the same one used by the original Angry Birds) and ran on a desktop computer. 

There were about 20 different aspects of the game that you could control: gravity, wind, lightning, etc. The idea was that participants would use Arduino to make a controller (there was a Fluxly Arduino library), and whoever could make the best controller would have an edge in the game. It was pretty fun; we launched it at the first RI Maker Faire, Foo Fest, and a bunch of other events. It even got a mention in Boing Boing.

The original Fluxly character was based on a whiteboard drawing by my daughter Louise that I turned into a sketch. For the game Arley Rose Torsone turned it into lovely pixel art.

Arley and I and Brandon Edens began meeting on Tuesday afternoons to teach ourselves how to make apps. The first project we worked on was another Louise-inspired game called Space Wozzle, an episodic sci-fi collection of mini games riffing on classic arcade games. We made a few iterations but we realized the truth about making video games is that they have to be fun and they have to jell, and I'm afraid Space Wozzle ended up having too any moving parts.

Later, we used the same "meet every Tuesday for a year and see what happens" business model with Elliot Clapp and Shawn Greenlee to work on some musical apps. Elliot and I had just finished the first year of Fab Academy and had made a few hardware musical projects at that time, like the Fluxamasynth and the Fluxalodian. Meanwhile Shawn had taught a great three week workshop on Pure Data called the Pure Data Club, which was how I started learning the tool. Another happenstace was the release of Peter Brinkmann's libPd, which allows you to enbed Pure Data as a sound engine in embedded or mobile devices. Shawn, Elliot and I made about 6 libPd apps in a year, two of which we thought were interesting enough to launch in the app store: Noisemusick and DR-OM.

The Noisemusick app was based on a hardware kit I had made at AS220 that we used in Learn To Solder workshops.  

After that I was at O'Reilly Media and Maker Media for a few years, during which I had the pleasure of working with Peter Brinkmann on a book about libPd called Making Musical Apps. I also worked with Ray Wilson on his book Make: Analog Synthesizers, which is when I spent more and more time exploring electronic musical instruments. 

I made a bunch of little instruments and sonic appliances at this time, one of which was the Yak Shaver. The Yak Shaver was a resitive touch interface to a Pure Data patch that was inspired by the Rhythmicon. The Rhythmicon (below right) was essentially the world's first electronic drum machine, which Henry Cowell commissioned from Leon Theremin. It was optical and operated on spinning tone wheels; each key on the 12 note keyboard would play a pulsing tone at a multiple of the previous key. My Pure Data patch emulated this, but let you read in a WAV sound sample file and play it back at different multiples of tempo.

Since 2014 I've been at AS220 Industries, which is a Fab Lab, Printshop and Media Lab in Providence RI. It's a great place to make friends and meet interesting people. Fluxly is directly influenced by many members and coworkers at the Industries: Jacque Bidon's moire pattern books, Aisha Jandosova and Melita Morales' Maker Camp activities (mutant instruments, gear trains, and more moire!), Tycho Horan's risograph color experiments hanging next to my worktable, Chris Anderson's experiments with the Benjolin synthesizer we made in a workshop, and more. Singer Laurie Amat (and coordinator of the PVD Loop festival) is a frequent Industries visitor and offered testing, vocal recordings, and moral support.

All this background may seem tangential to Making Fluxly but it isn't. Looking back at the various projects on my hard drive I think I tried 3 different times over the years to turn the original Fluxly "make your own controller" wizard duel game into a mobile app. Each time I hit a couple of road blocks; the major one was how to make it fun as a one player game. There was also a technical limitation of the physics engine running on a mobile device; I found it could only handle a hundred or so objects (with my primitive coding) where I was imagining a larger play area with hundreds of objects. 

The latest version of Fluxly finally started coming together when I thought about making the game more about the objects than about the characters. At first I was thinking it would be a physics puzzler with a soundtrack, but after hooking up the Yak Shaver Engine everything finally jelled and it became obvious that it should be a musical looper. Adding the physics interface to the Yak Shaver freed the instrument from its integer intervals and opened up room to explore randomness. Everything else seemed to fall into place because of the years of incremental influences and projects that had come before.

The 15 built-in samples may seem a little random, but each also has some background:

1. A sample from Aisha's Banu Musa Music Box, a programmable cardboard xylophone with chimes made from the scrap of Larry Zargosky's geodesic domes.
2. One of Laurie Amat's vocal snippets contributed for promotion of Modern Device's Reverbalizer.
3. A bytebeat sample from the Modern Device Shempscape Bytebeat Shufflebox.
4. A snippet from Chris Anderson's Benjolin playing.
5. A recording of the Shopbot spindle as Hannah Liongoren works on a project n the fab lab.
6. Another Laurie Amat sample.
7. Geoff Griffin on bass in the intro to "Ask Me About Being a Bog Person".
8. A classic public domain thunder sample.
9. The introduction to "The Best Sauce is a Good Applesauce".
10. A sample of the DROM app.
11. A recording of the RISD Jacquard loom from a project with Jungil Hong.
12. A bit of generative marimba music from the Fluxamasynth.
13. A sample of the Noisemusick app.
14. A recording from a Maker camp participant's mutant instrument.
15. The background music from original Fluxly game.

Next: How the Software Works