NixieBot

As it's the season of goodwill ( and hackers covering everything in LEDS ) Nixiebot now has a string of WS2811 based RGB LEDs added. Since it's nixiebot they are, of course, controllable by hashtag too. An arduino nano is driving them using the wonderful fastLED library .

The arduino is connected to the USB on the pi and appears as /dev/ttyUSB0 , I added a routine to nixiebot to pick up on a new #lights hashtag and the arduino has a routine to check serial input and receive commands from nixiebot.

The lights are purely USB powered so I enabled the max_usb_current=1 parameter on the pi after making sure that the 5V regulator on nixiebot's power board can take the extra current. It's a 2A regulator and with the max_USB_current flag set then USB can deliver 1.2A.

Current for a raspberry Pi 2B and camera module should be under 700mA so we're good... as long as I don't turn on too many LEDs. There are 50 LEDs in the string, each of which can take up to 60mA so you can see there's ample possibility for exceeding the 1200mA avalable from USB. However the code just twinkles a few LEDs at a time so we're good there.

To change them just add the hashtag #lightsX to your nixiebot command, where X is a number from 1 to 7, to choose from some preset colour schemes:

1: Red, White, Blue, Green

2: Purple, Green, Red

3: Green, Gold, Blue

4: Purple, Pink, Yellow, Blue

5: Green

6: red,

7: Blue

The ardiuno code (see files) runs a loop that:

• fades all Leds towards black a bit
• If it's time, picks a random LED to turn a random colour from the current set
• Checks to see if there are any characters waiting in serial input and, if so, interprets the command.
• delays 50ms then loops back

So the overall effect is twinkling leds that each fade out over a few seconds. Of course this is all going on according to the arduino timer loop and the pictures taken by nixiebot are timed according to nixiebot's code which spoils the twinkle a bit in the movies that nixiebot makes. So I might have a go at making the code synchronous with the Pi when composing movies next, the idea is to enter a mode where the arduino waits for a "do the next frame" command from nixiebot whicih is issued after each frame of a movie is taken. This will mean quite a rewrite of the arduino loop ( which is recycled from another project of mine ) as all the timers currently run on the arduino millis() function.

The arduino runs a simple command protocol, all commands are two characters and have ':' as the third, followed by any arguments the newline. Currently Nixiebot only issues the "SC:" to choose a colour set.

Here's the python side of things, it's based off a copy of the glitch setting function:

def setLights(tweet) :
    #sets fairy ligts according to #Lights:[1-6] 
    print("setting lights")
    level = re.compile(r'lights[1-7]')
    for tx in tweet['entities']['hashtags'] :
        t=tx['text'].lower()
        if level.match(t) :
            try :
                gl = int(t.split("lights")[1])
                print("lights req to ", gl)
                lights = gl
                lcom = serial.Serial("/dev/ttyUSB0",baudrate=19200,timeout=1.0)
                time.sleep(2.1) #allow time for arduino to reboot when port opened
                lcmd = "SC:"+str(lights)+'\n' 
                print(lcmd)
                lcom.write(bytes(lcmd,"utf-8"))
                time.sleep(0.5)
                lcom.close()
            except :
                print("lights setting exception text = ", t, "split = ", gl)
                pass

The only thing that had me mildly stumped was the fact that opening the serial port was resetting the arduino and subsequently the command was getting lost during arduino reboot. To cover this I just added a time.sleep(2.1) . In future I'm going to need to keep the port open globally if it's ever to be used for frame sync and so on. But for now this quick hack does the trick!

The arduino code should be available over in the files section of this project by the time you read this, merry Xmas and happy new year to all on hackaday!

One of those jobs that I have been meaning to get around to for ages is that of making sure that the NixieBot code would automatically be run whenever the pi was booted. Recently a couple of weekend power cuts that left the bot down for a few hours until I noticed its absence prompted me to get this task done.

First off, in normal operation nixiebot.py is run from a screen session. Screen is a text based window manager that allows command line processes to be run up, dismissed from view and then reattached to later to see what's going on or to interact with them. it's very handy indeed for a machine that has no keyboard or monitor that is only ever connected to by ssh session over the network. With screen you can start a process and terminate the terminal session without also terminating the process you ran from that session. Then later you can log back in and reattach to see what's going on with the process. It's a really handy utility that gets used in my day job all the time. So any script that aims to start nixiebot should start it in a screen session so that I can interact with it later if I want to change the clock routine's behavior.

To keep things neat we first need a script that will run up nixiebot, here it is:

#! /bin/bash
cd /home/pi/nixiebot
python3 nixiebot.py

Pretty straightforward, this script is what will be run in the screen session. I saved it as a file named startNixieBot.sh, put a copy in /usr/bin so that it is in the path and made it executable with:

chmod +x /usr/bin/startNixieBot.sh

so it can be invoked just by typing startNixieBot.sh from any directory.

The command to start up a screen session (named nixiebotScrn to identify it among any other screen sessions that might be running) and run the nixiebot code in it is this:

screen -S nixiebotScrn  -d -m startNixieBot.sh

So that deals with starting the code, what about stopping it? If you are at the console of nixiebot and type a 'q' then hit enter it will wrap up things nicely, terminating the connections to twitter's API in a polite fashion and closing down all the threads properly (more on those threads in a later log). So, rather than just killing the process, it would be good to send a 'q<cr>' sequence to nixiebot whenever the pi is being shut down.

Luckily screen has a way of doing this with the -X stuff command, here's how to stop the nixiebot that was started by the above command:

screen -S nixiebotScrn -p 0 -X stuff "q$(printf \\r)"

Notice the use of -S nixiebotScrn again so that the key gets sent to the right session. The rather abstract looking "q$(printf \\r)" is just shell script-ese for "q then the return key". I got this handy tip from this question on stackexchange. Stackexhange nearly always delivers the goods if you have a unix scripting question!

Having worked out the commands for startting and stopping a background nixiebot process from the command line, the next task is making sure that these commands get run on startup and shutdown.

There are a few ways of ensuring that a process runs on bootup with linux systems, the best way (IMHO) is to write a proper init script. If you go looking in /etc/init.d you'll find a whole bunch of scripts that deal with starting and stopping the various services that might be running on that machine. There are some nice refinements that distro builders have worked out over the years to make sure that services only get started when any other services that they depend on are ready and so on.

A little light googling produced a nice template script from here , all I had to do was edit the lines that actually did the work of starting and stopping the script plus change the descriptions in the header section.

#! /bin/sh
# /etc/init.d/nixiebot

### BEGIN INIT INFO
# Provides:          nixiebot
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Simple script to start nixiebot process at boot
# Description:       A simple script from www.stuffaboutcode.com which will start / stop a program a boot / shutdown.
### END INIT INFO

# If you want a command to always run, put it here

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting nixiebot"
    # run application you want to start
    /usr/bin/screen -S nixiebotScrn  -d -m  startNixieBot.sh
    ;;
  stop)
    echo "Stopping nixiebot"
    # kill application you want to stop
    /usr/bin/screen -S nixiebotScrn -p 0 -X stuff "q$(printf \\r)"
    ;;
  *)
    echo "Usage: /etc/init.d/nixiebot {start|stop}"
    exit 1
    ;;
esac

exit 0

Nearly there! Once this file was created as /etc/init.d/nixiebot then made executable with :

chmod +x /etc/init.d/nixiebot

all that remains was to tell the init system on raspbian to run the file on startup with this command:

update-rc.d nixiebot defaults

And that's it, rebooting now runs nixibot up and shutdown terminates it properly (you can run into problems with twitter's API connection and rate limiting if you just crash out the code without terminating connections properly).

Recent new feature: As well as tweeting user requested images NixieBot will send out a daily movie tweet about "how my day went". This movie is composed of one frame taken every 15 minutes throughout the day so you can see how lighting changes and weather affect the images the camera produces. The word to display is either picked from the last user requested word from the previous 7.5 minutes or else, if there was no request made during that time period, the most popular word (of four or more letters in length) used in the random tweet feed is displayed. Certain very common words:

boringWords=["this","that","with","from","have","what","your","like","when","just"]

are filtered out to make it more interesting. The movie attempts to summarize the twitter 'ZeitGeist' for the day.

How it works:

The time interval between frames is kept in the timeLapseInterval variable, every time round the loop in the main runClock() function this happens:

    if  int(t.minute) % timeLapseInterval == 0 :
                doTimeLapse()  #either choose a frame from recent first frames or, if none available, take one from random stats
                               #if it's the appointed hour, generate and tweet the time lapse movie. 
            else : 
                lapseDone = False 

The minutes value of the time variable t (set at the top of the loop) is checked to see if it's a multiple of the required interval, if so the doTimelapse() function is called. The lapseDone variable acts as a flag to make sure that doTimeLapse only gets called once per interval. Without this, if the timelapse process takes less than a minute to run, it would be called multiple times.

So what does doTimelapse do then? here it is:

def doTimeLapse() :
    global cam
    global lapseDone
    global makeMovie
    global effx
    global effxspeed
    if lapseDone :
        return
    print("doTimeLapse called")    
    #delete all lapse*.jpg older than (lapseTime / 2)
    #pick youngest lapse*.jpg file and copy to lapseFrames directory    youngestName = ""
    youngestTime = time.time()
    youngestFile= ""
    timeLimit = time.time() - ((timeLapseInterval/2) * 60)
    files = glob(basePath+"lapse*.jpg")
    for f in  files :
        fileTime = os.path.getatime(f)
        if fileTime < timeLimit :
            print("deleting ", f, " age =", (time.time() - fileTime)/60)
            os.remove(f)
        elif fileTime < youngestTime :
            youngtestTime = fileTime
            youngestFile = f
    if youngestFile != "" :
        print("moving file", youngestFile , "into frame store")
        move(youngestFile, basePath+"lapseFrames/")
    else :
        #take frame of most popular word in random tweet sample of four or more letters
        words=randstream.allWords()['wordList']
        bigEnough=[]
        for w in words :
            if len(w) >= 4 and w not in boringWords and "&" not in w:
                bigEnough.append(w)
        c = collections.Counter(bigEnough)
        topWords=c.most_common(20)
        theWord=topWords[0][0]
        print(topWords, theWord)        
        makeMovie = False
        stashfx = effx
        stashspeed = fxspeed
        setEffex(0,0)
        lockCamExposure(cam)
        displayString(theWord)
        cam.capture(basePath+"/lapseFrames/lapse-"+time.strftime("%Y%m%d-%H%M%S")+".jpg",resize=(320,200))
        unlockCamExposure(cam)
        setEffex(stashfx,stashspeed)
        
    lapseFrames = glob(basePath+"lapseFrames/*.jpg")
    #if there are now 96 files in the frames folder, make a movie and tweet it out #NixieLapse
    print(len(lapseFrames),"lapse frames found")
    if len(lapseFrames) >=96 :
        print("making daily time lapse")
        delay=20
        mresult = call(["gm","convert","-delay",str(delay),"-loop", "0", basePath+"/lapseFrames/*.jpg","Tlapse.gif"]) 
        print("Make movie command result code = ",mresult)
        if mresult == 0 :
            uploadRetries = 0
            while uploadRetries < 3 : 
                try:
                    pic =open("Tlapse.gif","rb")
                    print(">>>>>>>>>>>>> Uploading Timelapse Movie ", datetime.datetime.now().strftime('%H:%M:%S.%f'))
                    response = twitter.upload_media(media=pic )
                    print(">>>>>>>>>>>>> Updating status ", datetime.datetime.now().strftime('%H:%M:%S.%f'))  
                    twitter.update_status( status="This is how my day went: #NixieBotTimelapse", 
                       media_ids=[response['media_id']] )
                    print(">>>>>>>>>>>>> Done  ", datetime.datetime.now().strftime('%H:%M:%S.%f'))
                    uploadRetries = 200
                except BaseException as e:
                    print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Tweeting movie exception!" + str(e))
                    uploadRetries += 1 
            move(basePath+"Tlapse.gif", basePath+"lapseFrames/Tlapse"+time.strftime("%Y%m%d-%H%M%S")+ ".gif")        
        for f in lapseFrames :
            os.remove(f)
    lapseDone = True        
    return()
    

In between calls to doTimeLapse the word displaying and picture taking routines save the image as a file with name composed of the string "lapse" then a timestamp.

doTimeLapse() first iterates through all files named lapse*.jpg, it discards any that were created in the first half of the current lapse period and keeps track of the youngest file it finds that was created in the second half of the lapse period.

If this process finds a youngest file it will move it into a subdirectory where all frames for the day's movie are kept.

If no file is found then it retrieves a list of all words used in current buffer of random tweets by invoking the allWords() method of the randstream object (this TwythonStreamer object is in charge of receiving random tweets and keeps a circular buffer of the last 1000 tweets received, this buffer is a deque ).

This word list is first iterated through to remove words that are too small or in the boringWords list.

The resulting pruned word list is then fed into another of python's many handy collection types , the Counter.

A counter accepts values and compiles a dictionary of unique values against a count of how many times that value occurs.

Counters also have a handy most_common() method which is used in this case, to extract the most used word (actually for reasons lost in the mist of debugging time it extracts the top twenty words then picks the number one from those ... that should probably get neatened up one day).

Having found the most popular word it then displays it, takes a photo, then stores the image in the directory where the other timelapse frames are kept.

Next job is to see if there are a full day's worth of frames yet (and, in explaining all this to you I have found a potential bug, there's a hard coded value for number of frames per day when it should be calculated from the timeLapseInterval variable ... explaining your code to someone is a great technique for optimising and debugging! )

If a full day's worth of frames have been recorded then they are assembled into an animated gif with the "gm convert " command and that is posted to twitter (here there is substantial code duplication as there are other places where movies are posted to twitter... one day it might get refactored out into a separate function but this was a quick and dirty feature addition )

Finally the lapseDone flag is set so that the routine doesn't get called again next time round the main clock loop.

So there you go ... an insight into what happens when I start coding and keep adding things without a refactor ... lots of global variables serving mysterious purposes and code duplication, it still works though!