Mark Traverse's pages

Python Peer to Peer MicroService Bus for IoT Messaging Framework

Latest work:

I am currently well along a personal project to build a simplified data-driven MicroService Bus in Python to integrate RPI HAN nodes, ESP IoT wifi devices., communicating freely with the world of telemetry and cloud messaging service capabilities (think ESB for the PI).

Why?

Well, I started integrating everything I could on Raspberry PI. Sensors, i2c, 1-wire, GPIO, SPI, USB X10, API's, webscrape etc etc. Then started to think about how to federate within a Home Area Network. I didn't want everything going back to a central computer in the home, more wanted to be able to have satellite control points with say a PI in the kitchen dealing with communications for appliances, another dealing with living room,a nano for weather, satellite arduino mediators and of course small IoT devices on ESP etc. MQTT was the obvious answer for M2M communications because of the lightweight nature of the clients BUT you end up with a jungle of hard-wired scripts to maintain on each node. Every setup needs the tinkering with GPIO for sensors and displays, the 1-wire bus setup, the I2C setup, - and of course all the setup of pub/sub topics. So the idea was to create a metadata-driven wrapper around it all, harvesting environment setup by interogating the OS and adding data to allow it all to be consistently addressed through an abstrated driver layer. As this data is organised it is used for automated setup of publishing (sensors, inputs, web data) and subscription for controls (displays, connected devices, home automation controls)

Again, I know there are many available products for this that may get near to where I need (such as openHAB) but I really wanted total flexibility for extensibility. The world is moving fast and I need to be able to plug into iot services quickly (and sometimes those that wouldnt be supported in a product). So this will remain strictly HOMEGROWN with a healthy re-use of Open Hardware and Open Software.

The main components (per HAN node):

1. ENVIRONMENT DATA SCHEMES: Metadata & agent wraps over sub-systems
2. BULLET PUBLISHERio: Data-driven, resilient, bullet-proof configurable Scheme Publisher for MQTT (configurable thru topic controls)
3. DRIVERPACKi: returning 'readings' from multitude of Sensor types, API's, and all Input-bound interactions
4. AGENT MEDIATORio: Configurable Subscriber / Mediator responding to topic controls and orchestrating actions with sub-systems for all connected devices inc displays (OLED / LCD), I2C output devices, GPIO drivers
5. DRIVERPACKo: for data-driven actions on all connected devices, O/S commands, API's
6. RULESio: Lightweight configurable M2M rules engine for simple orchestrations (major orchestrations handled in NODE-RED)
7. STATEio: a lightweight persistent state store for short-term persistence
8. BRIDGEio: for handling M2M & Broker bridging between HAN MQTT and external / cloud iot services. (Adafruit.io, Arduino.io, Artik, PubNub, ThingSpeak, IBM Watson)
9. API G/W: Currently supporting Weather Underground, Google Docs, Twitter, Email.

Fun with Soviet era Vacuum tubes, originally used in missile guidance systems.

Banging the bits and bytes of the IoT age.
Upstairs at Erics
Starring Raspberry PI / Node MCU / MAX 6921 / IV-18 / Python3 multiprocessing and Queuing / MQTT / I2C

One of the reasons for doing this initially on RPI was to challenge the popular belief that these displays can only really be implemented with MCU's (stability of muxing etc).. I think many peoples experiments on the PI were based on the earlier / older models and of course we now have much faster processors with the PI 3 B and while RPI will never be, and is not an R/T platform, results are reasonable.

There's plenty of vfd iv-18 clocks out there including the lady Ada ice clock but these are all atmega /arduino and pretty constrained by size of the mcu's, and (until recently) the lack of internet on these platforms - so implementations have tended to be 'just' standalone r/t clocks rather than IOT objects presenting interactive web information with text animation - which is what I wanted.

I didn't want to just take someone else's work anyway, but really go back to first principles myself to learn Python and push the boundaries of the text trickery that could be used with these displays (of course noting the constraints of seven segment). And that's where we are currently - if still at the rudiments of the prototypes.

The first incarnation in software uses Python3 and its multiprocessor library to modularise separate scripts for VFD mux and a 'supervisor' which spits out 'character frames' of 9 digits over a Python Queue to the MUX which controls the bit banging to the MAX6921 HV shift register.

Recent additions add MQTT telemetry transport between the supervisor and mux and a general notification channel into the supervisor module that allows display from anywhere. Modularising around mq telemetry will ultimately allow me to port the mux to node mcu / ESP and the device become a standalone wi-fi connected general purpose display working with my home automation.

The possibilities are endless for these beautiful displays - controlling internet radio, music playback, temperature monitoring etc and I will be building on the basic concepts to get myself fully up to speed with IOT platforms.