Project Logs

Collapse

• Rebuild Server to .NET Core with improvements

  deandob • 03/04/2017 at 06:45 • 0 comments

  I am doing a major re-write of the server code for the HomePortal automation system.

  The new version will have the following (most architectural) features:

  1) Written in C# instead of VB.NET. No real benefit to be gained in swapping languages except that C# is a better supported language, and as I also write in C and JavaScript it is easier to jump between languages. Although I think VB is an expressive language, I do find C# enjoyable to

  2) Using the new Microsoft .NET Core open source technologies. This will make the solution truly portable across different operating systems easily and being open source makes .NET (and solutions like HomePortal) accessible to a wider audience. This will be the first (significant) .NET Core home automation system available.

  3) Using modern frameworks. The VB version is around 5 years old when there wasn't frameworks like Entity Framework and logging frameworks that the new .NET Core HomePortal version is using. I expect to retire about 50% of the codebase due to this, and it will be easier to maintain, faster and more accessible to others who may want to extend the solution.

  4) More consistent use of design patterns like PubSub, MVC and plugin architectures. The extensibility of the solution is significantly improved (eg. to add another plugin language like Python) by adding more features through 'Extensions' which are .NET Core DLLs that can be added to the main program and bound at runtime. It will also be possible to replace the default functionality with your own (eg. a different rules engine) by dropping in a new rules DLL.The plugins are even simpler to use, with pub/sub interfaces and are compiled at startup and the code will be editable in the client.

  5) Integrated security. The VB.NET version does not have security built in, and the new Core version has a security authorization / authentication layer in the subscribe function that controls the access to channels.

  6) MQTT pub/sub for browser access (still via websockets) with a simpler and more consistent implementation. REST and Sockets interfaces will remain and additional interfaces can be added via extensions as the event driven architecture is consistently accessed vis publish/subscribe.

  7) Not for the initial release but the next version will have Iotivity support (for device discovery and taxonomies), voice support (Cortana skills) and AI using LUIS cognitive services and Bots.

  8) Use of InfluxDB as the timeseries message store, which should speed up queries and history requests (for graphing) as well as provide additional history functionality applicable to timeseries datasets (ie. recording sensor events)

  The VB.NET version has a lot of functionality to be ported, however I do think the effort will be worth it as the solution will be faster, more reliable and more extensible. I will do a proper open source launch when it is completed (ETA 2 months - I'm about 40% through the porting).

  You can see the work in progress in my GitHub folders: https://github.com/deandob/HAServer

  If you have any feedback or ideas about the new version, do post comments.

• Online Demo

  deandob • 01/02/2017 at 04:27 • 6 comments

  Been a while since I posted an update.

  The software has had numerous fixes including better compatibility for Firefox and Chrome browsers.

  If you are interested in seeing a fully functional online demo system running where you can interact with preconfigured screens and also create / edit your own screens, please post a comment as I'm considering deploying the server into the Cloud if there is enough interest.

• Architecture

  deandob • 08/08/2016 at 13:16 • 0 comments

  A well thought out architecture and API is critical to the performance, flexibility, maintenance and extensibility of any software solution.

  The general design breaks the solution into 3 main subsystems:

  • Front end user interface. HTML5 / JavaScript leveraging Twitter's Bootstrap front end framework, using HTML objects to embed widgets as active and passive elements into HTML5 screens, and uses websockets to communicate to the automation engine using a topic based publish / subscribe model.
  • Automation engine. This is the brains of the system using Visual Basic .NET and uses event driven messages that pass through several modules including a state store, database, rules processing and console.
  • Plugin Manager. This is the integration layer that connects to the external world using Node.JS or .NET and uses a plugin design so that connection semantics to each device / service are managed in discrete JavaScript modules.

  Events & Message Queue

  Events are the heart of the automation engine where all events are represented as messages managed by a message queue. Being event driven enhances performance and allows use of modern design patterns like publish / subscribe and modern tools like Node.JS / JavaScript (which are inherently asynchronous) and .NET async.

  The message format is critical to ensure it is rich enough to cater for all types of messages and not too big to be difficult to work with and consume space. The format chosen is topic based similar to MQTT messages, and has proven to be very durable and flexible, as follows:

  <NETWORK> / <CATEGORY> / <CLASS> / <INSTANCE> / <SCOPE> / <DATA>

  • Network allows partitioning of messages and segmenting of networks if implemented in a large environment with several hubs (I only use one hub so the network name is common for all messages).
  • Category is the general type of message, eg. SYSTEM for system messages like timers, LIGHTING for all messages related to lights. There is an editable category list that covers all general automation and other categories.
  • Class is the entity that is interacting with the automation system and generally is matched to the plugin which represents the device or service being integrated with. For example, I use CBUS as my lighting automation system so there is a CBUS message class instantiated from the CBUS plug-in that deals with the semantics of connecting with the CBUS interface (serial with an ASCII protocol).
  • Instance is the interface on the class entity, and there may be many of these. In the CBUS example all the lights have a unique instance name (eg. Master-Bedroom). The interface definition is managed by the plug-in.
  • Scope defines the type of message sent to/from the instance. With the CBUS example, when a light switch is toggled on the wall, the CBUS plug-in receives this message from the CBUS interface as a update and fills the scope field out as STATE_CHANGE. It also allows the instance to have several functions (eg. can be used to send commands as well as status).
  • Data is the value of the message and for the CBUS example of receiving an update that someone switched the light on will have the value of 100 (100% dim level). All values are managed and stored as strings but the rules and transformation modules will recognise numbers within strings.

  Scheduling is performed via the rules engine using time based triggers.

  HTML5 widgets and plugins subscribe or publish to channels. Channels are bi-directional names for message class instances used with widgets and plugins. For example, a light widget would subscribe to the LIGHTING/CBUS/Master-Bedroom channel and all messages on that channel will be sent to the light widget and it can also publish messages on that channel (eg. widget can turn on the master bedroom light).

  
  Why Microsoft .Net as the platform?

  The automation engine is written in Visual Basic .NET for several reasons but mostly legacy.

  The code started as VB 6 in its first version back in 2003, when it was small system not unlike the average DIY home automation...

  Read more »

• Automation Engine

  deandob • 08/08/2016 at 11:05 • 0 comments

  The automation engine uses Microsoft .NET and runs on the home hub server providing and orchestrating all the automation services for the user interfaces and plug-ins. This is the most complicated sub-system and runs the following services:

  • Message management. Handles the incoming and outgoing messages via a multi-threaded message queue.
  • Database management. SQLite is used as the relational data store as it has decent performance and functionality without the need to administer the database.
  • State store. Maintains the state of the messages so that message state persist after the message has passed through the message queue.
  • Rules module. Every message handled by the central queue is passed through the rules module where the trigger rules are compared and if matched, triggers any associated events.
  • Transformation module. Transformations are similar to rule triggers except they apply calculations to the triggered message and act as virtual devices.
  • Queries module. Provides the ability to store and execute queries on current and historical data.
  • Console. Event logging and debug messages, displayed on the server console or remotely.

  Lets have a closer look at the implementation and lessons learnt of a couple of these features not discussed in other logs.
  
  

  Database and Queries

  The SQLite database stores all non system messages in the database as a time-series log (time as the index) using the raw message format. Each message category (eg. Lighting) has its own SQLite file and as the database structure is simple performance is good even with categories with 10's of thousands of messages logged. Database size grows over time - for example polling the solar inverter every 6 seconds for current power generated quickly adds up even if we are only sending messages when the state changes. The relational aspect of SQLite is used for the rules module, where an automation event is associated with triggers and actions, cached in memory when the automation engine starts and edited by the settings function on the HTML5 client.

  SQLite has a rich SQL syntax that means queries can be built to answer simple questions like who keeps turning lights on at 3am in the morning, or more complex ones like has the average power increased this month compared to last month. Queries can be run using the current state (from the state store) or historical (from the database) transparently and like transformations can use aggregate time-series functions like average, max and min.
  
  

  Rules Module
  

  Rules are managed through events, which have associated triggers and run associated actions when fired. Below is an event screen (click for higher resolution), which automates the CBUS lights in my home theater when a CBUS light switch is pressed (triggered by the relevant light switch ON message, then runs actions to manipulate 8 banks of lights to ramp up the lights slowly from the front to the back over 15 seconds - a custom light scene),

  The rules module is constantly evaluating new messages for a match against a trigger. Triggers are flexible and can capture state changes as well as date / time changes (eg. sunset, specific time of day or day of week) or both together (to create AND conditions). Below is the trigger record (click for higher resolution) that is listening for the CBUS light switch that the event above uses to fire the lighting scene actions.

  Events can associate with multiple triggers to create OR conditionals (activate with any of the triggers firing). Events can also have multiple actions associated with them so actions and triggers can be reused in multiple events.

  Triggers are defined by the message taxonomy and wildcards are allowed (eg. trigger on all Lighting messages). Actions either create new messages or run a script. Below is an action record that activates one of the dimmers in my home theater (click for higher resolution).

  Transformation Module

  Transformations are a powerful and unique feature of the automation engine. A transformation will transform raw messages...

  Read more »

• Plug-ins

  deandob • 08/07/2016 at 21:59 • 0 comments

  Plug-ins are the glue that binds external devices and services together, monitors and controls them and converts their specific protocols to a common message format. Integration is a key aspect of a home automation system and the plug-in manager makes it all happen.
  
  

  Design

  A modular plugin design is critical for separating the concerns of managing the integration of the device / service with the general services of the automation engine. It also means that we can easily add and maintain the semantics of the device / service we are connecting to.

  Converting the format of messages the external device / service uses to a common message format

  <NETWORK> / <CATEGORY> / <CLASS> / <INSTANCE> / <SCOPE> / <DATA>

  'normalises' the message so that it can be manipulated and integrated with other devices / services. See the architecture log for more description of the message format.

  Plugins are isolated snippets of code the perform the following functions:

  • Connect, disconnect and authenticate with the device / service
  • Provide any specific protocol services needed to communicate (eg. SNMP for routers)
  • Monitor the device by collecting status information and keeping relevant local state
  • Control the device (if it has actuation)
  • Translate the device / service message format into the standard format used by the automation engine
  • Handle any error conditions and recovery with the device
  • Schedule any tasks particular to the device (eg. poll for status). Any general automation task scheduling is managed by the automation engine not at the plug-in.

  Plugins can be written in .NET or JavaScript and are stored in directories according to the plug-in category (eg. Lighting). The plug-in Manager has the task of loading and validating the plug-in, calling any initialisation tasks the plug-in may need and providing the mechanism to communicate between the plug-in and the automation engine via common functions.

  Each plug-in has an associated settings file in the INI file format (as it is easy to read & edit) that describes the following:

  • Plug-in attributes
  • Specific settings required by the plug-in to connect to the device / service
  • Defines the pub/sub channel semantics for other plug-ins or widgets to subscribe to.

  The settings file avoids the need to hard code parameters like IP addresses so that configuration changes can be made on the fly without restarting. Plug-in settings file can be edited within the automation console.
  
  

  Plug-ins are not just about devices

  Just as importantly as being able to integrate with the hardware devices around the home, plug-ins are also used to connect and consume Cloud services. The weather dashboard outlined in another log is a good example of mixing local weather data from the home weather station with weather information from the Cloud. The combination of the local device data along with complementary Cloud data make for a more engaging and useful user interface.

  Another example of integrating Cloud services is being able to pull in stock data from a source like Yahoo Finance. A plug-in connecting to the Cloud works the same way as connecting to a local device, but this time the inbuilt Node.JS HTTP functionality is used to connect. Once a stock feed is running, many things are possible like alerts if the price exceeds a certain range (eg. generate a SMS from a trigger), history graphs or a stock ticker widget.

  Why Node.JS is well suited for Home Automation

  One of the main disadvantages of writing my own automation system is that I don't get to leverage a plug-in library from existing automation packages like HomeSeer and OpenHAB. However this hasn't been a problem in practice because of the power and ecosystem of Node.JS.

  Node.JS as the platform to host plug-ins has made a huge difference to the plug-in flexibility and enables quick development of even complex device / service integration. The ease of writing in JavaScript & its asynchronous nature along with the almost infinite number of NPM packages available (think libraries for those who...

  Read more »

• User Interfaces

  deandob • 08/07/2016 at 07:32 • 0 comments

  The user interface is an often overlooked but highly important aspect of a home automation system. A lot of experimentation and effort has gone into the user interface for the system I'm describing here as it evolved over the years. Initially implemented in Visual Basic Forms which was limited to simple display and control, the next version used Microsoft Silverlight which allowed a more functional interface including animation.

  When Microsoft ditched Silverlight I had to re-think the interface again. A native app would have been an easy option but I chose HTML5 which at the time was looking promising and has now evolved to a rich platform for web interfaces and widely implemented now.
  
  

  Web Interface

  The browser is the main interface to the system via a webserver on the hub (an integrated Node.JS Express webserver) which serves up a HTML5 / JavaScript rich client that communicates with the hub server over websockets. The main page displays the header and footer and handles core services like security, navigation tabs and network. JQuery and Twitter Bootstrap libraries are used to make the coding easier. This design pattern is a proven approach for a SPA (single page application) but popular SPA JavaScript frameworks like AngularJS aren't used here as they would complicate things - this system should be considered a framework for home automation as it has an opinionated implementation tuned specifically for home automation use-cases.

  What is unique is the use of HTML5 objects to represent all the entities inside a dashboard iframe, there is nothing hard coded, the user picks widgets from a toolbox and drags them to wherever on the screen, resizing if needed and setting attributes via a right-click menu or typing on the widget itself (eg. click on a widget's text and start typing to edit). These widgets can be simple drawing objects like a line or a panel, or richly functional widgets like a graph or lightbulb graphic that subscribes or publishes messages to and from the hub. It is a 'design surface' and works similar to Inkscape or Microsoft Powerpoint giving the user full flexibility about how they want the screen to look. The widgets are fully compliant HTML, CSS and JavaScript leveraging the full power of HTML5 (like SVG for vector graphics) so widgets can look as simple or as complex as you want them to be, including animation.

  Above is an example of one of the dashboard screens (click to see it in higher resolution). This is the weather tab and it combines feeds from the home weather station (temperature, wind direction and speed, rain and humidity), as well as services from the cloud (the local Bureau of Meteorology 5 day forecast and rain radar), day, month and year highs and lows calculated real time by the transformation module and graphs with data provided by the history (database) module.

  The graph widget is particularly interesting as it uses the excellent D3 JavaScript library and the user on a touchscreen can swipe left or right to scroll through history, or pinch to expand the view for more detail. There is also subtle use of animation - the weather radar is a moving slideshow of rain over the last hour, the needle on the dial and compass mimics an analog meter (accelerating from start to final position with some overshoot), the graph line is animated and the windmill rotates proportional to wind speed.

  By combining information from different sources with a layout that is sympathetic to the information presented, as well as rich graphics and animation all makes for an intuitive and compelling page. Nothing on this screen except for the header / footer is hard coded - it is all user configured and can be easily re-arranged if needed.
  
  

  Designing and Configuring Screens

  Above is another dashboard screen in design mode for editing (click for higher resolution). On the left you can see the widget toolbox that the user can scroll or search for a widget, drag it onto the design surface and place it or resize it as needed with a...

  Read more »

View all 6 project logs