• Device Cloud Hello World

    tdicola • 05/19/2014 at 01:38 • 0 comments

    This is a video of how to create a 'hello world' for the device cloud--blinking an LED from a webpage over the internet: 

  • Service Architecture

    tdicola • 05/18/2014 at 22:52 • 0 comments

    Here's a deeper look at the services for the DIY device cloud:

    This diagram shows a simple device cloud 'hello world' application that turns an LED on from a web page over the internet.  You can see in the bottom left a web page which is hosted on the broker server.  When a button is clicked (1) it communicates to a python web application which is hosted by gunicorn (python WSGI web app server) and nginx (main web server).  That web application talks directly to the MQTT broker on the server (2), which will inform any connected clients of MQTT messages.  A device runs a device service which is just a simple python script that talks to the broker (3) and turns on/off the LED with GPIO libraries (4).  The device service is hosted by the excellent supervisor tool so it will automatically run in the background on the device--all the messy details about properly running a background process are abstracted away by supervisor.

    This diagram highlights the two main extension points I see where you can plug in your own code to build your device cloud:

    • Web services, written as python WSGI applications (using a simple framework like flask).
    • Device services, written as simple python scripts that are run automatically by supervisor.

    The ansible deployment scripts for both the broker server and devices take care of setting up all the services, configuring them, etc.  You only need to place your web service and device service code in a certain location and follow a few simple conventions.

    I made a video of creating this simple hello world application and will post it shortly when Youtube is finished processing it.

  • Server Deployment Updates

    tdicola • 05/15/2014 at 08:33 • 0 comments

    I've done some cleaning up and refactored the Ansible playbooks to use roles as suggested by Ansible's best practices.  I'm not super happy with how the system is configured--right now global configuration is in the inventory file under the [all:vars] group.  This is a good spot to configure the entire system in one file but feels a little unwieldily.   I'll be looking more into other options here to see if there's a simpler and more flexible way to configure the system.

    I've also added quite a few roles to form the base of the server, including:

    • MQTT server: installs mosquitto server and configures it to use SSL secured MQTT communication.
    • MQTT client: installs mosquitto client tools, Paho MQTT python client library, and the SSL certs for communication with the server.  This role can be installed on both the Ubuntu-based server and Debian-based devices like the Raspberry Pi or Beaglebone Black.
    • Security: installs and configures fail2ban brute force attack protection package.  Ubuntu server is also pretty well locked down by only allowing SSH login with keys (i.e. no password auth allowed), and denying root the ability to login.  I want to look more into other useful security software / configs, but am pretty happy with security for now.
    • Web server: installs nginx web server.  There's not much configured right now, but I plan to put some basic web apps on to help manage the server & communicate with devices.
    • SMTP email relay: installs postfix mail transport and configures it to relay mail to an SMTP server like gmail.  This is handy for allowing the server to send emails, like from fail2ban warnings or perhaps notifications of MQTT events.  Surprisingly this was by far the most complex package to automate with Ansible, mostly because installing postfix requires answering some question prompts which are difficult to do with automation.

    One more major package I want to install is a Python WSGI app server that can host web applications written in Python.  I plan to make a few simple web apps for reading and writing MQTT messages and would prefer to write them in Python with the Flask web app framework.  I've looked into some options here and it looks like it comes down to either uWSGI or Gunicorn.  From my testing uWSGI is kind of painful to setup so I might go for Gunicorn.

  • MQTT Broker Setup On Amazon EC2

    tdicola • 05/12/2014 at 21:54 • 0 comments

    In the past couple days I've added playbooks to the project github source which allow you to provision an MQTT broker server on Amazon's EC2 cloud and set up a basic device cloud.  Check out this video to see a walkthrough of setting up a server and communicating with a Raspberry Pi: 

    I'm still a little new to using Ansibile, but so far am really happy with how simple it's making the process of standing up a server from scratch.

    Also I purchased the 'diydevicecloud.net' domain name and am making subdomains of it available to anyone who wants to host their device cloud from an easily reachable address with the excellent FreeDNS.afraid.org dynamic DNS service.  You can find the domain here and just need a free account on FreeDNS to get a subdomain.  I've even integrated into the Ansible playbook a step to register a regular cron job that will keep your broker server IP address up to date with your FreeDNS domain.  Watch the video to see more details of the process.  

    You aren't limited to using this domain either--there are many other domains on FreeDNS, or you can even buy your own domain and get a static IP in Amazon's cloud.

    The next area I want to focus on are client apps.  As a part of this project I want to document a few end to end examples of MQTT applications, such as measuring sensor data or remotely controlling devices.

  • Github Home & Documentation

    tdicola • 05/10/2014 at 02:07 • 0 comments

    I've started a repository on github that will be the home for all the software output from this project.  Since documentation will be an important part of this project I've also created a base for the documentation at http://diy-device-cloud.readthedocs.org/ using the excellent readthedocs.org service.  The documentation is actually built from files in the github repository for the project (under the docs folder) based on the sphinx tool.  Because the docs are in the repository it's easy to keep them versioned and up to date.  For now I've put up a couple docs that recap what I've covered so far in the project logs here.

  • Architecture Sketch

    tdicola • 05/09/2014 at 18:55 • 0 comments

    Here's a high level look at how I see different components fitting together to form a device cloud:

    The MQTT broker, mosquitto, is at the center of all communication and lives on a server in Amazon EC2, Microsoft Azure, etc.  Along with the broker other applications like visualizations or control apps are hosted on the same server and can communicate with the broker directly.  I have some early thoughts on these apps and will save that for a followup post.

    Devices in your home network, or really anywhere, connect to the MQTT broker using an encrypted SSL MQTT channel.  Normally SSL is a tremendous pain to configure and manage (especially with self-signed certificates), however I've found it's easy to automate the certificate creation tasks using Ansible so it should be painless.  I want the device cloud to be secure by default and not require someone to be an expert in security to setup or run the system.

    Devices can be any hardware that's powerful enough to speak MQTT's protocol.  Embedded Linux boards like the Raspberry Pi, Beaglebone Black, Arduino Yun, Intel Galileo, etc. will be very easy to configure and use with the device cloud.  I hope to even provide Ansible tasks that can deploy and configure MQTT client tools and libraries on any embedded Linux device automatically.

    What about devices which don't support SSL, like an Arduino & CC3000?  In this case mosquitto has a concept of a bridge server which can act both as an MQTT broker and client.  Any MQTT messages sent to the bridge will be relayed back up to the parent broker and vice-versa.  With this setup it will be possible to build a small, low power Arduino + CC3000 device that periodically connects to the bridge over an unencrypted channel to send/receive commands.  Because the bridge lives inside your home network (which is secured with wireless encryption, right?) it's still relatively secure.  Certainly more secure than opening the broker in the cloud up to unencrypted MQTT traffic!  Again Ansible tasks should be able to turn any Linux gadget into a bridge for the device cloud easily.

  • Ansible - Automation That's Easy

    tdicola • 05/08/2014 at 23:42 • 0 comments

    Ansible is part of a new generation of open source IT automation tools which aim to simplify the automation of tasks like installing software and setting up servers.  You can see more about Ansible from its creator in this recent PyCon talk: 

    Ansible strives to be simple and requires very little support infrastructure or complex setup.  Even though the project is only a few years old, it is already one of the largest on github and home to a healthy ecosystem of modules that can automate almost any server task.  

    I've been investigating Ansible and expect to make heavy use of it as the primary means of automating the setup and maintenance of the device cloud.  The primary output of this project will likely be a collection of Ansible playbooks and tasks that help you build and manage your own device cloud.

  • MQTT - Broker That Ties Everything Together

    tdicola • 05/08/2014 at 23:31 • 0 comments

    MQTT is a publish-subscribe protocol that's perfect as a broker for device to device communication.  Created by Andy Stanford-Clark and Arlen Nipper, it has been used in projects like Andy's own 'Twittering House'.  You can see more from Andy in this TEDx talk: 

    Mosquitto is an open source MQTT broker implementation that's under active development, has an established reputation, and is well documented.  I've done some informal testing with Mosquitto and it seems like the perfect broker to use in this project.

  • Motivation and Goals

    tdicola • 05/08/2014 at 23:06 • 0 comments

    What is a device cloud?

    I consider a device cloud to be a service that provides both a broker for communication between devices and a host for applications that interact with devices. The broker allows devices to communicate to other devices or applications without having to be tightly coupled or directly connected to each other. The device cloud is also an easily accessible host for applications that interact with devices, such as visualizations of data or control of device functionality.

    Why do you want a device cloud?

    • Discovering and connecting to your devices over the internet is hard! Dynamic IP addresses, NAT, firewalls, and more prevent you from easily accessing your home network and devices. Hosting your own device cloud on the internet allows devices to connect and communicate without the trouble of opening connectivity to your home network.
    • Not all devices can be connected and online all the time. A device cloud allows for asynchronous communication between devices. For example a sensor or actuator can periodically connect to your device cloud to record data or receive new instructions; because the device isn't connected all the time it can greatly reduce its power consumption.
    • You have total control over the data and services available to your devices. You aren't limited by what a 3rd party provides or concerned about what they might do with your data. You own the entire infrastructure and can mold it to your needs.

    Project Goals

    • Automate the setup and maintenance of services that form a personal device cloud. Setup should be an easily repeatable process from a single command or script.
    • Target running the device cloud on cheap, low end cloud infrastructure like Amazon EC2's micro instance free tier.
    • Build on existing tools, services, and protocols--don't reinvent the wheel! Many of the services to build a device cloud exist today, they just need to be put together into an easy to use package.
    • Document how to connect popular development hardware like the Raspberry Pi, Beaglebone Black, and Arduino to a device cloud.