NOTE: these build instructions are for the first prototype with the ESP8266 and the 2.7" ePaper display. Will be moved later to another project, and the instructions here will be updated for the new ESP32 module and the new 4.2" ePaper.

Build a new NodeMCU firmware

NodeMcu has a lot of modules you can use, as you can see in the documentation:

https://nodemcu.readthedocs.io/en/master/

If you buy a NodeMcu module from eBay, chances are good that it comes with an old version and not with all modules you might need. So for the latest version and with all the goodies you need, it is best to compile your own firmware.

The easiest way to do this is to use some cloud build service, as described on this page:

https://nodemcu-build.com/index.php

You can stop reading now if you use this service.

But you can compile it from source as well. You can do this with a Docker image:

https://hub.docker.com/r/marcelstoer/nodemcu-build/

Docker is a nice system to create runtime environments with all the programs and libraries inside your OS. There are many different environments. To quote the webpage:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

If you've ever tried to setup a cross-compiler environment, you know how much work this can be, and chances are good that you break something other on your system. Docker solves all these problems.

I'm using Debian Jessie as my main working operating system. Unfortunately it is not just "apt-get install docker" to install the Docker environment, and there is even another package named "docker", which makes it a bit confusing, but there is a manual at the Docker homepage how to install it and you need to to this only once:

https://docs.docker.com/engine/installation/linux/debian/

Obviously you don't want to do "apt-get remove docker", but the rest worked. This were my steps:

sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce

"docker-ce" is the free community version of docker. There are instructions for Windows and Mac, too. After the installation you can check that it works with "sudo docker run hello-world" on Linux.

Now you can install the NodeMcu Docker build environment:

git clone https://github.com/nodemcu/nodemcu-firmware.git
sudo docker pull marcelstoer/nodemcu-build
cd nodemcu-firmware
docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build

If you want to enable different modules, change the file "app/include/user_modules.h" and recompile your image with the last "docker run" command. The result is in the "bin" directory. There are two version, one integer and one float version, which determines what number type Lua uses. I use the float version, because this is the default in Lua and makes programming easier. The image name in my case was "nodemcu_float_master_20170410-1703.bin".

Flash a NodeMCU firmware image

For flashing a new image, I use esptool. Plugin your NodeMCU module to an USB port, clone the esptool project and flash your firmware. Usually a new device will appear, like /dev/ttyUSB0. Step by step:

git clone https://github.com/themadinventor/esptool.git
cd esptool
sudo python2 ./esptool.py --port /dev/ttyUSB0 write_flash 0x00000 ../nodemcu-firmware/bin/nodemcu_float_master_20170410-1703.bin

(if there is an error "ImportError: No module named serial", then do a "sudo python2 -m pip install pyserial". If you have only Python 2.x installed, you can use "python" instead of "python2", this is just my system, where I have Python 3.x installed, too. Don't ask how long I needed to Google for all the errors and to have both versions of Python installed in parallel)

Flashing needs less than a minute and the output should look something like this:

esptool.py v2.0-beta2
Connecting....
Detecting chip type... ESP8266
Uploading stub...
Running stub...
Stub running...
Attaching SPI flash...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0040
Compressed 436208 bytes to 279594...
Wrote 436208 bytes (279594 compressed) at 0x00000000 in 24.6 seconds (effective 141.8 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

You can test it with "minicom --device /dev/ttyUSB0 --baudrate 115200". You might need to disable hardware flow control (the usual Telix commands: ctrl-a, o, "Serial port setup", F and G for disabling hardware and software flow control). Hit the reset button on the module (small button near the USB connector) and you should see something like this:

NodeMCU 2.0.0 build unspecified powered by Lua 5.1.4 on SDK 2.0.0(656edbf)
lua: cannot open init.lua
> 

It has a Lua REPL, which means you can execute commands from the minicom terminal, which is nice for fast prototyping and testing things:

> = 1+2
3
> = wifi.sta.getip()
192.168.1.123   255.255.255.0   192.168.1.1

Install a Lua script

If you want to install a Lua script, which is started when the module starts, you can write this in a file called "init.lua". With the luatool.py you can upload it to the NodeMCU module:

git clone https://github.com/4refr0nt/luatool.git
python luatool/luatool/luatool.py --port /dev/ttyUSB0 --src init.lua --dest init.lua --restart

You need the git clone only once. Sometimes the luatool script doesn't work. Reasons for this could be that you have still minicom open in another terminal, or you need to reset the module and start the script again. Note: if there is an error in init.lua, the module might get into an infinite reboot loop. To fix this, you can re-flash the firmware image, which deletes all scripts.

This is an example script, which loads another script init2.lua from a webserver and executes it:

function executeString(s)
    local fun = loadstring(s)
    fun()
end

function stripHeader(s)
    local pos = string.find(s, "\r\n\r\n")
    return s:sub(pos + 4)
end

function runScript(host, url)
    receivedPage = ""
    conn = net.createConnection(net.TCP, 0) 
    conn:on("receive", function(conn, data) receivedPage = receivedPage .. data end)
    conn:on("disconnection", function(conn, data) pcall(function() executeString(stripHeader(receivedPage)) end) end)
    conn:connect(80, host)
    conn:send("GET /" .. url .. " HTTP/1.1\r\nHost: " .. host .. "\r\nConnection: close\r\n\r\n")
end

wifi.setphymode(wifi.PHYMODE_N)
wifi.setmode(wifi.STATION)
wifi.sta.config("SSID","PASSWORD")
wifi.sta.eventMonReg(wifi.STA_IDLE, function() print("IDLE") end)
wifi.sta.eventMonReg(wifi.STA_CONNECTING, function() print("CONNECTING...") end)
wifi.sta.eventMonReg(wifi.STA_WRONGPWD, function() print("WRONG PASSWORD!!!") end)
wifi.sta.eventMonReg(wifi.STA_APNOTFOUND, function() print("NO SUCH SSID FOUND") end)
wifi.sta.eventMonReg(wifi.STA_FAIL, function() print("FAILED TO CONNECT") end)
wifi.sta.eventMonReg(wifi.STA_GOTIP, function() runScript("192.168.11.27","init2.lua") end)
wifi.sta.eventMonStart()

Replace "SSID" and "PASSWORD" with your WiFi SSID and password and the address "192.168.11.27" with your webserver address. For faster development it is nice to install a local webserver. Then you can edit the init2.lua file locally on your PC or Mac, and to test it just hit the reset button on the module, which reloads the script. This helps also if you have errors in your Lua script, because you don't need to re-flash the firmware in this case. Leave minicom open to the serial port and you will see the output of the script (with "print") and when it reboots. Once your script works, you can rename it to init.lua and permanently install it on the module with luatool.py.

Note: the default max string length is 4096 bytes in Lua, which is the limit for your script. You can change this with 'collectgarbage("setmemlimit", 64)' in your init.lua to e.g. 64 kB. But when WiFi is connected etc., you have only about 40 kB available RAM. If you need more: the ESP32 has 520 kB RAM.

Installing an MQTT broker and using it from NodeMCU

For IoT devices a lightweight connectivity protocol like MQTT makes it easy to implement a sophisticated and reliable cloud service without much code. There are some free online brokers available which you can use, but sometimes it might be better to install your own broker. I'm using Mosquitto, an open source MQTT broker. For testing your cloud services from the command line, the mosquitto clients are useful as well. In Debian Linux or Ubuntu you can it install like this:

sudo apt-get install mosquitto
sudo apt-get install mosquitto-clients

For testing it, you can subscribe to a "topic". If some other client then sends something to this topic, you can read it. Open a shell and listen to a topic:

mosquitto_sub -t "foobar"

Now open another shell and publish something for this topic:

mosquitto_pub -t "foobar" -m "Hello World"

You will see the message in the first window. If you send something to another topic, you won't see it. See the manual page for more details, e.g. for debugging you can listen to all topics.

Now you can write a Lua script for the NodeMCU. I'm saving it on a webserver as described earlier:

-- init mqtt client without logins, keepalive timer 120s
m = mqtt.Client("clientid", 120)

-- on publish message receive event
m:on("message", function(client, topic, data) 
  print(topic .. ":" ) 
  if data ~= nil then
    print(data)
  end
end)

m:connect("192.168.11.27", 1883, 0, function(client)
	print("connected")
	-- subscribe "message" with qos = 0
	m:subscribe("message",0, function(client) print("subscribe success") end)
	-- publish a message to "foobar" with data = hello, QoS = 0, retain = 0
	m:publish("foobar","hello",0,0, function(client) print("sent") end)
end, function(client, reason) print("failed reason: "..reason) end)

This code creates a new MQTT client, then connects to your MQTT broker and in the callback function that is called with connected, it subscribes to the "message" topic and sends a message to the "foobar" topic. Replace the IP with the number of your PC where the MQTT broker is running. On the PC you can listen to the topic "foobar" with mosquitto_sub. When you run the script on the NodeMCU module, you can see the message "hello" on your PC. And when you send something to the topic "message", you can see in minicom that the NodeMCU module receives it:

> CONNECTING...
init2.lua loaded
connected
subscribe success
sent
message:
Hello World

On the module you could run a timer, read the temperature and then send it each minute to a topic, which then can a server program save in a database, or you could just log it with the Mosquitto client tools. That's all, no need to parse HTTP messages etc., very easy to use.