Linux ESP-NOW finally integrated on Solo12, our open source quadruped research platform !

More info on the robot : https://github.com/open-dynamic-robot-initiative/open_robot_actuator_hardware/blob/master/mechanics/quadruped_robot_12dof_v1/README.md#quadruped-robot-12dof-v1

It's alive! Thanks to our ESP-NOW linux library, we are now able to do closed loop control of fast brushless motor at 1kHz wirelessly !!

For that we designed a small PCB hosting an ESP-32 and exposing 8 SPI lines. With this setup we can control up to 8 custom made dual BL motor drivers. On the experiment below, position of 6 motors where sent via ESP-NOW to a RT-Preempt PC. The PC where 10m away from the motors, computing torque (current) to servo 5 motors on the 6th one position.

Additionally, The board is including a Ethernet port. This provides a backup solution for non WiFi friendly environments.

The board is open source and will be integrated to a VERY exciting robotic project. 

More on that latter...

I think this board could also serve as a base for many other remote controlled robotic project. With the Ethernet cable, it makes a flexible solution for fast and real time communication with hardware.

I installed (and patched) a real-time Linux on a computer, to do some 'more serious' testing. I chose Ubuntu 16.04 (with 4.14.109 kernel). And I patched it using RT-PREEMPT.

DISCLAIMER: I am no expert! And I probably did a lot of mistakes/inappropriate things. The following paper does not explain a universal method to install a real-time Linux on your machine. It just traces what worked for me.

Here is what I did :

https://github.com/thomasfla/Linux-ESPNOW/blob/master/Misc/RT_Preempt_patch_Linux.pdf

The computer I used was a Dell Precision (with an Intel Xeon).

I tried with 2 different PCIe Wireless Network Interface Cards :

• TP-Link TL-WN881ND (Seems to work really bad in this current setup).
• ASUS PCE-AC51 (The one that I used in the end because it seems to work the best in his current setup).

Finally, after tuning these parameters, and modifying a little my ESPNOW code on the computer (to take better advantage of the real-time OS), here is what I got from the testing :

These results show more consistency in the round trip time. In fact, the histogram shows only 1 sharp peak. Moreover, I did this test several times, and each time the average round trip time was very close to the ones I measured earlier (ranging from 1100µs to 1400µs).

Whereas, in my previous post, the average round trip time could vary from 1000µs  to 2100 (or more) within one test...

I don't notice any big changes, concerning the packet loss, other than it is more consistent over the test (same as for the round trip time).

All the results presented here have been obtained following this procedure (using the code mentioned in the previous Log) :

• Every 1ms, the ESP32 sends a packet to the computer with a payload of 127 bytes. This payload contains the time of emission.
• A C++ code running on the computer (with the highest priority) receives this packet, copies the time of emission in a new packet (with a payload of 127 bytes), and sends it back to the ESP32
• The ESP32 receives this response and measures the round trip time.

Effect of the data rate on round trip time and packet loss :

The figure 2 shows the round trip time and the number of packet loss depending on the data rate. Each point of the graph was obtained by averaging the values of 3 batches of 10,000 packets. These tests have been done both when the surrounding WiFi activity was 'very low' and 'normal'.

It seems that the data rate does not impact much the packet loss. A much more significant factor for the packet loss is the surrounding WiFi activity.

Moreover, as expected, the trend is that the higher is the data rate the shorter is the round trip. However, this relationship is not linear.

Further analysis on packet loss :

Since we choose to disable ACK to speed up the connection, we need to check that the packet loss is reasonable. However, the most important is that the loss is scattered. In fact, it would be very harmful in our robotic application to have big batches of packets being lost. It would mean that there is no feedback for 'long' periods of time.

The figure 4 shows that the size of the 'groups' of lost packets seems to follow a normal law centred on 0. The variance seems to be very low, so it is very rare (less than 0.1% in this example) to lose 3 or more packets in a row. Therefore the loss of packets is more acceptable.

Future improvements :

The next steps are :

• Try different Wireless Network Interface Cards on the computer to see how it impacts the performances
• Patch Linux on the computer to be real-time.

On the computer

Library :

A C++ library has been created to handle ESPNOW packets with Linux. You can find it here : ESPNOW_lib.

The code also contains a main() that send back every ESPNOW packet received (it acts like an echo).
If you want to use this code don't forget to turn on the monitor mode on your wireless interface and also tune it to the right channel (because the library can be set-up to use any channel). As a reminder, this is how you do it :

sudo ifconfig wlan0 down
sudo iwconfig wlan0 mode monitor
sudo ifconfig wlan0 up
sudo iwconfig wlan0 channel 1 

Berkeley Packet Filter (BP Filter) :

The BP Filter that has been attached to the socket was generated using the tcpdump command :

sudo tcpdump -i wlp5s0 'type 0 subtype 0xd0
and wlan[24:4]=0x7f18fe34
and wlan[32]=221
and wlan[33:4]&0xffffff = 0x18fe34
and wlan[37]=0x4
and wlan dst 11:22:33:44:55:66
and wlan src 77:88:99:aa:bb:cc' -dd

It filters packets according to :

• Type and subtype : Action frames (0xd)
• Category code of the action frame : Vendor specific (0x7f)
• OUI : Espressif (0x18fe34)
• Type of the vendor specific action frame : ESPNOW (0x4)
• Source and Destination addresses

The Assembly code generated has been modified a little bit afterward to fit any source & destination addresses. You can see it at line struct sock_filter temp_code[this->bpf.len]  in ESPNOW_manager.cpp.

On the ESP-Module

Acknowledgment (ACK) :

The aim of this project is to have a fast wireless connection between 2 devices, with the least delay.

However, the ESP-NOW protocol is built to wait for an ACK frame after every packet sent. It ensures the reception. And, while the ACK is not received, the ESP-NOW library tries to send the same packet over and over again.

This slows down the whole process !!

Moreover, in our robotic case, if a packet is lost there is no point on trying to send it again. In fact, this packet is now outdated and it is possible to send a newer one (which is more interesting for our system).

The easy way found to avoid ACKs, is to send the packet over the broadcast address (and not directly to the other device address). So the ESP will neither send or wait for ACKs.

In the code, you simply need to declare the peer with the FF:FF:FF:FF:FF:FF mac address :

esp_now_peer_info_t peer;

for (int i = 0; i < 6; ++i ) {
    peer.peer_addr[i] = 255;
  } 

Remark: Setting the ESP to send over broadcast to avoid ACKs is only available on ESP32. That is what motivated the choice to change from ESP8266 to ESP32.

Data rate :

The modification of the data rate was decisive to decrease of the round trip time. The default rate for ESP-NOW on the ESP32 is 1 MBps. At this rate, physically send a packet (header + 127 Byte of payload) takes approximately 1.5ms. So the round trip cannot be less than 3 ms!

By disabling some default 'safeties' in the WiFi library, it is possible to change the data rate directly at the WiFi level (and bypass the default 1 MBps in the ESPNOW library). Now the data rate can be set up to 54 MBps which correspond to a physical sending time of 0.05 ms. To change it, here is what you need to do :

#include <WiFi.h>
#include <esp_wifi_internal.h>
#include <esp_now.h>

int main() {

//...
WiFi.disconnect();
WiFi.mode(WIFI_STA);

/*Stop wifi to change config parameters*/
esp_wifi_stop();
esp_wifi_deinit();

/*Disabling AMPDU is necessary to set a fix rate*/
wifi_init_config_t my_config = WIFI_INIT_CONFIG_DEFAULT(); //We use the default config ...
my_config.ampdu_tx_enable = 0;                             //... and modify only what we want.
esp_wifi_init(&my_config);                                 //set the new config

esp_wifi_start();

/*You'll see that in the next subsection ;)*/
esp_wifi_set_channel(CHANNEL, WIFI_SECOND_CHAN_NONE);

/*set the rate*/
esp_wifi_internal_set_fix_rate(ESP_IF_WIFI_STA, true, DATARATE);

/*rest of the code :*/
esp_now_init();
//... 

} 

Remark: Of course, increasing the data rate too much can diminish the robustness of the transmission.

Frequency / Channel :

Finally, as you saw in the previous piece of code, it is also possible to change the channel of transmission.

Choosing the right channel, with the fewest devices using it, better a lot the performances :

• It reduces the perturbations and noise on the channel, and so, decreases packet loss.

• It reduces the time of the round trip. Since the frequency is less used, the network card doesn't have to wait long for the frequency to be free. And so more packets can be sent almost immediately.

• It tends to better the consistency of the performances.

Examples :

• Linux C++ code (does a simple echo using the 'library' we wrote) : https://github.com/thomasfla/Linux-ESPNOW/tree/master/ESPNOW_lib
• ESP32 code to send packets at 1kHz and measure round trip time and some other things : https://github.com/thomasfla/Linux-ESPNOW/tree/master/ESP32-Test
• (the PYTHON_serial folder contains some scripts to plot what the ESP sends back over the serial port).

Project status: The code is now able te send arbitrary ESPNOW data to a chosen MAC address.

What we need now is an efficient way to filter incoming packet. This is possible by software in the user space via a simple function that check the destination MAC, the type frame and some ESP now specific filed. But there is a better solution called Berkeley Packet Filter (BPF).

BPF can be attached to a socket. A filter is a program that will analyse any incoming packets and return True or False whether the packet should be accepted or rejected. This program is executed in kernel space in a virtual machine (after some analysis of the code). How cool is that?

To program a BPF, we have to write the filter with assembly, given a specific instruction set.

An exemple can be found here https://gist.github.com/oro350/8269805

In this example, we can see how a filter looks like:

static struct sock_filter bpfcode[6] = {
	{ OP_LDH, 0, 0, 12          },	// ldh [12]
	{ OP_JEQ, 0, 2, ETH_P_IP    },	// jeq #0x800, L2, L5
	{ OP_LDB, 0, 0, 23          },	// ldb [23]
	{ OP_JEQ, 0, 1, IPPROTO_TCP },	// jeq #0x6, L4, L5
	{ OP_RET, 0, 0, 0           },	// ret #0x0
	{ OP_RET, 0, 0, -1,         },	// ret #0xffffffff
};

Writing this program from scratch can be a bit tricky, if we need to adapt it to non fixed length header, compare a full MAC, etc.. 

But tcpdump can generate this for us with the -dd option.

Going further...

So now we have a hint for filtering packet in the kernel. But we can do better! The BPF could be done directly at the reception of incoming packet in the MAC80211, before the radiotap header is calculated etc..

This is not officially implemented in the current MAC80211 driver yet, but here is a patch and discussion about it

https://patchwork.kernel.org/patch/9679455/ 

I must admit that I never thought I would have to go so deep into linux for this project, but it's quite fun.

After a lot of testing with different lib and radio driver, together with @Florenc Caminade, we managed to capture and send ESP now packet! And it turns out to be quite simple, we just use linux sockets to send and receive packet. The tricky part was to generate a legit packet the card would accept and transmit as a vendor specific action frame. We had to include a radiotap header...

At the moment, packets are hard coded raw data, so it will only work if your ESP has the same MAC address, and it will send always the same data.. So not very useful but a good proof of concept

The code is on GitHub: https://github.com/thomasfla/Linux-ESPNOW/ and need a bit of cleaning.

To test it, you have to use this MAC address:

ESP: 84:F3:EB:73:55:0D
PC:  F8:1A:67:B7:EB:0B

you also have to set your interface in monitor mode, and it has to support packet injection.

compile and run using:

cd wifiRawSender/
make
sudo ./bin/sender wlan0

 where wlan0 is your interface up in monitoring mode (same applies for receiver example).

What's next?

• Generate or parse the packets (Host MAC, Destination MAC, Payload and payload size, ...)
• Simplify the radiotap header
• Find an efficient way to implement a MAC filter in monitoring mode
• Write a clean library
• Measure the round-trip time

Notes:

About the ACK behavior, I switch to an other wifi interface that does send ACK in monitoring mode !
I did not find if and where this is documented. For my application I don't mind to use a specific card.

To implement the ESP-NOW Protocol, we first need to understand how it works in details.

The datasheet doesn't say much on the really low level, especially on the acknowledgment behavior.

To understand what ESP-NOW does, I use Wireshark with my PC Wifi card in monitoring mode:

> sudo ifconfig wlan0 down
> sudo iwconfig wlan0 mode monitor
> sudo ifconfig wlan0 up
> sudo wireshark

With one ESP8266 sending ESP-NOW to a non existing MAC adress, I got the following packets:

The same packet is sent multiple times, with a decreasing datarate :

• 2 tries at 6Mb/s
• 2 tries at 2Mb/s
• 7 tries at 1Mb/s (maybe this last number of tries depends on a timeout...)

First bad news: ESP-NOW continues to send the packet again and again for 20ms !

I hope their is a way to disable this resend behavior... Indeed, for a robotic control, we prefer to wait for the next up-to-date packet than getting an outdated one, and jamming the channel...

Let's look at the packet itself:

0000   00 00 12 00 2e 48 00 00 10 0c 6c 09 c0 00 d5 03   .....H....l.À.Õ.
0010   00 00 d0 00 3c 00 84 f3 eb 73 55 0d 86 f3 eb 73   ..Ð.<..óësU..óës
0020   ca 61 84 f3 eb 73 55 0d a0 09 7f 18 fe 34 17 71   Êa.óësU. ...þ4.q
0030   47 8c dd ff 18 fe 34 04 01 62 12 12 12 12 12 12   G.Ýÿ.þ4..b......
0040   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0050   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0060   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0070   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0080   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0090   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00a0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00b0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00c0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00d0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00e0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
00f0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0100   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0110   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0120   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
0130   12 12 12 1d 2c ba 98                              ....,º.

 The payload is "0x04 0x12 0x12 ... 0x12" (on my test I choose to begin with a packet counter to distinguish different packets.)

The frame format seems to follow the documentation from expressif, and the header should be easy to generate:

https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/wifi/esp_now.html

Now, if I turn on the destination ESP8266, the packets are sent only once, and a quick acknowledgement frame is sent back in (about 60us):

I guess this acknowledgment frame will be generated by the low level MAC layer, so we shouldn't take care of managing it... 

The next test is then to send ESP-NOW packets to my laptop MAC address, and check that ACK are sent automatically even in monitoring mode... 

Hum... After a quick test, when I Sending a packet to my laptop's MAC address, it does't send an ACK back. Maybe the monitor mode disable this in the MAC layer ?