Chirppp, Serial over LORA

I've been working on the project little by little, but not enough to post on here. It's gotten cold, so range testing isn't a very tempting prospect at the moment.

I've retired the vocore and got it running on a RPI 1 B+, I was having some connection issues with the unsoldered pins stuck into the vocore's GPIO via throughholes, and I want to use it in some future projects, so I don't want to gunk it up with solder.

I've got some false positives coming in from the radios that can cause a desync issue, which I'm trying to track down. I'm not completely done with the project as is stands though, so check back every month or so and see if I've made any headway!

Daily living has gotten busy for me! Between not so successful interviews, dealing with insurance after getting rear-ended, and leaving for a cruise the 18th - 29th, I won't be able to work on chirppp for most of the month of August. Stay tuned though! I'll be getting right back into it come September.

I sorted out the problems I was having with the pty master and slave. As it turns out, echo is on by default. This means that if the master writes something to the slave and nobody reads it before the master reads, the slave will echo the data right back... not sure who decided to make that option the default, but for my purposes it served as a terrific wrench.

I was making... okay... serial connections. I was losing some packets, duplicating others, I think in order to transmit more reliably I'll have to wire up my vocore better. Here's what it looks like now

The wires are just hanging in the through-holes, I managed to exert enough pressure on the side of them to create a decent connection, but under load I don't think it's handing itself too well. I've been seeing some malformed input and output that wiggling the serial connection to the radio effects, I may have to whip out the soldering iron before too long.

But the connection was stable enough for me to stay up past my bedtime. I went ahead and did some digging on pppd and managed to create a command that would work on both devices.

Here's the command to run on my Vocore (server):

pppd -detach xonxoff noauth lock 192.168.69.1:192.168.69.2 /dev/pts/1 38400 eap-restart 100 lcp-echo-failure 10000 lcp-echo-interval 10 

And here's what I need to run on the CHIP (client)

pppd -detach xonxoff noauth lock 192.168.69.2:192.168.69.1 /dev/pts/2 38400 eap-timeout 100 lcp-echo-failure 10000 lcp-echo-interval 10 

As you can see, the only difference is the eap-timeout vs the eap-restart. The serial connection is, obviously, very slow, so in order to get ppp to maintain a connection I had to make it a little more error tolerant.

After that, I issued my first ping... I pinged the 69.1 address (server) from the 69.2 address (client), and much to my surprise, I actually got a response!

Now I couldn't do much more than that. I may need to recompile the kernel on my CHIP device to support PPP filtering, and I didn't have any firewall or iptables rules set up on the Vocore to allow any internet passthrough. To be honest I didn't think I'd get this far today.

I tried quite a few combinations of device speed and timeout duration as far as the serial emulator is concerned, but without a solid electrical connection for the devices to work with, there's no sense trying to fine-tune anything just yet.

I'm starting to see the light at the end of the tunnel though! Really all that's left to do is wire up the devices correctly, take it outside for some range testing, and see what kind of internet usage I can get out of it.

It's worth noting that the point to point protocol does have some error correction for dropped packets. It seemed at faster speeds with quicker timeouts the radios would be working harder, but could manage enough speed to allow ppp to pick itself up when the radios stumbled a larger portion of the time.

speed 38400 baud; line = 0;
ignbrk -brkint -icrnl ixoff -imaxbel

Looking better.

However, after having the pty slave be recognized properly, the data transfer between the radios is behaving strangely.

I'm going to have to go through and debug what is actually happening.

I have a feeling the iflags and oflags of the tty aren't set correctly. I'm going to have to read up on what they each do to try and get data sent back and forth correctly again.

Version 0.1 is well on its way!

I've still got a few kinks to iron out, I had to adjust the flowchart I posted in the previous log a bit to properly handle retry and heartbeat packets.

Occasionally I'll get a false positive that the radio has received a packet, and 0 bytes of data will be read from serial. This is tolerable at lower speeds, I can add some retry logic in to handle it well, but at higher speeds this can end up with only half a packet read. I'll have to pore over the driver logic again to see if it's a logic error of mine, or just an erroneous signal that is managing to make it through the AUX wire to the device...

If I end up not being able to fix it, I'll be able to make it send a retry packet if it gets corrupt data to just try receiving it again. Not the most elegant solution, but when you're trying to pick up hardware interrupts with epoll on a slow embedded device, there's only so much you can do :P

I may also have to move from using the libc binding of openpty to using the rust nix crate. When I use stty (a tool for looking at serial connection information) on one of ssh's ptys this is the output I see:

root@chip:/dev/pts# stty -F 2
speed 38400 baud; line = 0;
eol = M-^?; eol2 = M-^?;
-brkint ixany

Looks pretty good. We can see it's running at a virtual 38400 baud and we can see special end of line characters and some other flags that have been enabled.

This is what mine looks like..

root@chip:/dev/pts# stty -F 1
speed 0 baud; line = 0;
intr = <undef>; quit = <undef>; erase = <undef>; kill = <undef>; eof = <undef>;
start = <undef>; stop = <undef>; susp = <undef>; rprnt = <undef>;
werase = <undef>; lnext = <undef>; flush = <undef>; min = 0; time = 0;
ignbrk -brkint -icrnl ixoff -imaxbel
-opost -onlcr cr2 tab2
-isig -icanon -iexten -echo -echoe -echok -echoctl -echoke

 yeeg... not so good. We can see that apparently the speed is 0, we've got a LOT of undefined things, and for some reason only ixoff is enabled when we should be seeing ixon as well.

Hm. I'll have to poke around with that further to try and clean up how our pty is being exposed.

I've begun working on the actual program flow to send data through the serial ports seamlessly.

Here is the flowchart I've thought up for the initial design.

It's a big ugly, forgive me. But as you can see from the flowchart, I'm trying to keep the logic between the communicators the same. The only difference between the "client" and "server", is one starts at the "Receive data" stage, and the other starts at the "Read from serial and send packet" stage.

I did a quick implementation of this in a dirty state machine, and here are the issues I've had thus far that I need to look into fixing/improving:

• Currently my serial read function blocks until some data appears on the pty slave port. I need to find a way to make this process time out in order for the flow above to function correctly.
• The time it takes to send a heartbeat packet MUST be smaller than the time it takes for the Receive packet stage to time out. Otherwise one party things he's dropped a packet, but in reality the other party just didn't have any data to send.
• With this system we need to reserve a byte as a command area so we know if a packet is a heartbeat or a retry (duplicate data) so we know how to behave when we get those packets. Because serial sends data by bytes and not bits, this means we're left with 57 bytes in a packet, I only need 2 bits to send my command flags, so 6 bits are wasted.

Other than that, I think this a good basis for my design for v1 of chirppp, once I fix any bugs that rear their ugly head, of course.

If anyone is actually reading these, and likes to minimize program flows in their spare time (not judging), I'd love to hear any ideas on a flow that won't lose or accidentally duplicate packets, or if you spot any problem areas in the flow above, please let me know!

In my silence I've been working on getting pseudo-terminals (ptys) working with Rust.

For those that have never heard of a pseudo terminal before (I was one of them), this is essentially an emulated null modem connection for use in software.

SSH is one of the more popular program that makes use of this feature.

Essentially a pair of software-defined terminals are created, one master and one slave. The slave is the public facing portion of the pair, viewable in the linux filesystem, it can be read from and written to. The other end of the connection is the master pty, this one is not visible in the filesystem. It is a serial endpoint that behaves like a file that you can then interact with as you would a file in your program, reads, writes, etc.

To get this working in rust I had to use a crate that directly interfaces with libc, so through rust I was directly able to use the "openpty" C function binding, and it worked without a hitch.

This method did create some ugly platform-dependent code that I'd rather do without. I may visit how to rid myself of that later.

But for now I did an ugly little demo I screencapped and edited, so bear with me...

the left side is one device (the vocore), the right is the other (the chip), the top terminals are the output of the VERY simple version of chirppp I have working right now, the bottom is what a regular user would see with the chirppp service running on their machine.

I don't have a pretty serial terminal program that will run on both machines, so I had to communicate with the port using echos and cats, but you should get the idea...

Right now it only works as a simple back and forth with NO logic, so it does not yet operate as a true serial connection would. That's the next step. I have to mull over how I want to design the pseudo-protocol of handing packets back and forth in a way that won't chew up too much power or have too much latency.

I'll hopefully have some variations of how I want it to work in the next coming days.

I'm still a bit stuffed up but overall feeling much better from whatever summer sickness managed to take hold of me.

I decided to bite off a what seemed to be a small piece of work to do this evening, but here I am, typing this waaaay past my bedtime.

I'd like to start doing some testing of multi threading on this project. I think I might get some better numbers in terms of packet loss if I have a dedicated thread set up to ONLY listening for gpio interrupts and sending a message through a channel to the main thread saying it's okay to read from the serial port.

If I'm still seeing some issues with packet loss I may do some cpu profiling and see what I can make faster. I know that the library I'm using uses epoll under the covers to do edge detection on the gpio. In the end I might end up doing some research on how to get some quicker interrupt detection through sysfs...

I'm hoping that isn't the problem, but when I send 100 packets at the slowest speed from the faster device to the slower device, 3-8 go missing. When I send the same 100 packets from the slow device to the fast device they tend to all make it, so that gives me an uneasy feeling.

Anyhoo, moving on from speculation, tonight I spruced up the error handling and generation of a lot of the functions dependent on detecting these interrupts so I can have a better idea of what's going on, and more importantly, have a thread act intelligently and exit accordingly when an error is detected so I don't have to worry about stranding and children out there.

I wrestled a bit with the way rust returns Results and managed to create one or two new bugs for myself to iron out tomorrow or the next day, but we're on the move again!

Oh, and I ordered this from ebay, if all goes well by the time it arrives I'll have a good reliable codebase to do some range testing with the chip!

Here's hoping the distance numbers for these little radios will look a bit better than their speeds...

I've stayed home sick today. My throat seems to be tearing itself apart from the inside out... never fun.

I'm not on top of my mental game, so I wrote some quick code that I intend to throw away now that I've logged its results.

The first thing I decided to to was the look at the latency of a round trip packet. That is, sending it the receiver, and waiting for the receiver to send it back. This is the way the serial emulator is going to work, with this back and forth communication, so I figured that's how I would log the numbers.

Here's a graph of the latency and throughput for all the radio's speed modes

And purely for curiosity's sake, here's the same graph in a stacked format

To be frank these are the results I was expecting, but not the results I was hoping for. It seems after 10k, the throughput of the radio falls largely under the margin of error. Either that, or there's some sort of bottleneck on one of the devices... but these are what I'm going to have to use, so I'll just have to live with it.

Also it's worth noting that the speed if the connections do NOT compensate for the latency in any way. I wanted to keep them as-is, because if I plug in the devices to my computers and send some data, what you see should be what is recorded on these graphs.

So unfortunately it doesn't even look like we could break into the kB/s range... hell we didn't even break 200 B/s... but from the beginning I always knew this was going to be nowhere near fast. Hell I wouldn't even call a data link of this speed "slow" you could use some more exciting words like "glacial".

But I'm not discouraged, I still think that this is a really cool project and I'm certainly going to continue with it, I had considered all this before I even started knowing that these numbers would probably be what I was seeing.

I also logged some data on how the packet size effects the latency and throughput of the devices (no pretty graph, libreoffice crashed -.-) however the conclusion that was drawn was "it doesn't"

The latency of the connection from what I can tell BARELY depends on how much data the radio is actually sending. I'm talking about a difference of ~30ms (which is a drop in the bucket compared to pings of normal operation).

At 58bytes per packet we're seeing a speed of ~170B/s at a latency of ~340ms

and at 2 bytes per packet we're seeing a speed of ~6B/s at a latency of ~311ms

After 58 bytes what I noticed in a previous log proved to be the case, the radio splits the data up into packets of 58 bytes and we see no further performance gain.

Gathering this data did turn out to be a larger pain than I was expecting because I dropped a surprising amount of packets. Sure it was maybe 4 in 100, but to be honest I wasn't expecting any with the radios 2 feet away from each other.

I do not yet have any code to handle a lost packet, that will come later when I flesh out the serial emulator for these devices.

So what have we learned from all this information...

• setting the radio to a data rate of >10k is probably not worth the packet loss you'll see
• 1k, though it will achieve the longest range, has a painfully slow latency and speed on it.
• Some code will have to be implemented to intelligently and QUICKLY deal with lost packets
• The serial connection's speed when we're done won't be blowing anyone's socks off.

And what's next?

If I were doing this professionally the next thing I'd be doing would be grabbing one of the devices, strapping a battery onto it, and heading off outside to see what kind of effective range we get with this solution before deeming it a worthwhile project.

Good news is, I'm not doing this professionally (and I don't have a battery)

The C.H.I.P. device can power the radio from its 3.3v line at maximum power, and the chip itself can be powered off a regular microusb connection. My plan is to order a power pack used to recharge cell phones on the go, and take the whole assembly out with me and get some gps distance readings from the other device and see what kind of communication we can get.

In the meantime, I need to start thinking about the design of the serial emulator itself, and before I can do that there are some rather broad design decision that need to be made.

Should I use some sort of checksum (in addition to what LORA already has) to verify the data?

Should I implement any sort of retry functionality on a lost packet? or leave it up to the point-to-point protocol to handle those sorts of errors.

I'll be sitting back on the couch slamming water into my body hoping it can take care of these bacteria lurking in my throat, all the while thinking about this.

If anyone is reading this and has a good idea/previous experience about what might be a good idea to implement in a simple back and forth data transfer protocol I'd be glad to hear it!

So as you may know (if you've been keeping up), one of the devices I'm using in this project is a Vocore I got back in 2014.

This is a version 1 vocore, and the dock it comes with a microsd card slot for some external storage.

I figured I'd give that a go, since right now I've got a usb dongle stuck into it and it adds a lot of bulk that doesn't need to be there. (You can take a look here)

I got my hands on a nice cheap microsd card and I figured I'd get to work.

Well I started working, and I realized that the creator of the Vocore had quite a lot of trouble getting the sd card to work with SPI with OpenWrt. With the help of a community member, he was able to get it working with some adjusted config files and some patched driver code I found here... cool!

I did what the instructions said and uploaded my new openwrt image aaaannnndddddd

"kmodloader panic"

and we had a boot loop... oh dear

as it turns out, this (right here) is the method the community member used to get the microsd working.

I'm using bleeding edge chaos calmer, and according to this wiki page

"This mod does not work on the new trunk and Chaos Calmer 15.05 (since the revision r47045)."

Fantastic... So now I have a choice... I can pull that revision of the source down and use that as my baseline to get that sdcard hack to work... but before that, let's take a look at what is actually in that patch...

"kernel: backport some SPI layer improvements from 4.1 to 3.18, significantly improves flash speed on ramips"

Ramips happens to be the architecture I'm using... fantastic.

So for now I think I'm going to keep the usb dongle plugged in for some extra storage. I'd rather keep the flash access speed (the vocore has spi attached flash) than the ability to read and write to an sd card.

The speeds would've been rather awful keep in mind... the speeds over spi would've been around 100kB/s.

However here's the whole backwards progress of the entire process.

Like a moron I didn't back up my wireless config files... and that was a hell of a process.

I need to get the wireless on the device working again so I can ssh in to continue my work.

I guess that's life teaching me a lesson to back things up before diving in headfirst :P

Oh well... I need to get that reconfigured and the gpio and uart hooked back up to the vocore.

By the time I post another log I SHOULD see some forward progress graphing some of the speed and latency I'll see over different send speeds and packet sizes... lest I get any other brilliant ideas to try out.

EDIT:

Turns out it's not as hard the second time you do it. Both platforms and radios are back in business. Fingers crossed I'll get some time to work on this tomorrow!