ESP8266 with True USB

I still didn't get the flash to work correctly, but I had some sleepless night, so I sketched up a PCB for the first prototype. This is the top where the ESP-12F module goes:

And this is the bottom, where a Pro Micro goes:

There is also an extra button on the opposite side than the reset, for selecting the USB mode. That's pretty much it, since the Pro Mini already hast he voltage regulator and all that, and the programming pin can be held down by software.

Today I experimented a little bit with the mass storage demo. I created the Dataflash.h and flash command header files based on the modified demo from https://github.com/JayconSystems/EPaper-Mass-Storage-Demo-LUFA and I got it to compile. Then I changed the pins and the flash id to fit my project. Flashed it and lo and behold — it enumerates! Unfortunately, I was unable to write anything to the flash — I suspect I need to provide the commands for this particular model of flash from its datasheet. More debugging to come.

I also didn't try to use the USBtoSerial to actually flash the ESP8266 — I will do that later.

I didn't have much time this week, since I was travelling (and got stuck in Dublin due to cancelled flights), but I did take a Pro Micro with me for experimenting, and after some poking around the LUFA project's directories, I found a ready to use USBtoSerial demo. It is in fact very similar to the Arduino code I was trying to get to run earlier, but this one comes with a specific version of the library that it works with.

The project is in the Projects/USBtoSerial directory, and all you need to change to make it work on the Pro Micro is the makefile — the usual MCU, BOARD and F_CPU changes. Then make, objcopy to make a hex file, and avrdude to flash it. And it seems to be working!

I didn't actually have a USB serial dongle to test it properly, but I shorted TX and RX pins to get to test the echo, and that seems fine. When I get back home I will try to program the ESP8266 with this, but I don't expect any problems.

Next step is getting the mass storage demo to work with my own flash chip — since the Leonardo board definition misses the header files for flash, I will need to write those — hopefully I will be able to figure them out from the files for other boards. When that works, I will need to figure out how to make it look for the filesystem at an offset, and what the offset is for MicroPython (or even better, how to find it).

Today I tried to actually compile the arduino firmware for the atmega16u2, however, without much success. It seems to be written for an unspecified version of LUFA different than the current release. Rather than try to guess what version it should be, I decided to instead try the demos included with LUFA. That turned out to be remarkably simple.

All you have to do is unpack the LUFA zip file, open a terminal in the directory of one of the demos, edit the Makefile to replace "at90usb1287" with "atmega32u4", "8000000" with "16000000", and "USBKEY" with "LEONARDO", edit the sources to remove all references to joystick and instead make it spew some text constantly, and then do:

make all
avr-objcopy -j .text -j .data -O ihex VirtualSerial.elf VirtualSerial.hex
avrdude -c usbtiny -p atmega32u4 -U flash:w:VirtualSerial.hex:i

(I have and USBTiny ISP programmer connected to pins 14, 15 and 16 of the Pro Micro.) 

And it works — I get spam from the serial on the /dev/ttyACM0 device. Yay.

Sadly, I can't see any demo that would be reading from the serial, but I hope I can steal that from the Arduino firmware code.

The first thing I want to make is a USB to serial converter, so why not look at something that does it already. The ATmega16u2 chip on Arduino UNO is programmed as just such a device. A quick search, and we have the source code for its firmware. As I expected, it uses LUFA to do the USB work, and hardware serial to do the UART work. Pretty straightforward stuff, two ring buffers are used to keep the data before it is sent on its way, one for each direction.

What I'm disappointed about, though, is that the baud rate of the physical UART is hardwired as 9600 in that code. I was hoping for some AT code parsing and setting the speed. This is much more primitive.

Why? Because to program ESP8266 you need to talk several speeds. At least I think you do. The default serial speed after boot is some weird value like 74880, and then, I think, esptool changes that to whatever you have configured. And that means that the operating system drivers need to know how to change speeds, and that is usually done with AT commands, since USB to serial converters usually emulate a modem device.

There is a chance that it will work if I set 74880 hardwired, and tell esptool to use that speed for everything. It might just work, who knows. More testing required.

The plan for now is to create a development board with the ESP12F module on it, and one of the USB-enabled microcontrollers. The board would have a special "files" button on it. If started without the button pressed, the microcontroller will act as a regular USB to serial adapter, pretty much like the ATmega8u2 o the UNO, letting you program and debug the ESP8266. However, if during the startup the button is pressed, the microcontroller will keep the ESP8266 in reset, connect to its SPI flash through the pins on the bottom of the module, and make it available as an USB disk. This is the minimal plan.

Now, how to get there? I'm thinking about the following steps:

• get the blinky demo to run on the microcontroller,
• get the USB serial echo demo to run,
• get the USB to serial working,
• add AT commands for changing speed etc.
• get the reset and programming logic working,
• program the ESP8266 with it,
• get SPI working,
• communicate with the on-board flash chip,
• come up with a way to find the beginning of the filesystem on the flash (may require manual configuration),
• use a FAT library to verify we can read the filesystem correctly,
• get the MSC demo to work,
• modify the demo to use the flash chip,
• make it use the correct offset,
• run both the serial and the MSC depending on the button pressed.

By the way, if you find this project interesting, I would very much welcome collaboration. There is a lot of ground to cover, and a lot of experimenting to do. This can be easily spread to more people, each picking a different direction. And I also think that a lot of good projects could branch from this.

Also, I have zero experience with programming anything USB-related, so any help with that is appreciated too!

I have been looking through the cheap microcontrollers with USB support, and I have narrowed my list to two, with maybe possible third.

The first candidate is the venerable ATmega32u4. It has been around for a long time, there is a LUFA library for handling USB with some examples for the MSC device, and I have a whole bunch of cheap Pro Micro development boards I can experiment with. It also has a lot of GPIO pins, so it could also be useful for projects like #LAMEBOY for handling extra functions, like the buttons and backlight.

The second candidate is the SAMD11. Much more modern, cheap and comes in an easy to solder package. It also requires fewer passives — I think it can even work without the crystal. There is also a MSC device example for in the ASF3 library. It is however more of an unknown for me, as I have never before programmed any. I got a #DiXi board to experiment with it.

Finally, a distant third is the good old STM32F103 chip. It comes on the "blue pill" dev boards, and it also has some MSC examples. I have some limited experience programming it, too, so maybe it will turn out to be easier than the other two.