Close
0%
0%

ESP8266 with True USB

Make it act as a USB disk and just copy files on it.

Similar projects worth following

The ESP8266 (and ESP32) is a great chip: powerful, versatile, cheap, easily obtainable. But it's a little bit fiddly when it comes to programming the development boards and transferring the files onto and from them. You have to install the drivers (good luck if you have a mac), install the flasher software, get the flash parameters just right, get the boot pin low (or make sure the auto-reset circuit works), etc. Individually, each of those steps is easy. Together, they make it hard to simply just start coding.

Wouldn't it be great if there was a development board that, instead of the dumb USB to serial converter, would sport a small and cheap USB-capable microcontroller, just like the Arduino UNO or Micro:bit, which would let you simply copy files over to a USB drive it would expose?

That idea is not mine, of course, I heard it for the first time from @tannewt. But I really want this to happen.

However, I know almost nothing about USB, and this is very much a research project. I expect extremely slow progress and possibly shelving of this project later on. I believe it is possible to do what I want (at least the filesystem and serial access), but I'm not sure I have enough experience to pull it off.

All help is greatly appreciated!

  • First PCB

    deʃhipu03/23/2018 at 21:31 3 comments

    I still didn't get the flash to work correctly, but I had some sleepless night, so I sketched up a PCB for the first prototype. This is the top where the ESP-12F module goes:

    And this is the bottom, where a Pro Micro goes:

    There is also an extra button on the opposite side than the reset, for selecting the USB mode. That's pretty much it, since the Pro Mini already hast he voltage regulator and all that, and the programming pin can be held down by software.

  • MassStorage Demo with LUFA

    deʃhipu03/07/2018 at 23:54 0 comments

    Today I experimented a little bit with the mass storage demo. I created the Dataflash.h and flash command header files based on the modified demo from https://github.com/JayconSystems/EPaper-Mass-Storage-Demo-LUFA and I got it to compile. Then I changed the pins and the flash id to fit my project. Flashed it and lo and behold — it enumerates! Unfortunately, I was unable to write anything to the flash — I suspect I need to provide the commands for this particular model of flash from its datasheet. More debugging to come.


    I also didn't try to use the USBtoSerial to actually flash the ESP8266 — I will do that later.

  • USBtoSerial with LUFA

    deʃhipu03/05/2018 at 01:55 0 comments

    I didn't have much time this week, since I was travelling (and got stuck in Dublin due to cancelled flights), but I did take a Pro Micro with me for experimenting, and after some poking around the LUFA project's directories, I found a ready to use USBtoSerial demo. It is in fact very similar to the Arduino code I was trying to get to run earlier, but this one comes with a specific version of the library that it works with.

    The project is in the Projects/USBtoSerial directory, and all you need to change to make it work on the Pro Micro is the makefile — the usual MCU, BOARD and F_CPU changes. Then make, objcopy to make a hex file, and avrdude to flash it. And it seems to be working!

    I didn't actually have a USB serial dongle to test it properly, but I shorted TX and RX pins to get to test the echo, and that seems fine. When I get back home I will try to program the ESP8266 with this, but I don't expect any problems.

    Next step is getting the mass storage demo to work with my own flash chip — since the Leonardo board definition misses the header files for flash, I will need to write those — hopefully I will be able to figure them out from the files for other boards. When that works, I will need to figure out how to make it look for the filesystem at an offset, and what the offset is for MicroPython (or even better, how to find it).

  • LUFA Demos

    deʃhipu02/25/2018 at 23:06 4 comments

    Today I tried to actually compile the arduino firmware for the atmega16u2, however, without much success. It seems to be written for an unspecified version of LUFA different than the current release. Rather than try to guess what version it should be, I decided to instead try the demos included with LUFA. That turned out to be remarkably simple.

    All you have to do is unpack the LUFA zip file, open a terminal in the directory of one of the demos, edit the Makefile to replace "at90usb1287" with "atmega32u4", "8000000" with "16000000", and "USBKEY" with "LEONARDO", edit the sources to remove all references to joystick and instead make it spew some text constantly, and then do:

    make all
    avr-objcopy -j .text -j .data -O ihex VirtualSerial.elf VirtualSerial.hex
    avrdude -c usbtiny -p atmega32u4 -U flash:w:VirtualSerial.hex:i

    (I have and USBTiny ISP programmer connected to pins 14, 15 and 16 of the Pro Micro.) 

    And it works — I get spam from the serial on the /dev/ttyACM0 device. Yay.

    Sadly, I can't see any demo that would be reading from the serial, but I hope I can steal that from the Arduino firmware code.

  • Arduino's ATmega16u2 Firmware

    deʃhipu02/24/2018 at 00:44 1 comment

    The first thing I want to make is a USB to serial converter, so why not look at something that does it already. The ATmega16u2 chip on Arduino UNO is programmed as just such a device. A quick search, and we have the source code for its firmware. As I expected, it uses LUFA to do the USB work, and hardware serial to do the UART work. Pretty straightforward stuff, two ring buffers are used to keep the data before it is sent on its way, one for each direction.

    What I'm disappointed about, though, is that the baud rate of the physical UART is hardwired as 9600 in that code. I was hoping for some AT code parsing and setting the speed. This is much more primitive.

    Why? Because to program ESP8266 you need to talk several speeds. At least I think you do. The default serial speed after boot is some weird value like 74880, and then, I think, esptool changes that to whatever you have configured. And that means that the operating system drivers need to know how to change speeds, and that is usually done with AT commands, since USB to serial converters usually emulate a modem device.

    There is a chance that it will work if I set 74880 hardwired, and tell esptool to use that speed for everything. It might just work, who knows. More testing required.

  • The Plan

    deʃhipu02/20/2018 at 11:51 0 comments

    The plan for now is to create a development board with the ESP12F module on it, and one of the USB-enabled microcontrollers. The board would have a special "files" button on it. If started without the button pressed, the microcontroller will act as a regular USB to serial adapter, pretty much like the ATmega8u2 o the UNO, letting you program and debug the ESP8266. However, if during the startup the button is pressed, the microcontroller will keep the ESP8266 in reset, connect to its SPI flash through the pins on the bottom of the module, and make it available as an USB disk. This is the minimal plan.

    Now, how to get there? I'm thinking about the following steps:

    • get the blinky demo to run on the microcontroller,
    • get the USB serial echo demo to run,
    • get the USB to serial working,
    • add AT commands for changing speed etc.
    • get the reset and programming logic working,
    • program the ESP8266 with it,
    • get SPI working,
    • communicate with the on-board flash chip,
    • come up with a way to find the beginning of the filesystem on the flash (may require manual configuration),
    • use a FAT library to verify we can read the filesystem correctly,
    • get the MSC demo to work,
    • modify the demo to use the flash chip,
    • make it use the correct offset,
    • run both the serial and the MSC depending on the button pressed.

  • Collaboration

    deʃhipu02/20/2018 at 11:30 0 comments

    By the way, if you find this project interesting, I would very much welcome collaboration. There is a lot of ground to cover, and a lot of experimenting to do. This can be easily spread to more people, each picking a different direction. And I also think that a lot of good projects could branch from this.

    Also, I have zero experience with programming anything USB-related, so any help with that is appreciated too!

  • Chip Candidates

    deʃhipu02/20/2018 at 11:27 3 comments

    I have been looking through the cheap microcontrollers with USB support, and I have narrowed my list to two, with maybe possible third.

    The first candidate is the venerable ATmega32u4. It has been around for a long time, there is a LUFA library for handling USB with some examples for the MSC device, and I have a whole bunch of cheap Pro Micro development boards I can experiment with. It also has a lot of GPIO pins, so it could also be useful for projects like #LAMEBOY for handling extra functions, like the buttons and backlight.

    The second candidate is the SAMD11. Much more modern, cheap and comes in an easy to solder package. It also requires fewer passives — I think it can even work without the crystal. There is also a MSC device example for in the ASF3 library. It is however more of an unknown for me, as I have never before programmed any. I got a #DiXi board to experiment with it.

    Finally, a distant third is the good old STM32F103 chip. It comes on the "blue pill" dev boards, and it also has some MSC examples. I have some limited experience programming it, too, so maybe it will turn out to be easier than the other two.

View all 8 project logs

Enjoy this project?

Share

Discussions

PakHacK wrote 10/06/2018 at 11:39 point

Interesting project!! What about using DAPlink and use it for the ESP8266 or ESP32?

Best regards

  Are you sure? yes | no

deʃhipu wrote 10/06/2018 at 11:54 point

I'm afraid DAPlink is only for ARM devices.

  Are you sure? yes | no

PakHacK wrote 10/07/2018 at 16:00 point

Ok I see. I have found some interesting projects as UF2 and this:

http://bbs.esp32.com/viewtopic.php?t=6534

Hope this help you!

  Are you sure? yes | no

deʃhipu wrote 10/07/2018 at 17:57 point

Thanks.

  Are you sure? yes | no

Jarrett wrote 03/08/2018 at 03:57 point

Looks like I might be a month too late for slipping in another architecture, but at the very least you might want to look at the Microchip XPRESS Loader. It's for PIC, but it's pretty awesome.

Open source, you plug it in, and it enumerates as a flashdrive and a COM port. Dropping in a hex file programs it to another PIC. USB to serial is also kept intact.

  Are you sure? yes | no

deʃhipu wrote 03/08/2018 at 10:09 point

Thanks. I would still need to adapt the code to work with the esp8266 and to acces the flash, and I already made some progress with the AVRs, so I think I will stick to that, at least until I hit some kind of a roadblock.

Of course I see no reason why multiple people couldn't try doing this for different chips, if you have the time and energy to try!

  Are you sure? yes | no

Michał Moskal wrote 03/05/2018 at 17:55 point

You might take a look at https://github.com/mmoskal/uf2-uno - MSC firmware for UNO. You would just need to swap the serial interface. However, honestly, probably better to go with SAMD21 - the smallest versions are cheaper than ATmega and more versatile. You could port https://github.com/microsoft/uf2-samd21 to do what you want I think. You could have one instance of it to flash the SAMD21 and another one, modified to flash the ESP.

  Are you sure? yes | no

deʃhipu wrote 03/08/2018 at 09:46 point

Thanks, I saw your work, and I think it's amazing. I was looking into using the samd chips (samd11 to be exact), but even to just see the MSD example you have to install Atmel Studio, and I don't have access to any Windows boxes. The AVRs are ancient, but at least the toolchain is well established and documented. This is a proof of concept anyways, if it works well, it can be ported/rewritten for any chip.

  Are you sure? yes | no

deʃhipu wrote 03/08/2018 at 10:14 point

By the way, maybe you would like to try your skills on a similar UF2 bootloader for the ESP8266? Not quite what I'm trying to do here, but would also be great.

  Are you sure? yes | no

Morning.Star wrote 02/26/2018 at 06:31 point

That sounds highly useful, I had a lot of trouble with the ESP. Enough to put me off the things a bit anyway...

Are you thinking of a WiFi link, so it acts like a remote thumb drive? That would be a handy thing to have, external storage I can carry like a thumb drive but works like a network drive for mobiles. Does such a thing exist? :-)

  Are you sure? yes | no

deʃhipu wrote 03/08/2018 at 10:11 point

Well, there is this: https://www.aliexpress.com/item/Wifi-Wirless-Card-Reader-for-Micro-SD-SDHC-TF-Flash-Wireless-Storage-Device-XL6G/32645146472.html — might work for what you are looking for. It even runs Linux, and can be rooted!

  Are you sure? yes | no

Morning.Star wrote 03/08/2018 at 14:44 point

That looks pretty neat, and close to what I was thinking. The USB port is handy, I hadnt considered that. I was more interested in the WiFi, like a network drive but portable and with removable storage based on flash cards, and battery operated so anything anywhere can read and write it in my pocket. The AP would be handy for the phone too...
Cheers, nice find! :-)

  Are you sure? yes | no

deʃhipu wrote 03/08/2018 at 19:22 point

Well, the USB plug on that card reader is just for power, so that you can use it with a regular power bank. You access the files via WiFi.

  Are you sure? yes | no

Clara Hobbs wrote 02/20/2018 at 15:11 point

Keep in mind that you don't need all the functionality provided by esptool.py.  You only need the functionality required for the board the microcontroller is on, so you can drop at least ESP32 support and SPI flash detection.  Maybe more stuff as well (e.g. do you really need to be able to flash ELF files directly?).

  Are you sure? yes | no

ajlitt wrote 03/08/2018 at 02:16 point

Detection and geometry mapping is super easy if you lift the device table from spi-nor.c in the Linux kernel.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates