wESP32: Wired ESP32 with Ethernet and PoE

Revision 8 was mostly started as a bug fix revision to fix an incorrect pin strap on the RTL8201FI Ethernet PHY.  This didn't used to matter, but a new revision in the chip seemed to have "fixed" the fact that it didn't used to matter, so now it mattered, and needed to be fixed. :)

I was very fortunate that one of my customers who had licensed my design for their own use ran into this problem first and contacted me about it.  Since they run custom firmware, they were able to fix the issue in software.  I on the other hand want the wESP32 to work with various firmwares that customers may want to use, so a hardware fix was in order.

Since just doing a bug fix revision would have been boring, I decided to add a cool new feature that should not affect backward compatibility in any way.  Revision 8 boards now come with a Sparkfun Qwiic connector! :)

This should make it much simpler to use the wESP32 for simple PoE sensor nodes by just connecting Qwiic modules, no soldering required!

You can buy the new revision 8 on Tindie right now, stock on Amazon and Mouser will transition to revision 8 in the near future.

I should also give a shout out to Christopher Greenlee who is running a campaign on Crowd Supply right now for his Hornet Nest Alarm Panel, which is based on the wESP32 for its Ethernet / PoE / WiFi connected brains.

If you are interested at all in home automation and DIY security systems, go check it out!  It integrates with Home Assistant for a completely billionaire-free home security system! :)

A little late posting here, but just a heads-up that revision 7 has arrived and is making its way to various distribution channels!

New 4-layer PCB, much more decoupling capacitance on-board, RTL8201FI Ethernet Phy and now 16 MB of flash!

Tindie: https://www.tindie.com/products/silicognition/wesp32/

Mouser: https://www.mouser.com/ProductDetail/Silicognition/CS-WESP-04?qs=sGAEpiMZZMu3sxpa5v1qrriU3DaWTQdbHa4MxnML6FE%3D

Amazon: https://www.amazon.com/Silicognition-wESP32-Isolated-Ethernet-802-3at/dp/B084RV91NQ/

Note that the lead time shown on Mouser is bogus, they had 100 units shipped to them already.  Should ship much sooner than the lead time indicates.

A quick heads-up that I finally found the time to update the wESP32.com website with new information related to the changes in revision 7 such as 16 MB of flash and RTL8201 Ethernet PHY.  The product brief has also been updated, make sure to check the code examples to see the changes you may need to make to your software for this revision!

A quick progress report.  MicroPython 1.17 dropped on September 2 2021 which meant that I finally had an officially released binary with support for the new wESP32 with 16 MB flash.

To remove the biggest bottleneck in production called "me" from the equation as much as possible, I had my CM Kingtop build a fixture so they could program and test them, instead of me still having to do that with every board.  They built a nice fixture:

Much better looking than my fixture hacked together from two perf boards. ;)  That worked, but hurt your hand when running too many boards.  Big improvement.

Kingtop finished populating the through-hole parts and programmed and tested all boards:

They reported two boards that failed on initial test, but they were able to correct the problem.  So an initial failure rate of 0.2%, and 0% after rework, that speaks very well to their build quality and processes.  Believe me, I've seen much much worse!

They are ready to ship them now and it's still looking good for getting stock mid-September.

Now I only have to harass Mouser some more to try and get them to send me a PO for the new part numbers generated for this revision, after begging for months already.  But why do something ahead of time when you can do it when it's too late, right? 🙄

After waiting for four months since I ordered them, Mouser finally shipped the SI3404A PoE chip to my CM, so they have started production.  I received this picture of the new PCBs:

Looking good.  I think there shouldn't be confusion about which PHY this uses, since I have it in pretty big letters on the bottom.  Last night, I received this picture with surface mount components mounted:

Nice progress!  You can see the new RTL8201FI Ethernet PHYs and the ESP32-WROOM-32E module with 16 MB of flash.

Meanwhile, I have also managed to get a custom board config merged into upstream MicroPython so you can actually make use of all that flash memory!  Up till now, I used the generic ESP32 images but they are made to use only 4 MB of flash.  The flash memory will be partitioned with two nice 2.4 MB OTA partitions so we should be safe for a while as MicroPython keeps growing, and a nice 11 MB of flash will be available for the MicroPython file system!  I'm hoping that version 1.17 will be released soon so I can give an official release to my CM to program in to the boards.  As usual, the boards will also come with a custom boot.py file that will set up the LAN for you.  So if you use the built-in MicroPython, the change to a different PHY chip will be completely transparent.

I also ran some more performance testing and compared performance of the old LAN7820AI with the new RTL8201FI using the latest IDF's ethernet iperf example.  It looks like the LAN8720AI is marginally faster with UDP, while the RTL8201FI is marginally faster with TCP.  Overall, performance seems to be just fine.  The following numbers are approximate:

It had come to my attention that some customers had problems that some of their boards would stop working.  When they returned them and I did a postmortem, it was always the LAN8720AI Ethernet PHY that had died.

Eventually I was able to recreate a setup that would reliably kill the LAN8720AI.  If I had a wESP32 connected to a particular PoE switch, with the USB programmer connected to a particular laptop, the chip would die when plugging in the Ethernet jack.  So I set out to try and determine root cause and find a solution.

Doing some measurements on what exactly happened when plugging in the Ethernet using a battery powered oscilloscope, I could see some spikes occur on the 3.3V supply and ground when the cable was plugged in.  My theory was that maybe the charges on the various EMI caps connecting primary to secondary in the PoE switch power supply, the laptop power supply and the wESP32 would redistribute on connection and cause ground bounce that would for a very short time expose the LAN8720AI to out of spec voltages.

Now doing these kinds of measurements, you're never really sure about what you're seeing, what's real and what's just a measurement artifact.  So I wasn't really sure, but it was something to go after.  I would try to improve supply filtering, add TVSes for ESD protection on the supply and signal pairs, and add filtering in the ground connection between the EMI cap and the circuit ground.

I first changed supply filter caps on an existing board.  I tried larger bulk caps, and smaller close-to-the-pin caps for better response to high frequency spikes.  Didn't help.  I ordered TVSes, and added one to the 3.3V supply and filtered analog PHY supply.  No difference.

I decided to create a new, 4-layer board layout with footprint for ESD protection on the Ethernet differential pairs, the supply, and a differential filter that would including filtering ground spikes that might be coming through the PoE supply's EMI cap.  I went to 4-layer with a 3.3V power plane adjacent to GND through 0.1 mm prepreg to provide improved HF power supply filtering.

Here's the prototype:

Here's the ESD protection chip I added to the data pairs, the little chip above the wasp:

Aaaaand it didn't help squat. 😠  The PHY died just the same as before.

Now what?  Stock was running low, and I really would rather not make another 1000 potentially troublesome boards that might die in some setups.  My goal is for the wESP32 to be powerful and reliable, able to take whatever you can throw at it.  Dying when connected to certain equipment just won't do.  I've only had a few customers complain about it, usually in some situation where there's external power, so it doesn't seem to be a widespread problem.  But still, I don't find it acceptable.

I did some investigating, and found that in the years since the wESP32 was first released, the ESP-IDF has come to support more PHY chips.  It looks like IDF 4.x added more options, and the ESP32 Arduino core also already supports them.  In the mean time, MicroPython has dropped the IDF 3.x based images that supported Ethernet, but support may be coming in the IDF 4.x based images.  So, this opened possibilities.  IDF, Arduino and MicroPython support are the most important for me, other software is likely to follow as it gets migrated from IDF 3.x to IDF 4.x.

I had tried to avoid changing PHY chips because I don't like forcing customers to have to change their software.  It's only a change in the PHY definition, so it's minimal effort, but it's still a change that needs to be made.  I had considered it a last resort, but after all this effort and killing tons of chips while testing, I had to finally admit defeat in my efforts to stop the LAN8720AI from dying.  I had really tried hard, tried everything I could think of to protect the chip, to no avail.

So I set out to create a prototype with the RTL8201FI Ethernet PHY instead, to see if that one would keep working or if it would die as well.  Because at this point, I didn't really know if another PHY would actually solve the problem or suffer the same fate.  I redid the 4-layer layout with all the other improvements to give it the best chance of surviving, and created a prototype:

Slowly going through tests, working up to the worst case scenario that would always kill the LAN8720AI, this prototype with the RTL8201FI just kept working!  Hurray! 😁

Since then I have done a lot more testing.  I have tried it with various networking equipment, externally and/or PoE powered, I have replicated external power scenarios that caused problems for one customer, and nothing I have tried has killed this chip.  It's working beautifully and reliably, and is just as fast.  I have removed the extra ESD protection, done a lot of testing that way, and it keeps working just fine.

Which is how it should be, of course.  Seriously, this isn't rocket science.  It's a fully isolated system.  I know how to do power supply design, I follow best practices for filtering and decoupling.  I even went to 4-layer with a layer stackup optimized for decoupling HF.  And nothing I did would save that lousy LAN8720AI.  It's just a troublesome chip.

I remember how it had been troublesome in another way.  Remember all the trouble I went through to get it to reset correctly?  I had to add a cheap micro just to make that work reliably.  The RTL8201FI spec doesn't mention any nonsense about needing the oscillator to run before releasing the PHY reset like the other chip did.  It's a simple "wait at least 80 us after power is applied before releasing reset, then wait 150 ms before accessing the registers".  So, while I had kept most of the board the same for the RTL8201FI prototype, including the PMS150C-559 to do the reset, I decided to try if I could remove the PMS150C-559 and just tie the PHY reset and oscillator enable to the ESP32's EN signal:

And what do you know: it works just fine.  So easy.

So, bye bye LAN8720AI, and good riddance.  You were difficult and overly sensitive.  Hello and welcome RTL8201FI, you seem to be a much better part, not such a snowflake, and ready to do your job without any fuss.

"Silicognition" PMS150C-559

As mentioned in the last update, I ordered factory programmed Padauk PMS150C-U06 chips from Sino-Mos to provide a reliable reset for the PHY chip on every board.

I expected everything to be OK with them, but you never know 100% for sure if everything was understood correctly when dealing with China, until you actually have something in-hand. For one thing, I was relieved that they had indeed re-reeled them. It would have been a big bummer if I would have received a bag of loose SOT-23’s. :)

Then I had to make sure they were indeed programmed. So I reworked another 12 boards from the first batch that didn’t initialize the PHY correctly with chips from one of these reels, and after rework they all work flawlessly!

Getting these chips has actually been a smoother experience than I had expected. It’s kind of cool to now have a low-cost custom chip that does exactly what I want it to do, with its own partnumber and everything. PMS150C-559 is now forever an “ESP32 Ethernet PHY reset manager”. Maybe I should write a datasheet for it. :)

Also, since I have 9,000 of them, I can definitely spare some if anyone is making their own ESP32 board with Ethernet and wants a canned solution to deal with the LAN8720A PHY reset. There’s an updated schematic that shows how the chip is used on the new wESP32 revision. So get in touch if you want to buy some.

Rev 3 PCBs

I took a little risk and ordered new PCB panels that use the PMS150C-559 before I had verified the factory programmed chips. But it had all been taking much longer than expected already and some backers have been wondering when they’d receive their reward, so I tried to recoup some time that way.

Having everything in-house I made a kit to send to a CM I was going to try in Mexico.  That's when things started to go really sideways.

The scourge that is customs

The first customs issue I ran into: how to do the paperwork correctly. This was going into Mexico temporarily just to have it assembled and then it was coming right back to the US. I sought advice from the CM and a local export company, and yes, I received information on some generalities, but no one really gave me any concrete answers on what codes to use, exemptions to claim or checkboxes to tick. So after wasting almost a week waiting for guidance I gave up and shipped the box with the documentation as good as I could figure it. I marked the shipment with the AES 30.37(q) to indicate a temporary export that would be returned within a year. Whether that is correct or not I don’t know. Nobody had told me what to do and it seemed the right thing to do.

DHL got the box to Mexico after 4 days or so. Then the trouble started. First they claimed they couldn’t contact the CM at the telephone number I had been given and had put on the forms. They wouldn’t tell me why they needed to contact them. DHL was giving me options like “we can send your package back, or send it somewhere else for an extra charge, or we can destroy it for you (!)”. This nonsense took a week or so, a week of anxious freaking out for me. Then after wasting what they must have deemed sufficient time, they somehow did manage to get in touch with them and supposedly needed a tax ID from me. I run a passthrough LLC and certainly wasn’t going to give them my SSN! That sort of blew over after some work done by the CM and then they found that the contents of the box didn’t match the PI I had included. This was partially my fault: I had based the PI on the BOM, but even though neither me nor the CM cares about the exact part number of a reel of resistors, of course customs paper pushers do. I also had included some full and partial reels in the kit and amounts on the PI didn’t match. I really didn’t think anyone cared just how many parts were left on a partial reel, but of course they did.

What a joke. None of it should have mattered of course. The parts are temporarily exported and will be sent back to the US again. But, that’s not how they see it apparently. They just wanted to drive up the value to get the maximum amount to slap import tax on. After wasting more than 3 weeks I finally received word that the CM had received the parts. And also that customs had slapped $1300 import tax on my shipment!

I almost fell off my chair. This is highway robbery, pure and simple. Customs is nothing but piracy, hijacking of another person’s property for ransom, a scourge brought upon those trying to accomplish something constructive by leeches who don’t accomplish anything other than making money by ripping people off and making them miserable. It’s a purely artificial complication designed to hamper progress and cause friction to those who want to get things done.

Think about it. What have they accomplished? They have wasted 3 weeks, causing everyone who is waiting for product to have to wait 3 weeks longer. Nothing useful was accomplished in that time. They’ve managed to get $1300 from me. Wooptidoo, have fun with it guys because it’s the only thing you’ll get from me. Because suddenly, there’s no point to building my product in your country and dealing with all this hassle anymore. You’ve just more than doubled the cost of building my product and it’s now a similar price to building locally. So why would I bother next time and go through this agony again?

The CM in Mexico is actually the biggest victim in this. They have not done anything wrong but the pirates at the border think they should take as much money just for letting my parts through as the CM is going to make actually doing the useful work of building boards for me! In doing so, they’ve made the CM uncompetitive and there’s nothing they can do about it. Their government has conspired against their business for its own benefit. It’s so wrong.

OK, enough of the rant. Where does that leave the project?

I thought I had added plenty of padding to the schedule the last time the projected ship date was adjusted. I had of course never expected that customs would eat up this much of it right off the bat. Now it depends on how quickly the CM can do the build and then how fast they can get the PCB assemblies back to me. If we don’t have yet another customs hijacking on the way back into the US, late May may still be feasible.

Since there’s a lot of uncertainty associated with when I will get the 500 boards back from Mexico, I’ve also scheduled time at Colorado Tech Shop to have the other 500 boards built there and they’re going to try to cram me into their busy schedule around May 15. So at least now we have parallel production paths and some redundancy on getting boards to ship from one or the other before the end of May.

This is going to be the last time I’ve collected all parts here for building the wESP32. To be able to meet future production targets and reduce logistics headaches I will need to outsource much more of the production process. I will likely have the next batch made at PCBWay and let them take care of PCBs and assembly. I’m currently evaluating them by having them produce the next round of LiFePO4wered/Pi+ boards, and up till now it has been a much smoother process. I also am evaluating KingTop Technology by having them do a build of my LiFePO4wered/Solar1 product.

Figuring out supply chains and good partners to work with as production scales up is a painful process with ups and downs. Sometimes you win and sometimes you lose. Eventually you find good partnerships that work to be able to meet demand and make sure customers are satisfied and get the products they need. I can only thank them for sticking with me through the process!

Finally, a long overdue update! The silence doesn’t mean we’ve been taking it easy, on the contrary: we’ve been hard at work to make the wESP32 even better!

A yield problem

After the initial build of 245 units, we were able to ship 205 units that were good right away. There were various issues with the remaining 40, but the vast majority of them had the same problem: the Ethernet PHY would not get initialized properly.

The LAN8720A Ethernet PHY is kind of a difficult part. It has several pins that have double use: during reset their logic level indicates various configuration settings, while after reset they are used in the RMII interface to communicate with the ESP32. An extra difficulty is that the chip expects the 50 MHz clock to be present during reset, so the chip can clock in these configuration settings.

Unfortunately, the ESP32 by default uses the IO0 pin for the Ethernet MAC clock. (More options have become available for dealing with the Ethernet MAC and PHY clocks, but because IO0 was the first option available in the SDK, it enjoys the broadest firmware support so that’s why I keep using it. Plus to keep as many other ESP32 pins as possible available to the user!) The ESP32 IO0 pin is kind of a difficult pin as well, since it is used on reset to determine whether the ESP32 should go into programming mode or run the current firmware. So we have a perfect storm of an IO0 pin that needs to be high at boot to run normally, but low at boot to go into programming mode, and needs to have the 50 MHz clock on it after boot to allow Ethernet communication, and that before the PHY comes out of reset.

When I designed the wESP32, I just looked at how Olimex dealt with the IO0, ESP32 reset and PHY reset on their ESP32-GATEWAY and copied that for the wESP32, figuring they must already have found a good solution since this was a product in production. On my first hand-built prototypes, it all worked well, so I went ahead with it in production. Then I found that more than 10% of the boards had an issue. Interestingly, in their latest revisions, Olimex has changed their product to use IO17 as PHY clock output to bypass the whole problem! This might be a smart thing to do, but I am loath to make backwards incompatible changes and take a precious GPIO pin away from the user. So I started looking at other ways to deal with this.

Looking for a solution

I needed a solution that would act as a reset manager for the 50 MHz oscillator and Ethernet PHY and would be triggered by the EN signal to the ESP32. When EN is low on power-up and when triggered by a programmer, the 50 MHz oscillator needs to be off, so the programmer can drive the IO0 pin low to enter programming mode, or the IO0 can stay pulled high and the system boots. The PHY needs to be kept in reset as well, because its reset can only be released after the 50 MHz clock has started up. After the ESP32 has had the opportunity to determine the boot state, the 50 MHz clock needs to be turned on, and only after the clock signal is present can the PHY reset be released.

I had assumed there would be plenty of low cost, SOT packaged voltage detectors and reset managers that could replace my existing solution in the space I had available, but to my great surprise, nothing I could find in the market really seemed to fit the bill. There are plenty of supervisors that monitor multiple voltages and generate a reset, but I couldn’t find any low cost options that monitor one input and generate two reset signals with separate delays.

So I started thinking outside the box: did I really need to add a little microcontroller to do this? There are several microcontrollers that are cheaper than higher end supervisor chips, and it would give me full flexibility to generate the reset signals exactly how I wanted them. On the other hand, I didn’t really want to deal with programming this micro on every board. I would need to add programming pads and a manufacturing step to perform programming. Ideally I would be able to order factory programmed chips and avoid all that. To me, this would just be a simple fixed function chip that does what I need it to do.

The PMS150C solution

In looking for really low cost micros, I came across some EEVBlog videos about a three cent microcontroller. Extremely low cost OTP chips, with factory programming available: this seemed perfect for this! So I ordered the emulator, programmer and some Padauk PMS150C chips in SOT-23-6 package and once I received them I got to work.

I agree with Dave Jones that the tools for these chips are really quite good! In a weird, quirky sort of way, with a C-like language that has additional shortcuts and directives, and lots of Chinglish instructions. But with all the GUI settings screens and examples provided, I had no issue getting my simple program done with a minimal amount of fuss. I didn’t even have the emulator yet at the time so I just burned a chip, hacked it on to one of the errant wESP32 boards and gave it a try.

And it works beautifully, pretty much from the first try! The only thing I had to adjust was adding a pull-down resistor to the enable pin of the 50 MHz oscillator, to make sure it stays off during the brief period after power-up when the PMS150C is still in reset and not driving the oscillator’s enable pin yet. Which is not a big deal for the PCB redesign, because I have an unused 10K resistor available in resistor array R6 right next to the chip!

Below is a trace of the EN signal to the ESP32 releasing in yellow, and the enable pin of the 50 MHz oscillator being driven high from the PMS150C after a delay of about 200 ms:

And here is the reset to the PHY chip in blue, being release about 100 ms after the 50 MHz clock in yellow is started:

Of course, I had to gain a little more confidence before I went ahead with this, so I hacked eight more boards this way to verify whether everything worked:

And they’re all working great!

So then I needed to actually be able to get factory programmed parts. This would only be a practical solution if I could get reels of these chip factory programmed and ready to use by the CM, just like any other fixed-function part. That’s where I hit a snag. According to Dave Jones, LCSC had told him they could provide this service. However, when I got in touch with them, they said they don’t do it. I tried to contact several of Padauk’s listed authorized resellers, but never got any response. In desperation I contacted Padauk directly and they responded and got me a contact at Sino-Mos, who I’m working with now to get these custom programmed parts manufactured. The chips are a little more than three cents each when factory programmed, but nothing to complain about. I figure that with shipping included, they will still be less than eight cents each. I do need to buy 9,000 of them at once though, so I hope the wESP32 will keep selling well after this. :)

Once everything is straightened out and I have an order confirmation, I will go ahead and order 1000 revised PCBs and hopefully get the next batch of wESP32s built soon. In the mean time, thank you for your patience as the shipping date has been slipping. But I think we’re out of the woods now!

You may have noticed a warning at the bottom of the MongooseOS demo in the demo code repo, about accidentally blowing up the Ethernet phy while playing with MongooseOS demo code.

I didn’t find any time to pursue this further, but I have to give the people at MongooseOS credit for getting to the bottom of this! I received this message from them:

We did some research, and found the problem with ethernet. It happened to be quite stupid: the default config for the built-in LED used pin 22, which conflicts with the ethernet TX.
So, this command should set the board correctly:
mos flash esp32
mos config-set eth.enable=true eth.mdc_gpio=16 eth.mdio_gpio=17 board.led1.pin=-1 board.btn1.pin=-1

This highlights an issue that can happen with other firmwares as well: misconfigured pins doing damage to the phy. By default, the wESP32 ships with MicroPython installed and it is configured correctly out-of-the-box for the Ethernet phy as it is configured on the wESP32. If you load different firmware though, it is your responsibility to ensure the ESP32 pins are configured correctly and will for instance not configure an ESP32 pin as an output that connects to a phy output, and thus cause a short. Please refer to the schematic to make sure. The wESP32 website section on Software provides more information on wESP32 support in different environments.

If you have ordered one or more wESP32's, I recommend you try it with the default MicroPython firmware when you first receive it, to confirm the hardware is sound and it receives an IP address (check with lan.ifconfig()) from DHCP over Ethernet, which only happens if the phy is working. All units are tested to do this before they are shipped, but this way you can prove to yourself the hardware is OK and nothing bad happened during shipping. After that you are free to use whatever firmware you want of course, but I will not replace defective units once they have been flashed with different firmware, as I cannot know what happened to them.  Thanks for understanding!

On Monday 1/7, we finally received completed panels from Colorado Tech Shop!

So we spent most of this week doing final assembly (installation of two through-hole electrolytic capacitors and the Ethernet jacks) and testing for all 245 boards in the first batch. I am quite pleased with the result, I hope you backers will be happy as well!

Some boards were used for internal testing, some had to be sent back to Colorado Tech Shop for rework after finding issues during test, but today we were able to ship the first 205 units to Crowd Supply for delivery to backers next week.

One of the boards was used for an experiment. I wanted to see if the board worked with the AI-Thinker ESP32-S module instead of the ESP32-WROOM-32:

I’m happy to report it seems to work just the same, so this module could be used to provide a future U.FL antenna connector option!