This clock has been operating just fine for the last 11 months until about a week ago. We had a power outage due to an electrical storm. When the power shut down, it oscillated on and off rapidly for about 30 seconds. Apparently, this fried the buck regulator chips on the power supply board and sent +12V out on the 3.3V and 5V rails. The stuff running on the 5V rail had LDO (linear) regulators between the 5V supply and the loads, and they survived. The display driver boards were running directly off the 3.3V rail and when it went to 12V, all 9 of the display driver boards were killed.

Rather than repair the display driver boards, I have substituted different design driver boards from the work that I did on the display drivers last year. This was less expensive and fairly simple to do. The mechanical mounting plate for the displays needed to be re-designed and fabricated. The replacement display boards use the MAX7221 driver chips, so they have a SPI interface rather than the I2C interface of the original boards. The cable change  for the SPI interface was simple to do. The display driver software for the MAX7221 chips is substantially different below the top level than the I2C software, but the top level function calls for the driver library are almost identical. I swapped in my replacement library and only had to do minor edits in a few spots. The MAX7221 drivers run on +5V, so the whole system is operating on +5V now.

The most difficult part was to get the physical layer SPI drivers to work. My driver library calls the Atmel/Microchimp ASF low level drivers to deal with the hardware interface. The ASF libraries have been a big disappointment: the documentation stinks and the code is strangely architected and buggy as hell. The serial driver was so bad that it was easier to code my own than use the ASF library. The SPI functions are close to that point. 

Another project was running on the same design filter/regulator board as this clock was, and it also failed in a similar manner in this storm. Since both systems failed, there is a regulator board design problem. Now that the whole clock system is running on +5V, I have another filter/regulator board design using different regulator chips that may be suitable. The switching regulator on the substitute regulator board is an older design which runs at much lower frequency than the one being replaced. Right now, there are significant noise spikes on the +5V power rail, and when using the 12V wall wart power supply that the system was running on, the GPS is having problems getting a fix. When running on a good quality bench power supply, it is OK. The noise spikes may be from the older regulator, or the different display driver chips. This is still to be determined.

I think that I am not done with the power supply issue. In addition to fixing the noise problem, I need to look at adding an overvoltage crowbar circuit or a hot swap power controller between the 5V regulator and the rest of the system. This is to protect the system from similar failures in the future. Shorting the switching transistor is one of the most common failure modes on a buck regulator system. The rest of the system needs to survive this.

Hot Swap power controllers make excellent protective devices. Originally designed to go on PCBs that get inserted into a powered up system like a server rack and protect the system and the PCB from transient events related to plugging in the PCB. They have undervoltage and overvoltage sensing to disconnect the PCB power from the input power supply. They have overcurrent sense to disconnect the output from the system in the event of a problem on the PCB. Lastly, they have a linear "soft start" feature to control the current while the capacitors on the PCB charge up to the system voltage.

Once I have the power supply problems sorted out, I will edit this log and post the results.

9/10/2021 Update

Here is the schematic for the power supply board that has been causing problems. The capacitors and inductors in the upper left quadrant of the schematic are all part of an experimental input filter. The connector P1 is the DC input to the system. It goes directly to a barrel jack on the case of the clock. The purpose is to prevent noise from the switching regulators from getting out onto the external wiring and radiating from there. To a lesser extent, it is also to prevent noise from entering the regulators from the outside world, but the environment that I have been running these boards in does not have a lot of opportunity to pick up noise outside the system.

A couple of things that I have discovered while looking through this system:

1) The inductors in the input filter are capable of generating a pretty stout transient when the external power supply is plugged into the box.

2) The saturation current on the regulator inductors (L1 and L5) needs to be above the max current that the regulator can switch.

The scope shot above shows the voltages at the input of the board, the output of the filters and the output of the 5V regulator when power is applied to the board via P1. The input voltage is 12V and there is a modest spike when the energized 12V supply is connected to the board. Looking at the filter output, that modes spike goes up to 25V or more. The data sheet absolute maximum input voltage on the ADP2302 regulator chips is 24V. It is not difficult to imagine that transient on the filter output exceeding that value and killing the regulator chips.  The output of the 5V regulator does not start coming up until after the transients have settled down.

There are 2 possible fixes for this problem. The simplest is to remove the input filter inductors (L2, L3, L4) and bridge the sites with wire or metal strips. This will reduce the filter roll off, but has a high chance of solving the transient problem. I need to build a jig to measure the conducted emissions and see if they are acceptable after removing the inductors. It would also be interesting to see what the emissions look like before removing the caps too. The second fix is to put a voltage clamp like a Zener diode across the filter output. There is already a clamp across the filter input (D2), and that is likely why the input voltage transient only goes up to about 16V on power up.

An additional advantage of removing the filter inductors is that the inductors are expensive, about  half the cost of the complete board.

When testing the power supplies, some 12V  1.5A wall wart power supplies would not successfully power the system. In some situations, they would destroy the regulator chips instantly, and in some situations, they would just sit without the input voltage coming up far enough to start the regulators. Here is a scope shot of the wall wart output voltage and current while attempting to drive the regulators.

The channel 1 trace is the wall wart output voltage and the channel 4 trace is the current going into P1. The input voltage is swinging more than 20V below ground on these pulses. The blocking diode D3 protects the board from these negative pulses. There is some kind of interaction between the wall wart and the LC input filter. I tested the wall wart with a resistor capacitor load and it sucessfully comes up to its rated output voltage in that situation. I can only assume that the LC filter is causing problems with the wall wart regulation during start up.

Reading the data sheet on the ADP2302/3 regulators, there is a mention in the section on inductor selection that suggests that the saturation current rating on the regulator inductors (L1 and L4) should be above the maximum current output of the chips in some situations. The regulator current is under 1A in normal operation. During startup, it can get higher while charging the capacitors spread around the system on the power supply rails. I have not observed it going above 1.5A. The first regulator inductors I tested were rated at 2A. The reliability of the board with these was very poor. I swapped in some inductors rated at 4A and things worked much better. This clock operated for many months with those inductors in place. The peak current output of the switch from the data sheet is 6.4A. I found some inductors rated at 8A that are close to footprint compatible. It took a little scraping of the solder mask to fit the new inductors on the existing PCB.

I revised the board to easily use these new inductors and include clamping diodes on the filter output voltage and the regulator output voltages.  The clamp diodes on the regulator outputs are intended to blow the input power fuse F1 which has been downrated to 1.5A in the event that the regulator chips short again.