weeBell - personal central office for POTS phones

I took the gCore demo enclosure and expanded it to hold an 18650 battery and the gCore POTS shield.  My partner took one look at my freshly printed prototype - that I was proud of - and sniffed "it sure is boxy".  I guess that's the outcome of only knowing OpenSCAD as a 3D design program.  The loved-one-of-my-life's opinion not withstanding, I'll put the design and STL files in the repo when I create it in case anyone is interested.  Maybe someone else will create some awesome 3D printed design.

It can be assembled with some M3 x 0.5 screws and standoffs (better instructions will be in the repo).

More I2C Debugging
I have,  in the past, jumped to conclusions about the root cause of some issue I'm debugging and it's bit me.  So I've learned to try to be very sure before doing something like blaming a failure on a chip bug.   The I2C failure bothered me so I spent another couple of days trying to figure it out.    I started by writing a test program running on a Teensy 3.2 that accessed both the gCore's EFM8 and the SGTL5000 over I2C to see if it could duplicate the failure.  It didn't.  This was very interesting and made me wonder if it wasn't an ESP32 driver problem of some sort.  That sent me down a rabbit hole of changing parameters in the Espressif I2C driver, but without effect.  I added filters to the I2C bus in case there was noise I didn't see confusing one of the I2C state machines.  That didn't work either.  I looked at a lot of scope traces.

Finally I had the idea of trying to figure out who was holding SDA low after the read should have finished (kind of a Homer Simpson DOH' moment...).  I put a 470 ohm resistor in series with the SDA line going to the SGTL5000 chip and a scope probe on either side of the resistor.  The idea is that there will be a small voltage drop across the resistor due to the pull-up resistor current and the voltage will be slightly lower on the side with the active SDA pull-down.  Bingo!!!!!   In the picture below the green trace is the scope probe on the ESP32 side and the blue trace is the probe on the SGTL5000 side.

I could clearly see the ESP32 releasing SDA after SCL went high to signify an I2C STOP condition but the SGTL5000 continued to hold SDA low leading to the timeout.  I don't know what's going on in the SGTL5000 and I don't know why I couldn't reproduce the failure with the teensy but I felt I had enough proof that it was the cause of the bus failure.  Confident enough to decide to move forward replacing it with another codec.

New Codec Selection

By this time I was really only considering two replacement parts, the MAX9867 and the ES8388.  They seem to be pretty comparable parts with the MAX9867 having better availability for me to build a few of these boards for sale on tindie. I also found some linux kernel drivers for the MAX9867 which I figured I could port.  The less expensive ES8388 is available from lcsc.com but shipping is more expensive which obviates a lot of its price advantage.  But the ES8388 is also found on both Espressif's audio dev boards and in the AI Thinker A1S module.  In fact I had gotten some of Espressif's hands free demos running on an AI Thinker dev board at the beginning of this project.  Espressif has an ES8388 driver as part of their Audio Development Framework (ADF) with a permissive license.  That driver is what ultimately swayed me to use the ES part.

Rev 2 Board

After creating a new part for the ES8388 I was off modifying the hardware.  I could get rid of a 1.8V regulator but added several more capacitors.  Also took the time to clean up a couple of other minor things that bothered me in the rev 1 layout.  

Proto board order from PCBWay and a lcsc.com order for ES8388 parts are going out today.  I'm also buying some more IP101GR ethernet PHY chips from lcsc for my tCam-POE.  Since Group Gets doesn't seem interested in that I may build a few for tindie to see if anyone is interested.

Additional parts from Mouser and a stencil from Osh Stencils can go out later in the week.  In the meantime I'm going to look at the other firmware issues with a hack in place that ignores I2C timeouts from the SGTL5000 in my existing prototype.

Today I got some time to dig into the I2C issue and am pretty convinced the SGTL5000 is clock stretching when it shouldn't be doing anything at all.

As I  mentioned in the previous log entry there are three devices on the I2C bus.

• SGTL5000 Codec chip at I2C 7-bit address 0x0A
• gCore EFM8-based PMIC/RTC at I2C 7-bit address 0x12
• FT6236 Touchscreen controller at 7-bit address 0x38

Different tasks access the chips as part of their normal processing.  The I2C bus access functions are protected by a mutex so only one task gets to run a transaction at a time.

• The SGTL5000 is configured at start-up and then not accessed unless the volume of either the microphone or speaker changes.
• The EFM8 is accessed every 100 mSec primarily to look to see if the power button has been pressed (to turn the device off) and get the charge/battery state for display in the GUI (the battery voltages going into an averager).
• The FT6236 is polled every 30 mSec by the GUI task looking for touch activity.

What I see is that some reads of the EFM8 get an I2C timeout.  According to the Espressif documentation this can only occur if the slave stretches the clock (holds SDA low) too long (default about 400 uSec).  The failure was intermittent until I found one situation where it occurs on every read.

The EFM8 has a register called GPIO that contains the charge status bits and also a bit indicating if a Micro-SD card is installed or not.  When the charge status is "Charge Done" and a Micro-SD card is installed then every single read times out. 

The scope trace below shows the interleaved EFM8 and FT6236 accesses with the timeout.  You can see the timeout at the beginning of the longer bursts which are accessing the EFM8.

The failing read looks like the following (I had increased the default timeout to 1200 uSec as an experiment).  The EFM8 can stretch the clock but only before the first data byte and only for a very short time (I wrote its code).  The failing clock stretching occurs after the first data byte.  You can see the Espressif driver generates 9 clocks when it times out in an effort to clear any hung slave I2C state machine and then ends the cycle.

What is interesting is that if - while the system is running - I unplug the gCore POTS shield to disconnect the SGTL5000 (which, at this point is not getting any I2C cycles) the read magically starts working.

The read data = 0xA in the failing case which is curious because the SGTL5000 address is 0xA.  I suspect  the more random failures may have occurred when various voltage or current data being read had the value 0x0A in a read data byte.  But the correlation between a data byte and the SGTL5000 address is a bit tenuous because when the 7-bit address is clocked out, it's actually shifted up one location to make room for the read/write bit.   So I'm not entirely sure what's going on but I am sure that the SGTL5000 is interfering with reads going to the gCore EFM8.

I did another experiment where I look at the data the Espressif I2C driver returns when it is also returning a timeout error and it appears to be correct (e.g. it read in the correct data to its buffer before noting the timeout).  So I have a possible work-around: Ignore the timeout error and just pretend the cycle finished correctly.  It's a one line addition to the low-level I2C access routine.

esp_err_t i2c_master_read_slave(uint8_t addr7, uint8_t *data_rd, size_t size)
{
    if (!is_initialized) {
        return ESP_FAIL;
    }
    if (size == 0) {
        return ESP_OK;
    }
    
    i2c_cmd_handle_t cmd = i2c_cmd_link_create();
    i2c_master_start(cmd);
    i2c_master_write_byte(cmd, (addr7 << 1) | I2C_MASTER_READ, ACK_CHECK_EN);
    if (size > 1) {
        i2c_master_read(cmd, data_rd, size - 1, ACK_VAL);
    }
    i2c_master_read_byte(cmd, data_rd + size - 1, NACK_VAL);
    i2c_master_stop(cmd);
    esp_err_t ret = i2c_master_cmd_begin(i2c_master_port, cmd, 1000 / portTICK_RATE_MS);
    i2c_cmd_link_delete(cmd);
    
    // SGTL5000 I2C bug work-around
    if (ret == ESP_ERR_TIMEOUT) ret = ESP_OK;
    
    return ret;
}

But of course this is unsettling.   I could try to qualify it by looking for 0xA in one of the read data bytes but I'm not sure if that might not introduce additional issues.

Alternatively I could bite the bullet and replace the SGTL5000 with another codec chip which means writing and debugging another codec chip driver.  I'm already planning to spin the board for a couple of minor changes so I would include the new chip there.  

Codecs that look like a good fit include the MAX9867, some TLV320 parts, and the ES8388.  I'm ignoring parts like the DA7212 because they're wafer-level bump mounting and I don't want to deal with that. The ES8388 is available to me in the USA through LCSC which might take some time to get here but is the least expensive.  The MAX9867 is available through normal US distribution but 3X the cost.

So decision time.  What to do?  I need to sleep on this.  I am going to include the kludge while I try to debug some of the other issues.

Enclosure Prototype
I took the basic case I designed for gCore and tried to extend it for weeBell.  I think I've run into the limits of my methods for using OpenSCAD for enclosure design.  I missed some linkage in the OpenSCAD "code" describing the box and it's just a tiny bit too long.  But this should be easily fixed and I'm going to also add support for a foot so the front of the box can sit at an angle.

Bluetooth Code

The first "application" for weeBell uses the Handsfree Classic Bluetooth protocol to allow POTS phones to answer or initiate calls over a linked cellphone. It currently has the unsexy name "gcore_pots_bt" but I think I'll rename it "Blue POT" like my original gadget.  It is written in C using the Espressif IDF v4.4.4 and is quite functional but with a few things I'm dealing with at the moment which is why it hasn't been uploaded to a repo yet.

1. I have an issue initially pairing with my iPhone.  I'm using Secure Simple Pairing and configuring it so both device show a PIN with yes/no confirmation buttons to prevent man-in-the-middle attacks.  However the iPhone will sometimes show a dialog box asking if I want to download contact info (which it must assume any handsfree device wants to do).  
2. I'm still having some issues automatically reconnecting with a cellphone which make me think I'm still not quite using the Espressif API correctly.  The code can connect on power up but if the iPhone was connected and then disconnects (e.g. leaves the house) the code won't always reconnect when the iPhone comes back in range.
3. I have an occasional issue synchronizing the I2S streams to and from the codec.  The echo cancellation algorithm must always see data from both streams with a constant offset but unfortunately the Espressif I2S driver runs both paths, and their associated DMA engines, asynchronously and even though I go through great pains to start them at the same time and they are the only process running on one core, sometimes they get out of sync.  I may have to write some low level code to replace the existing driver.
4. I am having an issue with some I2C cycles failing.  I think I have found a hardware bug in the NXP SGTL5000 Codec chip.  It sits on an I2C bus for configuration along with the touchscreen controller and gCore's built-in power management/RTC controller.  The code only really talks to the Codec at start-up when it configures it and then only if a volume level is changed.  However it constantly polls the other two devices (the touchscreen controller is polled every 30 mSec and the gCore controller every 100 mSec).  Once in a while when the first read data byte in an I2C read is 0x0A the I2C transaction seems to be stretched and won't complete until the ESP32 I2C peripheral times out.  This only occurs when the SGTL5000 is connected to the bus even when no cycle is running to it.  Unfortunately there is a particular status value from the gCore controller that is frequently 0xA.

The software GUI looks as follows.  The keypad can be used to dial a number (enabling, someday I hope connection to really old phones that lacked a dial) or can be used to dial DTMF tones during a call from a rotary phone (for example to navigate automated systems).

The Mute button mutes outgoing audio from the Phone's microphone and Do Not Disturb prevents the phone's ringer bell from ringing (although the phone can still be answered).

The Settings screen allows some adjustments and pairing control.  The code supports generating different dialing tones and ringer frequency/cadence so that weeBell can simulate phone exchanges from around the world.  I'm currently collecting information for some various countries.  The Auto Dim switch causes the display to dim way down when nothing is going on.  This is because I hate gadgets that blast out a lot of light at night.  A phone call or touching the screen causes the brightness to fade back up.

I hope over the next week or so to figure out the current issues and be able to release the first version of the code.  I'm also making a minor spin to the PCB to incorporate a few slight changes although I'm holding off on that in the case I have to replace the Codec with a different chip.

Ultimately I'd also like to add caller ID generation to the code.  This requires the implementation of a Bell 202 compatible software modem (or, for some countries caller ID DTMF tone generation).