I have finally fulfilled a goal I had since my original BluePOT project back in 2019 - to add Caller ID support.  I know that it's not used by very old phones but I always felt it had to be there for me to claim my gadget was a good replica of a real phone system.  Plus I wanted to learn how it was done.

Took longer than I thought it would

Actually, thanks to spandsp, adding Caller ID support itself wasn't that hard.  A new state machine in pots_task and some new audio paths.  I also had to add support for date and time as Caller ID sends this along with the phone number so I added a new GUI screen to set the time and some timekeeping utilities.  Unfortunately it uncovered some confounding bugs as I would return to my desk in the morning to find the weeBell clock was massively slow.  Turned out there were two bugs.  One of which I fixed and one I am currently working around.  The gCore RTC is used in a traditional HW RTC mode.  It sets the time when weeBell boots.  The system clock maintained by FreeRTOS is used as the operational time while weeBell runs.  I found that both would be slow after a night where my cellphone was across the house and the Bluetooth connection between weeBell and the phone was very "iffy".  At first I was confused why both would be slow but by different amounts.  One of those situations where something confusing makes it hard to look in the right place to identify a problem.  But after a bit I realized they were unconnected.  Turns out the gCore RTC bug was because heavy I2C activity could cause it to miss once/second RTC interrupts.  While weeBell runs it is constantly polling the gCore RTC/PMIC EFM8 micro-controller to see if the user has pressed the power button to turn the device off, as well as looking at battery and charge state for the GUI (and low battery shutdown).  All of that polling was slowing down gCore's RTC.  With some restructuring of the EFM8 code I was able to fix that bug (described here).  However the system clock slowdown is still not entirely understood but I am pretty sure it has to do with the Espressif Bluetooth stack perhaps blocking timekeeping interrupts in the ESP32 under conditions where it is working excessively hard to initiate or maintain a Bluetooth connection. I am still debugging this (it is a slow process) but put in a work-around that seems to be ok.  Every hour the gcore_task checks the HW RTC time against the system time and adjusts the system time if it finds the system time has drifted.

I also tuned up overall operation a little (the backlight fade-up and fade-down are much smoother now) and added a few countries (discussed below).  All-in-all I'm really happy with this release.

Audio Architecture

The primary addition to the audio subsystem is a FSK Modulator capable of either Bell 202 or V23 frequencies.  This is essentially a 1200 baud modem implemented in software that is one way Caller ID is transmitted to the phone.  The other way Caller ID is transmitted is using DTMF tones.  The block diagram below shows the new audio architecture.  In fact I have two DTMF encoders.  The original is still used to generate DTMF during a phone call (for example a user of a rotary phone can type DTMF codes for '#" and '*" on the GUI to access a phone tree).  The spandsp library also includes another DTMF generator in the Caller ID support functions.

GUI Changes

GUI changes were easy.  Another control on the Settings screen takes you to a Set Time/Data screen.  I copied most of the logic for the Set Time/Date screen from my own tCam firmware.  I also copied over RTC and time support utilities from that project and modified them slightly for weeBell.

The time and date are also displayed on the main screen while it is connected to a cellphone.

The quick&dirty about Caller ID

Wikipedia informs us that Caller ID got its start in 1968 in Greece when an engineer named Theodore Paraskevakos began development of a system for a company working in the air transport industry.  In 1971, working with Boeing he created the first practical system and demonstrated it to several telephone companies.  He was followed by Japanese inventor Kazuo Hashimoto's 1976 prototype Caller ID display device.  Although the phone companies initially wanted Caller ID to be a voice announcement (for which they would charge a per-call fee) another engineer, John Harris, working for Northern telecom argued it should be a feature of telephones and displayed instead of spoken.  Ultimately the first commercial trial for Caller ID (plus some other services) was conducted in 1984 by BellSouth in Orlando, FL.  It would take an additional decade for it to be rolled out across the USA.

In the USA the service eventually became known as the Bellcore Calling Number Delivery (CND) standard (TR-TSY-000030 and TR-TSY-000031) and is a based around transmission of information encoded using a Bell 202 modem (1200/2200 Hz FSK modulation) between the first and second rings.  Initially the message only included a timestamp and the originating number (SDMF format).  Over time additional message types were defined to include information such as the calling party's name (MDMF).  Because the Espressif Bluetooth stack does not currently deliver the calling party name I only implement SDMF at this time.

Of course telephone companies around the world were also developing their own Caller ID systems and just like the early days of telephony there ended up being a whole bunch of protocols.

European systems used a V23 modem (1300/2100 Hz FSK modulation) instead of Bell 202.  Some system used DTMF tones to transmit numeric data instead of FSK.  Some systems would transmit the information before the first ring.  In this case they used a variety of signaling mechanisms to indicate to the Caller ID receiver that a message was coming.  They could reverse the line polarity and/or send a special 2130/2750 Hz dual-tone alerting tone (DT-AS).  Some systems, like Ireland's, even sent a very short ring (RP-AS).  The actual message formats varied between countries and there were at least four variants of DTMF encoding.  In Japan the Caller ID system required the telephone to actually pick up the line for the Caller ID information then hang it up prior to the subsequent call.

Humorously the ETSI European Standards organization belatedly documented all the various methods in a document ETSI EN 300 659-1.  Essentially it says, "here are all the ways it was done but we're writing it down now".  

Currently I think weeBell can support most formats except the Japanese system.  Unfortunately I don't yet have a way to test any system but the USA Bellcore system (I have an idea for a universal Caller ID box which I will work on if I find some time).

Supporting more Countries

One of the most fun aspects of all this for me is is getting to listen to the dial tone and rings of foreign countries.  It's like a quick trip abroad.  Supporting an international audience was a prime goal for weeBell.

To add support for a specific country I have to know the following things:

1. Dial Tone
2. Re-order (or Congestion) Tone
3. Off-hook (or Howler) Tone
4. Ring Frequency and Cadence
5. Caller ID protocol

I have easily found Dial Tones although often a country would have many dial tones.  I also can generally find the Re-order Tone.  Finding Off-hook tones is much harder (or even if a country had one at all - in some countries it appears the dial tone just went quiet after a while).  Mostly I work from a document I found in the spandsp library which presumably was used by Asterisk systems which lists tones by country.  I also bought a short subscription to the 3amsystems website to look at their tone database (and download some of the tones I would implement using samples).

Finding Ring Frequency and Cadence is also problematic.  Most online resources show the audio one gets in the receiver listening to a remote ring.  This often, but not always, mimics what the real ringer sounded like.  I did peruse youtube a bit finding various recordings of old phones ringing.  And I found some European standards documents which listed the ring frequencies as part of test verification parameters.

But Caller ID information is the most difficult to find.  This is because there was so much variation in things like the alerting phase as well as the fact that many countries supported multiple implementations (initially you had to buy a phone that matched your provider although eventually phones would support multiple standards).  This is information I fear is being lost over time as well.

I spent countless hours researching but not always being sure of a specific country implementations.  Ultimately I decided for version 1.0 of the Bluetooth firmware I would support a "representative" sample.

1. Australia - For the interesting modulated dial tone.
2. Europe - Call this "Typical" Europe.  Most of Europe uses a 425 Hz dial tone and I think DT-AS Caller ID alerting before the first ring so this would be my test of that configuration.  
3. Germany Pre 1979 - When the dial tone was a Morse Code 'A'.  And I'd test my code when configured for no Caller ID at all.
4. India - Another country with a fun modulated dial tone and I could test Line Reversal alerting with one of the DTMF Caller ID protocols.
5. New Zealand Rev - Supporting the reverse rotary dial phone (and testing my rotary dial map function).
6. United States - Well, natch, my home country (plus we have a great Howler).
7. United Kingdom - The Brits too have a great Howler plus they are the only place that uses one particular form of Caller ID (so called "SIN277" with Line Reversal + DT-AS alerting tone before the first ring).

It's pretty easy to add new countries to the firmware if you want to compile your own.   You just add to the country_info data structure in the components/utility/international.c file (described in the international.h file).  I'm also open to adding countries if anyone would like to reach out.  I'll need the information above.

Firmware is here for anyone who cares.