I tried several other approaches with breakpoints

• temporary breakpoint - fails when hit
• software breakpoint added through GDB - not supported since the code running is in flash
• adding a breakpoint, then stepping one instruction - code runs until a breakpoint is hit, then fails
• adding a software breakpoint via __asm__("BKPT"); line fails as well

It's likely something obvious I'm missing since I'm by no means knowledgeable in this area. So it's time to learn the basics of what's going on under the hood. I found a nice article to get me started on GDB, it's remote serial protocol (used to talk to a debug probe), and breakpoints in general: https://interrupt.memfault.com/blog/cortex-m-breakpoints

With what I learned there, I could print out GDB commands and conclude that installing a breakpoint is not what causes a problem. Rather, failure happens consistently after a breakpoint is hit and GDB is trying to read general registers

Following the printout:

1. Code is paused (Ctrl-c)
2. Temporary breakpoint is installed at in function loop() at line 23. ('tbreak ...')
3. Request for resuming code execution ( '(gdb)c' )
4. Request made to install a hardware breakpoint ($Z1) at address 0x47a
5. Request made to continue execution ( '$c...' )
6. MCU reports a breakpoint has been hit ('Packet received: T05')
7. Request made to read registers( '$g...' )
8. Request fails

Playing with the debug firmware - A breakthrough!

To further isolate the problem after a few days of hitting my head against the wall, I went on to poke at the blackmagic firmware. I could see above that the error always happens when requesting the registers after a breakpoint, during "$g" command. So I went in https://github.com/vedranMv/blackmagic/blob/kinetis_mk20/src/gdb_main.c#L117 file and tried to remove handling of that command from firmware in the way shown below:

/* execute gdb remote command stored in 'pbuf'. returns immediately, no busy waiting. */
int gdb_main_loop(target_controller_s *tc, char *pbuf, size_t pbuf_size, size_t size, bool in_syscall)
{
    bool single_step = false;

    /* GDB protocol main loop */
    switch (pbuf[0]) {
    /* Implementation of these is mandatory! */
    case 'g': { /* 'g': Read general registers */
        ERROR_IF_NO_TARGET();
        const size_t reg_size = target_regs_size(cur_target);
        /*if (reg_size) {
            uint8_t *gp_regs = alloca(reg_size);
            target_regs_read(cur_target, gp_regs);
            gdb_putpacket(hexify(pbuf, gp_regs, reg_size), reg_size * 2U);
        } else {
            gdb_putpacketz("00");
        }*/
        gdb_putpacketz("00");
        break;
    }

Flashing this on, the issue was no longer there. I could hit a breakpoint and resume from it without errors. The only problem now is that, the registers are no longer being updated.

Conclusion

After getting help from maintainers of blackmagic project, the issue turned out to be in GDB itself. My platformio project somehow ended up using a very old version of GDB that had a bug causing it to fail in special cases when reading registers from the MCU. Bug report in GDB's issue tracker was the final confirmation.

In short, the bug was in the handling of a response GDB receives from a debug probe when querying target registers. If the registers look such that first three bytes look like an error code ('Exx....'), GDB would falsely interpret this as a probe encountering an error and would simply terminate the session. In some of the screenshots above, there was no "remote failure" as GDB states, there was just an unlucky situation that the first register returned in the reply had an unfortunate value that would be converted to "E09...".

The fix was to simply upgrade GDB to a version where the bug is fixed, and since in platformio GDB is part of the compiler, that meant updating the compiler itself. To bump the compiler in platformIO, it's necessary to update the configuration with the following line:

platform_packages = platformio/toolchain-gccarmnoneeabi@1.90301.200702

Version can be any of the supported versions, as long as it's higher than 1.50401.190816.

After establishing a connection, I could poke further at the GDB console. Using a simple blinky project for test

#include <Arduino.h>

// Set LED_BUILTIN if it is not defined by Arduino framework
#ifndef LED_BUILTIN
    #define LED_BUILTIN 2
#endif

void setup()
{
  // initialize LED digital pin as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop()
{
  // turn the LED on (HIGH is the voltage level)
  digitalWrite(LED_BUILTIN, HIGH);
  // wait for a second
  delay(1000);
  // turn the LED off by making the voltage LOW
  digitalWrite(LED_BUILTIN, LOW);
   // wait for a second
  delay(1000);
}

I got used to platformIO in VS code, so the example is made through Arduino framework. Before jumping into the platformIO's debugging integration, I thought to give it a test in CLI. Seems like there's no problems with loading the code...

..setting a break point, then running to it...

However, continuing after a first breakpoint seems to fail. Before the failure, windows gives an audible sound that a USB device (Teensy, I assume) has been disconnected.

Any attempts to resume the program, short of full restart ('run' command) fail.

Attempt 2

When pausing the program via ctrl-c during a run, and not using any breakpoints, sessions seems to work fine and there's no errors or device disconnects

Few more tests isolated the error to the use of breakpoints, with the following observations:

• Setting a breakpoint that will never be hit (e.g. in a setup() function after it has already run) will not produce an error. Unsetting that same breakpoint will
• Setting a breakpoint in a place where it can be hit will cause an error seemingly before GDB reports that a breakpoint has occurred

Unfortunately, blackmagic currently doesn't support MK20 chip used by the Teensy.

There's an old PR where someone tried to contribute to blackmagic by adding support for MK20, but sadly it never got in. That can be used as a base with the hope of pushing it upwards if it succeeds.

From here on, I'm in a new area for me. Even though I've used several debuggers in development, my knowledge of what makes them tick and GDB console is pretty much none-existent. This is going to be fun!

Pin definitions for JTAG can be found in MK20 datasheet.

Based on Teensy 3.2 schematics, the pins needed for JTAG are already connected to MKL0 bootloader chip. 

This great article outlines a nice trick for using the JTAG lines, without having to physically cut the lines between teensy and a bootloader chip. Namely, there's a test point with a reset signal for for bootloader chip, preventing it from interfering with the signal lines.

Testing it all

Hooking it up to a blackmagic probe running my branch, I could confirm the approach works since the JTAG scan lists a device correctly. This simply starts GDB on a computer, connects to a blackmagic probe on a serial port and instructs the probe to scan JTAG for devices.

And a test setup