Project Logs

Collapse

• One web command, one reboot: hunting a 480-byte stack overflow on the RP2350

  Carlos Valdes • 24 minutes ago • 0 comments

  My rooftop Meshtastic node hung twice, once in September and again in October, and the watchdog reset it. No crash message and no fault output: the log just stopped, and about 15 seconds later the node booted again.

  The rooftop node is the sibling of this board: a plain Pico 2 with a separate W5500 module. It runs the same firmware with the same RP2350 build settings, and the fix at the end applies to both.

  Making it happen on demand

  Both hangs came right after admin messages from the Meshtastic web client, sent through the node's HTTP/HTTPS API (my fork adds that API so the web client can reach the node over Ethernet). One of those messages was "set fixed position". I sent that same command from a script, and the node hung every time: once after 22 hours of uptime, and again 83 seconds after it came back. So it had nothing to do with how long the node had been running.

  Then the same command through three different doors:

  • HTTPS (port 443): hangs.
  • Plain HTTP (port 80): hangs. So it is not TLS.
  • The stock Meshtastic TCP API (port 4403): works.

  Where it stops

  The last line in the log was always Use AES128 key. What comes next in the code is the node checking the signature on a local copy of its own outgoing packet: an XEdDSA verify, about 90 ms of curve25519 maths and a lot of stack. In a healthy send, the next log line appears about 90 ms later.

  A red herring

  The HTTP API handler kept a 512-byte buffer on the stack. Making it static was my first fix. The node still hung.

  Measuring instead of guessing

  So I added marks that print the free stack of the running task (from the stack base that vTaskGetInfo() reports and the current stack pointer), right before and after the verify, only for the node's own packets:

  • TCP API: it entered the verify with 2,080 bytes free, and the lowest point inside was 232 bytes. The verify needs about 1,850 bytes.
  • HTTP API: it entered with 1,368 bytes free, about 480 bytes short. The "after" mark never printed.

  Root cause

  In FreeRTOS mode, arduino-pico runs setup() and loop(), and with them the whole Meshtastic packet path, inside a task called "CORE0" that it creates with a fixed stack of 1024 words: 4 KB, with no build option to change it. That line is still the same in arduino-pico master today (freertos-main.cpp, line 150).

  Stock Meshtastic builds for the RP2350 also run arduino-pico in FreeRTOS mode, so they have the same 4 KB. Through the TCP API they get through the verify with 232 bytes to spare.

  The fix, without patching the core

  The linker can redirect every call to a function: with -Wl,--wrap=xTaskCreate, the framework's call to xTaskCreate lands in __wrap_xTaskCreate, which can still reach the original as __real_xTaskCreate. The wrapper bumps the CORE0 task to 8 KB and creates every other task as asked:

  extern "C" BaseType_t __wrap_xTaskCreate(TaskFunction_t pxTaskCode, const char *const pcName,
                                           const configSTACK_DEPTH_TYPE uxStackDepth, void *const pvParameters,
                                           UBaseType_t uxPriority, TaskHandle_t *const pxCreatedTask)
  {
      configSTACK_DEPTH_TYPE depth = uxStackDepth;
      if (pcName && strcmp(pcName, "CORE0") == 0 && depth < MESHTASTIC_CORE0_STACK_WORDS) // 2048 words = 8 KB
          depth = MESHTASTIC_CORE0_STACK_WORDS;
      return __real_xTaskCreate(pxTaskCode, pcName, depth, pvParameters, uxPriority, pxCreatedTask);
  }

  • A 41-hour soak afterwards: no reboots, and the lowest free stack in 5,769 samples was 3,504 bytes.

  That last number is worth a second look: 3,504 bytes free out of 8 KB means 4,688 bytes used at the deepest point, more than the whole old stack.

  What this does not show

  • Long runs on the V2.0 board: the boards are not here yet. Its firmware build includes the same wrapper.
  • Whether this explains every hang. I have seen another one with the same last log line and a different trigger, and I have not proven it is the same bug.

  The fix is in my firmware fork: core0Stack.cpp and the commit.

View project log