I was trying to track down some memory issues, that turned out to be cache access errors from the ISR accessing memory it's not supposed to.  After dramatically simplifying the ISR and creating a new task to handle calculating the data sent to the panel, that error - which was seen when trying to work with more Arduino libraries - is gone and I was able to put together a quick demo of SmartMatrix Library working with WiFi.

I took Jason Coon's awesome ESP32 FastLED Web Server sketch, stripped out the FastLED driver code (as it uses WS2812 LEDs) and merged in the FastLED_Functions sketch from the SmartMatrix Library examples.  Even after initializing SmartMatrix at the end of the sketch, it requires ~30kB memory free for mallocs used by the WiFi and web server portions of the code.

The sketch is here.  If you want to try this, make sure you have the latest code downloaded from the SmartMatrix Library github (TeensyLC branch), and follow the instructions in Jason's README:

https://github.com/pixelmatix/esp32-fastled-webserver/tree/SmartMatrix