Scope Creep and Memory Management

After continuously having issues with any network requests failing 20-80% of the time, I figured out that the issue was running out of memory. I didn't suspect this was the issue, as the biggest request I would be doing is only 1-2Kb right?

Well after checking micropython's "mem_info()" output, I realized that the heap was entirely fragmented to the point there was no contiguous space for a 1-2Kb chunk of data to be stored.

So I set to work to figure out how to fix this, and that's when the scope creep was fully realized. Currently the program has these classes:

MiniPico: Contains device functions and any system functions, oled output, etc
SecondCore: Handles playing sounds over buzzer, reading key presses, and timers
KeyPad: Handles the logic for what characters to return based on what buttons are pressed
Console: The console
Interpreter: The interpreter program
Screensaver: a simple screensaver program

I tried to figure out how to have MiniPico as one file and have these other programs in other files then import them, but as each of these classes call functions in other classes, I was not able to figure that out. It wouldn't work. I need to research that more, it would be a good thing to know how to do.

On to the fragmentation issue. At first I started wondering if it was almost entirely because of all the string operations I had going on. I think this was a huge part of it. This lead me onto a path of playing around with gc to see what is using memory and when. It was very informative, and i'll cover that more in a bit.

Back to the string thing... This software is mainly text based software. I am creating a console for this device, so I have to use strings right? Maybe not, I am not opposed to using strings, but I realized I could have fixed size byte arrays to store the buffers for the console and screen text. This will let me reuse the same memory space for that. Another thing I can do, instead of having string operations happening in each class, and string data stored in each class, put it all in one class or object at the beginning of the file so it's all in one chuck and not scattered throughout the entire heap. I think this was the main issue, but not entirely sure.

Once I realized my program needed to be rewritten, I was on the verge of switching to and learning the C SDK. Mainly because you have to have complete control over memory in C, so you know how much memory is allocated and where (generally). I love C programming and want to learn more, and want to try C on the Pico, but getting the environment setup and rolling is something I haven't visited yet. I did install VSCode and the Pico Extension, and was about to start experimenting (I will soon), but then I had the thought. Why just switch to C when this is a great opportunity for learning more about how micropython manages memory, and it will be a good challenge to try to write a large program that doesn't have memory issues.

On to my findings while experimenting with micropythons garbage collector: Here is the whole code that I was playing around with.

# test micropython program for experimenting with memory usage,
# and byte arrays. On P Pico W

from micropython import mem_info
import gc, random, time

ROWS = 32 # Integers use 8 bytes each
COLS = 16
PERCENT_100 = 100
PERCENT_25 = 25
PERCENT_50 = 50
buf = bytearray(ROWS * COLS) #main buffer byte array uses ~550 bytes
out_buf = bytearray(16) #output buffer

gc.enable()
MAX_MEM = gc.mem_free() #set upper memory threshold at current level
gc.threshold(int(MAX_MEM / PERCENT_100 * PERCENT_25)) #collect when memory is 25% full


def get(buf, row, col):
    return buf[row * COLS + col]

def set(buf, row, col, value):
    buf[row * COLS + col] = value

def free_mem():
    #mem = ""
    #mem += 'mem: '
    #mem += f'{round(gc.mem_free() / MAX_MEM * 100)}'
    #mem += '%'
    #return mem
    #return round(gc.mem_free() / MAX_MEM * 100)
    #return 128 * random.randint(0, 64) + random.randint(0,64)
    return gc.mem_free()
    
def main():
    
    start_time = time.ticks_ms()
    print(mem_info(1))
    while True:
        for y in range(ROWS):
            for x in range(COLS):
                #gc.collect()
                set(buf, x, y, random.randint(37, 37+37))
                out_buf[x] = get(buf, x, y)
                now_time = time.ticks_ms() - start_time
                if now_time > 5000:
                    print('############################################')
                    print(mem_info(1))
                    print(free_mem())
                    start_time = time.ticks_ms()
main()

My discoveries were these:

Basic Math operations such as adding, subtracting, or multiplying don't really add any trash in memory
Doing anything with a Float or a string starts clogging up the memory fast, and gc.collect() must be used after doing those.
Setting the gc.threshold() is a good idea. Otherwise gc will wait til the memory is completely run out before collecting garbage.
Doing anything with a string starts adding garbage to the heap. This takes up a lot more memory than a float. You may say everybody knows this, strings in Python are immutable so any work done on a string creates a new string, and you would be right. It's just interesting to see it visualized.
I haven't even experimented with playing around with popping or appending to lists. I imagine that can clog up the memory.

Below is the mem_info() data before doing anything. The free_mem() function I am using is where I am testing different operations and seeing how they affect the memory. If I use the bottom two options where the return is just an integer, then the available memory never changes.

With the following operation:

return round(gc.mem_free() / MAX_MEM * 100)

The division creates a float, then we use round to turn it back into an int. Running this for 50 seconds, you can see the difference in the memory:

Not bad... But if we aren't careful with collecting garbage, then it wouldn't take too long for this to become an issue... Also I was trying to figure out what is using the memory between where the program started inserting data and where it says "176 lines all free"... This is a small example of fragmentation. On my current MiniPico software, that's what it looks like all through it... I will put the data for that at end of this post.

Now while running this good example of bad string operations:

mem = ""
mem += 'mem: '
mem += f'{round(gc.mem_free() / MAX_MEM * 100)}'
mem += '%'
return mem

We will notice in the results below that in 50 seconds, it was filling up the memory a lot faster:

I am not sure what my final conclusion is, but i feel like I know a bit more about the pico's memory usage now using Micropython.

Here is the mem_info() data for my current "main.py" for the MiniPico that I hit a wall with. This is the memory without even doing anything after loading all classes... It is apparent that I need to do something different:

stack: 588 out of 7936
GC: total: 189952, used: 55936, free: 134016
 No. of 1-blocks: 503, 2-blocks: 69, max blk sz: 256, max free sz: 503
GC memory layout; from 20011a00:
00000000: MDMLhhhTTDhhT==DBBBBTDhTh===TDBDh====B=BBBBBBTB=BTB=BBBTB=TBTB=B
00000400: h===TB=h===========B=MDhSh===================h========h=========
00000800: ========h=======================================================
00000c00: ========h=======================================================
00001000: ========h=h=hhhBhhBhFFSShhBTDhhhhhhhMDSTBh=h==hhhhBhhhhSTTTTTTSS
00001400: hB=hB=hhhhhhhhh=======h========Sh=Sh=SShT===MDhBhBBhBh=======h==
00001800: ==============T===T===T===T===T===h=============================
00001c00: ==T===T===T===h=======hh=h=BhhTTTTTTTTTh====BBBhTTBBTh===BDhBhT=
00002000: ======h==Sh==Sh=T=h====h===h====Thhhh=======T==SFh==============
00002400: ======h=hhB=TSBh==BDhhhTBBhhhhhhhhhTB=hhhB=hhh==================
00002800: ==================h===TB=ThT====h=======B=BBTBBB=BBBBBBBh===BDBB
00002c00: BBBBBDh=h=h===BBh===============================================
00003000: ==============h==================h==============hhhh=ShLhLh=hLLh
00003400: LhLh===BDBBSh=======h====================================h======
00003800: =====h==h===BTTBBBDB=B=h===h===BBTBTB=TB=B=BBBTBB=Bh=FT==Sh=====
00003c00: ================================================================
00004000: ==BBBBBTB=BBBBBBBBBh===BDBBh=h===Lh==============h==============
00004400: =========AhAhhh=LhLhLh=h====hLhLhLhLhLhLhLhh=========h===LhLhLhh
00004800: ==h=T=======LhLhLhLhLhLhLh=====Dh==hh=====h========h============
00004c00: ===================================================LhLhLhLhLh===
00005000: ====h==============================hLhLhLh==DDDLhh==============
00005400: hhLhLh==Lhh=====h=========h=h=h=h==h======h===h==h=Lhh==========
00005800: ===================h=======h====h==TTTTTTTTTTTTLhh==hLh====h==hh
00005c00: hhLhh=hTTTh========TTTTTTTTThh..................................
00006000: ................................................................
00006400: .............................................h=======...........
00006800: ..............................T====h============================
00006c00: ================================================================
00007000: ================================================================
00007400: ================================================================
00007800: ===================================.............................
       (5 lines all free)
00009000: ..................h=======................Sh=ShSh=Shh...........
       (4 lines all free)
0000a400: .............h=======...........................................
       (3 lines all free)
0000b400: ..............Fh=======.........................................
0000b800: ......................................................h=======..
       (3 lines all free)
0000c800: ......................................h=======..................
       (2 lines all free)
0000d400: .......................................h=======.................
       (7 lines all free)
0000f400: ......................................h=======..................
0000f800: ..........................................................Sh====
0000fc00: ===T====........................................................
00010000: .................................h=======.......................
       (2 lines all free)
00010c00: ...........................................h====================
00011000: ================================================================
00011400: =====================================...........................
00011800: ..........h=....................................................
00011c00: ......h=======..................................................
       (5 lines all free)
00013400: ....................................h=..........................
00013800: .............................hhh................................
       (3 lines all free)
00014800: .......Sh..............F...........................h=======.....
00014c00: .......................F........................................
00015000: ................................................................
00015400: ...........F....................................................
00015800: ................................................................
00015c00: ......................................F.........h=======........
00016000: .Sh.............................................................
       (2 lines all free)
00016c00: .........SSh=.....................................h........Sh===
00017000: ====Sh=........Sh=........Sh.........h=.........................
00017400: ................................................................
00017800: .......................................................Sh.......
00017c00: ....Sh..........................................................
       (2 lines all free)
00018800: ........................................................F.......
00018c00: ................................................................
00019000: ..............Sh..............h=======..........................
       (3 lines all free)
0001a000: ..............................Sh................................
       (2 lines all free)
0001ac00: ......T==...............................Sh......................
0001b000: ...........................................Sh................Sh.
0001b400: ...............................h=======..............Sh=........
0001b800: ...................................Sh...........................
0001bc00: ...............................Sh.........Sh.........Sh.........
0001c000: .......................................h........SSh.............
0001c400: ......................h...............Sh........................
0001c800: h=======.........................Sh=............................
0001cc00: ...............................................Sh...............
0001d000: ................................................................
0001d400: ....................................Sh.........T........SSh.....
0001d800: ......h=.........................Sh=............................
0001dc00: .........Sh..................S..................................
       (3 lines all free)
0001ec00: .........h....................................................Sh
       (2 lines all free)
0001f800: .........................S........hh........h...................
0001fc00: .h=h=..........h=...h=...hh....hh=hhh=...h....................h.
00020000: ................................................................
00020400: .......................................................h=======.
       (6 lines all free)
00022000: ...........................................Sh...................
00022400: ................................................................
00022800: .........Sh=.........T=.........h=======..T=.........T==........
00022c00: ..........................F.....................................
00023000: ......F...................................h=====================
00023400: =================================================...............
       (3 lines all free)
00024400: .................Sh=............................................
00024800: h=...h..........................................................
       (4 lines all free)
00025c00: .............................................h.....h............
00026000: ..............h.....h.....h....................h.....h.....h....
00026400: .h....................h....................h.....h.....h.....h..
00026800: ...hh.............h.....h.....h.....h.....h..hh..........h.....h
00026c00: .....h.....h.....h.....hh.............h.....h.....h.....h.....h.
00027000: .hh==........h.....h.....h................................h===..
       (2 lines all free)
00027c00: .....................................h====......................
00028000: ................................................................
00028400: ...........h===.................................................
00028800: ...............h=========================h========h======h======
00028c00: =======h=====h=====h=======h==========h==============h=======h==
00029000: =====h========h=========h====h====h================h============
00029400: h========.......................................................
       (2 lines all free)
0002a000: ............h==========h========================================
0002a400: ================h================h===h===h===h=========h======h=
0002a800: =h===========h=====================================h==h========h
0002ac00: ==========================================================h=====
0002b000: ==h=============h===================h===========================
0002b400: ================================================================
0002b800: ===============================h==========h===h=================
0002bc00: ========h==================h==========h=========================
0002c000: ===================================h====h=======================
0002c400: ======================h==================h======h===============
0002c800: ==========================h=======h===h============h============
0002cc00: ================================================================
0002d000: ============================h===h===h======h=========h======h===
0002d400: =====h=======h====h====h=====h==================================
0002d800: ============================================================h===
0002dc00: ===========================================================.....
0002e000: ................................................................
0002e400: ................................

A Little Progress

Discussions

Become a Hackaday.io Member