Well it happened: I went down a yak-shaving rabbit hole.

The thinking went like this:

• As I write more words in Python generated assembly I'm having to spend a lot of time on both getting stack manipulation code right, and counting the number of instructions. The more I lean on Python to make my code simpler, the more complicated the counting gets.
• I'd therefore like to be able to assemble a piece of code into a buffer, and just measure the length of the buffer.
• This means that I'm going to need to make changes to asm.py, which I've copied into my private area.
• If I make changes to this file I need to be able to merge upstream changes, which is difficult because git doesn't really understand that the files are related.
• I need a script to help with this: make a branch which contains unmodified upstream files, but in the right locations, and merge that into my branch. This is Yak 1.
• If I'm going to make big changes to asm.py, I really don't want any big upstream changes to happen which I have to merge. But there was a ticket to port all of the files to Python 3, which I anticipated might be such a change.
• So perhaps I should port asm.py myself. This is Yak 2.
• I won't really know if my porting work is correct unless dev.py is ported too, and all of the other files it uses. Yak 3.
• Then there's the work to refactor asm.py itself. 4.
• Once this is done I can look at writing helpers for stack manipulation. This is not as simple as writing push() and pop() macros - at least not if I want the code to be close to as good as hand-written. Because of the way that the Gigatron instruction set works, most memory accesses use the X register, but the X register can only be set, used to produce a memory address, or incremented (and only then in combination with writes), and can never be read. To write efficient stack code you need to look at several operations at once: My hand written code sometimes updates the stack pointer variable, and sometimes doesn't, and if I can batch several writes together, I fill the stack top down (or up really, the stacks grow downwards) using the st [y, x++] instructions. So writing Python code that can work out what the appropriate instructions to use are amounts to a small optimising complier. A big five!
• It would be really handy if someone wrote a nanopass compiler framework for Python: The rest of the herd.

I actually shaved yaks 1 to 3. The git script was fun, and has proved useful, and taught me a lot. The Python 3 port is also nice - and is useful to the wider Gigatron project.

I spent a lot of time vacillating between the "compiler" and the asm.py changes, and worrying that both are a waste of time. On the one hand it's kinda stupid to fee guilty about wasting time when the whole project is just for fun; why not follow an impulse to write some code if I want to? On the other hand, I would actually like to complete this at some point!

I think a big problem for me right now is that I don't really know how hard it's going to be to get this finished without shaving the yaks, I don't know how much work these side-lines would be, and so I don't know how much benefit they'll provide. But if I push on without them and it turns out that they would have been useful, I have missed an opportunity to benefit from the effort.

For now I've put both projects to one side, and am trying to make progress on implementing more words. More on this in the next update. I think I might well come back to the instruction counting work soon, but I'm going to go without the stack operation optimiser. I think this may come back in the form of an inliner and peep-hole optimiser to convert words written in Forth to Gigatron code. I've had an itch to write a compiler for a year now, and this might be the opportunity to scratch it.