Close

Somewhere in the Once Upon a Long Time Ago

A project log for Let's Build a Modern LLM - Entirely from Scratch

While there are several open source versions of various AI applications out there, there is still very little out there for doing training.

glgormanglgorman 04/29/2026 at 02:360 Comments

Right!  Whatever I just said. 

Well, anyway.  

What if this project isn't really about trying to predict the Lottery, and it is especially not about trying to predict any current game using a database of numbers from as long ago as 2012, which hasn't been updated in at least that long?  

No, that's not the point.

That is only the beginning.

Really, an LLM, or so we are told, is not so much about words or numbers as it is said to be about "tokens".  Whatever those tokens represent is, well, whatever they represent.  Maybe macroblocks from a JPEG of a medical X-ray, or maybe waveform data from a seismograph, or "whatever."  That, of course, was the big surprise, with the discovery that "maybe attention is all you need."  Yet somehow, the "pairs of pairs" meme seems to have stuck with researchers as one of the best predictors, at least as far as text generation is concerned.  So does it make sense to try to, in effect, create some kind of "lottery-like" application that increases the number of balls in a particular game to some "reasonable number" like 65535, or something like that, along with an appropriate increase in the size of the "reading frame?"

Maybe.  Maybe there are a lot of things that need to be tried in parallel, and it is just as important to come up with methodical techniques for formalizing the parallelization of whatever processes are being put to the test.  Something else comes to mind, such as whether to try to store the topology of a neural network in a traditional fashion, such as by extending the capabilities of Spice, or Ki-Cad, or whether it is actually much easier to do something like that with a relational database like MySQL . Yet, that of course is where the newfound meme of "vibe coding" might come in handy, that is to say, as things grow, and grow, and grow.  Which they will.

Likewise, we might want to add or delete a source from the training set.

So a "make system" is also needed.

This thing is huge, but not impossible.  

Yes, there will be back propagation and gradient descent.

Maybe even a seemingly infinite number of monkeys, all leaping about.  Having private "conversations" with each other, and so on.

This is an ambitious project.

In the meantime, thank you for choosing Johnny Cab.  Hope you enjoy the ride!

Discussions