Let's Build a Modern LLM - Entirely from Scratch

Well, sometimes we need try and catch.  They are convenient.  Pascal, on the other hand, has an arcane way of exiting a function by using an EXIT function.  So function A can call function B, which calls procedure C, which calls D, I think, and then procedure D can call EXIT(B) if it wants to, and then even if we are 999 levels of recursion deep with C calling D, calling C over and over again, it should all unwind back to wherever we would have been if we unwound everything normally, via the normal return paths.  This can be emulated in C++, in those situations where you have some legacy Pascal code that needs to be rewritten.

class EXIT_CODE
{
public:
    bool m_edit;
    int     err;
    char *m_str;

public:
    EXIT_CODE(char*str);
    EXIT_CODE(int n, bool edit);
};

 Just create an EXIT_CODE object and populate it with a string or an integer for the error code, if that is what is required, or else set a flag to invoke the editor.   Maybe.  Yet for me, the really fun thing is still knowing that I can get away with having a mostly Pascal-compatible version of write, and writeln, that not only works from regular Visual Studio Code but also seems to work from CUDA code.  Which might seem to defy explanation, but it does seem to work!

Believe it or not.  So, naturally, I think I need to do some more work on this concept.  Like maybe it should be able to do certain types of asynchronous IO automatically.  So that the user does not need to worry about locks and semaphores, buffering, and so on.  Most of that I am already doing in the Propeller Debug Terminal, but there are still some differences between how the CUDA interface works and the way I did things in a C++ port of a version of the UCSD Pascal compiler from 1979, and so on.

Likewise, when thinking about tokenizing an input stream, I have an Eliza-based method, as well as a complete rewrite of the UCSD Pascal compiler's INSYMBOL function to choose from.  Maybe.

This version shamelessly uses the use of goto in C++, but it also makes for a good use of a C++ implementation of Pascal-style sets to check for different character types.  All in good fun.  So you thought that writing a tokenizer is trivial?  Wouldn't it be nice if we could specify something that works like BNF, without having to use LEX or YACC or Regex?

Be very afraid.

I think it can be done.

But not quite there yet.

So, in the meantime, back to brute force methods.

Well, then again.  There is a version of Propeller Debug Terminal that I wrote back in 2019, which is able to host an instance of Eliza, as well as an instance of Mega Hal.  So, maybe now is a good time to work on also getting it to host an instance of Lotto Pro, when built as a library instead of as a standalone application.

So that will be the next step then.  Get "Lotto Pro running", as close to as-is as possible, inside Propeller Debug Terminal, that is, as a statically linked library.  Then it will be possible to start the aforementioned complete rewrite of the core of the application, so as to perhaps try to make some kind of collection of "mixture of experts models" that all run in parallel, and which can make use of each other's strengths and weaknesses.

"Do, or do not!"

Alright then.  Let's try this!  It is deep.  Dark.  Messy.  Be very afraid!

For a very long time now, like ever since Windows 95, there has been this technology called "Microsoft Foundation Class", and one of the very nice things that is available in MFC is something known as "Document View Architecture".  Which in turn is a VERY nice piece of tech, even to this day.  That is, when you can get it to do whatever it is that it is supposed to do.  Unfortunately, in practice, that often requires a minefield of tricks and hacks to get the framework to jump through all of the desired hoops.  Things like manually editing resource files across multiple projects, to try to get them to play nice with each other, and that is just the beginning.

Whether it is obvious, as to what is happening here, may not be so obvious.

When a multiple-document, multiple-view application is initialized, there are a bunch of data structures that are created, called "Document Templates," that tell the framework what to do when we want to open a document, whether from a file, or as a new instance.  So we need to specify the type of document, according to the class that handles that particular type, as well as associate that document with a view, and also an appropriate menu bar. When it works, it works very nicely, since we can have a document such as a "Terminal Document" that might have multiple views associated with it, such as a text view, or an oscilloscope view, or a spectrum analyzer view.  Likewise, each combination can have its own unique set of menus, or they can run off of multiple copies of the same menu system, with or without variations.

So this is how I think I am finally going to get Eliza to talk to Megahal, or to have instances of Megahal that can chat with each other, such as an instance trained on "Alice In Wonderland" that might be set up to have conversations with "The Federalist Papers"

So, yes.  Just getting to a "Hello World State" with "Lotto Pro" being hosted inside another application is a pretty big deal.  Not just for what it does, in and of itself, but for what it makes it possible to do with other applications, according to the time-honored method of "figure out how to do it" then "copy the technique.

To be continued.  Then again.  At least now it is doing something!  Right now I have come to realize that there is a massive amount of "accumulated" technical debt that needs to be overcome before any new "real work" gets done, and who knows?  That is probably a major problem across the industry as a whole.

Yet I have managed to get quite a bit done in the last couple of days.  Lotto Pro now works with Unicode, without crashing when I try to use the std libraries for things like iosstream, fstream, and so on, because I am no longer using those libraries to load the lottery history file.  Not getting 9999 or more linker errors from having incompatible libraries, duplicate functions "already defined" errors, with warnings to "use NODEFUALT LIB option," which then results in a few 1000 more "unresolved external symbol" errors when the linker can't find the right version strcpy, or whatever.  Also nice.  So what I did of course, was to "borrow" some code that I wrote for the C++ port of the UCSD Pascal compiler, in order to get around a nasty problem of the iostream libraries not wanting to accept the UNICODE string format when being passed a file name as generated by the file picker.  And that, of course, is just one of many rabbit holes that have to be dealt with when needing an ASCII-aware terminal mode, in an otherwise purely UNICODE application.  One method is to make shadow copies of everything, and only generate the UNICODE version of a string when it is time to display it.  The other is to want to go "full Monty" and completely re-invent string handling altogether, so as to have a concept of being able to have "text objects" that themselves might be managed linked lists of dictionary pointers, and so on, which can hopefully be extended to more fully embrace other "token" types, that is based on certain LISP concepts.

Thus, the original Pascal INSYMBOL technique retains some appeal, since it, or something like it, might be more easily adaptable to a more generic implementation, i.e.. one that can recognize not just Pascal keywords, operators, identifiers, numbers, etc., but something that can handle more general tokenizing problems, as if it were more general, like regex, on the one hand. but ideally we would like to be able to "infer when a flat file might just as well have been written out as a CSV", so why don't generate some JSON on the fly, which can be used to create the BNF-like metadata that the lexer needs to import the data.

Or just hard code it, again, and figure out the more generic problem later.

Or perhaps a bookshelf.

Well, anyway.  It has been a while since I started a new log entry.  At least I have a catchy title for this one.  Now I think it is time to get busy with some serious project integration.  Elsewhere, I have suggested that this project is actually something like 20 or maybe more projects, all of which need to be brought together into a cohesive whole.  So perhaps the best way to go forward is to take an inventory of all that has been accomplished so far.  Make a list.  Check it twice.  Chase down some of the naughtier issues that need to be addressed, as well as showcasing some of the nicer things that can be done within the current framework.  Then spool up another homegrown AI that will be trained, of course, on every article that I have ever written for Hackaday, so that I can have yet another conversation with it, as if to ask, "Where do we go from here?"

Just looking at the code base that went into creating an entry for the "One Hertz Challenge" reveals the magnitude of the problem.

Yet, as long as vibe coding is the new meme, how hard should it be to integrate all of that, along with the "world model" stuff, into an LLM, perhaps by using some of recently available online tools to rewrite all of this?  Or else, there is yet another question that needs to be addressed, and that is "whether any of the latest development tools can handle project integration?"

That is, if nearly all of the heavy lifting has been done.  The next step, therefore, is simply, dead simple, so crazy simple that it will drive a whole bunch of DIY types crazy, when they realize what they have been missing out on, all of this time.  Let's take the code base for the program "Lotto Pro" that I was talking about earlier, at the beginning of this project.  The one that looks at pairs and pairs of pairs of numbers and their respective probabilities in Lotto-type games.  Yes, that one.  Let's turn that from an application that is Lotto-specific into a library that works with many different types of data.  Don't worry.  Eventually, source code will be provided, but for now, just enjoy the ride.  Just in case you want to make your own LLM+AI aware version of DOOM, this is where you are going to start.  LIke this:

In pretty much any version of Visual Studio, you will find a project properties menu, and what you want to do is navigate from the "General Project Defaults" list to the "Configuration Type", where you will want to select whatever it is that you now want instead, whether that is an application, static library, DLL, or whatever.  So, let's select "static library" for now.  Then, just to see what happens, we compile it, and get something like this:

Now we have something.  The same thing that we had earlier, only different.  An old project that can be dropped into another larger project that allows us to build applications like the previously mentioned "One Hertz Challenge" project.  It is of course, where it can get really weird, really fast.  Creating workspaces that turn dozens of projects that were originally created as standalone applications into libraries, that might later be turned into ActiveX controls under Windows, or DLL's, or perhaps they might get ported to Type-Script or another language for use as browser add-ons.

Remember: A 1000-mile journey begins with a single step, and a 100,000 line project begins with a single idea and a few lines of code. But in the meantime, here is one little tidbit, to help indicate the direction that this project is headed.

Pretty simple, at first blush.  Take the old application.  Change the project type from application to "static library", then add the "new" library project to the existing workspace used to develop the codebase for the "One Hertz Challenge", as previously stated.  Then the fun begins.  Lots of other things to do, such as remembering to comment or #ifdef out the CWinApp-derived object that the original application relied on.

In any application that actually trains an AI on user-provided content, we are going to need to have some methods for getting the training data into the model, obviously.  Thus, there is a very important chore that is going to be one of the next things that needs to be done, and that will be to rewrite a bunch of stuff that looks like this:

draw::draw (char *in)
{
    char *pos;
// example
//1618 Oct 05, 2002 4 12 14 28 42 m07

    pos = in+28;
    strncpy (str,in,28);
    str [28] = 0;
    sscanf (in,"%d",&number);
    sscanf (pos,"%d%d%d%d%d%d",&x[0],&x[1],&x[2],&x[3],&x[4]
//#ifdef MEGA
    ,&x[5]
//#endif
    );
    sort ();
}

... so as to eliminate any project-level dependencies on functions like strcat, strcpy, sscanf, and so on!

Thus, I will want to revise the coding style to use the same method that I used in "One Hertz Challenge" to parse NMEA strings as they were being obtained from a GPS module connected over USB, and THAT of course, was being done with a completely rewritten version of Eliza, which in turn had be reworked to use a "tokenizer" contained within another library called "framelisp"

And that, of course, looked something like this:

bool ELIZA::detect_nmea (const char *target)
{
    char *str;
//    check for "$" as the first token
    m_process.rewind ();
    if (m_process.eof())
    return false;

    m_process.get (str);
    int test;
    test = btext::compare (str,"$");
    if (test==0)
    writeln (debug_term,"$ Token Found:  Possible NMEA message!");
    else
    return false;

    if (m_process.eof())
    return false;

    m_process.get (str);
    test = btext::compare (str,target);
    if (test==0)
    writeln (debug_term,"\"",target,"\"","token indicating NMEA message!");
    else
    return false;

    if (m_process.eof())
    return false;

    m_process.get (str);
    test = btext::compare (str,",");
    if (test!=0) {
    writeln (debug_term,"Invalid Sequence, \",\" expected");
    return false;
    }
    if (m_process.eof())
    return false;

    return true;
}

 Yes, it is quite a busy, simple, arcane, and brute force way of doing things.  But this tries to make good use of routines that in turn a designed to be adaptable to make use of text objects that can store data as ASCII strings, UNICODE strings, linked lists of dictionary pointers, or simulated linked lists that hide inside vectors, or else "whatever", where "whatever" might someday include Huffman coded fully searchable text objects, that support operations like insert, delete, find and replace, or whatever else might get added, without needing the full capabilities, or bloat therefore, of REGEX.  Even though that would be nice to have also.

Other niceties when building using "the AI aware framework" include the easy-to-use capabilities of the library to do Pascal-style IO, using write and writeln functions, from within C++, no less.  

Still, this is going to take a while.

I will figure this part out later.  At least I found a "famous movie quote" that is different from one of my own favorite lines to perhaps someday begin every chapter of a book filled with short stories with, such as "It was a dark and stormy night, and as the swamp thing staggered from the crypt, suddenly there was a need for words."  In the meantime, instead, before I say anything else about any of that, I learned that there is yet another approach to AI that might be in vogue, so to speak, even if "Vogue", the magazine, hasn't taken notice of it yet.  All in good fun, of course!  

Somewhere, I still have this thing.  Maybe it will turn out to be useful?  In the meantime, I was spending quite a bit of time watching YouTube videos about AI, and in particular, there is yet another approach to problems like image recognition that has been gaining traction, and that is the one that some are referring to as "JEPA", which in turn is an acronym for  "Joint-Embedding Predictive Architecture."  So I suppose that I need to figure out what that seems to imply, just in case it does in fact imply something.  Something quite specific, I think, insofar as how the same technique might have application within a broader "mixture of experts model", such as the one that I am trying to create here.

So now I have this idea, which I have had for a while, actually, and a plan seems to be coming to fruition as to an interesting approach to text generation, chain of reasoning, and all of that.  Really, it seems pretty simple.  Maybe as simple as this:  Well, almost.  This, I mean:

Of course, we could put some simple spheres inside a sphere, and let them bounce around, as if to simulate the kinetic theory of gases, or else we could simulate a Lotto-type game.   Or we could imagine replacing the sphere with a human brain, and imagine a way of modelling random neurons firing, and perhaps even tracing out the path that some of the sodium and potassium ions might be taking, or else we might want to think in terms of specific ganglionic structures, and how the flow of different neurotransmitter substances might affect things. Really, it is all just physics.  Or is it?

Not that things might not get a little bumpy, of course, if we wanted to model something else, perhaps with nearly exactly the same code.  Like when we decide to put a sphere inside a sphere, which might be inside another sphere, and so on.  Setting the stage perhaps for other such fun things as "thermonuclear initiation", or maybe just "plate tectonics".

But that is not where this train is headed.  Yet.  Or else, maybe in the brain we might have the hippocampus, and the cerebellum, and a bunch of other hierarchically connected, as well as "otherwise connected" structures.  Yet one approach is still to take the Lottery simulation and subdivide a three-dimensional space into a collection of interconnected realms, which would, of course, allow multiple simulations to run in parallel, on the one hand, while at the same time providing some completely general methods for partitioning.

So maybe I need to change the subject.  Let's look at a real-world problem, along with a possible application for AI in order to solve it.  Let us consider the utility, therefore of the "kinetic theory of gases" and its application to a potentially very simple, and hopefully not dangerous problem, such as "Can I put a window air conditioner in a fireplace?"

As I said earlier, hold on, it is going to be a bumpy ride!  I do, after all have an ever increasing list of AI blunders, of varying degrees of severity or seriousness.  Now what is happening here should be obvious.  The AI in this case, I think, is suffering from a form of near-sightedness, and it is actually a particularly dangerous one, or could be, whether in this situation or not, but definitely in other circumstances, there is a certain kind of "near sightedness" that creates a particularly grave danger.

For example, it is well known that in ancient times, the Roman Senator Cato was in the habit of ending every speech with the phrase "Carthago delenda est," which loosely translated means "and as for the rest, I hold that Carthage must be destroyed".  Now imagine the ancient generals in Carthage had received "intelligence reports" that Cato was "contemplating an ideology of conquest", but perhaps those notions were merely "aspirational", and so on, and that that was further paraphrased to suggest that "there is no immediate risk of invasion" but that "maybe the Roman Senate will take up the subject "next year."  Now what if history repeats itself? 

In the meantime, the Titanic is still equipped with five watertight compartments, any two of which can flood with no risk that the ship will sink.  This gets paraphrased to "this ship is unsinkable", but out of an abundance of caution, we want to transfer as many of the passengers as possible to a rescue ship because of the possibility of a power failure. Now imagine that you are a first-class passenger.  Do you wait for a rescue ship to arrive, and wait comfortably in your stateroom, or do you brave the otherwise 'unnecessary discomfort of the icy cold conditions of a lifeboat?"

Yet if attention is all you need, and if the fireplace problem as it is stated, is actually a problem in "optics", as well as a problem with "the kinetic theory of gasses", since it would seem to me, that "political thinking is often full of hot air", and "hot air rises", well the conclusion should be obvious, as long as you are not lighting a fire in the fireplace while the air conditioner is in use, or otherwise, "in the fire box", potentially blocking the flow of air.

Clearly the AI does not have a world model.  Neither does its attentional system accurately understand what is and is not a real-world risk based on the apparent distance to its "analytic horizon".

In other words, it is "shortsighted".  And while air-conditioners are not supposed to emit creosote or carbon monoxide, assuming that building codes do not prohibit it, I don't see why it would be inherently unsafe, or not worth trying in a pinch, especially when 100's of elderly or otherwise fragile persons might be at risk of dying during a heat wave if there are no other practical means of cooling.

That does not mean, however, that "building codes" are just another form of hot air.  Yet it might imply that ABS has already been achieved, as in "Artificial Bureaucratic Stupidity!"

A world model is therefore needed that the AI can interact with, and even be a part of.

Stay Tuned!

But I like to begin my log entries with a catchy title, nonetheless.  So, in any case, maybe it is time to look at some code.  For Lotto-type games, I created a source file called lotto.cpp, which does all of the heavy lifting, insofar as looking at a bunch of stuff that involves examining pair histories, looking for triplets containing particular pairs, and so on.   So the C++ class for "draw" that is going to be rewritten for that part of the model starts out looking like this:

class draw
{
    int score;
    int number;

    int x [6];
    char str [32];

public:
    
    draw ()
    {
        memset (this,0,sizeof (draw));    
    };
    void sort ();
    draw (int a, int b, int c, int d, int e, int g = 0)
    {
        memset (this,0,sizeof (draw));
        x[0]=a; x[1]=b; x[2]=c; x[3]=d;
        x[4]=e; x[5]=g; sort ();
    }
    draw (char*);

    bool operator > (draw &d);
    int &operator [] (int arg)
    {
        return x[arg];
    }
    void test (char *, size_t max);
    bool increment (int limit = _G::MAX_NUMBER);
    bool match_copair (int p, int q, int r, int s);
    void dump (char *buff, size_t max);
    void calc_pairs ();
    void calc_trips ();
    
};
class draw
{
    int score;
    int number;

    int x [6];
    char str [32];

public:
    
    draw ()
    {
        memset (this,0,sizeof (draw));    
    };
    void sort ();
    draw (int a, int b, int c, int d, int e, int g = 0)
    {
    memset (this,0,sizeof (draw));
    x[0]=a; x[1]=b; x[2]=c; x[3]=d;
    x[4]=e; x[5]=g; sort ();
    }
    draw (char*);

    bool operator > (draw &d);
    int &operator [] (int arg)
    {
    return x[arg];
    }
    void test (char *, size_t max);
    bool increment (int limit = _G::MAX_NUMBER);
    bool match_copair (int p, int q, int r, int s);
    void dump (char *buff, size_t max);
    void calc_pairs ();
    void calc_trips ();
};

There is, of course, a lot of room for improvement, and a lot of things that need to be changed, since clearly it is nowhere nearly as simple as using find and replace to change the word "game" to "context" and "draw" to "reading frame" as one might want it to be.  One of the first things that I want to get rid of, therefore, is the reliance on the use of sscanf to parse a line from a Lottery history file into an entry into a game history database.  That needs to go badly!  Especially if we are going to be "tokenizing" things like the Federalist Papers, or articles from Wikipedia, or whatever else we might encounter.  So, this is going to present an interesting challenge, in that respect, i.e., make improvements to the original code so that it will work with more history files for more types of games. At the same time, enabling the algorithms that we are using to try to make "predictions" about what pairs with whatever, to work with other types of tokens, that is according to a more robust probability model.

void CLottoProDoc::find_copairs (int p, int q, int SEARCH_DEPTH)
{
    _int8 copairs1 [64][64];
    _int8 copairs2 [64][64];

    int i,j,k,l,N,p1,q1;
    char str [DEFAULT_BUFFER_SIZE];
    sprintf_s (str,DEFAULT_BUFFER_SIZE,"searching for copairs %02d-%02d ... \r\n\r\n",p,q);
    strcat_s (outbuff,OUTPUT_BUFFER_SIZE,str);    

    for (i=0;i<_G::MAX_NUMBER+1;i++)
    for (j=i;j<_G::MAX_NUMBER+1;j++) {
        copairs1 [i][j]=0;
    copairs2 [i][j]=0;
    }
    for (N=SEARCH_DEPTH;N>=0;N--)
    for (j=0;j<DRAW_SIZE;j++) 
    if (history[N][j]==p)    // found first number
    {
    for (k=0;k<DRAW_SIZE-1;k++)
    for (l=k+1;l<DRAW_SIZE;l++)
    {
        p1 = history[N][k];
        q1 = history[N][l];
        copairs1 [p1][q1]++;
        if (p1!=q1)
        copairs1 [q1][p1]++;
    }
    }
//    OK BUIlT A LIST OF PAIRS THAT HAVE OCCURED WITH THIS NUMBER
    for (N=SEARCH_DEPTH;N>=0;N--)
    for (j=0;j<DRAW_SIZE;j++) 
    if (history[N][j]==q)    // found second number
    {
    for (k=0;k<DRAW_SIZE-1;k++)
    for (l=k+1;l<DRAW_SIZE;l++)
    {
        p1 = history[N][k];
        q1 = history[N][l];
        copairs2 [p1][q1]++;
        if (p1!=q1)
        copairs2 [q1][p1]++;
    }
    }
//    ALRIGHT NOW HAVE A LIST OF COPAIRS FOR THE SECOND NUMBER
    int found;
    for (i=1;i<_G::MAX_NUMBER;i++)
    if ((i!=p)&&(i!=q))
    for (j=i;j<_G::MAX_NUMBER+1;j++)
    {
    if ((j!=p)&&(j!=q))
        {
        found = 0;
        if ((copairs1 [i][j]>1)&&(copairs2 [i][j]>0)||
        ((copairs1 [i][j]>0)&&(copairs2 [i][j]>1))
    )
    for (k=SEARCH_DEPTH;k>=0;k--) {
        if (found==0)
        {
        sprintf_s (str,DEFAULT_BUFFER_SIZE,"FOUND COPAIR %02d-%02d\r\n",i,j);
            strcat_s (outbuff,OUTPUT_BUFFER_SIZE,str);
        found = k;
        }
        if (history [k].match_copair (i,j,p,q))
        history[k].dump (outbuff,OUTPUT_BUFFER_SIZE);
        }
    }
    if (found!=0)
    strcat_s (outbuff,OUTPUT_BUFFER_SIZE,"\r\n");
    }
//    now search for rings.
    for (i=1;i<_G::MAX_NUMBER+1;i++)
    if ((i!=p)&&(i!=q))
    for (j=i+1;j<_G::MAX_NUMBER+1;j++)
    if ((j!=p)&&(j!=q))
    if ((copairs1 [i][j]>0)&&(copairs2 [i][j]>0))
    for (k=j+1;k<_G::MAX_NUMBER+1;k++)
    if ((k!=p)&&(k!=q))
    if ((copairs1 [i][k]>0)&&(copairs2 [i][k]>0)&&
    (copairs1 [j][k]>0)&&(copairs2 [j][k]>0))
    {            
    sprintf_s (str,DEFAULT_BUFFER_SIZE,"Found ring %02d-%02d-%02d",i,j,k);
    strcat_s (outbuff,OUTPUT_BUFFER_SIZE,str);
        if (history.check(draw(p,q,i,j,k))==true) {
        strcat_s (outbuff,OUTPUT_BUFFER_SIZE," ***");
    }
    strcat_s (outbuff,OUTPUT_BUFFER_SIZE,"\r\n");
    }
}

 And to be quite honest, I have no idea what will happen if I rewrite this so as to be able to chow down on "Alice In Wonderland" or "The Adventures of Tom Sawyer."  Lots of things will need to be parallelized.  Lots of things need to have neuronal weights calculated based on relative probabilities, and of course, a lot of loops need to be replaced with some other kind of iterator that works with ordered or unordered sets, and so on.  Yet this is at the heart of at least one type of method for trying to find the best "co-pair" to match with a given pair of tokens, so as to then try to make a "forward prediction" within a reading frame, as if to also enable "predicting the next word in a sentence."

Maybe.  Right now this is VERY messy.

I was thinking about doing another log entry about Lottery systems, and the probability that certain groups of numbers will come up, so as to take a closer look at certain types of problems, like what is the actual probability that someone's birthday will come up, on any particular Lotto type game, as well as the more general problem with respect to the probability that any particular set of 2, or even 3 numbers might come up, like 3 and 14, of course.  Yet that leads to the more general problem of "how long will it be on average for 3-14 and 15 to come up", depending on the number of balls in the particular game, and the number of numbers picked in a particular draw.  But that will have to wait.  Maybe I will write a short C++ program that prepares a chart for the waiting time for each variation, with fans of e = 2.71828, which can be played on a Lottery as 2-7-18-28, along with "whatever" taking the absolute longest, that is, besides matching all five or six numbers, according to the game.

Instead, I want to look at something else, something that is, that I hope we can get some help with, that is, in the realm of "vibe coding," perhaps.

Yet if in the previous example, I was thinking about how the phrase "I LOVE A MYSTERY" might encode the sequence 5-8-30-14, or else the phrase "I LIVE OUT MY DREAMS" might encode the set 5-8-13-14-30, it is easy to see that neither of these came up on the California Super Lotto last night, even if the five and eight did, but oh so sorry, no 13, and no 14, no hidden pi, or any pie for that matter to be had.  No pi, pie, too bad so sad, and better luck next time?

Or else wait a minute.  I didn't even play the Super Lotto last night, and not just because I am not even in California right now.  At least the 5 and 8, and then I noticed something that I find interesting, nonetheless.  Something that fits in nicely with my examples "DOWN THE PRIMROSE PATH", "DOWN THE HATCH", "DOWN THE RABBIT HOLE" and of course "DOWN THE DRAIN" and that would of course also mean that I should have mentioned something along the lines of "DOWN BY THE RIVER SIDE" as an example of something that can be, and perhaps should be done when we have "lists of words" that we somehow "need" to use in a sentence, or in a composition, as a whole, and where we need to be a bit more flexible with word order, or else when we need to accommodate the need to have an occasional word or phrase inserted into the response.

So I spotted something, something kind of fun, since I was working on a meme that begins with 5 and 8 representing the word "LOVE", yet what about "FLOWERS" which encodes the sequence 8-5-40, which of course does not work, with last night's Super Lotto, yet "A FLOWER WILTS!" does encode the sequence 5-8-45-10. Just as easy to think of as the "easy e = 2-17-18-28", and just as hard to predict, of course!

A flower wilts.
A flower wilts when it has no water.
A flower wilts when it has no air.
A flower wilts in one year!
A flower wilts.

Ah poetry!  Using "random numbers as seed values!"

Easier said than done?

Q.E.D

For now!

Looking at the Fantasy Five Birthday problem, vs. the Power Ball birthday problem, and how to calculate the "Birthday Problem" for any Lotto-type game.  In the California Fantasy Five game, there are 39 numbers to choose from.  So, if someone's birthday was 12-25, then what is the chance that 12-25 will come up on the California Fantasy Five, which, as stated, has 39 numbers, vs. the Power Ball, which has 69 numbers?  According to my classic HP-15 calculator, the birthday probability for the California Fantasy 5 game is approximately 1 in 74.10, whereas the "Birthday Probability" for the Powerball is approximately 1 in 234.60.

One way to do this calculation is, of course, to use the "choose function C" to compute the value C(39,5)/C(37,3) for the California Fantasy Five, or to likewise simply compute C(69,5)/C(67,3) for the Powerball, obtaining the 234.60 value in that case, as stated.  So this would appear that we could find the general case for any Lotto-type game as C(M, N)/(M-2, N-2), which uses either a bit of multiplication and division to get done, or else we could perhaps generalize it a bit further, by discovering a pattern that seems to exist in the calculations, when it is written out by hand, so that it also presents an opportunity to perform some optimization. 

Thus, it would appear that the problem of finding the probability of having two specific numbers come up in a five-numbered draw set actually generalizes to the value: N*(N-1)/20.  This, of course, might have some interesting implications.

With an LLM, of course, we are going to want to replace some Lottery concepts like "draw set" with "reading frame," and we are going to want to think about the idea of grammars that might have 10's of thousands of symbols or more, so that maybe problems of this type are going to turn out to be important when we think about memory allocation, thread pool size, and so on.  Yet drawing from the examples of "down the rabbit hole", "down the primrose path", "down the hatch," and "down the drain", it should be easy to contemplate that there should be some way of developing metrics for measuring the probability that two randomly chosen symbols will occur within some context, so as to be able to do some work in the problem of scaling things like "neuronal weights" associated with relevance, or significance, that is, within the framework of the hopefully more general transformer based attentional systems.

Maybe.  Or we can go back to asking "What happened to the other dollar?"

There is a whole bunch of stuff that I want to do, like talk about the "phone book trick from the movie Rain Man", and how there is an urban legend that it was actually done by the magician and memory expert Harry Lorayne, who was also one of the authors of a best-selling book entitled "The Memory Book."  Even if Lorayne didn't actually memorize the entire Manhattan telephone directory, it was said that he did memorize quite a few pages, just to prove that it could be done.  

This is something that I want to talk about in a different context, i.e., with respect to Lottery systems, whether they work or not.  Yet they will have utility, I think, in finding interesting and novel ways to approach the problem of designing an LLM from scratch.

Since this is going to take a while to explain, perhaps it would be best to start with an excerpt directly from the book, or just simply borrow a description for one popular system for memorizing numbers from the Wikipedia article on the so-called "Mnemonic Major System."

So here is "The system", shamelessly borrowed from Wikipedia:

--------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Now what do we do with it?  Well when taking a short phrase like "I LOVE A MYSTERY", or "I LIVE OUT MY DREAMS", we can easily see that the first sentence encodes the sequence 5-8-30-14 and the second sequence encodes the pattern 5-8-13-14-30, that is to say, if we have a constraint that what we want to do is to try to find a way to create some kind of "magic cookie" as it were, that somehow encodes a set of numbers that can be played on a Lottery game.

Yet that implies something, of course, and that is a whole "new" kind of "mining", that is to say, "new" for most people.  Since the order in which Lottery numbers are drawn usually doesn't matter, for some games, that is, like the Power Ball or the Mega Millions, which creates the situation of perhaps generating "sets of single consonants, as well as pairs and pairs of pairs of consonants" as a starting point, that is to say, for random text generation!  

I mean, sure!  Why not?  It should be easy to see how the word pair "MARTHA COMPONENTS" might be playable on some Lottery game as the set 34-17-39-22-10, if I am reading it right.  Now this implies some things, such as the idea of searching the space that contains every possible choice of the type "1-2-3-4-5" though "43-44-45-46-47" and generating all 120 permutations within that space, and then generate every possible set of split points for each combination and permutation, and then of course do a dictionary lookup to see which combinations have word mappings, and then generate potential "magic cookies."

I don't want to tell you how many CPU's I might have melted, or how many hard drives I destroyed trying to generate text in this way.  Yet maybe now is a good time to explore some interesting possibilities with a modern GPU.

Warning!  This project could melt your PC!  You have been warned!

The way that I figure it, what if I could write 3000 words each day, of "new material" which is somehow relevant to training an LLM, by whatever means?  Then it would take "only" 333.33 days to get to one million words of hopefully, new, original, and just as important, potentially "useful" content.  Yet there are other things that an LLM might be trained on, like MIDI data derived from audio input.

Or patterns in DNA, maybe?

I mean, why not?  If it seems like a reasonable thing to try, and if the advent of vibe coding might somehow make the otherwise gargantuan task of retooling something like 100,000 lines of code, spread across at least 20 libraries, actually seem feasible.  Then maybe that is yet another domain for AI, which is not only highly important, but also relevant, even if at the same time it might seem like such explorations are perhaps being underserved, at least within the Open Source community.

I know that I am going to want to be doing things with "transformers"; however, they actually work.  Yet, there is also the obvious, yet seemingly all too often overlooked, task of getting the data into the program, to begin with.  Maybe that isn't as hard as it might seem.

Yet there is also this idea of generating as much "metadata" as possible, maybe, based on traditional methods.  Not just because it is possible to do, but because, maybe, just maybe, the transformer model would be a lot more efficient if some of the initial heavy lifting were done according to standard coding techniques.

Interestingly enough.  When I took some Atari BASIC code and experimented with using a "genetic" algorithm to approximate a mathematical function, I found that I got better results if I ran 32 models in parallel, and let them compete with each other, let's say for 64 iterations each, than I was getting if I ran a single model for 2048 iterations.

So I think that there are even more avenues that need to be explored, as far as how this sort of thing might work out in the long run.  Clearly, at least in this particular case, a competition among the so called "mixture of experts", multi-model approach, seems to work best.

Right!  Whatever I just said. 

Well, anyway.  

What if this project isn't really about trying to predict the Lottery, and it is especially not about trying to predict any current game using a database of numbers from as long ago as 2012, which hasn't been updated in at least that long?  

No, that's not the point.

That is only the beginning.

Really, an LLM, or so we are told, is not so much about words or numbers as it is said to be about "tokens".  Whatever those tokens represent is, well, whatever they represent.  Maybe macroblocks from a JPEG of a medical X-ray, or maybe waveform data from a seismograph, or "whatever."  That, of course, was the big surprise, with the discovery that "maybe attention is all you need."  Yet somehow, the "pairs of pairs" meme seems to have stuck with researchers as one of the best predictors, at least as far as text generation is concerned.  So does it make sense to try to, in effect, create some kind of "lottery-like" application that increases the number of balls in a particular game to some "reasonable number" like 65535, or something like that, along with an appropriate increase in the size of the "reading frame?"

Maybe.  Maybe there are a lot of things that need to be tried in parallel, and it is just as important to come up with methodical techniques for formalizing the parallelization of whatever processes are being put to the test.  Something else comes to mind, such as whether to try to store the topology of a neural network in a traditional fashion, such as by extending the capabilities of Spice, or Ki-Cad, or whether it is actually much easier to do something like that with a relational database like MySQL . Yet, that of course is where the newfound meme of "vibe coding" might come in handy, that is to say, as things grow, and grow, and grow.  Which they will.

Likewise, we might want to add or delete a source from the training set.

So a "make system" is also needed.

This thing is huge, but not impossible.  

Yes, there will be back propagation and gradient descent.

Maybe even a seemingly infinite number of monkeys, all leaping about.  Having private "conversations" with each other, and so on.

This is an ambitious project.

In the meantime, thank you for choosing Johnny Cab.  Hope you enjoy the ride!