The mane motivation for using the budge library for glxgears64 was it didn't have a general purpose 3D transform.  There could be a way to fake perspective by hard coding the size of the polygons.

The mane problem with the budge library was smoother rotation.  GLXgears needs 128 rotation values to achieve a useful gear visualization.  Keyboard input actually has only 64 rotation values.

The rotation  tables actually have magnitude * sin baked into the same lookup value.  To search the entire space of magnitude * sin, he creates hashes of the magnitude & angle.  A hash of bits 0:3 of magnitude & angle searches 1 table.  A hash of bits 4:7 of magnitude & angle searches another table.  The 2 lookup values are added to get magnitude * sin.  Since the angle only changes once per frame & magnitude changes once per vertex, the hash mangles the angle bits but not the magnitude bits.

Compressing the angle into 4 bits required reducing it to 15 possible values.  It was 15 because presumably, 360 was evenly divisible by 15.  The 15 values are used twice to cover a complete rotation.  1 addition operation saves a lot of memory compared to a single table with all 256 possible magnitudes & all 15 possible angles.  It allows all 16 magnitudes of the least significant nibble to be reused with every value of the most significant nibble without creating unique table entries.

128 angle steps with 16 magnitudes would need a bigger hash with a 6 bit angle joining the original 4 bits of magnitude.  64 * 16 for RotTabLo & 64 * 16 for RotTabHi.  The 4 RotIndex tables would be 256 bytes to address the larger lookup space so another 1024. That's 3072 bytes for the rotation/magnitude tables. 

That arrived at a higher resolution angle table which produced roughly the same values for magnitude 0x7f but could not properly render the image anymore. 

Things fell apart with magnitude 0xd0.  Small negative magnitudes like 0xff looked similar but large negative magnitudes like 0x80 & 0xd0 were falling apart.  Lions hoped negative magnitudes would work themselves out naturally in that nibble algorithm.  Bits 0:3 must have always been magnitudes 0-16 but bits 4:7 were magnitudes -128 - +128.

This got high res rotation working & the framerate was unaffected.  There is a lingering problem of negative magnitudes from -16-0.  These give a different table for the low nibble, but somehow the number theory manages to solve negative magnitudes with only bits 4:7 being signed.  -113 * cos(0) or 0x8f resolves to 0x80 from RotTabHi + 0x0f from RotTabLo.

Helas, the video made it obvious that some table values were repeated, creating a stuttering.  It's not so easy to see since the emulator has stuttering of its own.  The total angles has to be 127 instead of 128.  Still, these demos show a level of 3D performance on the commodore we never imagined 40 years ago.