Unlimited tile world on a Commodore 64

Young lion knew a lot about sector access on the 1541 but not anymore.  There were undelete tools & disk editors in Compute magazine.

There are some bits about raw sector reads.

https://codebase64.org/doku.php?id=base:reading_a_sector_from_disk

https://wpguru.co.uk/2016/01/how-to-use-direct-block-access-commands-in-commodore-dos/

https://www.lemon64.com/forum/viewtopic.php?t=55010

I/O doesn't get as much love as graphics & sound, probably because the mane focus of retro computing is games & much I/O has been replaced with RAM expanders or custom hardware. 

You have to specify the track & sector.  The tracks start on 1.  Sectors start on 0.  Track 1 sector 0 is byte 0 of the .d64 file.  The last sector on track 1 is 20.  The number of sectors changes based on track.  The world table of contents thus needs to define tracks & sectors instead of monolithic sectors.

track       # sectors
1 – 17      21     
18 – 24     19     
25 – 30     18     
31 – 35     17     

-------------------------------------------------------------------------------------------------------------

It's highly desirable to have the world map on drive 9 & the mane program on drive 8.  The mane problem for animals is lockups after calling CHKIN for drive 9.  You have to read a status register $90.  If bit 8 is 1, no device is attached. 

~/.vice/vicerc needs to have a line enabling drive 9 to get around the lockup

Drive9Type=1541

Then x64 needs a -9 argument to attach a disk.   Noted the -attach commands don't work.  Only -9 works.

-----------------------------------------------------------------------------------------------------------------------------

There are some long delays when calling OPEN.  Some of the internet notes it's waiting for the device to assert a clock signal.  It also has a hard time printing to the printer while accessing  a disk.

Noted in the asm version of opening the data channel

open 5,9,5,"#"

You have to call SETNAM to set "#" as the filename.

The only way to read a sector was to call OPEN for the control channel with "u1 2 0 1 0" as the filename.  With the CHKOUT API, it's not possible to read consecutive sectors without calling OPEN & CLOSE for each sector & having a long delay.  A lower level API is required to have any hope of asynchronous reads.  The big challenge with what young lion envisioned was the simultaneous loading of tiles in the background while scrolling the world map.

It seems the serial API only goes 1 way at a time.  You can't CHKOUT & CHKIN simultaneously to print to a debug port while reading from a disk port.   It would be easier if the screen could be used for debug output.  Then CHROUT could simultaneously write to the screen while CHRIN read from a serial port.

https://www.pagetable.com/?p=1031

https://retro-bobbel.de/zimmers/cbm/programming/serial-bus.pdf

There are some notes about using a lower level talk/listen API instead of the CHKIN/CHKOUT API but no examples.

There's a reference manual for the 1541

https://www.mocagh.org/cbm/c1541II-manual.pdf

The kernal source code is

https://github.com/mist64/c64rom/tree/master

Inspecting the kernal source code & peeking addresses revealed how to use the talk/listen API. Basically call LISTEN with 9 to access drive 9.  Call SECOND with (15 | 0x60) to specify the secondary address for the control channel. Call CIOUT to send the u1 command to the control channel. Call UNLSN to execute the command. The mane trick is the secondary address sent to SECOND has to be ored with 0x60. The OPEN workflow is still necessary to open the drive but it's only needed once in the program lifecycle.  Then, the talk API can send consecutive "u1" commands to read sectors.

This API was able to read all 256 bytes from each sector.  Some sources say only 255 bytes per sector are accessible.

Also noted UNLSN is only required once to execute the u1 command.  No other I/O operations need to call UNLSN or UNTLK.

The key development is going to be a polling mechanism for drive communication & a state machine which can be interleaved inside the drawing routines. It seems to bit bang the serial port.  It's a synchronous serial port but it also requires disabling IRQ's.  This could limit the granularity of multitasking.

The more time gets sucked into this obsession & the more lions desire access to logic signals not possible with emulation, the more enticing a vintage commodore becomes.  The only thing currently produced which has the original logic signals is the EVO 64 & it has a $340 tag.

https://retrospective.shop/evo64-rev1-diy/

The bare board with no IC's is $285.  The problem with this is it surely has bugs because it's a much lower volume.  It doesn't have the RF modulator.  Lions still probably have ancient WinTV PCI boards which could capture RF video but nothing that supports PCI.

There are still commodores in unknown states of disrepair on fleebay for a lot less money.  The mane problem is various IC's are known to have perished & the peripherals are all dead.  They also take a lot more space than the bare board.  The cases could be discarded but lions are loathe to waste anything.  The lion kingdom would have to bodge a keyboard, disk drive, & joystick replacement, probably out of an arduino.

The fleebay offerings show how so much of the internet has become spectators.  There just aren't that many animals into buying retro artifacts.  If they buy the stuff they see on the tube at all, they're never using it.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Switching to bitmap mode increases the tile size to 2500 bytes.  The maximum world size drops to 68 tiles.  Fortunately, the current art can fit in 64 tiles.  The ocean sections would have to be chopped.  To get the maximum scrolling speed in bitmap mode, there aren't enough clockcycles for any significant decompression.  It can only service a serial port interrupt for the disk drive.  It almost needs some form of multitasking.  There could be reuse of tiles but that wouldn't meet the goal of unlimited tile worlds.

There could be RLE decompression on the fly, in the serial port interrupt.  A pure line segment world is coming in under 64 chars.  There could be codes to repeat the last char a certain number of times.  A simplified variable length coding could be a big win, where most commonly used characters have a 4 bit code & less commonly used characters have an 8 bit code.  Because the most repeated characters are also the most commonly used & clockcycles are limited, RLE might be the only useful route.

Character set optimization is a difficult problem.  It has to search all previous characters with all combinations of colors.  If it creates a new character, it has to delete all previous characters which the new character can replace with a color combination.

With RLE encoding, the test world had a theoretical maximum of 380 tiles or 19x19 tiles.  It's pretty low detail though. The largest tile was 652 bytes.  A table of contents pointing to the start of each tile has to go in the mane program.  The tiles have to be padded to multiples of 1 sector to make seeking practical.  Space could be further optimized by packing multiple tiles in a single sector & eating the extra read time.  Without padding, the world grows to 23x23 tiles.

It takes 2 seconds for an unoptimized 1541 to load 652 bytes at 400 bytes/sec.  It's expected to take 5 seconds to scroll 1 tile horizontally & 4 seconds to scroll 1 tile vertically.  It would need a fast loader capable of random sector access if the tiles got more detailed but testing doesn't require it. 

As far as lions know, no 1541 replacement today replicates the head seeking time.  We'll never know if the tile loading would have been fast enough.  In the day, the tiles would have had to be stored in the same relative positions on the disk as the world to minimize head seeking.  Neighboring tiles would need neighboring tracks & sectors.

The constrained line drawing tool got the giant, complicated island borders drawn.  It still took a long time, but it would have been impossible the old way.

The zoomed out image looks surprisingly smooth on today's monitors, but the commodore would in reality be zoomed in on a tiny 320x200 pinhole.   There isn't much detail in each tile, but the scale was dictated by the minimum player size which could convey enough detail.  Most tiles won't have any clues to where the player is.

Color assignment is a big problem.  This world was sketched back when lion memory had 3 unique colors per cell with 1 global background color.  That was true for bitmap mode but not character mode.  Character mode allows primary colors for the foreground color & pastel colors in 3 background colors.

It seems the entire world needs the same 3 background colors.  You can't change background colors between tiles because where a background color changes, there has to be 1 tile which doesn't use the background color which changed.  Blue Max did this by only having green & brown in the border tiles.  The blue/grey color was not used in the border tiles & the border tiles ended up looking sparse.

Young lion loved light green, which costs 1 background color.  If light green, brown, grey are the background colors, the foreground has to be black, white, blue, yellow, light blue, not possible with black lines.  If black lines are taken out & the buildings are all grey, the challenge becomes just white & blue in the same cells but it looks terrible without black lines.  A raster interrupt could allow every row to have different background colors, not very useful.

There are ways to count clockcycles & change color palettes by column. Can't do anything else during the vertical scan.

If it's going to be anything close to what young lion imagined, it has to be bitmap mode.

Some experiments with loop unrolling showed it could get bitmap scrolling up to acceptable levels, but it might run out of memory or have to abandon page flipping.

A minimal constrained line drawing program ended up being quite difficult.  The algorithm ended up requiring a lot of lookup tables to determine the possible lines from every starting point.  The result will hopefully be art that can be created with a commodore character set & there will be enough characters to define all the possible lines.  The tool can only generate a PNG with the line art or a screencap.  A past lion might have grown it into a full character based paint program.

The details possible with this tool would be impossible by manually entering 1 character at a time or drawing in PC paint, as would have been done 40 years ago.  It would have taken forever.  The tool would have been much harder to write for a commodore in those days & 10 year old lion would not have been able to program it, but it's fun to imagine what could have been in those days.  10 year old lion knew nothing of the math operations.

There are also a lot of simplifications young lion didn't know about, like using the keyboard & text console for input in a full time project display instead of a menu system.  Programs which used keyboard input all flipped between menu screens.  Most by 1985 only had graphical menu input.  Young lion didn't believe the more primitive ways had any future once the graphical ways arose.  He could have also used the VIC1525 for a text console but like young animals today, he was more prone to following common practices than leading.

At least the erase tool is resizable.  Despite all its successes, the GEOS paint program only had a fixed sized erase tool.  It always annoyed young lion that the tools used fixed sprites instead of XORed brushes.  You most often used the paint tool's variable size brushes to erase.

Drawing double wide multicolor pixels again & all the tricks to account for that in the algorithm tickled the original memory of discovering fat pixels on a CRT.

Playing around with the bitmap scroll demo, it didn't seem horribly slow & the color sacrifices in character mode were pretty bad.  It could have a 2x scrolling mode.  But the low fidelity doesn't justify more than character mode.  The Blue Max strategy of changing color palettes over water & land or based on current tile would eliminate the need for more colors.  A text file defining rectangles with the background colors might work.

The problem is the amount of time required to make the most minimal game artwork is not on a path to ever being finished.  Just a rough SDF-1 sketch with no elevation information, no greeblies, not drawn to scale took 2 hours.  Overlaying TV show art or freepawing before drawing character line segments seems to help. 

The map evolved to constrained line art on an 8x8 grid.  A character set with just line segments of different slopes & a drawing system based on these constraints might work. Line intersections are difficult.  

Facing the prospect of converting the complicated island outline to line segments, it became clear that a custom drawing program for drawing constrained lines on top of a photo is required or it's never going to be finished.  The constrained wireframe image can then be loaded in Gimp to fill colors.

There's no doubt if it paged instead of scrolled, RLE compressed bitmaps would be the way to go.  A modern game engine could just copy assets from the TV show.  Goog uses parallax distortion so copying their maps won't work.

--------------------------------------------------------------------------------------------------------------

The ideal way to store a map would be raw sector access.  The commodore could access any sector on disk the same as a modern block device. The other formats either didn't have random access or reserved a lot of memory for an inode & directory, which isn't very efficient if you're only reading.

It seems concurrent disk I/O while running a program was achieved with a serial port interrupt handler.  It wasn't free.

The key need is an algorithm which compresses a PNG into an optimum character set & color map. 

Using character graphics with its limited colors simplifies the map compiler.  It needs to just match every 8x8 with a growing character set.  The 3 background colors are fixed.  It only needs to assign the 4th color to color memory.  Then it needs to write a double size proof showing the character codes in every 16x16 so the user can fix bugs.

If it used multicolor bitmap mode, the map compiler would have to search for an existing character which could match the current 8x8 by reassigning 3 colors.  It could save memory.

Because of the limitations on Y scrolling, smooth scrolling would be only in the X direction.  There are ways to use the raster interrupt to scroll both ways in Y, but they reduce vertical resolution to 23 rows.  It's doubtful if reducing X to 38 rows is worth it, since lower frame rates & choppy scrolling have gotten normal in the last 40 years.

----------------------------------------------------------------------------------------------------------------------------

They achieved vertical scrolling by showing a score panel on the blanking area.  That also had a large looping map.

This world map looked like 100% bitmap drawing with no page flipping.

Multicolor character mode was more limited than lions remember.  Each character has only 1 unique color & it can only be the 1st 8 palette colors.  The other 3 colors in each character are global background colors.  5 bits in color RAM are discarded.  It wasn't much better than 4 color CGA.

Only multicolor bitmap mode allowed 3 unique colors in each 8x8 with a global background color.  The high 4 bits in color memory were never used.

Scrolling by copying the bitmap cells without page flipping was pretty sheered but a reasonable frame rate for the time.   All the smooth side scrollers were using multicolor character mode.  Double buffered bitmap mode is going to suck RAM.

After a heroic assembly language effort, there was another demo with page flipping & cursor key input, but still only copying bitmap memory.  Managing tiles & copying 2kb of color memory would probably lower the speed another 33%.  This speed might have been viable in a slow moving, walking adventure game.  There could be a raster interrupt handler to get smooth scrolling.  

Another way to speed it up is to scroll 1/2 page at a time.  That would involve quite a pause.  Neither method would be fast enough for a playable fighter jet game.  Because of the speed cost, bitmap mode would be better off using 1/4 screen bitmap tiles instead of characters.

Zaxxon was the best looking side scroller in lion opinion.  It obviously used character mode.  Grey, blue, red were the background colors.  Black, yellow, white were the color memory.  Dithering simulated light blue.  It's not intuitive for black to be local but it allowed a flame effect.  The limitations made a lot of objects hollow black outlines.

Blue Max had blue, grey, green, brown in background colors.  It changed between grey & blue when moving between land & river scenes.  Black was in color memory based on the explosions not changing it.  Young lion distinctly remembered the look changing between the land & river.  The road changed between brown & grey to conserve colors.

Interestingly, Zaxxon moved 25 rows in 7 seconds.  Blue Max moved 25 rows in 3.4 seconds.  The scroll test moved 25 rows in 3.7 seconds.  Diagonal movement made those games look faster than they were but would entail copying a 2nd character strip.  Zaxxon was damn slow.  Noted Y scrolling on the commodore could only go 1 way because  it only blanked 1 row.  Only X scrolling could go 2 ways because it blanked 2 columns.

Young lion envisioned a universal game engine with cinematic fidelity, but clearly the game engine would have to pick 1 compromised method to fit the gameplay & not be universal.  It would be visually similar to every other side scroller.

What's really needed is hello world for the serial port, since you can't debug graphics by printing on the screen.

Lions kind of dread having to get too involved in the specifics of C64 programming.  It's kind of expected nowadays to have libraries abstract all that.

For its fame in retro computing, there's hardly any developer documentation.  chatgpt can produce very little useful assembly output.  There's no stack overflow for it, very few programming examples. The glxgears demo was possible because internet examples for graphics abound, but not for I/O.

CC65 has a graphics library tgi.h but it just draws text, polygons in high res bitmap mode.  There's a device library dio.h which allows random sector access.  There are standard C functions in stdio.h for accessing files.  There's a debug library dbg.h which just dumps RAM.  There's a decompression library lz4.h.  There's a serial port library serial.h which supports single character I/O.  Don't think these are all implemented for every target.   The only real magic seems to be in cbm.h

cbm.h has C64 functions for file I/O: cbm_open, cbm_load, cbm_save but no seek.  It has a few kernal functions: cbm_k_ckout, cbm_k_bsout, cbm_k_chrout, cbm_k_getin, cbm_k_load, cbm_k_open, cbm_k_talk.  It looks like you have to make some chrout calls to seek.  It doesn't really abstract anything.

The addresses of the kernal routines are abstracted to names in cbm_kernal.inc

There are some hits for printing the serial port output on the host: https://vice-emu.pokefinder.org/wiki/RS232#connecting_to_local_modem_simulator

The kernal used a device number to select the serial port for character output:  https://www.c64-wiki.com/wiki/Device_number

There's a better reference for the kernal: https://sta.c64.org/cbm64krnfunc.html

Maybe it would revive a positive memory to have the original reference manual, smell the glossy paper while cleaning up the hanging chads created by the plastic binder.

http://cini.classiccmp.org/pdf/Commodore/C64%20Programmer%27s%20Reference%20Guide.pdf

 A PDF of it is easy to obtain nowadays but not searchable.

https://archive.org/details/c64-programmer-ref/mode/2up

A hideous javascript viewer with limited searching exists.

The mane things not in it were always subroutines in BASIC & KERNAL routines, a map of memory below byte 1024.  Young lion only ever read the beginning of the manual which covered graphics.   Old lion now focused on the end of the manual which covered I/O.

Sadly, getting serial port text out of VICE proved intractable.  There seemed to be a mismatch in baud rate or a missing step.  It might be easier to have a physical commodore with logic signals that could be probed.  The RS232 signals were made with shift registers on the CIA chips.  Waveforms from physical hardware would allow an animal to make a custom serial driver or use bit banging, but it's not possible with emulation.

After much browsing of the reference manual, VICE was able to print hello world to the console through printer emulation.

.autoimport    on              ; imports C library functions
.forceimport    __STARTUP__ ; imports STARTUP, INIT, ONCE
.export        _main           ; expose mane to the C library
.include        "zeropage.inc"
.include        "cbm_kernal.inc"


.segment    "CODE"
.proc    _main: near

; open the printer page 338
    lda #1 ; logical number
    ldx #4 ; device number
    ldy #7 ; secondary address
    jsr SETLFS

    jsr OPEN

; direct CHROUT to the printer
    ldx #1 ; logical number
    jsr CHKOUT

; print something
    ldx #$00          ; initialize X register for indexing
LOOP:
    lda MESSAGE,x     ; load the character from the message
    beq DONE          ; if character is zero, we are done
    jsr CHROUT         ; call CHROUT routine to send the character to the serial port
    inx               ; increment X register
    jmp LOOP          ; repeat the loop

DONE:
; direct CHROUT to the screen to print ready on the screen
    ldx #0 ; logical number
    jsr CHKOUT
    rts               ; return from subroutine


MESSAGE:
    .byte "hello world"
    .byte $0a
    .byte 0           ; null terminator for the message


.endproc

Getting VICE to print hello world on the console required editing ~/.vice/vicerc.  The key options were

PrinterTextDevice1="/dev/stdout"

Printer4=1

PrinterUserport=1

The text needs to end in $0a to get the buffer to flush.  The assembler converts the text to PETSCII. Then the emulator converts the printer output to ASCII.

It seems the printer was the mane way an ordinary consumer would have debugged graphics programs in the old days.  Berkeley Softworks actually cross compiled on UNIX workstations.

---------------------------------------------------------------------------------------------------

Kind of impressive what kinds of 2D tile worlds were created back in the day with just character sets, character objects, & the available RAM.  Seeing the caliber of 2D worlds that were created kind of defeats the appeal of making yet another one.

Always wondering if a disk mechanism is necessary to achieve the desired map.

Another childhood dream was an unlimited world that dynamically loaded tiles from disk in the background as a user navigated it.  Games in those days loaded the entire map into RAM, which made for some small worlds with few details.  Loading tiles from disk would increase the world size from 48kb to 170kb.  Conceivably, a small part of RAM could be constant, making a somewhat over 200kb world possible without swapping disks.

A small concept of Macross island was enlightening.  Even 40 years later, lions still found exactly the same simplifications to be the easiest way to draw it.  1st of all, it would have to be character based in order to scroll reasonably smoothly.  It could load data from disk in the background, but drawing it needs to go fast for it to scroll.  Methods involving bitmap mode, compositing reusable bitmap objects would be too slow.  They would require moving a full page at a time.

At the scale required for a reasonably distinguishable player sprite to navigate it, the world map would be low enough in resolution to only require an enhanced multicolor PETSCII.  The player would have to be 5 characters tall.  Also, the amount lions could afford to invest in artwork would seriously limit the amount of detail.

At most, the global character set would have greeblie characters for metal seams, windows, port holes, checkerboard characters to add detail to the ground, higher detail segments like the battleship bridge, trees, waves.  Another revelation was how low the detail in the TV show was.  It wasn't far beyond PETSCII.

Step 1 would be freepaw sketching the world in the required dimensions, in a 2880x1600 image. Then assembling the world on a grid overlay out of reusable 8x8 character cells.  Then a program would generate the character set & tiles from the world image.

An elevation collision map of equal size would have to be made.  If the elevation map had just 4 possible elevations, it would add 250 bytes to each tile.  170k would store 77 tiles.  Each tile would be 1000 bytes for character memory, 1000 bytes for color memory, & 250 bytes for elevation.

77 tiles would give 9 tiles by 8 tiles.  RLE compression could be a big enough win to justify any speed impact.  It really has to fit all the updates to screen memory in a raster interrupt while devoting the rest of the time to decompressing tiles into offscreen memory.  If it needs 4 offscreen tiles, 1 onscreen tile, memory for sprites, character set, it could have 32k of leftover memory to store 14 more tiles.  It would be a 10x9 world instead of a 9x8 world, diminishing returns to be sure.

The Commodore couldn't read raw blocks from disk without diabolical hacks to the drive firmware.  They normally had to be in a .REL file in a filesystem.  There has to be a memory resident directory translating tile numbers to disk blocks.

No matter what, it would be a very degraded world compared to what was in young lion's mind, partly because of the limitations of the technology & partly because a complete world with the lowly detail of the TV show would have been impossible to sketch out even by a child lion with unlimited time.  Drawing a 2880x1600 image on any home confuser of the time was a tall order.  A 360k floppy could probably store it with compression, but software of the time couldn't edit it in 640k of RAM.  Even if it was done on a PC, the output couldn't be transferred to the commodore.