LED_height * (LED_with*character_count-1)
I'm settling for a 12 5x7 character display with a row of LEDs between characters. That makes 7 * (6*12-1) = 497 LEDs. Meaning 71 rows, so at least 5 HT16K33 controller, if not better 6 for simplicity, each controlling 12 (except the last 11) columns.
So you are going to need that diode for changing the i2c address after all?