So after getting tired of messing around with the delay & loop count and running into latency or key chatter, I came across Matt Vilim's Kinesis Controller project. His firmware could handle multiple key state changes, where previously I was sending a HID report per state change.

After porting his firmware the keyboard latency was fixed without incurring any key chatter.

However, I was occasionally getting stuck keys due to debouncing. Luckily I found Stapelberg's more recent project, KinX controller and ported the debouncing algorithm and that seems to have fixed it for now.