I was turned off by wasting a $200 raspberry pi 5 on a remote KVM and was curious if an ESP32 could do the same thing at a fraction of the cost. So after much fiddling, this project uses an ESP32 P4 (Rev < 3) and a Toshiba TC358743 HDMI to CSI adapter as an Remote IP KVM.
According to espressif's documentation, the esp32-p4 should be able to do this just fine using the build in h264 encoder which should be capable of 1080p@30 streaming. Unfortunately I learned that the esp32-p4 is currently stuck on an EOL revision, 1.3 and the new revision (3+) is still in sample status and modules are not readily available.
I ended up using MJPEG instead of h264 and I was able to achieve 20fps.