I'm writing this primarily for my future self, who may one day decide to return to the IMX477 driver implementation to add features like HDR support, 4 MIPI lanes, or 4K 60fps mode.

The issue is that the RPI implementation of the driver is locked at a maximum of 10 fps in 4056x3040 mode. While this may be suitable for the RPI with its slow processors and ISP, which can only handle 1080p images, it is insufficient for the RK3588.

https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/media/i2c/imx477.c#L945

To start, I want to provide some basic knowledge about how IMX sensors work.

On the Raspberry Pi forum, there are several topics discussing cameras and MIPI-CSI, but one thread is particularly interesting. It explains how the LINK_FREQUENCY and PIXEL_RATE values in the driver code work, why the Raspberry Pi is limited to 10 fps, and what the HBLANK and VBLANK values are: Raspberry Pi Forum Post.

For better understanding, I included an illustration from the datasheet. As you can see, while the raw image is being transferred, it does not completely fill the space within the packets, leaving free space horizontally (HBLANK) and vertically (VBLANK).

If you examine the VBLANK and HBLANK values using v4l-ctl, you will see that the horizontal blanking value is extremely high, occupying almost four times more space than the line width itself.

root@orangepi5:~# v4l2-ctl -d /dev/v4l-subdev2 --list-ctrls

Image Source Controls

              vertical_blanking 0x009e0901 (int)    : min=252 max=8380960 step=1 default=252 value=252
            horizontal_blanking 0x009e0902 (int)    : min=19944 max=19944 step=1 default=19944 value=19944 flags=read-only

The first thing you should try is to lower the horizontal blanking value. In the driver, this is defined by the line_length_pix constant for each mode and the corresponding registers at addresses 0x0342 and 0x0343.

    {
        /* 12MPix 10fps mode */
        .width = 4056,
        .height = 3040,
        .line_length_pix = 0x5dc0,
....
        .reg_list = {
            .num_of_regs = ARRAY_SIZE(mode_4056x3040_regs),
            .regs = mode_4056x3040_regs,
        },
    },


/* 12 mpix 10fps */
static const struct imx477_reg mode_4056x3040_regs[] = {
    {0x0342, 0x5d},
    {0x0343, 0xc0},

You can lower that value by approximately 10% and achieve around 12 fps at the stock clock speeds (don't forget to adjust timeperframe_min and timeperframe_default). However, further reduction will result in a "broken" image.

My initial thought was that by changing the register values, I somehow broke the horizontal synchronization trigger (as vertically, the frames were still correct!). However, further investigation revealed that the reason was completely different.

To proceed, I need to provide more understanding of how IMX sensors work internally.

While the IMX477 datasheet available on the internet is somewhat limited, it still contains plenty of useful information, including the Clock System Diagram, which explains how the data flow is designed.

On the left side, we can see INCK identified by the EXCK_FREQ. This is a base clock value used in all module subsystems. However, that value can be multiplied or divided by constants like IOP_PREPLLCK_DIV and IOP_PLL_MPY, resulting in multiple different clocks like ADCK, CPCK, IVTPXCK, and so on.

In the middle of the pipeline, there is a buffer called FIFO, which uses the IVTPXCK and IOPPXCK clocks for its operation. Considering that the RPI driver is initially limited by the slow MIPI lane on their processor, I started changing the IOPPXCK clock speed.

IOPPXCK can be calculated as INCK / IOP_PREPLLCK_DIV * IOP_PLL_MPY. While register names are not usually published, one of the developers on GitHub was kind enough to list all of the register names and their corresponding values.

REG_IOP_PREPLLCK_DIV=0x030D,  # The pre-PLL Clock Divider for IOPS ,
REG_IOP_MPY_MSB=0x030E,  # The pre-PLL Clock Divider for IOPS ,
REG_IOP_MPY_LSB=0x030F,  # The PLL multiplier for IOPS [7:0]  ,

I grab those values and wrote a simple tool to easily compare different register sets.
https://github.com/Monstrofil/sony-imx-controls

  REG_EXCK_FREQ             (00310 |    0x136) =>         24 (0x18)
  
  REG_IOP_PREPLLCK_DIV      (00781 |    0x30d) =>          2 (0x2)
  REG_IOP_MPY_MSB           (00782 |    0x30e) =>          0 (0x0)
  REG_IOP_MPY_LSB           (00783 |    0x30f) =>        150 (0x96)

Surprisingly, if you take the register values and perform the calculation 24 / 2 * 150, you get 1800, which is also known as 450 x 4 and was mentioned in the post on the RPI forum. I'm still not sure what that magic constant 4 represents, but throughout the whole process, I had no trouble defining IMX477_DEFAULT_LINK_FREQ as INCK / IOP_PREPLLCK_DIV * IOP_PLL_MPY / 4 * 1000000.

In my case, I changed REG_IOP_PREPLLCK_DIV to 3 and REG_IOP_MPY to 0x0140 (320), which gave me a LINK_FREQ of 24 / 3 * 320 / 4 * 1000000 = 640000000.

And it worked! After changing the LINK_FREQ, I was able to lower the HBLANK space and achieve around 25 fps. However, there was a weird ghosting effect on the right side of the frame.

Long story short, the issue was the balance between the IVTPXCK and IOPPXCK clocks. After I increased the IVTPXCK clock speed a little bit, the ghosting went away, and I was able to achieve a stable 30 fps.

It also turned out that while IVTPXCK is defined as EXCK / REG_IVT_PREPLLCK_DIV x REG_PLL_IVT_MPY / (REG_IOPPXCK_DIV x REG_IOPSYCK_DIV), which is 24 / 4 * 0x015e / (5 * 2) = 210 for the original driver, it also matches the PIXEL_RATE as PIXEL_RATE = IVTPXCK x 4 x 1000000. Again, the constant "4" appears, but this time it multiplies rather than divides the resulting value.

After some trials, I ended up with a 984M pixel rate mode specifically for my new 4K 30fps mode and modified IVT dividers.

#define IMX477_PIXEL_RATE_984M        984000000

  REG_IVTPXCK_DIV           (00769 |    0x301) =>          5 (0x5)
  REG_IVTSYCK_DIV           (00771 |    0x303) =>          2 (0x2)
  REG_IVT_PREPLLCK_DIV      (00773 |    0x305) =>          4 (0x4)
  REG_PLL_IVT_MPY_MSB       (00774 |    0x306) =>          1 (0x1)
  REG_PLL_IVT_MPY_LSB       (00775 |    0x307) =>         94 (0x5e)
  REG_IOPPXCK_DIV

And that finally allowed me to get these images:

While being relatively straightforward, this process actually demotivated me to the extent that I shelved this 360 camera project for almost a year.