So this took me a while. The library that I'm using stores the matrix data in a one dimensional array with the length of 40 * 16. 

Each LED matrix is 8x8 and are connected in that order: 

0 - 1 - 2 - 3 - 4
                |
9 - 8 - 7 - 6 - 5

 And each LED matrix is wired like this:

0 - 1 - 2 - 3 - 4 - 5 - 6 - 7
8 - 9 - . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . .  63

When I finally found the error in this function, I had already written a lookup table for all the 640 LEDs. This is fixed for my setup, but it should be possible to rearrange it for others.

int matrix_arrangement[] = { 0,1,2,3,4,9,8,7,6,5 };
void club_matrix_render(void)
{
    for (int i=0; i<led_count; i++)
    {
        int led_real_x = i % width; // 0 to 39
        int led_real_y = i / width; // 0 to 1

        int matrix_number_x = led_real_x / 8; // should be 0 to 4
        int matrix_number_y = led_real_y / 8; // should be 0 or 1

        int matrix_number = matrix_number_y * 5 + matrix_number_x; // 0 to 9

        int matrix_x = led_real_x % 8; // 0 to 7
        int matrix_y = led_real_y % 8; // 0 to 7

        ledstring.channel[0].leds[matrix_arrangement[matrix_number] * 64 + matrix_y * 8 + matrix_x] = matrix[i];
    }
}