Today I connected all boards together, stacked, turned on... And saw FAIL. Most noticeable bug was inverted output, that is, segments that should be lit are dark, and vice versa. Also last digit is odd, even with all signals the same, it outputs different pattern. (On inspection it turned out several wires connecting small board came out, and after resoldering all four digits have same pattern given the same input.)

Another problem is ghosting; it seems digit driver circuits are too slow.

Some photos:

So it seems I am not finished here, but into some bug hunting and rectifying them until the thing works as it should.