I know that the weakest part is currently my 1st LO so I tried to check what will happen if I replace it with a better synthesizer. I used temporarily STuW81300 which is single chip synthesizer with integrated VCO. It improved results noticeably. All random spikes disappeared from displayed spectrum and it was possible to achieve resolution down to about 1kHz.

STuW81300 on 1st picture (resolution down to 1kHz), my synthesizer on 2nd picture (no comment):

I also discovered that there are some unexpected frequencies about 30kHz away from the measured frequency. I couldn't figure out where they come from. I tried many changes: different power supply (linear instead of switching), changing LCD clock, slightly tuning main oscillator, tests at different input frequencies, different source of signal, different 2nd LO, different 1st LO (I couldn't sweep from microcontroller so I was sweeping it manually while observing level), removed input attenuator module different 5V for 1st IF amplifier. All the time there was some 30kHz modulation. When I provided 2nd IF (228MHz) signal from professional synthesizer  to input of IF board and swept manually I didn't notice anything 30kHz away from center frequency so it seems that the 30kHz adds somewhere in 1st and 2nd mixers board.
UPDATE: I think I found that - my signal generator (nanoVNA v2) generated the modulation or jitter and and also 2nd LO generated similar modulation or jitter. So as long as I tried to change one thing every time I saw the same effect but from different sources. Conclusion: use different source instead of nanoVNA v2 and improve 2nd LO.

I did some testing to see where noise on IF board comes from. It seems that IF amplifier makes the noise (of course, 1st stage is critical). When the board is shielded, there's not much difference if I completely disconnect input of the amplifier. Maybe some changes of amplifier circuit will help but I don't expect that big improvement is possible. Improving IF filters to loose less signal may be more effective.