It took a couple of days to write a program for avr processor capable of outputting sync signals good enough for the monitor to recognize them as valid.

As I chose to not use pre-made libraries, it takes some time to make things work properly.
As the starting point, I wanted to generate the standard VGA signal, 640*480px @60Hz. After this is achieved, the goal is to test other modes, the standard ones, as well as not so standard -- by playing around with timings and sync signal polarities.

Here is the first image that I generated:

On the left is logic analyser output - the red, orange and yellow traces show R, G and B color signals respectively, while below them are Vsync (green trace) and Vsync (blue trace) signals; on the right is the screen actually showing the image encoded in these waveforms.
This signal was far from perfect - the edges of vertical bands were fussy, and had very noticeable vibration to them. The problem was in the code, of course. After some adjustments and fiddling with it, I got a stable image, though the vertical edges were ragged:

Here is the reason, why the edges looked like this: the Hsync signal is generated by the timer/counter on atmega168 that is operating independently of anything going on on the cpu itself -- essentially I am generating PWM signal, that is very stable, and this way the display monitor is able to lock on it. But the generation of visible pixels is done via software running on this cpu. When the counter reaches pre-set value (508, this at 16MHz the processor is working on is almost exactly the line length for standard VGA signal), it triggers interrupt, and in its service routine the flag is set. This flag is then read/ polled in main loop, and there all the necessary computations are done. While setting the flag by interrupt service routine is consistent, the reading of it via polling loop is not, and this means that the drawing of each line stars with different offset to the Hsync signal, making vertical edges ragged.

After I moved all the line drawing logic into interrupt service routine, the ruggedness was almost gone, although the interrupts are served for almost all the width of drawn line, and vertical edges still showed short comb pattern:

This proved fairly hard to remedy, as the cause is that drawing logic has conditionals in it, and depending on condition, there is different number of cycles the cpu has to do to process them. Finally, with some fiddling with the code, I was able to produce the image with smooth edges - it took inserting another condition, and several 1-cycle instructions such so this combing was canceled out: