I've already investigated changing line frequency, and line count per frame independently, now it's time to see, how the display takes simultaneous changes. As previously, I am starting with standard VGA 640*480 @ 60 Hz.

Now, the line count and line frequency are changed together in such a way as to keep frame rate close to constant of 60 frames per second. This way, when line count per frame is decreased, the line duration is increased accordingly (thus lowering line frequency), and vice versa.

I am starting with decreasing line duration / increasing line number.

The recognized standard VGA 640*480 persists up to 530 lines / 31.796kHz. 

At higher line frequency and line number fallback to 800*480 occurs.

A switch occurs when line number reaches 559 and line frequency becomes 33.536 kHz -- now, the image is recognized as 1920*540px. This is exactly half of full HD (1920*1080), and I think it is actually the interlaced version 1080i.

Next switch is at 568 lines / 34.076 kHz  -- fall back to 800*480.

At 621 lines occurs switch to LCD native resolution of 1024*600.

At 625 starts to recognize as SVGA (800*600 @60Hz).

At 634 lock is lost, screen is black.

At 740 lines a new lock occurs, the signal is recognized as 1280*720px.

At 779 lock switches to 640*400.

When reaching 819 lines, image size becomes 1280*800.

In the vicinity of 890 lines, switch to 1152*864 -- somewhat strange resolution, I would say.

At 935 lines starts the zone of 1440*900.

Between 975 and 985 lines lies black zone, and then starts 1280*960.

Between 1035 and 1045 is another gap, and then starts 1280*1024.

At 1088 lines starts 1680*1050 @60Hz mode.

At 1115 lines the HD resolution 1920*1080 is recognized. This goes up to 1165 lines, after that switch occurs to (possibly default) 720*400px mode. 

At 1195 lines switch back to 1920*1080 occurs.

Maximum number of lines per frame is 1200, when over this value, "Not support" message is shown. This concludes the upwards sweep.

Going down in line number, starting from 525 of the standard VGA, through 522 lines, the image is recognized as 640*480, and starting from 521, it is now 800*480 @60Hz.

At 491 lines the lock is lost, and black screen is presented. Then, starting from 488 lines, lock is regained with resolution of 720*400.

Seems like the lowest line number per frame is 421, as lower values trigger "Not support" message.

Now, I tried to find out what would be a smallest frame supported, at lowest frequency -- it turns out 720 by 400 pixels, at 49 Hz, with line duration of 48.625 microseconds (frequency of 20.565 kHz). By measuring width of image blocks on screen ("pixels" generated by Arduino board) with ruler, and taking into account that to draw one it takes 11 clock cycles, I came to value of 35.44 microseconds, out of the total 48.625, as the active draw length; and having 720 pixels on it, the square pixel duration would be 49.2 nanoseconds, which means pixel clock should be 20.316 MHz. If using 20MHz crystal, these timings will yield image of 708 by 400 very slightly elongated pixels, which can be corrected in width adjustment menu on OSD of this display monitor.

----------------------------

RESULTS:

The cheap noname chinese monitor with LCD panel of 1024 by 600 pixels, and driven by RTD2660H chip can accept a variety of timings through its VGA port, leading to different display resolutions recognized. The line frequency and the number of lines per frame can be varied smoothly in a wide range, only condition is that resulting frame rate should lie in range between 49 and 75 frames per second. There is caveat in that sometimes, close to boundaries between recognized modes, colors can be incorrect, and some narrow ranges can not be locked on. All the most popular modes (640*480, 800*600, 1280*1024, 1280*720 (HD 720p), 1440*900, 1920*1080 (HD 1080i and 1080p)) are recognized, as well as several others, aligned to display's aspect ratio.