I saw a similar project about 6 years ago at Chaos Communication Congress in Berlin - so I can't take credit for the solar-cell-as-sensor idea. (Unfortunately, I don't remember who made that - some university I think. It was a bit strange architechture as well - realtime patched Windows PC with Matlab/Simulink etc.)
The algorithm is not that sophisticated - no modeling involved. 2 cascaded control loops - the outer one has wheel speed as input and ball position (solar cell target voltage) as output, the inner one has solar cell voltage as input and motor pwm as output (plus a feedback from motor speed to pwm - so the output from the inner controller is closer to acceleration than speed). Whole controller structure:
The spinning motion in the video comes from the fact that the feedback from motor speed was tuned at higher speeds and less load (table tennis ball). So for the steel ball I have quite a big control deviation (distance of the red an green line) on the inner controller. The i part of the outer controller "learns" this away, and after the spinning direction changes, the deviation is in the opposite direction:
Clever use of solar cell as sensor.
Maybe you could use short circuit current of solar cell as input, as current is linear function of illumination as opposed to voltage which is logarithmic function. I think you would gain a lot on sensitivity.
If you want more information about subject, you can go on:
http://pveducation.org/pvcdrom/solar-cell-operation/open-circuit-voltage for voltage info
http://pveducation.org/pvcdrom/solar-cell-operation/short-circuit-current for current info.