Once we had our labelled dataset, we faced a question: should the model learn from colour images or greyscale ones? Greyscale models can be more robust to lighting changes, while colour gives the model more information to work with. Instead of guessing, we trained both.

Our first model, trained on 410 colour images, scored a mAP50 of 0.982. The greyscale version came in at 0.964, which was close enough that neither was a clear winner. We even built a version that automatically switched between the two models depending on the camera feed. After testing both live on the webcam, though, we decided that running two models added complexity without a real benefit, so we went with colour.

Then we pushed further. The judges might pick a different Crayola marker than the one we practised with, so we added 124 new images of markers in different colours, for 534 in total. The score dipped slightly to 0.962, but that was expected: the model was now learning what any marker looks like, not just one specific colour. On real footage it spotted the marker in 18 of 21 frames, and the misses were mostly frames where the marker hadn't come into view yet.

Finally, we trained a combined model that recognises both the marker and the robot arm itself (765 images), which reached a precision of 0.989 and a mAP50 of 0.985. Knowing where the arm is means the system never mistakes its own gripper for an obstacle.