In this section I will show you the programming of the Nicla Vision, and the MKR WAN 1310 in the Water Sprayer system. We will also test the prototype.

Programming The Nicla Vision

Below I show you the code for the Nicla Vision board:

# AUTHOR: GUILLERMO PEREZ GUILLEN

import sensor, image, time, os, tf, pyb
from machine import Pin

pin0 = Pin("PG1", Pin.OUT_PP, Pin.PULL_UP)
pin1 = Pin("PG12", Pin.OUT_PP, Pin.PULL_UP)

redLED = pyb.LED(1) # built-in red LED
greenLED = pyb.LED(2) # built-in green LED

sensor.reset()                         # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565)    # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)      # Set frame size to QVGA (320x240)
sensor.set_vflip(True)
sensor.set_hmirror(True)
sensor.set_windowing((240, 240))       # Set 240x240 window.
sensor.skip_frames(time=2000)          # Let the camera adjust.

labels, net = tf.load_builtin_model('bee_or_spider_v2')
found = False

def flashLED1(led): # Indicate with LED when target is detected
    found = True
    led.on()
    pin0.on()
    img.draw_string(5, 12, label)
    pyb.delay(1500)
    led.off()
    pin0.off()
    found = False


def flashLED2(led): # Indicate with LED when target is detected
    found = True
    led.on()
    pin1.on()
    img.draw_string(5, 12, label)
    pyb.delay(1500)
    led.off()
    pin1.off()
    found = False

clock = time.clock()

while not found:
    clock.tick()
    img = sensor.snapshot()
    for obj in tf.classify(net, img, min_scale=1.0, scale_mul=0.8, x_overlap=0.5, y_overlap=0.5):
        print("**********nPredictions at [x=%d,y=%d,w=%d,h=%d]" % obj.rect())
        img.draw_rectangle(obj.rect())
        predictions_list = list(zip(labels, obj.output()))
        for i in range(len(predictions_list)):
            confidence = predictions_list[i][1]
            label = predictions_list[i][0]
            print("%s = %f" % (label, confidence))
            if confidence > 0.8:
                if label == "bee":
                    print("It's a BEE")
                    #img.draw_string(5, 12, label)
                    flashLED1(greenLED)
                    #pin0.on()
                if label == "spider":
                    print("It's a SPIDER")
                    #img.draw_string(5, 12, label)
                    flashLED2(redLED)
                    #pin1.on()

    print(clock.fps(), "fps")

How does it work? 

1. This code is similar to the code from chapter 5. The bee, spider and unknow prediction scores are printed through the serial port;
2. The conditional to activate the detection of a bee or a spider must be greater than 0.8;
3. If the camera detects a bee, then the green LED lights for 2 seconds and prints the message "It's a BEE" on the serial port; 
4. Additionally, pin0 (PG1 port) turns ON for 1.5 seconds and then turns OFF;
5. If the camera detects a spider, then the red LED lights for 2 seconds and prints the message "It's a SPIDER" on the serial port; and
6. Additionally, pin1 (PG12 port) turns ON for 1.5 seconds and then turns OFF;

Troubleshooting and solution: 

To make this script with MicroPython, I faced the problem of programming the output ports of the Nicla Vision since there is not enough information. Fortunately in the OpenMV discussion forum I found the solution: https://forums.openmv.io/t/controlling-gpio-pins-of-arduino-nicla-vision-board-using-openmv-ide/7533

Programming The MKR WAN 1310

Below I show you the code to upload to the MKR WAN 1310 board, but you can use any arduino board.

/*
AUTHOR: GUILLERMO PEREZ GUILLEN
*/

// We include the library to be able to control the servo
#include <Servo.h>

int Pin1 = 4;
int Pin2 = 5;

// We declare the variable to control the servo
Servo servoMotor;

int bee = 0;
int spider = 0;

void setup() {
  // We start the serial monitor to display the result
  Serial.begin(9600);
  // We start the servo so that it starts working with pin 6
  servoMotor.attach(6);
  pinMode(Pin1,INPUT);
  pinMode(Pin2,INPUT); 
  // initialize the servo
  servoMotor.write(100);
  Serial.println("ITS BEE OR SPIDER?");
}

void loop() { 
  if (digitalRead(Pin1)==HIGH){
    Serial.println("It's a BEE");
    bee++;
    Serial.print("BEE = ");
    Serial.println(bee);  
    Serial.println("--------------------");    
    delay(1500);
  } 
  else if (digitalRead(Pin2)==HIGH){
    Serial.println("It's a SPIDER");
    spider++;
    Serial.print("SPIDER = ");
    Serial.println(spider);    
    Serial.println("--------------------");
    servoMotor.write(175);
    delay(1500);
    servoMotor.write(100);
  }
  else {
    delay(200);  
  } 
}

How does it work?

1. Pin1 (D4) is used to detect the bees, so a counter keeps track of the bees and prints it on the screen each time it is updated;
2. Pin2 (D5) is used to detect the spiders, and in the same way a counter keeps track of the spiders and prints it on the screen each time it is updated;
3. Additionally, the servo (D6) is activated to press the water sprayer for 1500 milliseconds and then release it.

Test

Below I show you a screenshot during the test carried out with this interesting prototype. You can appreciate the bee and spider counts displayed on the serial port.

And a zoom to the water spray bottle when it emits a waterjet.

Below I show you the video during the tests carried out.

 I also show you a video capture on PC so you can see what is happening in detail.