Present! Availability sensing for Zoom meetings

After assembly, the code was uploaded to the Arduino and the python script was run, and everything was working perfectly. Video below.

Future improvements - add a choice to choose between apps (Teams / Skype etc.) since it only works with zoom as it is. 

The STLs were printed out, and apart from a few hiccups here and there, the print was okay. Not the BEST quality, but not the worst either. with local travel restrictions, it was hard to find tools to finish the print, but all in all, not bad. everything fits, and the clamping works well. hinge works well. Top holes are used to change the PIR settings. 

Now, we turn to trusty solidworks, Needed something that is easily printable with my ender and something that resembled a webcam, so I went with this design. Designing the clip was interesting, and hope that it works, and is flexible.. I only have PLA at the moment, and not sure whether it will be flexible.

I'll upload the STL files.. and On to printing, assembling and testing it out. 

Now that the micro controller is programmed, Remainder is to code for reading the serial input and turn the camera and mic on/off depending on the situation. Needed to find some way that Zoom application can be manipulated thru code. As an API for zoom did not exist, first thought was to turn the camera and mic off in device level. but, that seemed like an inefficient way, as there could be issues when reconnecting the devices to zoom app (automatically). the only remaining option was - automation. 

Python seemed like the best choice for this. simply due to the support , ease of use, and familiarity. There was no way to get the initial status of the camera and the mic, therefore, a simple GUI was created to launch the application and to see whether the camera and mic is open when the app is open (user input).

GUI indicates whether the app is started or not, and a radio button to indicate the status of the camera/mic. The code is below

import time
import tkinter
import tkinter as tk
from tkinter import messagebox
from threading import *
import sys,os
from main import PresentEx
from PIL import ImageTk


def Confirm():
    # print("Thread")
    if(t1.isAlive()):
        messagebox.showwarning(title=None, message="Program is already running")
        labelRunning['text'] = 'Running'
        labelRunning['bg'] = 'green'
    else:
        labelRunning['text'] = 'Not Running'
        labelRunning['bg'] = 'red'
        if (var.get()<=0):
            messagebox.showwarning(title=None, message="Please Select camera/audio status")
        else:
            Threading()
            labelRunning['text'] = 'Running'
            labelRunning['bg'] = 'green'

def work():
    MainCode = PresentEx(var.get())
    MainCode.main()

t1 = Thread(target = work)

def Threading():
    global t1
    t1.start()
    # t1.join()
    # time.sleep(5)

def Exit():
    global window
    global t1
    window.destroy()
    window.quit()
    os._exit(1)
    t1.terminate


window = tk.Tk()
window.title('Present!')
window.geometry
window.resizable(False,False)
screen_width = window.winfo_screenwidth()
screen_height = window.winfo_screenheight()
window_width = int(screen_width*0.2)
window_height = int(screen_height*0.1)

window.geometry(f'{window_width}x{window_height}')
var = tk.IntVar()

CameraStatus = tk.Label(window, text= "Is the camera and audio off/on right now?").place(x =(window_width*0.01),y=(window_height*0.09))

p1 = ImageTk.PhotoImage(file ='meeting2.png')
window.iconphoto(False, p1)

labelRunning = tkinter.Label(window, text="")
labelRunning.pack(side=tk.LEFT, pady=30, padx=10)
labelRunning.place(x=window_width * 0.02, y=window_height * 0.65)

if (t1.isAlive()):
    labelRunning['text'] = 'Running'
    labelRunning['bg'] = 'green'
else:
    labelRunning['text'] = 'Not Running'
    labelRunning['bg'] = 'red'

print("thread Status")
print(t1.isAlive())

R1 = tk.Radiobutton(window, text="On", variable=var, value=1)
R1.pack(side=tk.LEFT,pady=30,padx=10)

R2 = tk.Radiobutton(window, text="Off", variable=var, value=2)
R2.pack(side=tk.LEFT,pady=30)

B = tk.Button(window, text ="Confirm", command = Confirm)
B.place(x=window_width*0.6, y=window_height*0.4)

B = tk.Button(window, text ="Exit", command = Exit)
B.place(x=window_width*0.8, y=window_height*0.4)

print(window_height,window_width)


window.mainloop()

The GUI starts the background process on a separate thread, so as to avoid crashing the GUI. Tkinter has been used to create the GUI. the GUI calls the PresentEX class from main.py. which has the functionality of automation. The main.py is as below

import serial
import time
import keyboard
import win32gui, win32com.client
import pyautogui

testArray = []

class PresentEx():

    serialLine = ""


    def __init__(self,status):
        self.status = status

    def winEnumHandler(hwnd, ctx):

        global testArray
        if win32gui.IsWindowVisible(hwnd):
            count = 0
            a = win32gui.GetWindowText(hwnd)
            testArray.append(a)


    def SerialInitiate(self):
        serialLine = serial.Serial('COM3', 9600)
        time.sleep(0.1)
        return serialLine

    def ZoomFunctionality(self,ProgramArray):
        global testArray
        print(ProgramArray)

        if(self.status == 1): #2off 1on
            self.status = 2
        else:
            if "Zoom" in ProgramArray:
                ZoomInstances = ProgramArray.count("Zoom")
                print(ZoomInstances)
                if ZoomInstances >=2:
                    hwndZoomMeeting = win32gui.FindWindow(None,"Zoom")
                    print(hwndZoomMeeting)
                    win32gui.SetForegroundWindow(hwndZoomMeeting)
                    time.sleep(0.1)
                    pyautogui.hotkey('alt', 'a')
                    time.sleep(0.1)
                    print("audio Disabled")
                    pyautogui.hotkey('alt', 'v')
                    time.sleep(0.1)
                    print("video Disabled")
                else:
                    if "Zoom Meeting" in ProgramArray:
                        print("XXXX")
                        hwndZoomMeeting = win32gui.FindWindow(None,"Zoom Meeting")
                        print(hwndZoomMeeting)
                        win32gui.SetForegroundWindow(hwndZoomMeeting)
                        time.sleep(0.1)
                        pyautogui.hotkey('alt', 'a')
                        time.sleep(0.1)
                        print("audio Disabled")
                        pyautogui.hotkey('alt', 'v')
                        time.sleep(0.1)
                        print("video Disabled")

        testArray = []
        print(self.status)

    def main(self):

        serialLine = PresentEx.SerialInitiate(self)
        print("initiate")
        print(self.status)

        while (1):
            time.sleep(0.1)
            try:
                print(serialLine.readline().decode().rstrip() == 'PERSON DETECTED')

                if (serialLine.readline().decode().rstrip() == 'PERSON DETECTED'):
                    win32gui.EnumWindows(PresentEx.winEnumHandler, None)
                    print("IN LOOP")
                    serialLine.flushInput()
                    PresentEx.ZoomFunctionality(self,testArray)
                    while(1):
                        print("Person Still There")
                        if (serialLine.readline().decode().rstrip() == "NO PERSON DETECTED"):
                            win32gui.EnumWindows(PresentEx.winEnumHandler, None)
                            print(testArray)
                            PresentEx.ZoomFunctionality(self,testArray)
                            print("Zoom Turned OFF")
                            break
                        serialLine.flushInput()
                    time.sleep(3)
                    # serialLine.flushInput()
                    # print(serialLine.readline().decode().rstrip() == 'PIRSensed')
                    print("Wait Done")
                serialLine.flushInput()

            except (UnicodeDecodeError):
                print("Unicode Decode Error Exception Handled")
                pass

        if __name__ == '__main__':
            main()

Serial library has been used to read the serial data from the arduino,  and pyautogui has been used to automate the functions. The process is as follows. Zoom application has two windows. the main window, which is named "Zoom" and the meeting window is named "Zoom Meeting" and once you minimize the window, the video floats and the floating window is named "Zoom" as well. 

Zoom has hotkeys to turn on/off the camera, and turn on/off the mic. Alt+a for the mic, and Alt+v for the camera. 

Software chooses the window based on the name, brings it to top, and executes the hotkeys from pyautogui. Hopefully, this gets the job done.

Next, testing. 

Game plan was simple, send some serial data once the PIR detects a person. seemed simple enough, PIR was in the repeat trigger mode. and  I used repeat trigger mode, because system should ALWAYS be aware if the person is there or not. However, I still needed a timeout for turning off the camera and mic, or else it will keep turning everything off all the time. (PIR get some wrong readings and garbage values at times.) therefore, the code waits for 30 seconds until the person isn't there to turn the camera and mic off. 

I had to manage the serial data that was being sent to the PC. we obviously do not need the data all the time, just when we detect / undetected  someone. Therefore, only 5 lines were sent thru serial line to the PC. and used strings as opposed to binary values (1 or a 0) just so that we can confirm that its the correct data and not some random value, 

That was it for the Arduino section.  Code is below

int PIRSensor = 12;
unsigned long timeInLoop = 0;
unsigned long elapsedTime = 0;
void setup()
{
  Serial.begin(9600);
  pinMode(PIRSensor, INPUT_PULLUP);
}

void loop()
{

  int count = 0;
  while (digitalRead(PIRSensor))
  {
    if (count < 5)
    {
      Serial.println("PERSON DETECTED");
    }
    count = count + 1;
    delay(1);

  }
  unsigned long currentTime = millis();
  while (!digitalRead(PIRSensor))
  {
    timeInLoop = millis();
    elapsedTime = timeInLoop - currentTime;
    if (elapsedTime > 30000)
    {
      Serial.println("NO PERSON DETECTED");
    }
  }
}

Found a arduino nano and a PIR. the wiring was standard and just the connection to the PIR was needed. Arduino IDE was installed alongside python

Several methods came to mind.  The first thing needed was the integrating it with software. We decided to focus on Zoom since its the most used platform. After searching the internet, found that there is no SDK that works from user side. 

Next option was to run a python automation script on command. 

Next, the feedback, computer vision seemed like the best option but 1. it was expensive, 2. some people might not like having a camera (privacy concerns) therefore, we decided to go with a simple PIR sensor. 

used Arduino nano because that was available, and we did not need a lot of inputs and outputs. 

I did not see the Log update was required from the beginning, but i will explain all the steps here. The idea came to me and my friend sachith while brainstorming project ideas for the contest. mainly because we've witnessed some embarrassing situations of people that have been uploaded to social media platforms. Seems like with this invention we could ease the mind of people who are working from home.