Instructions | Virtual Steering Wheel!

1

Step 1: Install Python

If you haven’t already: Go to https://www.python.org/ Click Download Python Install it like any other app
2
Step 2: Install the Libraries
These are the special helpers for Python:
1. Open Command Prompt (Windows) or Terminal (Mac/Linux)
2. Type the following and press Enter:
pip install opencv-python mediapipe pyautogui

What Are These Libraries?

Let’s understand them like superheroes:
1. 🧠 Mediapipe – This smart tool sees your hands and finds points like your thumb, wrist, and fingers.
2. 👀 OpenCV – It helps us use the webcam and draw things on the screen.
3. ⌨️ PyAutoGUI – It presses keys like W, A, S, D for you.

Step 3: Copy the Code

this took me days to figuring out and i am giving you the code for free!!!

Here’s the magic program. You can open Notepad, or better: install VS Code.

import cv2
import mediapipe as mp
import math
import pyautogui
import time

mp_drawing = mp.solutions.drawing_utils
mphands = mp.solutions.hands

cap = cv2.VideoCapture(0)
hands = mphands.Hands(static_image_mode=False, max_num_hands=2, min_detection_confidence=0.7, min_tracking_confidence=0.5)

def calculate_angle(x1, y1, x2, y2):
    angle = math.degrees(math.atan2(y2 - y1, x2 - x1))
    if angle < 0:
        angle += 360
    if angle > 180:
        angle -= 360
    return angle

def is_thumb_up(hand_landmarks):
    return hand_landmarks.landmark[4].y < hand_landmarks.landmark[3].y

pressing_w = False
pressing_s = False

turn_sensitivity = 0.01
straight_threshold = 6
tap_duration = 0.01 # Still used, but for logic-based tap, not sleep

# Tap state for A and D keys
tap_a_time = 0
tap_d_time = 0
tap_in_progress_a = False
tap_in_progress_d = False

while True:
    ret, image = cap.read()
    if not ret:
        break

    image = cv2.resize(image, (480, 360))
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    results = hands.process(image)
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

    current_time = time.time()

    if results.multi_hand_landmarks and len(results.multi_hand_landmarks) == 2:
        hand1 = results.multi_hand_landmarks[0]
        hand2 = results.multi_hand_landmarks[1]

        if hand1.landmark[0].x < hand2.landmark[0].x:
            left_hand = hand1
            right_hand = hand2
        else:
            left_hand = hand2
            right_hand = hand1

        x1, y1 = left_hand.landmark[0].x * image.shape[1], left_hand.landmark[0].y * image.shape[0]
        x2, y2 = right_hand.landmark[0].x * image.shape[1], right_hand.landmark[0].y * image.shape[0]

        mp_drawing.draw_landmarks(image, left_hand, mphands.HAND_CONNECTIONS)
        mp_drawing.draw_landmarks(image, right_hand, mphands.HAND_CONNECTIONS)
        cv2.line(image, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2)

        thumb1_x, thumb1_y = left_hand.landmark[4].x * image.shape[1], left_hand.landmark[4].y * image.shape[0]
        thumb2_x, thumb2_y = right_hand.landmark[4].x * image.shape[1], right_hand.landmark[4].y * image.shape[0]

        cv2.putText(image, "Accelerator", (int(thumb1_x), int(thumb1_y) - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)
        cv2.putText(image, "Brake", (int(thumb2_x), int(thumb2_y) - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 1)

        angle = calculate_angle(x1, y1, x2, y2)
        cv2.putText(image, f'{int(angle)} degrees', (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 0), 1)

        # Tap simulation for left and right
        if angle < -straight_threshold and not tap_in_progress_a:
            pyautogui.keyDown('a')
            tap_a_time = current_time
            tap_in_progress_a = True
            cv2.putText(image, "Left", (10, 210), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
        elif angle > straight_threshold and not tap_in_progress_d:
            pyautogui.keyDown('d')
            tap_d_time = current_time
            tap_in_progress_d = True
            cv2.putText(image, "Right", (10, 210), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)

        # Release A key after tap_duration
        if tap_in_progress_a and (current_time - tap_a_time > tap_duration):
            pyautogui.keyUp('a')
            tap_in_progress_a = False

        # Release D key after tap_duration
        if tap_in_progress_d and (current_time - tap_d_time > tap_duration):
            pyautogui.keyUp('d')
            tap_in_progress_d = False

        # Acceleration
        if not is_thumb_up(left_hand):
            if not pressing_w:
                pyautogui.keyDown('w')
                pressing_w = True
        else:
            if pressing_w:
                pyautogui.keyUp('w')
                pressing_w = False

        # Braking
        if not is_thumb_up(right_hand):
            if not pressing_s:
                pyautogui.keyDown('s')
                pressing_s = True
        else:
            if pressing_s:
                pyautogui.keyUp('s')
                pressing_s = False

    cv2.imshow('Hand Tracker', image)
    if cv2.waitKey(1) & 0xFF == 27:
        break

if pressing_w:
    pyautogui.keyUp('w')
if pressing_s:
    pyautogui.keyUp('s')
pyautogui.keyUp('a')
pyautogui.keyUp('d')

cap.release()
cv2.destroyAllWindows()

4
Step 4: Explaination of Code and Important Settings in Code Before Using It
🔍 Explaining the Key Code Lines

✅ Importing Tools

import cv2 # To use webcam and show visuals
import mediapipe as mp # To detect hands
import math # To calculate angles
import pyautogui # To press keyboard buttons automatically
import time # To handle timing for button presses

✅ Initial Settings

hands = mphands.Hands(...)
1. This initializes MediaPipe Hands to detect up to 2 hands.
2. It finds where your hands are and tracks their movement.
✅ Detecting Angle Between Hands

def calculate_angle(x1, y1, x2, y2): ...
1. Calculates the angle between your left and right hands.
2. This tells whether you're turning left, right, or going straight.
🧠 Important Settings :

🛠 turn_sensitivity = 0.01
1. This was used in a previous version to make turning more or less sensitive.
2. BUT in your current code, it's not used anymore, so we can remove it or ignore it.
🔁 straight_threshold = 6

This value decides how much your hand angle must change before it counts as turning.
1. If the angle is less than -6, it means turn left (A key).
2. If the angle is more than +6, it means turn right (D key).
3. If the angle is between -6 and +6, it means go straight (do nothing).
This helps avoid small hand shakes being treated as turns.

⏱️ tap_duration = 0.01

This sets how quickly to release the A or D key after tapping it.
1. 0.01 means the key is held down for just 0.01 seconds (very short).
2. This makes the car turn in small steps, so you don’t turn too much with a small hand movement.
If you want smoother, longer turns, increase this value to something like 0.05.

🖐️ Detecting Thumb Up or Down

def is_thumb_up(hand_landmarks): return hand_landmarks.landmark[4].y < hand_landmarks.landmark[3].y
1. Checks if the tip of the thumb is above the second joint (on the screen).
2. This is used to detect:
3. Left thumb up = stop accelerating
4. Right thumb up = stop braking
🚗 Controls Using Hands

🟢 Accelerate (W key)

if not is_thumb_up(left_hand): pyautogui.keyDown('w')
1. If your left thumb is down, press the W key (accelerate).
2. If your left thumb goes up, release the W key (stop).
🔴 Brake (S key)

if not is_thumb_up(right_hand): pyautogui.keyDown('s')
1. If your right thumb is down, press the S key (brake).
2. If your right thumb goes up, release the S key (stop braking).
↔️ Turn Left or Right (A and D keys)

if angle < -straight_threshold: pyautogui.keyDown('a') # release after tap_duration
1. Angle < -6: Press A (turn left)
2. Angle > +6: Press D (turn right)
These are short taps, so the car turns briefly and doesn't stay turning forever.

🧹 At the End

pyautogui.keyUp('w')
pyautogui.keyUp('s')
pyautogui.keyUp('a')
pyautogui.keyUp('d')
1. When you exit the program, it releases all keys to avoid the car getting stuck going forward or turning.
✅ Final Thoughts

🎮 What You Can Control
5
Step 5: Important Settings in Code Before Using It
🧠 Important Settings :

🛠 turn_sensitivity = 0.01
1. This was used in a previous version to make turning more or less sensitive.
2. BUT in your current code, it's not used anymore, so we can remove it or ignore it.
🔁 straight_threshold = 6

This value decides how much your hand angle must change before it counts as turning.
1. If the angle is less than -6, it means turn left (A key).
2. If the angle is more than +6, it means turn right (D key).
3. If the angle is between -6 and +6, it means go straight (do nothing).
This helps avoid small hand shakes being treated as turns.

⏱️ tap_duration = 0.01

This sets how quickly to release the A or D key after tapping it.
1. 0.01 means the key is held down for just 0.01 seconds (very short).
2. This makes the car turn in small steps, so you don’t turn too much with a small hand movement.
If you want smoother, longer turns, increase this value to something like 0.05.

Virtual Steering Wheel!

What Are These Libraries?

🔍 Explaining the Key Code Lines

✅ Importing Tools

✅ Initial Settings

✅ Detecting Angle Between Hands

🧠 Important Settings :

🛠 turn_sensitivity = 0.01

🔁 straight_threshold = 6

⏱️ tap_duration = 0.01

🖐️ Detecting Thumb Up or Down

🚗 Controls Using Hands

🟢 Accelerate (W key)

🔴 Brake (S key)

↔️ Turn Left or Right (A and D keys)

🧹 At the End

✅ Final Thoughts

🎮 What You Can Control

🧠 Important Settings :

🛠 turn_sensitivity = 0.01

🔁 straight_threshold = 6

⏱️ tap_duration = 0.01

Discussions

Become a Hackaday.io Member