In Stardew Valley, even though I played it more than once already, I still needed to scrutinize guides to remember villagers' gift preferences while trying to increase my friendship with certain characters. Thus, I decided to employ computer vision to recognize villagers and display their birthdays and gift preferences (loves, likes, and hates) automatically while playing Stardew Valley.

Since I wanted this project to be as compact as possible, I chose to use a LattePanda Alpha 864s, a high-performance SBC (single-board computer), has an embedded Arduino Leonardo. To recognize villagers while exploring the Stardew Valley world, I deployed the OpenCV template matching. After detecting villagers successfully, to display their gift preferences and birthdays without interrupting the gaming experience or performance, I used the embedded Arduino Leonardo on the LattePanda Alpha. Arduino Leonardo prints the detected villager information on a 3.5" 320x480 TFT LCD Touch Screen (ILI9488) and notifies the player via a buzzer and a 5mm green LED.

Because I was usually struggling to increase my friendship with these eight characters below, I used the OpenCV template matching to recognize them:

• Abigail
• Emily
• Haley
• Maru
• Penny
• Alex
• Harvey
• Sam

After running this project and obtaining accurate results, it obviates the need for shrewd gift acumen for villagers while playing Stardew Valley. 😃 🎁

Huge thanks to DFRobot for sponsoring this project.

Sponsored products by DFRobot:

⭐ LattePanda Alpha 864s | Inspect

⭐ DFRobot 8.9" 1920x1200 IPS Touch Display | Inspect

Step 1: Recognizing villagers while playing Stardew Valley with OpenCV

I decided to employ template matching to recognize villagers in Stardew Valley while playing the game. Template Matching is a method for searching and finding the location of a template image in a larger (input) image. OpenCV provides a built-in function - cv.matchTemplate() - for deploying template matching. It basically slides the template image over the input image (as in 2D convolution) and compares the patches of the input image against the template image to find the overlapped patches. When the function finds the overlapped patches, it saves the comparison results. If the input image is of size (W x H) and the template image is of size (w x h), the output (results) image will have a size of (W - w + 1, H - h + 1). You can get more information regarding template matching from this guide.

There are several comparison methods implemented in OpenCV, and each comparison method applies a unique formula to generate results. You can inspect TemplateMatchModes to get more information regarding comparison methods, describing the formulae for the available comparison methods.

I developed an application in Python to recognize villagers in Stardew Valley and send their information (gift preferences and birthdays) to the Arduino Leonardo. As shown below, the application consists of one folder and five code files:

• main.py
• windowframe.py
• vision.py
• arduino_serial_comm.py
• pics.py
• /assets
• -- templates (cropped villager poses)
• -- villagers.json

I will elucidate each file in detail in the following steps.

To execute my application to recognize villagers successfully, I installed the required modules on Thonny. If you want, you can use a different Python IDE.

⭐ Download the opencv-python module.

pip install opencv-python

⭐ Download the pywin32 module, which provides access to Windows APIs.

pip install pywin32

Step 1.1: Deriving templates from in-game screenshots and defining villager preferences

First of all, to be able to use template matching to detect villagers in Stardew Valley, I needed to create templates for each villager (character) in my list shown above. Since there are different poses in Stardew Valley for characters, I utilized in-game screenshots to catch unique character poses - sitting, turning right, turning...