Goal

The goal with the project is to be able to get the robot to complete small tasks at home. Either through teleoperation, partly automated or fully automated. With the goal of automating tasks more and more over time. Also to have a friendly and easy to use interface to make it easy to use.

It would be especially useful for elderly or disabled people who need help but are at risk of being infected with COVID-19, influenza or any other illness. The robot can reduce human contact and relieve small tasks that can be difficult if, for example it is difficult to bend down or you sit in a wheelchair, some example tasks could be to help fill water and food bowls for pets, through object detection help to find things at home, or maybe watering the flowers.

Open source

So the plan is to make everything open source, however, i won't make it public while I'm developing the robot. For 2 reasons mainly, so i am going to change a lot while developing the robot, so i think it's better for me to take the developing costs instead of having the people wanting to build the project having to buy new parts as soon as i want to change something. Also i am doing video streaming and teleoperation, there is big consequences for having security holes in the software on these parts, so i really want to address any security issues before i make the code public.

Design

The design is kept quite simple to begin with to focus more on the software side. I will try to use cheap and easy to find components. Right now i have a simple base built on an existing robot platform with 20x20 aluminium extrusions attached to it as a frame for mounting components. On the "head" there is a 7" touchscreen for displaying an animated face and with the possibility to use as an input device.

The goal is to have a good base built for the robot first with the possibility for modular attachments.

Software

I'm using Tensorflow for Object detection and facial recognition. A python script to read sensor data from the arduino and to stream the live video to a server and use Socket.io to send real time commands to the robot so it's able to be remote controlled and start different directives, for example the only one i have now is the "Follow"-mode that use the object detection to follow the object of choice.

Mycroft AI is used as the voice assistent to give commands, play music etc.

Here you can see an early test of the follow mode. Further improvements have been made after this, mostly on making the object recognition faster and i will post more videos of that soon.

Project Logs

Collapse

Quick update
Erik Knutsson • 12/28/2020 at 12:00 • 0 comments

It's been a while since i last posted an update. A combination of having a lot to do at work, starting my own side business and moving to a new city have made the progress a lot slower. But a few things have happened since the last update that i thought i would share for now.
So in the last video I installed the Jetson nano to the robot and now I got it pretty much up and running so the robot can again recognize objects and follow me around. I would say that I got the desired results, the machine vision is much faster on the Jetson nano and is much more suitable for the robot.
The second thing i did was to replace the speakers, the old ones were really to big, didn't look good and didn't even sound that good. So i got a hold of a pair of smaller speakers with much better sound. I printed new cases for them and attached them lower to the ground so they don't stick out as much as the old ones.
I got some video material on all of this and hope to get it out as soon as i have the time!
Video log - part 2
Erik Knutsson • 09/02/2020 at 14:23 • 0 comments
Video Build Diary
Erik Knutsson • 08/09/2020 at 19:14 • 0 comments

Sooo... i desided to try something a bit new, thought it would be a fun experience to start doing a build diary in video format and in the same time learn more about filming, video editing and so on. So i did this first (really simple) video on some background on the project. So please subscribe to my youtube channel to be sure to catch all the videos. But of course i will continue to post everything i do here and post all the videos here as well.
Speech commands #2
Erik Knutsson • 07/12/2020 at 17:58 • 0 comments

I have continued working on more features for the voice assistant. Also i have been doing some work on improving the follow mode which i tried combining in this test. Turned out pretty cool! You can see it's much faster than the first video on the follow mode in the projects description. I can still do some improvements on the speed though, but i think i need to improve the hardware to be able to run the predictions for the object detection faster. So i think i will do this either by switching out the Raspberry Pi with a Nvidia Jetson Nano computer or using an accelerator like Coral. I would like to try and experiment a bit with a Jetson Nano, but im leaning more towards the accelerator right now since it would be cheaper and i don't have to transfer everything to the Jetson Nano which could possibly be alot of work.
Speech commands
Erik Knutsson • 06/29/2020 at 20:26 • 0 comments

Been working on the custom app for Mycroft AI to be able to control the robot by voice. I just got some basic commands down so far, but it's pretty cool to see it in action.

To give the robot a little more of a personal feel i also started training a new Wakeword/Name for the robot. So i named it David, seemed like a good name since two of my favorite robots from movies is named David :)
Adding sound
Erik Knutsson • 06/25/2020 at 21:27 • 0 comments

To fully integrate the new voice assistent i've been working on adding speakers to the robot, so used some speakers i had laying around from an old bluetooth speaker project and printed a holder for them. It's not the prettiest. But as you can see on my wiring in the robot, i try to get it functional at first and make it pretty later :)
Voice assistent capabilities
Erik Knutsson • 06/07/2020 at 20:51 • 0 comments

The weekend was spent on testing out different voice assistants. The setup i had before was to simply have my own text-to-speech where i just checked for certain keywords. Not the best, but it worked. But i wanted to add something like Alexa so i instantly could get way much more functionality without having to program everything on my own.
So i started out with Alexa, was not that hard to implement a test version from their examples so i could play podcasts and radio and so on and it worked great. It was fast and accurate. However i really wanted Spotify support so i could play my music and podcasts. But it turns out Amazon have disabled these types of applications if you are not developing a public application.
So i moved on to Mycroft AI that i found had a Spotify Skill, this was a little bit more tricky to get working at first, it was a easy install but the output audio would not play, after some work i finally got it working, it's slower than Alexa, but i read you can do some optimizations and i do really like that it is Open Source, it was really accurate and it seems easy enough to develop my own Skill to do a custom skill to control the robot. So the direction i will be going in will be with Mycroft AI.
Next step is to create the custom skill and to install a speaker, since im just using a Bluetooth speaker right now which is a bit unnecessary, but i have some speakers from an old build of my own bluetooth speaker that i will probably re-use.
You probably get the idea of how a voice assistant work, but here is a demo video anyway, since some content is always nice :) Something is weird with the sound in the demo, because it jumps into the middle of the first sentence. But i think that is some weird problem with my speaker since i don't get that with headphones connected.
Congratulations it's a box!
Erik Knutsson • 05/28/2020 at 20:25 • 2 comments

So while it's not the most important part of the project it's always fun to do some designs. But it's also really easy to just end up with a box with wheels. It's easy to build and everything fits great together. But... it is a bit boring, i will see if i can come up with something better than this and of course suggestions are welcome!
Some CAD work
Erik Knutsson • 05/28/2020 at 10:14 • 0 comments

Been doing some work in cad to try and get some more parts 3D-printed. Been doing these bottom plates that is attached to the frame for holding components. Had to do a total of 4 plates to be able to fit them in the printer.
In place it will look something like this. Could probably do a better connection between the plates to keep it nice and flat. So perhaps that will change later.
While i'm waiting on some components and more filament to arrive in the mail i will be doing some more CAD work to hopefully get a complete design and overview of the robot.
Remote interface
Erik Knutsson • 05/25/2020 at 20:50 • 0 comments

Put some more work into the interface for the teleoperation part of the robot. From here i can see the video feed with object detection, steering and some controls for making the robot autonomously find and follow objects. It's actually quite nice to make it play music and make it follow you arround the apartment :)