WJCarpenter's pages

https://getcrankshaft.com/ Is a software bundle for using a Raspberry Pi and attached touchscreen display as a DIY Android Auto head unit. The project has been dormant for a couple of years, but that hasn't stopped me from poking around at it. One of the clever things they do is mount the boot and root filesystem read-only to minimize the chance of SD card corruption, which is one of the banes of SD cards with Linux. The selectively make the filesystems writable when they need to store some persistent thing, like a config update. The usual reason for wanting to operate from a USB thumb drive is minimized. However, because of the physical geometry of my screen and the case I printed for it, it's a tremendous amount of trouble to get the SD card out and back in once things are assembled. 

So, I want to skip the SD card and operate directly from a USB thumb drive. I'm using a Raspberry Pi 3B+, which can boot from USB out of the box. Some things in the Crankshaft layer assume you are using an SD card and prevent booting and operating from USB. I've worked through the changes needed to get it working, and here are the details.

There are the usual two Linux filesystems involved: the boot filesystem and the root filesystem. Crankshaft assumes those are /dev/mmcblk0p1 and /dev/mmcblk0p2 (SD card partitions), respectively. The USB drive will typically have those same filesystems on /dev/sda1 and /dev/sda2, respectively. That's not guaranteed if you have other things involved, so it's better to identify them by the partition UUID. I assume you're doing your image-making on some Linux machine. With the USB drive plugged into that machine, you can get the partition UUIDs via the blkid command:

$ blkid
/dev/sda1: LABEL_FATBOOT="boot" LABEL="boot" UUID="BBBB-AAAA" TYPE="vfat" PARTUUID="abcdefab-01"
/dev/sda2: LABEL="rootfs" UUID="ffffffff-eeee-dddd-cccc-bbbbbbbbbbbb" TYPE="ext4" PARTUUID="abcdefab-02"

 The thing we're interested in is the PARTUUID. The following steps assume you have the partitions of your thumb drive actually mounted on some Linux machine. You may need to be root to do some of the steps, depending on your local configuration of that Linux machine.

Caution: It's possible to end up with partitions on different devices having the same partition UUID. Using both devices on the same host can result in a lot of confusion and chaos. The smart thing to do for that case is to change the partition UUIDs on at least one of them. This thread has some good information about how to do that: https://askubuntu.com/questions/1250224/how-to-change-partuuid

In the root directory of the "boot" filesystem (/dev/sda1 in my example), there is a text file cmdline.txt that supplies command line parameters to the kernel at boot time. It's quite long but looks something like this:

console=tty3 root=/dev/mmcblk0p2 ro rootfstype=ext4 ....

With a text editor, change the "root=" part to reference your root filesystem partition (/dev/sda2 in my example). Instead of a device name, use the partition UUID. For example,

console=tty3 root=PARTUUID=abcdefab-02 ro rootfstype=ext4 ....

With no SD card inserted, the RPi's boot sequence stuff will find the boot partition on your thumb drive, and the above change will tell it where to find the root partition. But more tricks are needed.

In the "etc" directory of the root filesystem is the standard fstab file that tells Linux what filesystems to mount where. For the Crankshaft image, it will look something like this:

proc        /proc        proc            defaults,noatime,nodiratime        0    0
/dev/mmcblk0p1    /boot        vfat            ro,defaults,noatime,nodiratime        0    2
/dev/mmcblk0p2    /        ext4            defaults,noatime,nodiratime    0    1
ramfs        /tmp            ramfs            size=128m,nodev,nosuid,noatime,nodiratime        0    0
ramfs        /var/tmp        ramfs            size=16m,nodev,nosuid,noatime,nodiratime        0    0
ramfs        /var/log        ramfs            size=16m,nodev,nosuid,noatime,nodiratime        0    0

(followed by several more "ramfs"  mounts, which are irrelevant for this discussion)....

This is a description of how I have set up commercial skipping during playback of my emby recordings.

I have a few ways of recording live/linear TV. I have both good antenna OTA reception for local channels and various (legal/paid) accesses to other channels. The two server applications I use most for playing or recording live TV are Channels DVR and emby. I run both of them in docker containers on the same Raspberry Pi 4.

There is a popular application called comskip that analyzes video files and uses several tunable heuristics to detect commercial advertisement breaks. It can output the list of breaks in any of several standard or quasi-standard formats. Probably the most popular output format is EDL, edit decision list. Commercial skipping is a two-part process. First, the video file is analyzed (which to some extent can be done while it's being played or recorded). Second, when the video is played, the player understands the results of the analysis and provides for either manual or automatic skipping of commercial breaks.

Channels DVR has support for comskip built in, both for the analysis and playback. It's easily configured via the web GUI. emby does not have comskip built in, but it can be added via the sophisticated Emby.Comskipper plugin. The plugin provides the playback skipping based on EDL files. (It also recognizes some specially designated advertising chapter markers sometimes present in video files, though that has nothing to do with comskip.) The author provides detailed and complete instructions for how to hook in the comskip analysis piece to the emby server.

Although I run emby in a docker container, I wanted to run comskip either in its own container or on the host system. I didn't find a convenient comskip docker image, so I used the glue described below to run comskip on the host RPi 4. Running comskip with 3 threads, the CPU cores are loaded at about 75% each during actual analysis. For now, I only run a single comskip job at a time.

comskip-inotify.sh

This is a short shell script that manages the kick-off of comskip jobs. Jobs can be triggered in one of two ways.

1. When the script starts, it looks for video files in the emby recording area that do not have a companion EDL file. This is a decent enough catch-up technique for crashes, reboots, etc.
2. After that, it waits for the arrival of new video files from emby. That's done via the inotify-wait tool watching for creation events in the entire emby recording area.

For either job type, a job is created using task spooler. It's a straightforward queuing tool available in most Linux package repositories.

There are a couple of odds and ends to deal with in the script.

• Creating of new directories is ignored.
• It only acts on ".ts" files. I think (but I'm not sure) that all emby recordings are TS files.
• Jobs created in the first step are marked with "FIND" in the queued job. Jobs created for new files are marked with "WAIT". This is mostly for the benefit of manual monitoring of the queue.
• "WAIT" jobs are pushed to the head of the queue, ahead of all queued "FIND" jobs. That doesn't interrupt any already running job.
• Task sppoler standard output files and "finished"  entries are deleted from time to time to get rid of clutter.

Without further ado, here is comskip-inotify.sh:

#!/bin/bash

# Top level directory under which emby puts the Live TV recordings (default can be overridden on the command line)
RECORDINGS=${1:-/mnt/red-drive/emby/programdata/data/livetv/recordings/}

consider_one_file() {
    DIRECTORY=${1}
    FILE=${2}
    HOWCOME=${3}
    case ${HOWCOME} in
	FIND)
	    # only process files with a .ts extension where there is not already an EDL file
	    if [ "${FILE}" != "${FILE%.ts}" -a ! -e "${DIRECTORY}${FILE%.ts}.edl" ]
	    then
		queue_one_comskip ${HOWCOME} "${DIRECTORY}" "${FILE}"
	    fi
	    ;;
	WAIT)
	    # only process files with a .ts-READY extension
	    if [ "${FILE}" != "${FILE%.ts-READY}" ]
	    then
		rm -f "${DIRECTORY}${FILE}"
		queue_one_comskip ${HOWCOME}...

((I'm moving the content formerly on this page to Instructables to reach a wider audience. And here it is: https://www.instructables.com/HOWTO-Moving-From-Google-Voice-GV-to-Google-Messag/))

In the Spring of 2023, a mallard duck hen made a nest in our yard. We knew that meant ducklings in a few weeks, so I scrambled to figure out how to stream it for the enjoyment of friends, neighbors, and strangers. Of course, I don't mean the kind of stream where you might expect to find a duck. I mean a video stream. It's all over now, and the ducks have gone. Here is my Youtube playlist of the highlights.

I already had a few video cameras scattered about watching the outdoors. Even though we live in a suburb, and even though we have a fenced yard, we do get occasional visits from wildlife: cats, raccoons, cats, a possum, cats, squirrels, cats, and birds. None of those cameras cover the area where the duck nested (I'm pretty sure she didn't think about that when she picked the spot, but who knows). None of my current cameras were suitable for exposure to the amount of rain we're getting this Spring, and none of them have a big friendly button with a label liike "click here to stream to the universe".

This is a write-up of the ultra-fast learning I did to get my live stream going. 

I had to work fast because we didn't know how long the duck had already been nesting when we found out about it. A duck will lay eggs for about 10 days before starting incubation. Then all of the eggs incubate for about 28 days. All of the ducklings will hatch within a few hours of each other. Ducklings are "precocial". The mother duck does not actually feed them directly. Instead, within a day or so of hatching, she leads them to some place where they can feed themselves.

The first thing I did was order an outdoor surveillance camera. My criteria were:

• at least full HD (1080p) resolution
• fast delivery
• cheap, cheap, cheap
• video stream accessible by 3rd party tools

Fast delivery and cheap requirements funnelled me to the many, many no-name Chinese cameras on Amazon that could arrive next day or the day after. Without time to really study the matter, 3rd party access to me meant a camera supporting Real Time Streaming Protocol  (RTSP). I later discovered that ONVIF is a sort of profile of conventions and discovery that uses RTSP. I only know a little bit about video codecs, but I had a hunch that H.264 was a good thing to have, and that H.265 was OK but not great to have (for reasons not completely clear to me). Most of the no-name Chinese cameras don't say anything about any of that stuff in their descriptions, but a few do. Most are locked into some proprietary access scheme developed by a chipset maker and need a dedicated mobile app to view the videos.

I'm not going to bother mentioning the particular model of camera that I got because they come and go anyhow, and it might not even be available by the time you read this. I'll just say that the Amazon description did explicitly mention RTSP support, and it met my other requirements. I'll also mention that the mobile app to configure it was Tuya Smart, which I was already using for other reasons.

OK, now I had the camera, and I'm skipping the step about running a long orange extension cord and mounting the camera to my fence with an amazing variety of boards and clamps. I was able to get the camera configured in the Tuya Smart app so that I could access the video stream over WiFi. How to get from there to live streaming?

One of the first things I did, after some Google searching, was install the Agent DVR application from ISpyConnect. This application is pretty cool and supports a huge variety of IP cameras. Since my camera supported ONVIF and RTSP, Agent DVR was able to work out the connection particulars with no trouble at all. Once that was working, it was also willing to tell me what it had discovered about RTSP streams. I found that my camera had a main stream of 1080p and a secondary stream of VGA (640x480). Agent DVR does have the ability to read an RTSP stream and relay it out to a server for live streaming, but that feature is not available in the free, unlicensed version. (That...

I'm surely not the first person to figure this out, but I didn't come across this solution while searching around. They are probably out there and I just didn't find them.

The problem, in a nutshell, is how to reliably map USB devices into a docker container. The more specific problem, for me, is that I run the Home Assistant docker container (with docker-compose) and want to map in the USB ports for my Zigbee and Z-Wave interfaces. (I use the popular HUSBZB-1 combo dongle, which slightly complicates things, but that's incidental.) Things were pretty easy when I could always count on them being /dev/ttyUSB0`and /dev/ttyUSB1 on the host machine. When I started using an unrelated USB device on the same host machine, things turned into a little dance of plugging, unplugging, restarting containers, pulling out hair, etc.

It's easy to find solutions for creating reliable, persistent names for USB devices on Linux. The answer is to use something in udev rules. udev even has a really easy way to create a symlink with whatever fixed name you choose. It's very simple, and you will find zillions of internet articles describing how to do it. It doesn't take long to discover, however, that docker can't cope with mapping those host machine symlink names to a container device name. When you start looking for solutions to that problem, you find that it's been a docker sore spot for years and years.

The solution is to use hard links instead of symlinks. It's not particularly difficult, but it's also not particularly obvious if your only exposure to udev is from reading canned recipes on various web pages.

Here's how I did it. I have 3 USB devices of interest: the Zigbee and Z-Wave devices mentioned, and a 3D printer that I use with Octoprint in an unrelated docker container. There are many articles around to tell you how to find the internal details (vendor, serial number), etc of USB devices for use with udev, so I won't go into that here. However, I will direct your attention to the very useful pseudo-directory /dev/serial/by-id/. (There is also /dev/serial/by-path/. You don't want that one.) Without you doing anything, it will contain symlinks to your serial USB devices. Here's part of mine:

lrwxrwxrwx 1 root root 13 Sep  1 18:18 usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0 -> ../../ttyUSB2
lrwxrwxrwx 1 root root 13 Sep  1 18:18 usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if01-port0 -> ../../ttyUSB3

For a given device, the name of the symlink is always going to be the same (at least as far as I can tell). If docker could map symlinks to devices, that would be all you would need for reliable names. But, alas. There are various Linux utilities for dereferencing symlinks, and the one we want to use here is realpath. It provides an absolute path to the real file. (A commonly-suggested alternative, readlink, would only give you the relative path mentioned in the symlink, which is not directly helpful.)

$ realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0
/dev/ttyUSB2

Armed with that, we can now stitch it into udev rules. If you want to know more about udev rules, there are plenty of explanations available, so this is going to be just a distilled description of how to apply them. The persistent names I use are /dev/ttyUSB-ender3v2, /dev/ttyUSB-zigbee, and /dev/ttyUSB-zwave. Those are the hard links I want to end up with. Here is a tiny shell script that maps one of the above symlinks to the applicable hard link, depending on its single command line argument:

#!/bin/bash

p=""
if [ "$1" == "ender3v2" ]
then
  p=`realpath /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0`
elif [ "$1" == "zwave" ]
then
  p=`realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0`
elif [ "$1" == "zigbee" ]
then
  p=`realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if01-port0`
fi

if  [...