Project Logs

Collapse

• Some capture measurements [UPDATED]

  WJCarpenter • 08/10/2024 at 20:50 • 0 comments

  I finally got around to doing some video captures with the dongles on a couple of Raspberry Pi boards. For these experiments, the OS was Raspberry Pi OS Lite (64 bit), aka Debian Bookworm, running from an SD card. The "lite" OS does not include a graphical desktop. I chose to avoid that overhead since my experiments would be command line driven anyhow. I also tested on HP EliteDesk 800 G3 i7-6700T 2.80GHz (8gb RAM, 8 threads).

  The dongle I used is the one with the MS2130 chip. The one I have has a USB-C connector, and I used a simple mechanical USB-C to USB-A adapter. The RPi 3 has only USB 2.0 ports, but I used a USB 3.0 port on the RPi 4. For the HDMI input, I used a Google Chromecast dongle (the HD version, not the 4K version). It was just repeatedly displaying various setup animations. I don't think there was any audio. In any case, I didn't try to capture any audio because I knew the video would dominate the performance challenge. CPU figures below are by eyeballing htop and averaging across all 4 cores for the RPi boards and 8 cores for the x86_64.

  Test 1:

  This first test tries to capture what I ultimately need: an MPEG-TS stream at FHD resolution. The capture dongle outputs 50 fps at that resolution for MJPEG. The command transcodes that into MPEG2

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -f mpegts out.mkv 

  RPi 3: Transcode 11 fps. CPU about 50%.

  RPi 4: Transcode 25 fps. CPU about 50%.

  x86_64: Transcode 50 fps. CPU about 13%.

  Test 2:

  This is the same experiment, but with YUYV input instead of MJPEG. The dongle's advertised rate is only 10 fps.

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format yuyv422 -i /dev/video0 -f mpegts out.mkv

  RPi 3:  Transcode 8+ fps. CPU about 35%.

  RPi 4:  Transcode 10 fps. CPU about 20%.

  x86_64: Transcode 10 fps. CPU about 4%.

  Test 3:

  This is a repeat of Test 1, but with the "ultrafast" preset for H.264.

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -preset ultrafast -f mpegts out.mkv

   RPi 3: Transcode 11 fps. CPU about 50%.

   RPi 4: Transcode 25 fps. CPU about 40%.

  x86_64: Transcode 50 fps. CPU about 14%, with one thread outlier of 35%.
  

  Test 4:

  The same as Test 3, but with YUYV input.

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format yuyv422 -i /dev/video0 -preset ultrafast -f mpegts out.mkv

   RPi 3: Transcode 8+ fps. CPU about 35%.

   RPi 4: Transcode 10 fps. CPU about 20%.

  x86_64: Transcode 10 fps. CPU about 4%.
  

  Test 5:

  This test captures the incoming video without transcoding. For some scenarios (not mine), this allows for later transcoding as a separate step. 

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -c copy out.mkv

  RPi 3:  Transcode 50 fps. CPU less than 5%.

  RPi 4:  Transcode 50 fps. CPU less than 5%.

  x86_64: Transcode 50 fps. CPU less than 1% with some threads completely idle.
  

  Test 6:

  ffmpeg -f v4l2 -video_size 1920x1080 -input_format yuyv422 -i /dev/video0 -c copy out.mkv

   RPi 3: Transcode 4+ fps. CPU less than 5%.

   RPi 4: Transcode 8+ fps. CPU less than 5%.

  x86_64: Transcode 50 fps. CPU less than 5% with some threads completely idle.
  

  Observations:

  It's no surprise that the RPi 4 performed better than the RPi 3. For both boards, the only things that could keep up were the captures of raw encoding of MJPEG. The ultrafast transcoding of YUYV kept up, but only to match the dongle's limitation of 10 fps, which is not acceptable for entertainment video. The x86_64 box was able to keep up at the full frame rate with plenty of overhead. In a future experiment, I'll connect multiple dongles to see how it handles it. The x86_64 box also has a single USB-C port, so I gave that a quick try with Test 1. It resulted in a higher frame rate (60 fps) at the cost of slightly higher CPU. It might be possible to operate multiple...

  Read more »

• An aside: adventures in desktop streaming

  WJCarpenter • 08/07/2024 at 19:16 • 0 comments

  One of the things I want to capture, and which sort of inspired this whole line of research, is something that can run in fullscreen mode on Linux in general and Raspberry Pi in particular. Since I am already on a Raspberry Pi, can I efficiently stream an entire desktop? I did a bunch of experiments, mostly on a Raspberry Pi but a bit on a Linux x86 machine. This is a description of some things I found out along the way. You shouldn't think of this as a monograph from an expert in this area. Rather, I am trying to describe a few specific things with enough detail for context that it is useful (including being useful for future me). If you are and expert in any of these areas, you will probably be chuckling at how much I struggled to figure out what you think is obvious.

  My use case requirements simplified the general question somewhat.

  • The desktop would be 1920x1080, aka "FHD".
  • The target framerate was 60 FPS, but I figured 30 FPS would also be acceptable. 
  • I wasn't concerned too much about progressive versus interlaced.
  • The stream format would be MPEG-TS.
  • The stream would include audio produced by an application.
  • There would only be at most a single consumer of the stream, though that single client would not always be consuming (it would come and go).

  There is a heck of a lot that I don't know about MPEG, MPEG-TS, and their friends. I could write a book. No, wait, I couldn't write a book; I mean that what I don't know could fill volumes. As it is, what I do and don't know fills untold quantities of software documentation, forum postings, and wikis. Quite often advice to "just try this" is offered by someone who doesn't really know anything about anything, except that that single "try this" worked for them under some unspecified circumstances. Consequently, there is paradoxically too much information available since a lot of it is wrong or obsolete with no way to tell the difference. It's the Fog of Wikis.

  The FFmpeg approach

  ffmpeg -h full

  Although there are billions of tools for doing things with video streams, almost all of them are built on top of FFmpeg or its constellation of libraries and their close friends. I decided to start by going right to the BMOC and see if I could use it directly. For a long time, I've put off learning a lot of things about FFmpeg simply because it can do so much. On my Linux desktop, just printing the "full help" from the ffmpeg command gives over 15,000 lines of text. The tool is ridiculously capable (and I mean that in a complimentary way).
  

  I soon discovered that there are a couple of ways to use the Linux display as the FFmpeg input. The simplest to understand is "x11grab", which in FFmpeg parlance is a format. The input device is something that identifies the X11 display (typically ":0.0"). The other way is "kmsgrab", which does something similar but at a different architectural layer (docs). My (unverified) understanding is that "kmsgrab" uses fewer resources but can sometimes be trickier to use. (Note: Before using ffmpeg with "kmsgrab", you need to set a capability on the ffmpeg binary: "sudo setcap cap_sys_admin+ep /usr/bin/ffmpeg".) This Wikipedia article gives a good overview of DRM/KMS. This article, though with some vendor specifics, is also a pretty good read: https://wiki.st.com/stm32mpu/wiki/DRM_KMS_overview Both "x11grab" and "kmsgrab" offer options for selecting portions of the screen, size of the screen capture, and so on.

  Because FFmpeg is available for so many platforms, and because there are lots of compile-time choices, either or both of those two formats might be missing from an environment. Some years ago, "x11grab" may have been deprecated in favor of "xcbgrab", but most of my copies of ffmpeg still have "x11grab". I don't really know the difference between the two or what their twisted history is all about. My suggestion would be to see which one is present and use it. I think the options are compatible. Find which one you have by running this...

  Read more »

• UVC and UAC

  WJCarpenter • 05/27/2024 at 19:28 • 0 comments

  I don't know whether to file this under "learn something new every day" or "teach an old dog new tricks", but I recently looked into a standard called USB video device class (UVC) published by the USB Implementers Forum. (There is a companion standard, UAC, for audio, but I haven't looked at that.) The dongles based on the MS2109 and MS2130 chips advertise UVC and UAC compatibility, but I didn't really know what that meant beyond "don't need to install custom drivers". The dongles also expose a USB HID device, and I had assumed that any configuration control of them would be via the USB HID. Not so.

  UVC is one of those standards where any particular device is likely to only implement some subset, and that is true of these dongles. UVC provides protocol elements for querying or changing lots of different configuration items. There are multiple open source utilities that can be used to interact with UVC devices. For example, yavta and v4l-utils. Here are some examples for my MS2130 dongle:

  This reports the available video modes, resolutions, and frame rates (figures for frame rates are quoted as frame intervals, so 1/60 is 60fps):

  $ sudo yavta --enum-formats /dev/video2
  Device /dev/video2 opened.
  Device `USB3 Video: USB3 Video' on `usb-0000:00:15.0-1' (driver 'uvcvideo') supports video, capture, without mplanes.
  - Available formats:
      Format 0: YUYV (56595559)
      Type: Video capture (1)
      Name: YUYV 4:2:2
      Frame size: 1920x1080 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1600x1200 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1360x768 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x1024 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x960 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x720 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1024x768 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 800x600 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 720x576 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 720x480 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 640x480 (1/60, 1/50, 1/30, 1/20, 1/10)
  
      Format 1: MJPEG (47504a4d)
      Type: Video capture (1)
      Name: Motion-JPEG
      Frame size: 1920x1080 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1600x1200 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1360x768 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x1024 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x960 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1280x720 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 1024x768 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 800x600 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 720x576 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 720x480 (1/60, 1/50, 1/30, 1/20, 1/10)
      Frame size: 640x480 (1/60, 1/50, 1/30, 1/20, 1/10)
  
  Video format: YUYV (56595559) 1920x1080 (stride 3840) field none buffer size 4147200$ sudo v4l2-ctl -d2 --set-fmt-video "pixelformat=MJPG
  

   This switches the video mode from YUYV to MJPG

  $ sudo v4l2-ctl -d2 --get-fmt-video
  Format Video Capture:
      Width/Height      : 1920/1080
      Pixel Format      : 'YUYV' (YUYV 4:2:2)
      Field             : None
      Bytes per Line    : 3840
      Size Image        : 4147200
      Colorspace        : sRGB
      Transfer Function : Rec. 709
      YCbCr/HSV Encoding: ITU-R 601
      Quantization      : Default (maps to Limited Range)
      Flags             : 
  
  $ sudo v4l2-ctl -d2 --set-fmt-video "pixelformat=MJPG"
  
  $ sudo v4l2-ctl -d2 --get-fmt-video
  Format Video Capture:
      Width/Height      : 1920/1080
      Pixel Format      : 'MJPG' (Motion-JPEG)
      Field             : None
      Bytes per Line    : 0
      Size Image        : 4147200
      Colorspace        : sRGB
      Transfer Function : Rec. 709
      YCbCr/HSV Encoding: ITU-R 601
      Quantization      : Default (maps to Full Range)
      Flags             : 
  

  These tools tend to think you already know what the various parameters mean, but they are quite handy once you get the hang of them. The settings seem to stick, at least for the specific dongle that I have. If I unplug it and replug it, it remembers what I last set. The bad news is that the video playing software that I have handy (vlc and ffplay) seem to want to tweak some of the UVC settings, probably to better suit how they handle...

  Read more »

• syslog for MS2130 dongle

  WJCarpenter • 05/24/2024 at 00:20 • 0 comments

  When plugging the MS2130-based HDMI capture dongle, /var/log/syslog says this:

  2024-05-23T17:05:32.366859-07:00 yogi kernel: [14895.900908] usb 2-1: new SuperSpeed USB device number 5 using xhci_hcd
  2024-05-23T17:05:32.398878-07:00 yogi kernel: [14895.933905] usb 2-1: LPM exit latency is zeroed, disabling LPM.
  2024-05-23T17:05:32.422841-07:00 yogi kernel: [14895.956505] usb 2-1: New USB device found, idVendor=345f, idProduct=2130, bcdDevice=31.00
  2024-05-23T17:05:32.422889-07:00 yogi kernel: [14895.956531] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
  2024-05-23T17:05:32.422895-07:00 yogi kernel: [14895.956540] usb 2-1: Product: USB3 Video
  2024-05-23T17:05:32.422899-07:00 yogi kernel: [14895.956546] usb 2-1: Manufacturer: UltraSemi
  2024-05-23T17:05:32.422902-07:00 yogi kernel: [14895.956552] usb 2-1: SerialNumber: 20210623
  2024-05-23T17:05:32.438897-07:00 yogi kernel: [14895.973543] usb 2-1: Found UVC 1.00 device USB3 Video (345f:2130)
  2024-05-23T17:05:32.498521-07:00 yogi kernel: [14896.032696] hid-generic 0003:345F:2130.000A: hiddev0,hidraw1: USB HID v1.10 Device [UltraSemi USB3 Video] on usb-0000:00:15.0-1/input4
  2024-05-23T17:05:32.503063-07:00 yogi mtp-probe: checking bus 2, device 5: "/sys/devices/pci0000:00/0000:00:15.0/usb2/2-1"
  2024-05-23T17:05:32.503640-07:00 yogi mtp-probe: bus: 2, device: 5 was not an MTP device
  2024-05-23T17:05:32.592287-07:00 yogi (udev-worker)[16747]: controlC1: Process '/usr/sbin/alsactl -E HOME=/run/alsa -E XDG_RUNTIME_DIR=/run/alsa/runtime restore 1' failed with exit code 99.
  2024-05-23T17:05:32.603188-07:00 yogi systemd[4030]: Reached target sound.target - Sound Card.
  2024-05-23T17:05:32.606696-07:00 yogi mtp-probe: checking bus 2, device 5: "/sys/devices/pci0000:00/0000:00:15.0/usb2/2-1"
  2024-05-23T17:05:32.606803-07:00 yogi mtp-probe: bus: 2, device: 5 was not an MTP device
  2024-05-23T17:05:32.715245-07:00 yogi wireplumber[4049]:  Failed to call Lookup: GDBus.Error:org.freedesktop.portal.Error.NotFound: No entry for camera
  

  A relevant detail is that it has both the video and audio device stuff, but it also has a HID interface. There is some reverse engineered work for doing things via that HID interface here: https://github.com/BertoldVdb/ms-tools

  In contrast, when plugging the device into a USB 2.0 port, it's logged as:

  2024-05-24T10:47:05.913518-07:00 yogi kernel: [42791.625745] usb 1-4.2: new high-speed USB device number 13 using xhci_hcd
  2024-05-24T10:47:06.093496-07:00 yogi kernel: [42791.807407] usb 1-4.2: New USB device found, idVendor=345f, idProduct=2130, bcdDevice=21.00
  2024-05-24T10:47:06.093516-07:00 yogi kernel: [42791.807417] usb 1-4.2: New USB device strings: Mfr=1, Product=5, SerialNumber=7
  2024-05-24T10:47:06.093520-07:00 yogi kernel: [42791.807419] usb 1-4.2: Product: USB2 Video
  2024-05-24T10:47:06.093521-07:00 yogi kernel: [42791.807421] usb 1-4.2: Manufacturer: UltraSemi
  2024-05-24T10:47:06.093521-07:00 yogi kernel: [42791.807423] usb 1-4.2: SerialNumber: 20210621
  2024-05-24T10:47:06.113028-07:00 yogi kernel: [42791.826992] usb 1-4.2: Found UVC 1.00 device USB2 Video (345f:2130)
  2024-05-24T10:47:06.183876-07:00 yogi kernel: [42791.897291] hid-generic 0003:345F:2130.0013: hiddev0,hidraw6: USB HID v1.10 Device [UltraSemi USB2 Video] on usb-0000:00:15.0-4.2/input4
  2024-05-24T10:47:06.187848-07:00 yogi mtp-probe: checking bus 1, device 13: "/sys/devices/pci0000:00/0000:00:15.0/usb1/1-4/1-4.2"
  2024-05-24T10:47:06.188320-07:00 yogi mtp-probe: bus: 1, device: 13 was not an MTP device
  2024-05-24T10:47:06.286059-07:00 yogi (udev-worker)[77117]: controlC1: Process '/usr/sbin/alsactl -E HOME=/run/alsa -E XDG_RUNTIME_DIR=/run/alsa/runtime restore 1' failed with exit code 99.
  2024-05-24T10:47:06.315144-07:00 yogi mtp-probe: checking bus 1, device 13: "/sys/devices/pci0000:00/0000:00:15.0/usb1/1-4/1-4.2"
  2024-05-24T10:47:06.315285-07:00 yogi mtp-probe: bus: 1, device: 13 was not an MTP device
  

  To round things out, here's how the MS2109 dongle is logged (it only...

  Read more »

• The five hurts

  WJCarpenter • 05/22/2024 at 01:39 • 0 comments

  I figured out what's up with the 5fps over the cheap dongle I have. In fact, I ordered a second, newer looking dongle, and it also gave me only 5fps.

  Essentially all of these low-cost HDMI-to-USB dongles are based on the Macro Silicon MS2109 chip. It took some sleuthing, but I eventually found a datasheet for that chip. The chip has two output modes: YUV422 and MJPEG. MJPEG is what we really want, and the chip can output up to 1920x1080@30fps. In YUV422 mode, it's limited to 1920x1080@5fps. The dongles (at least the ones that I have) are configured for YUV422 mode, and that can't be changed without modifying the firmware in EEPROM. I don't know why the manufacturers selected that very limited output format.

  Meanwhile, it looks like the go-to chip for the recent generation of those dongles is the Macro Silicon MS2130 (datasheet). In addition to the faster USB 3.0 interface, it supports output of 1920x1080@60fps for both YUV422 and MJPEG outputs. (The datasheet says the chip can be programmed to output 4K@15fps, but I haven't seen that mentioned in any advertisements for these things.

  The older MS2109 dongles can be had for around US$10, and the newer MS2130 dongles go for around US$20. For this sort of project, that price difference doesn't matter too much. I got a dongle with an MS2130 chip:

  Bus 002 Device 094: ID 345f:2130 UltraSemi USB3 Video

  It does, indeed, emit video at FHD@60fps. 

  I could wish that it emitted MJPEG, but at least I am seeing the full resolution and frame rate.

• Expensive-ish HDMI encoder

  WJCarpenter • 05/18/2024 at 22:54 • 0 comments

  When I started thinking about this cheap encoder project, I assumed it was more likely than not that it wouldn't be good enough for household TV watching. To hedge my bets, I ordered a standalone HDMI encoder box. It's this one:

  I selected this one according to the Cheap and Lazy algorithm. I saw it mentioned by some other people, and it was inexpensive relative to others mentioned. It was also readily available with just a short delay. The day I bought it, there was also an instant coupon for US$20 off the price.

  I ran the same sort of POC test with this device that I ran with the cheap encoder dongle earlier. Results were pretty good. Here is what vlc says about the stream:

  vlc was able to consume and display a very clear image with both H.264 and H.265 codecs. Despite best efforts, I was only able to get a frame rate of 30fps. I'm not sure if that's a limitation of the device, a limitation of the Fire TV stick, or some other settings thing. The video and audio were completely smooth in vlc.

  
  As a side note, there was some small inconvenience as a side effect of buying the least expensive device. The setup instructions are for Windows (whatever that is), and the device comes configured with a static IP address of 192.168.1.120. I already have a device with that address, so I had to temporarily turn it off while I configured this new device. Luckily, there is a simple web UI that lets you change the network configuration and various other things. There was an unexpected bonus (because I didn't pay attention to it while shopping). The device can record to an SD card for subsequent download. I don't know if I'll ever use that, but it's nice to have.

  Regardless of what happens with this overall project, I should be able to scratch my single-channel itch with this device after applying some software magic.

• Proof of concept

  WJCarpenter • 05/17/2024 at 22:28 • 0 comments

  Do those dongles actually work, and are they better than "marginally above crap"? Let's see what we can see.

  When I plug the dongle into my Linux machine, it shows up as this USB device:

  Bus 001 Device 038: ID 534d:2109 MacroSilicon

   Searching around for those ID strings leads to the MacroSilicon MS2109 chip that powers the dongle that I have. I haven't pursued the innards of this chip, but it was easy to figure out that it's supported by the Linux v4l subsystem and presents as a pair of video devices.

  crw-rw----+ 1 root video 81, 2 May 17 11:36 /dev/video2
  crw-rw----+ 1 root video 81, 3 May 17 11:36 /dev/video3
  

   At this point, I just needed any old HDMI video signal. I plugged in an Amazon Fire TV stick that was in my "send to the thrift store" box. I used vlc to open /dev/video2 as a capture device.

  Hey, that worked! You can't see it, but the audio also played correctly. On my 4k monitor, that window with the FHD video doesn't look completely sharp, but it's good enough for the POC. Here's what vlc says about it:

  The resolution is FHD (1920x1080), but it's only 5fps. The playing video has the choppy look that you'd expect from that frame rate. My quad-core low-powered Linux box reports about 30-50% on each core.

  It didn't really prove anything that I didn't already know was OK, but I also streamed that video. My first attempt was to use mjpg-streamer, which I had used in an earlier project., but some python thing was giving me a hard time today. Instead, I used µStreamer, which is part of the PiKVM project. It doesn't include audio for this kind of setup, but I was able to view the playing video in a web browser.

  Next steps: I've ordered a newer dongle that has a USB 3.0 interface, and I have some beefier Linux boxes to try. I am cautiously optimistic.

• The HDMI dongles

  WJCarpenter • 05/17/2024 at 22:01 • 0 comments

  There are dongles that take HDMI input and plug into a USB port for output, and they are extremely cheap. They are readily available online for US$10-20. I've had one that looks like this for several years:

  Until today, I'd never plugged it in. It was probably pretty cheap and I bought it for hazy future use after seeing it mentioned in some article somewhere. The future is now! The documentation that came with it claims it can accept an input of 4K@30fps and output FHD@30fps. I don't know if that claim is actually true. Even if it is, 30fps is less than desirable for watching TV. Newer iterations of similar dongles claim to output up to FHD@60fps. My TV is only FHD, not 4K, so FHD@60fps would be fine for now if it turns out to be true.

• The DVR

  WJCarpenter • 05/17/2024 at 21:52 • 0 comments

  DVR, I hear you ask. These days, it's relatively easy to create your own DIY DVR with inexpensive hardware. For example, a Raspberry Pi with a couple of USB hard drives is enough computer power. Digital tuners are available for receiving OTA programming. The most well-known is HDHomeRun, but there are others. Many paid programming services also make their live TV content available for streaming via a technology called TV Everywhere. For the software, I've experimented with Channels DVR, emby, and a few others. None are perfect, but several are good enough.

  One of the side-effects of these various DIY DVR efforts is that the people involved, both the developers and the users, have lots of interesting ideas for how to deal with programming. It is some of those ideas that I am exploring in this project. In particular, it's aspects of these very long discussion threads in the Channels DVR community forums that set me to thinking.

  BETA: Chrome Capture for Channels

  HDMI for Channels

  ADBTuner: A “channel tuning” application for networked Google TV / Android TV devices

View all 9 project logs