Project Logs

Collapse

• Streaming

  deʃhipu • 01/11/2023 at 22:10 • 0 comments

  It doesn't help much to have a camera on your robot, if you don't do anything with the video. Displaying it on a display is cool, but not really that useful when it has to be attached to the same device. Vision algorithms are still a bit too advanced to me, so I decided to try the other useful option: streaming it into a browser.

  Displaying a single jpeg image taken with the camera was easy enough, but we want live video - can we do that? Turns out that we can, by using something called M-JPEG (for motion jpeg), and the multipart/x-mixed-replace MIME type. All we have to do is send the JPEG images separated with a boundary, each with its own http header. After a few hours of tinkering, I finally got something like this working:

  import board
  import busio
  import esp32_camera
  import asyncio
  import socketpool
  import wifi
  
  from adafruit_httpserver.server import HTTPServer
  from adafruit_httpserver.response import HTTPResponse
  
  
  server = HTTPServer(socketpool.SocketPool(wifi.radio))
  PORT = 80
  BOUNDARY = "FRAME"
  
  i2c = busio.I2C(scl=board.IO35, sda=board.IO33)
  data_pins = (
      board.IO21, board.IO17, board.IO16, board.IO18,
      board.IO37, board.IO34, board.IO36, board.IO39,
  )
  cam = esp32_camera.Camera(
      data_pins=data_pins,
      pixel_clock_pin=board.IO12,
      vsync_pin=board.IO40,
      href_pin=board.IO38,
      pixel_format=esp32_camera.PixelFormat.JPEG,
      frame_size=esp32_camera.FrameSize.QVGA,
      i2c=i2c,
  )
  
  @server.route("/")
  def base(request):
      with HTTPResponse(request) as response:
          response._send_headers(content_type='multipart/x-mixed-replace; boundary=%s' % BOUNDARY)
          while True:
              jpeg = cam.take()
              response._send_bytes(request.connection, b'--%s\r\n' % BOUNDARY)
              response._send_bytes(request.connection,
      b'Content-Type: image/jpeg\r\nContent-Length: %d\r\n\r\n' % len(jpeg))
              response._send_bytes(request.connection, jpeg)
              response._send_bytes(request.connection, b'\r\n')
  
  
  async def poll(interval):
      server.start(str(wifi.radio.ipv4_address), port=PORT)
      while True:
          server.poll()
          await asyncio.sleep(interval)
  
  
  async def main():
      poll_task = asyncio.create_task(poll(0))
      await asyncio.gather(poll_task)
  
  asyncio.run(main())

  This works, but there is a small problem. As soon as we start streaming, the server stops responding to any other requests, and our robot would stop waking or doing anything else, because the loop that sends the video frames is not async.

  I will need to modify how this particular http server works to make the endpoints into async functions, so that we can add an await command in the loop that sends the frames. But that is something for another day.

• Everything Works!

  deʃhipu • 12/31/2022 at 20:01 • 0 comments

  The PCBs arrived in the break between holidays, and I got to assemble one of each. First the Raspberry Pi Pico shield:

  I tried it without the crystal first, with a bodge wire to a free pin to generate the master clock with PWM. Once I had it verified working that way, I removed the bodge and soldered the 24MHz crystal, and it works flawlessly. Then I moved on to the S2 Mini shield:

  I tested the previous version of this with the crystal connected with bodge wires, and it didn't work, so I had very little hope for this, but turns out it was the fault of the bodge wires, and it works perfectly fine with the crystal soldered on the PCB.

  I think that this pretty much concludes the project. I will make the design files available, and that's it.

• Version 4 and MooCam

  deʃhipu • 12/18/2022 at 22:58 • 0 comments

  Since I was ordering PCBs anyways, I went ahead and added the version 4 of the shield, with the footprint fot the oscillator added, to the order:

  Nothing really exciting there. I also ordered the oscillators, and they should arrive shortly, so we will see how well this works.

  The pull request for making the xclk pin optional got merged to CircuitPython, so that should work once version 8.0 is released.

  I also had a stab at a similar shield for the Raspberry Pi Pico and Pico W. It's an excellent moment for it, since the Raspberry Pi Foundation loves surveillance so much – they should be more than happy that their corrupted government is getting more tools to make sure nobody harbors Unapproved Opinions.

  Since there was room, and since extra storage might be useful with a camera, I also added a microSD card socket in there, and a button for the shutter on the front side, together with a police star.

  We will see how all that works, but unless I made some horrible mistakes, it should work.

• Next Steps

  deʃhipu • 12/15/2022 at 19:18 • 0 comments

  Now that I have it working, what is the next step? Well, right now the shield requires the microcontroller to provide the main clock signal for the camera chip. That means that the camera needs at least one PWM channel to work, and with only 8 channels available on the esp32-s2, and with 8 servo needed on my #Fluffbug robot, that doesn't work very well. But I also have an off-the-shelf camera module that doesn't even have the main clock pin broken out, because it has an oscillator that takes care of that. Could I add an oscillator to my shield, and make the PWM channel unnecessary? Probably. But how?

  Well, let's search around for schematics of those modules and see if we find any useful hints. The first one I found is already promising:

  It shows that we only need to connect a single part to the MCLK pin, the one marked as X1, and labeled "24M" underneath. So probably some kind of a 24MHz oscillator. There is some more detailed documentation at https://doc.embedfire.com/mcu/stm32/f407batianhu/std/zh/latest/book/DCMI_OV2640.html#id31, together with an image:

  Here the oscillator Y1 is connected through a resistor, but otherwise it seems the same. The text under the image says " 标号处的是一个24MHz的有源晶振", which translates to "The label is a 24MHz active crystal oscillator". So, active crystal oscillator. It's also powered with the 2.8V, so we need a CMOS one.  And I can probably test it without having to make a new PCB, as I don't much care about accuracy here, just that the clock is provided.

  Unfortunately I don't have any 24MHz active crystals oscillators in my drawer at the very moment, so I will need to order one to see how that works.

• Great Success

  deʃhipu • 12/15/2022 at 00:17 • 0 comments

  Last time I gave up on the ESP32-S2, so today I decided to see how this camera works with the RP2040 instead. That uses a different library, called imagecapture, and since that library makes the master clock pin optional, I decided to first try with an off-the-shelf module that I have, which I couldn't use previously, because it doesn't expose the master clock pin – it has a crystal oscillator on board for this.

  I got everything connected, I copied the example code, and... it just works, as advertised. Wow.

  But wait, does that mean my shield is faulty? Let's connect it and see!

  This time it didn't work as great – I got some data from the JPEG mode, but when I saved it in a file, it wasn't recognized as a JPEG file. So I looked closely. A normal JPEG file starts with bytes ff d8 ff e0 00, but the data I got starts with ff e8 ff d0 00 – the e8 and d8 are swapped! But how? Well, e8 is 11101000 in binary, and d8 is 11011000 – could it be I have two data pins swapped? I tried swapping them back, and lo and behold – a correct JPEG image file popped out!

  Could it be that was my problem all along? The diagram telling which pin is which on the camera was wrong? After all, datasheets have mistakes in them. So I quickly went back to the S2 Mini to see if my shield works after all if I swap the two pins. I connected everything back and... nothing. As before the 16MHz clock returns unintialized memory, and the 20MHz clock just sits there waiting for data indefinitely.

  So I went through the pins one more time, and I discovered that the stickers that I got with my S2 Mini boards, with pin numbers on them have the pins 12 and 13 swapped. And it just so happens I was using pin 12 as the pixel clock. Except I had it s pin 13 in my code, because of the mistake in the sticker. Changing it to 12 suddenly made everything work (with the 20MHz master clock, the 16MHz is still broken). Both the RGB565 and the JPEG modes work perfectly!

  I can't even say how happy that makes me. This shield was a lot of frustration, and it the end it all turned out to be due to mistakes in pin numbering.

  Here is the working code, if anybody is interested:

  import adafruit_ili9341
  import busio
  import digitalio
  import esp32_camera
  
  
  displayio.release_displays()
  display_bus = displayio.FourWire(
      busio.SPI(clock=board.IO7, MOSI=board.IO5),
      command=board.IO9,
      chip_select=board.IO3,
      baudrate=80_000_000,
  )
  display = adafruit_ili9341.ILI9341(
      display_bus,
      width=320,
      height=240,
  )
  
  i2c = busio.I2C(scl=board.IO35, sda=board.IO33)
  data_pins = (
      board.IO21, board.IO17, board.IO16, board.IO18,
      board.IO37, board.IO34, board.IO36, board.IO39,
  )
  c = esp32_camera.Camera(
      data_pins=data_pins,
      external_clock_pin=board.IO11,
      pixel_clock_pin=board.IO12,
      vsync_pin=board.IO40,
      href_pin=board.IO38,
      pixel_format=esp32_camera.PixelFormat.RGB565,
      frame_size=esp32_camera.FrameSize.QVGA,
      i2c=i2c,
      external_clock_frequency=20_000_000,
  )
  
  g = displayio.Group()
  bitmap = None
  while not bitmap:
      bitmap = c.take()
  shader = displayio.ColorConverter(
      input_colorspace=displayio.Colorspace.BGR565_SWAPPED
  )
  tg = displayio.TileGrid(bitmap, pixel_shader=shader)
  g.append(tg)
  display.show(g)
  display.auto_refresh = False
  while True:
      bitmap = c.take()
      tg.x = 1 # make the grid dirty
      tg.x = 0
      display.refresh()

• More Frustration

  deʃhipu • 12/13/2022 at 23:04 • 0 comments

  So I figured out the display problem – I connected the (unused) reset pin to the camera's master clock pin by mistake, so the display was being reset at 16MHz... After fixing that, I finally got an image on the screen:

  That... doesn't look like any swapped pins. To my untrained eye it looks like simply empty unintialized memory. Could it be that it "works" in the 16MHz mode simply because it does nothing? I tried switching it back to 20MHz, and now I'm getting a suspicious exception:

  Traceback (most recent call last):
    File "code.py", line 45, in 
  TypeError: unsupported bitmap type
  

  So it looks like the bitmap type changes depending on the clock frequency? How does that even make any sense? Madness! 

• Pure Frustration

  deʃhipu • 12/13/2022 at 22:20 • 0 comments

   I dug out this project recently, trying to see if I can get it working with the new esp32_camera module in CircuitPython. I managed to get the board to either just crash and hang, or crash into safe mode, but not to capture any images. Then I went looking at the source code, and noticed a mention that it uses edma mode when the master clock is set to 16MHz. So I changed it from the default of 20MHz, and lo and behold, I got some data coming in both the GREYSCALE and and RGB565 modes. The JPEG mode is still returning no frames, though, though it no longer crashes.

  So I thought: maybe I have the data pins in wrong order somehow, and that is why the JPEG mode doesn't work – because there is a certain header expected by the code, and if the pins are in the wrong order, it never gets the right data. But how do I verify that? Maybe I could at least display the data I'm getting in the RGB565 mode to see if it's correct, and then if it's not, I would at least know the problem lies there and I could try changing the pin order or something.

  So I connected the display to the board with the camera shield, and added all the code that initializes it, and... nothing. White screen.

  Which is very strange, because this is CircuitPython, and so after the code finishes running it should display the REPL, not a white screen. So I started commenting out pieces of code, and it turns out I get white screen as soon as I create the camera object... And then it won't go away until a hard reset. Just great.

  In the mean time I realized that even if I get this shield working, it will not work with the #Fluffbug robot, which was the main reason I was working on it in the first place. Why not? Because the esp32-s2 can only do up to 8 PWM channels, and I'm already using all of them for controlling the servos of the robot. And the camera driver uses at least one PWM channel to generate the master clock.

  So now I'm thinking about giving up on the esp32-s2 all together, and using the imagecapture library with rp2040 instead. I will be doing some further experiments to see if that works any better. At least it doesn't have to deal with the horrible Espressif code.

• Version 3

  deʃhipu • 09/15/2022 at 09:04 • 0 comments

  Another sleepless night, and another PCB designed. This time I put most of the components on the front, and I included all the capacitors and even the resistor between the analog ground and regular ground. I skipped the inductor, though, too much hassle.

  Will it work? Who knows! But if it doesn't, at least I will be reasonably sure the problem is not filtering.

• The Big Cap Conspiracy

  deʃhipu • 08/27/2022 at 22:07 • 0 comments

  I still didn't get any useful results with this shield. I put it away for a while, but recently the camera support for CircuitPython was rewritten, so I though I will try again. All I managed to get is some damaged JPEG frames:

  But that suggests, that indeed both the hardware and software do work and all the connections are correct, there is just a problem with reliability. So maybe some of those capacitors, resistors and inductors that are on those example schematics, that I completely ignored while designing my shield are actually necessary?

  You can see there are quite a lot of them, and I reduced it all to just two, because I didn't want to feed the Big Capacitor consortium that puts all those unnecessary capacitors on the schematics. Well, easy enough to test, what if I put back some additional caps and see if that affects the reliability?

  By the way, someone should really start to produce parts that contain several capacitors in parallel inside them, so that you don't have to do things like this. Anyways, I added two 220nF caps to those 1µF caps I already had there, to see if it would change anything. And yes, it does! Now my code crashes into a hard fault handler instead of outputting corrupted JPEGs unreliably. Which seems like a worse outcome, but the important part is that there indeed was a change, so the filtering actually has an observable effect.

  So the next step is to redesign this camera shield with the full complement of capacitors and other filtering paraphernalia, and see if that works. If it does, *then* I can see if some of them can be removed.

• 1.2V and Version 2

  deʃhipu • 12/02/2021 at 23:33 • 0 comments

  Today the correct 1.2V LDO arrived, so I replaced it on the shield, replaced the camera sensor (in case I fried this one with the incorrect 1.5V) and tried again. And... nothing again.

  With the help of Jeff Epler, I checked the signals on various pins, and as far as my toy scope can tell, they are all correct. In other words, it should all work, but somehow doesn't.

  Next I'm going to look for a different example, not involving CircuitPython, to see if maybe that will work. I also have a Chinese module with the same camera, that I want to try.

  In any case, the shield doesn't seem to need any electrical fixes, so I went ahead and ordered the second version, with slightly nicer traces, and i2c pins in a more traditional place — so it can be used together with other shields.

View all 15 project logs