Close
0%
0%

IronEar - On-device sound recognition

Sound recognition that runs on the device. Works with Home Assistant, Meshtastic, Meshcore, or your own software.

Similar projects worth following
239 views
0 followers
Acoustic sensor node for Home Assistant, Meshtastic, WebSocket, and REST API. Online and offline mesh capabilities. No audio is stored or leaves the device. ESP32-S3 with three PDM mics, LoRa module, GPS module, compass module, power module, and battery.

Pre-launch page on Crowd Supply: https://www.crowdsupply.com/ironear-technologies/ironear

Audio pipeline: energy gate and gain setting. Raw audio is classified on the device with Google's YAMNet model (>500 classes), followed by filter settings (background noise, confusion clusters, thresholds, and cooldown).

Learning: custom scenes (a sound fingerprint of a place over time) and individual sounds can be taught.

Broadcast: scene triggers, modes, and the filtered stream can be broadcast to Home Assistant or a Meshtastic channel, plus node-to-node mesh functions.

Audio Lab

• Live microphone status and a one-click recalibration of the three mics (five seconds, quiet room).

• Pipeline layers, each with an on/off switch and a question mark: energy gate with a floor that follows the room, classifier (always on, 521 classes), confusion clusters, class grouping, background subtraction, background calibration, per-class thresholds, multi-frame voting, cooldown.

• Per-layer settings behind each row: thresholds per class, votes needed and window, cooldown per class, grouping and cluster definitions.

• Config profiles: save the whole pipeline setup under a name, load one, return to default.

• Raw stream: every classified window with its top five classes, scores, level in dB and inference time, and the reason a window was stopped (gate, cooldown, threshold, voting) or passed.

• Filtered stream: what survived every layer, plus learned-sound and scene matches as they happen.

• Replay of the board's own history into both streams on page load, so the page never opens empty.

• Pause for the live streams.

Learning

• Learned sounds: teach up to five sounds of your own (doorbell, garage door, a particular alarm) from a 3 to 30 second recording; rename, delete, set the match threshold per sound; live matches shown as they happen.

• Learned scenes: record the sound character of a place over 30 seconds to 5 minutes (quiet night, dog park, busy street); the board recognises the scene later and keeps it as a state; rename, delete, see the live similarity.

• Learned motion: with presence sensing on, record a movement (car leaving, gate opening, hallway walk) as a radio-signal shape and match it later; up to eight.

• Learned sounds and motions become labels you can route on the Broadcast page.

Broadcast

• Triggers: a name, one or more labels (built-in classes, learned sounds, motion), minimum score, votes, cooldown, optional "only with motion"; routes to LoRa and Home Assistant with optional position.

• Filtered alerts: route whatever survives the full pipeline, by class, for the most accurate alerts.

• Scenes: route a learned scene as a state with similarity threshold, number of matches and window.

• Test button on every trigger, filtered alert and scene: a synthetic event through the real routing.

• Rate caps per minute for LoRa and Home Assistant; fleet merge so one LoRa line per event names the node that heard it loudest.

• Live broadcasts: what leaves the box, from this node and from linked nodes.

• Log of the last ten broadcasts with the outcome per transport (sent, capped, locked, off, failed).

Mesh and fleet

• Map with this node and every fleet node that has a position.

• GPS: fix state, position, altitude, satellites in use and in view, the surveyed position the node keeps across reboots, re-survey, and placing a node by hand on the map.

• Compass: heading, one-button calibration with a rotation check.

• LoRa: on/off, protocol choice between Meshtastic and MeshCore, region and frequency as fixed by the firmware, transmit and receive counters, channel create, join from a link or key, share as a Meshtastic link or a MeshCore key, radio test.

• House link: nodes on the same channel link over Wi-Fi at home or off-grid, relay each other's alerts to LoRa or Home Assistant, and share time.

• Presence sensing: off, slow or fast, using the Wi-Fi signal between linked nodes as a motion sensor, with the live motion index.

• Fleet table: every node heard, how it is linked (house, LoRa, both), battery, signal, last heartbeat, whether its settings match, copy settings from this node, place a node on the map.

• Texts received on the channel, from a phone or another board, shown as events.

Device (Settings)

• Board connection: Wi-Fi status, scan and join a network, the board's address with a pinned fixed address or release to DHCP, raise the setup access point for ten minutes, uptime, GPS summary, WebSocket address.

• Firmware update...

Read more »

  • 1 × ESP32-S3-WROOM-1
  • 3 × Infineon IM73D122
  • 1 × EBYTE E22-900M22S
  • 1 × Quectel LC76GAB
  • 1 × MEMSIC MMC5983MA

View all 7 components

  • My first video

    ststephen510 • 15 hours ago • 0 comments



  • Pilot - resolving the last design issues

    ststephen510 • 3 days ago • 0 comments

    This prototype is not strictly my pilot build. There are still a few minor board changes planned. But my last board issues have now been officially resolved:

    This board has now a ceramic antenna giving good indoor access to the GNSS satellites,

    The user and reset buttons are in different areas of the board and the toothpick solution is replaced by a more clever design (see video with sound),

    A JST PH connector for UART was added for the labs fixed-frequency testing. The connector will likely be removed for the production version.

    While the enclosure will not be shipped with a waterproof enclosure there is a relatively simple path forward for anyone to make their IronEar nodes waterproof.


  • Some core firmware functions

    ststephen510 • 4 days ago • 0 comments

    The firmware runs on the ESP32-S3 with ESP-IDF 6.0 and does all the listening on the board. Three PDM mics feed one second windows into a YAMNet classifier with 521 sound classes, quantized to int8 and running under TensorFlow Lite Micro. The model weights sit in PSRAM, which turned out to be faster than running them from flash. One window takes about a second to classify, so the board keeps up with real time with a little margin.


    Nothing leaves the box as audio. What comes out is a label, a score and a level above the room floor. On top of the fixed classes you can teach the board your own sounds. It records the classifier's embedding of your doorbell or your garage door for a few seconds and matches each new window against it from then on. Scenes are a different thing. You let the board listen to a place for a minute or more and it keeps a fingerprint of which sounds occurred and how often. Later it compares the last two minutes of what it hears against that fingerprint, so "quiet night" or "dog park" become states the board can tell you it is in.

    Alerts go out three ways. Over LoRa on a private Meshtastic channel, so a phone with a radio sees them without any network. Over MQTT into Home Assistant, where the board shows up as a device with its sensors and an alert event. And between boards in the same house over ESP-NOW, so a node without a radio can hand its alert to one that has one.

  • RC2 - One board with 1.5 issues

    ststephen510 • 5 days ago • 0 comments

    RC2, my second release candidate :). So much better than RC1. Charging and charge status now work beautifully. I'm using a Texas Instruments BQ24295RGER, a TLV62569DBVR, and an ADC [for battery voltage]. No singing caps. I switched the LoRa module from NiceRF to an EBYTE module, and the Quectel module from L76K to LC76GAB. The antenna matches the EBYTE modular approval for FCC. All the changes are mainly driven by FCC and CE RED certification.

    The issues with RC2:

    - My Rainsun GPS1003 is just not seeing any satellites. I'm so convinced it must be an issue with the new LC76GAB that I message Quectel. I receive a firmware update for the GNSS module, but still nothing. I hold a big ceramic antenna to the module, and then and there I receive an abundance of satellites. I decide that the next prototype version needs a ceramic antenna.

    - Do you see the three tiny holes on the top of the enclosure? These are to press the reset, boot, and user button with a toothpick :). Never liked that solution, so the next prototype has this solved.

  • Prototype RC1 - My first PCB

    ststephen510 • 5 days ago • 0 comments

    In my job, RC means release candidate. This means that after my proof of concept, I thought this project was cool and that maybe someday I could convince others that this is a pretty useful sensor to have. My wife and I were staying with my parents and parents-in-law during a period of parental leave (thank you, California), and that is when I was able to learn so much about PCB design. I used KiCad (also thank you, KiCad, for providing this awesome tool for free) and redesigned my board three times, because by the end of each redesign I had learned so much that I thought it would be best to just design the PCB one more time. I learned about these amazing datasheets for each component, about PCB manufacturers, about trace widths for data and power lines. And probably a few hundred other lessons.

    In this prototype, I designed a LoRa breakout board, a mic breakout board, a controller board, and an enclosure. The main issues on this design were:

    • Battery charging not working
    • Singing caps
    • Hot inductor
    • A missing pull-up on EN, which kept the board from working. I had to bridge EN to +3V3 on each board to make it work.

  • Prototype Alpha - Proof of Concept

    ststephen510 • 5 days ago • 0 comments

    It was relatively easy to create models that could classify a few sounds. I used Edge Impulse to train them. At that time the most difficult task was finding training data and cutting it into the correct window size. My favorite sounds to teach and to show the proof of concept were "Cough", "Clap", and "Whistle", sounds that I could easily make to test my models. My wife learned quickly that if there was clapping, or coughing, or whistling in the house, I was testing my project.

    Anyways, my hardware setup for the first proof of concept was an ESP32-S3 dev board and 3x INMP441 modules, and at that time I also had some LD2450 and LD2451 radar modules that I was testing.

    Here you can see my "sentry tower" setup. A slice of pool noodle with holes for the mics and a channel for the wiring, covered by a sock.

    At that time I also came across Google's YAMNet and TFLite. YAMNet already knew 521 sounds, far more than I could ever collect training data for, and TFLite Micro was the way to run a model like that on a microcontroller - except it didn't fit on my microcontroller. So from then on the task was fitting it onto an ESP32-S3.

  • The Beginnings

    ststephen510 • 6 days ago • 0 comments

    Hi, as I'm writing this now my prelaunch page on Crowd Supply is live. This is a huge milestone for me and not something I anticipated a year ago. The start of the project was triggered by a few thoughts that merged into this idea: (1) I learned about the huge human trafficking industry and the work of https://techagainsttrafficking.org/. I always wondered if there was a way I could contribute as a manufacturing engineer. I wished I had other engineering skills, such as hardware or software engineering. (2) I read about Ukraine's Sky Fortress and how they were able to successfully counter Russian drone attacks with a relatively cheap audio sensor mesh. (3) I had a conversation with a Bay Area company that builds sentry towers to surveil industrial perimeters. Those sentry towers are a bit creepy and super expensive.

    All this somehow contributed to starting a project to detect sounds.


    Here a picture of my early project phase:

    Raspberry with Respeaker 2-mics, USB GNSS Module, and RTL-SDR. RPI was browning out constantly and I understood little. Trying to create a sensor with useful information.

View all 7 project logs

Enjoy this project?

Share

Discussions

Does this project spark your interest?

Become a member to follow this project and never miss any updates