AND!XOR DEFCON 24 Badge

Wow. What a DEF CON that was!

We are blown away by the popularity of our unofficial badge. Our goal was to put a badge in people's hands that we liked and hope that others like it too. We've been putting a lot of our work out in the public to ensure enough people know about it before DEF CON that we can make our money back on production. We hit that goal and even a little above it with some donations from various folks. We are pouring the extra into next year's badge to help do things right and seed some of our development. At one point we had about 300 people lined up to buy 70 badges. Watching the elevators on the 9th floor open at once and seeing everyone poor in was humbling. We really wished we had more and were disappointed not everyone could get some.

Our favorite comment we received from owners of the badges was the fact that they had difficulty walking around the conference without being stopped. Sorry everyone! We were stopped a lot too but that usually led to some great conversations. We went home with tired feet and no voices but we wouldn't have it any other way.

Next year we won't be hand assembling, nor will our learning curve be as steep. Our team is already brainstorming what to do. Right now the ideas are grand, eventually we'll narrow it down and start our design but for now it's all about big ideas! We also plan to do a run of a lot more badges.

In the near term, you won't see anything from us on development. We'll probably startup another hackaday.io page for DC25. We'll be quietly brainstorming and doing proofs of concepts in between real-life responsibilities. DC25 will be bigger and better.

Thank you everyone at DEF CON, you made all this work worthwhile!

It's DEF CON week! We finished production of the badges over the weekend and have packed everything up. We'll be bringing spare parts and tools for the possibility that a few Benders break during the conference. Looking forward to seeing everyone there!

We've wasted so many hours fixing radios and LEDs on dozens of badges, turns out we were able to fix through software. We've posted some details on Twitter and complained/celebrated in IRC over the past week but wanted to write down exactly what happened.

Radio

Roughly 20% of all badges were having radio failures on startup. Through some built in debug features in code we could easily see the PLL was not locking on these badges during startup. Very strange behavior. In some cases this was due to a loose load capacitor for the radio's 32Mhz oscillator. Easy fix, but not enough on most of the failed radios. We then discovered that the radios on failed badges would start after a flash (about 30 seconds from cold startup). Turns out our badges were starting to quickly and not allowing the transceiver to startup in the radio. Moteino DualOptiboot has a similar fix and sets a delayed startup fuse. Updating our bootloader has since fixed a dozen badges. We tried adding a delay during setup() in the main badge code, sadly this is too late in the execution. The delay needs to occur very early in the bootloader.

LEDs

40% of assembled badges were having LED failures. We're using WS2812B LEDs (at least that's what we ordered) that require specific timing. In code we bitbang to send the colors and meet the timing requirements. No issues during prototyping or the first 20 badges. Until we ran out of LEDs and switched to our large batch from China. We should have known better when the reels came labeled as WS2815B. Most of the badges with these WS2815B failed. Usually the left eye (LED #1) would work but 2-8 would flash or show full white randomly. Our fix was to desolder the left eye and replace with a new LED. Most of the time this fixed it as the LED would get a better connection to the pad.

Ultimately the fix was in software. Our timing of the 1s and 0s to the LEDs was just outside spec. The higher quality LEDs easily dealt with it but not the WS2815Bs. We've messed with the High/Low timings in code before but nothing helped. Eventually we tried lowering the latch time to 45usec, suddenly all the broken badges worked! 17 badges off the rework pile and into the complete pile.

Production was in full swing over the weekend. We started Friday evening and continued into the afternoon on Sunday. Once the solder-smoke had cleared, 15 bling badges were completed, all main badges had been reflowed and assembled. Of the main badges 36 have been fully tested and set aside for final badge code flashing and kitting.

During the evenings I will be reworking badges that failed testing and running each through full functional tests.

We made a couple minor tweaks to the firmware so that will need to be flashed on all badges during kitting.

If you've been following us on Twitter (@ANDnXOR) production of the final badges has begun. The goal is to produce:

• 100 White Badges
• 20 Black Badges
• 50 White LED-only "Bling" Badges

This upcoming weekend the three of us will converge on San Diego and spend the weekend furiously soldering, flashing, and testing badges. To help ease the workload we have started production a little early. To date, we have reflowed 60 white badges with 22 of those fully assembled and flashed. This has helped us work out the physical space and process required to pull this off.

If you follow us at all on Twitter, you've probably heard us complaining and celebrating success with the RFM69 radio we're using on the badge. Through many frustrating hours of tedious debugging of the radio's registers and a lot of trial and error we believe we've narrowed down an issue related to production of the radios themselves.

Issue: Occasionally we would notice the badges unable to receive data. It turns out the radios were getting stuck in a frequency synthesizer mode waiting for PLL to lock. RFM69 has an automatic sequencer (that we're using) that does the job of walking the radio through standby, receive, synthesizer, etc modes as our application switches it. Frequency synthesizer is an intermediate mode between standby and receive. According to the data sheet the radio should spend about 60 usec in this mode before jumping to receive. In our case it was getting stuck.

Hours and hours of debugging lead us to be able to recreate the issue (cycle the badge several times by removing the battery) and detect the issue (opmode = synthesizer and irqflags ready = 0). Every register we tried to manipulate to get it into a receive mode failed. Even re-initializing the radio failed. As a workaround the badge now detects the issue on startup and prompts the user to cycle the badge by removing the battery. Not ideal, but it works.

Eventual Solution: Over the past few months my four year old has become interested in what I've been working on. He likes to watch the animations and flashing lights. This past weekend I let him play with it only to have him drop it on the floor. Upon picking it up and restarting, the radio failed every time it was restarted. I inspected the radio and found this:

Notice the capacitor is dislodged from one of its pads.

After fixing the capacitor the PLL lock failure went away. I wasn't able to repeat anymore either. Not even once after many battery removals. This leads me to believe the dislodged capacitor is a load capacitor for the 32mhz crystal (also pictured). Without it, the radio doesn't have a good reference clock and can't lock.

We've started producing a few of the badges now for DEFCON. Part of the production process includes a functional tests of each component. The third badge produced came up with the PLL Lock error when first booted. Upon inspection, its capacitor was also dislodged. It turns out the next four radios inspected also have this issue to various degrees. Most are good enough. There is clearly a manufacturing QA issue here.

Fortunate for us, this is a simple fix. The capacitor is extremely small but doesn't take much work with a soldering iron to straighten it out.

Over the weekend we took two prototype badges (rev 2 and rev 3) to Layer One in Los Angeles. This was our first time at the conference and enjoyed the small size compared to DEFCON. Throughout the conference and Saturday Night activities we sported both the Layer One badge and our Bender badges. This was as much of a test of how well they performed as much as an early peek to folks at the conference.

The badges were great conversation starters, people were very impressed with the OLED screens and animations. We also identified a fair number of bugs we haven't seen before, the biggest being random freezes.

Revisions 1 and 2 of the badge had noise issues related to the layout of the board. This would make the badge seem to be deaf. What we also found was that revision 3 also would occasionally drop packets although with much higher gain due the hardware being fixed. In addition we determined that the badges could not pass data if within 10 feet of each other. After hours of debugging and monitoring registers in the radio we found a race condition during initialization and misuse of a sensitivity setting that was reducing the dynamic range by about 8dB.

While frustrating, this is very good news. The software fixes we employed for the dynamic range and race condition issues have improved the reliability of the radios immensely. Even on revision 2 which has the hardware issue and has about a 35dB loss it is able to send and receive data with no issue (at reduced range).

At one point we were leaning towards dropping the radios and social features from the badges altogether. Even with revision 3 we couldn't send / receive data reliably. Fortunately that won't be happening.

There's something oddly satisfying about a shell. Especially when it's coming from your badge.

It's high time we give everyone a peak at what we're doing. Two short videos, one is the startup sequence of the badge and the other is a Flappy Birds clone we created.

The month of April was spent developing the final design of the badge. We just completed the third revision and are confident in the design. We are ordering an initial round of 20 PCBs to perform a final validation before placing an order for the remainder. The 20 PCBs will serve as the special badges for friends and will be in a different color.

The focus now shifts completely to software and production. Software has reached 4000 SLOC and using about 75k of the badge. All hardware components are fully integrated and in use. The badge now includes a 2MB SPI flash chip, of which we are using about 200kB for animations, graphics, and settings. USB is fully functional and we routinely flash the badges over USB. Although, debugging over USB is difficult so we keep our USB to Serial adapters close by.

As far as bling goes, the badge has 14 different animations utilizing the display and LEDs with a few more planned.

As recently as last week we added a basic shell that runs over serial or USB for control of the badge and additional user interactions. There is something oddly satisfying seeing a terminal prompt come from an embedded device.

Since the last update, we've added a light sensor (to save power by dimming the LEDs) and a tilt sensor that flips the screen. This will make the badge useable when worn around the neck.

Next major features to add to the badge will be to build upon the social features and games (several planned).

On the production front, we have about 1/3 of our material in hand. Capacitors, resistors, flash chips, displays, tilt sensors, and crystals. With last of our capacitors and buttons in route.

Finally, we've produced three revision 2 badges that will be headed to Layer One in Los Angeles at the end of the month. Looking forward to showing them off.