Project Logs

Collapse

• We're nearly done.

  John Adams • 06/21/2017 at 23:58 • 0 comments

  We'd like to share a video with you of Macrofab programming the badges.

  PCBs are done, everything's come off the pick and place machines, and we're doing really well on our schedule. I think we're going to make it.
  

• On writing network code

  John Adams • 05/14/2017 at 06:00 • 0 comments

  Hello! It's been a couple of months since we've posted here, but here's some fun technical tidbits for you to think about. Our generic postings have moved to out Kickstarter Updates, but I'll try to keep the more technical ones here.

  ---

  I've written and rewritten the networking code about five times for the fight game.

  The last attempt, which I am still on now has taken about three weeks of off and on, but very serious critical thinking. What have we learned?

  Well, first off, you should be very grateful that you get to use TCP/IP on the Internet every day. It is a remarkably well designed protocol and has served us well for many years. When you are attempting to recreate the same protocol on a tiny 48Mhz system with barely any stack space or RAM (16KB, OMFG.) you learn to appreciate what you have in your average TCP/IP stack.

  Currently the protocol on the badge could be described much like TCP/IP, minus, well, the good stuff in TCP/IP. We don't have the RAM for sliding window ACKs and generally our protocol is closer to Stop and Wait ARQ. As you may know from your college networking textbooks (I had to go back and read mine), SaW ARQ is very inefficient but can be written with a very small memory footprint.

  Our protocol is "Stop and Wait ARQ, with Time Outs (TO), and a small (four packet deep) Circular TXQueue."

  Writing that was hard, but writing a version that played nice with our graphics was harder.

  Things I would have done differently;

  1) Move packet handling to a different thread. I did it in the game thread, this was stupid.

  2) Written my animations in a way that didn't impact networking.

  Okay, this right here is the #1 reason our badge had problems. Because to do animations I would paint a frame and then do a chThreadSleep() -- Nothing in your device should go to sleep if the same thread is processing the damn network. DO NOT put your thread to sleep, you will drop packets and life will suck.

  3) Written in the txqueue ring buffer in from day one. Only being able to have one packet in the queue was killing us, but needing the queue at all was an indication that Stop and Wait really wasn't Stop and Wait. We needed the txqueue, because sometimes we have to tell one badge more than one thing in a row (i.e. "I have moved, and after you get that understood this game round is over.")

  In any event, we now have some very nice radio networking code that works well on ARM machines. The fight sequences are fast and tight, and for the most part the badge doesn't deadlock anymore waiting for moves. We're still torture-testing the badge though and will probably have a few more sets of changes before DEF CON.

  *Crosses fingers* I think I got it this time.

• Many problems, but we're getting there.

  John Adams • 03/23/2017 at 20:06 • 0 comments

  Our Kickstarter has been incredibly successful, and now that we're sold out, I've got this long, long spreadsheet with about thirty items on it that need finishing. Fortunately, on the hardware side of things, we're about 90% there.

  I spent nearly three days not writing software and revising the PCB. I believe that I've fixed the chip mounting issue, repaired the faulty power switch issue, and rerouted audio and grounds enough to eliminate the popping issue on RF transmit. Hopefully.

  The KW01 is a ridiculous chip. It has non-matching solder and paste layers, via requirements, ground requirements, and all of that is down to four decimal places in mm. Our translation of that data from NXP's Altrium files to KiCad was a disaster, causing bad foot placement and a number of failed connections in production. So, we have to do one more spin. That takes us from 3/23 to April or May, which is not where I want to be production-wise.

  We'll probably engage Macrofab to do most of the programming and assembly work, if we can get parts to them on time and such. At least then if our boards are DOA they can help with rework.

  It is frustrating and unfortunate that I have to do a 3rd prototype this late in the schedule, but it is necessary to ensure a proper final product. At least we will be able to expedite this through Macrofab and possibly get a board back in a couple weeks.

  More updates as I get them.

• Updates, and our Kickstarter!

  John Adams • 03/16/2017 at 18:34 • 0 comments

  The last week has been pretty amazing, and while I haven't been updating here, I have been updating a few other places.

  Rev2 boards came back. one works 100%, the other works

  There appears to be an audio issue on all boards. I am not sure how this happened by I had a cap from the audio amplifier, C42, on the far left of the board over by the RF circuit. That cap should not have been there. When we transmit, we get a digital click in our speaker. Now, you're never going to hear this at DEFCON but it is damn annoying.

  I've moved C42 back to the right side of the board and tore up a pile of vias and traces that were along that path in the hopes that it fixes this issue.

  On the second board, we had a few bad/melted WS2812B's. One was so dead it couldn't pass any signal through DOUT. The other, well, it was just dead. Using the hot air station to rework those parts restored function on the second board.
  

  However, the second board's pin 49 (DIO) which we use as an RF Interrupt wasn't connected at all, making it impossible to receive packets. I had an excellent phone call with Macrofab earlier today to discuss why the KW01 isn't mounting to the board correctly, and their engineers are going to take a look at the footprint today.

  I cannot recommend Macrofab enough. They have been giving me stellar customer service!

  In funding news...

  I spent quite a few hours making a video and created a Kickstarter with it. That's online now, and KS has approved our page so you can actually preorder the badge on kickstarter now!

  Funding has been incredible. 50% of our funding goal in under three hours.

  Now, obviously, $6,000 is not enough to make this project happen. But, I expect that the badge will sell out and we made the goal purposely low. That's just what you have to do to make Kickstarter go. Low expectations, High rewards.

  This project is actually around $18,000 - $20,000 to make at quantity 225. We'll get there. I know it.
  

  Thank you all for your support.

• A delay, and some mechanical things.

  John Adams • 03/03/2017 at 00:40 • 0 comments

  Rev2 Update

  It looks like there's been some sort of problem at Macrofab and their pick and place machine (they only have one?!) isn't working.

  We expected to have rev2 of the prototype on it's way to us today but unfortunately it looks as if we will not have our boards on the pick and place until March 10th, which possibly puts our prototype shipping out on March 15th after the fab does some hand soldering on our through-hole parts.

  It's a delay, one that we've padded our schedule for but a delay nonetheless!

  I've tried to reduce the through hole parts as much as I could but they're still on the board. I only recently discovered that I could order 40-pin SMD connectors, so maybe that's an option for another revision. Moving to that means only the speaker would require hand soldering and this could improve the time to produce a board.

  Chinese parts are starting to come in

  We've also received another batch of East Rising screens from Shenzen. I'm very pleased with these screens as they are sharper and have a far more responsive touchscreen than the MCU Friend screens we were using. The best part is that they don't require another set of pins to support the SD card, and(!), they use microSD instead of standard SD. The finished appearance is much more professional and clean.

  MCU Friend (top) vs East Rising (bottom). East Rising is so much nicer . Also notice the EFF supporter logo, activated by secret unlock code!

  Speaking of "ordering parts from China": Batteries have been ordered (ETA 3/28), Standoffs have been measured and a sample has been ordered (ETA Friday), Screens and MicroSD cards remain to be ordered. Mostly, I'm ordering based on a build of 300 units on these parts, assuming that some percentage will fail. Our final unit count will be in the 200-300 range depending on pricing, sponsorships, and part availably. We still need more sponsors. Please help!

  We sort of got screwed on the batteries as the minimum order was 500, so if you get to DC25 and want an extra battery for your board, we will absolutely sell you one if you'd like to buy one.

  How's the game coming along?
  

  On the software side, I think I've repaired the last deadlock/hanging bug in the fight network code. It's very hard to get the fight to deadlock now, but it still deadlocks once in awhile. I need to sort that out and it's very hard to debug.
  

  I wrote a new command line interface to edit the system preferences, as well as an unlock screen to unlock special things via secret codes.

  Matt has just about finished the artwork, and it looks really, really good. Now it's up to me to incorporate all of the characters and their respective animations into the game.

  UI Detail for the default badge application. I'm Caesar. Deal!
  

  
  

  What's left on the game now is the leveling system, code cleanups, some UI tweaks, and implementation of all of the magic unlock codes. The UI is there for unlocking, but it doesn't do much yet.

  Bill has also managed to get some very amazing video surprises working, but I'm not going to spoil that just yet.

  I'll have a new update around 3/10 and I hope to have boards by 3/15!

  Thanks for your interest, you're awesome!

• Spin #2: Electric Boogaloo...

  John Adams • 01/30/2017 at 07:45 • 0 comments

  Most of my weekend was spent laying out version two of the badge and working out issues with the USB battery charging circuit. The battery charger works now, and we have a new layout which will probably be much closer to the final version. I like it a lot, it feels nice in your hands, and the shape is modeled on a roman bath, which is entirely appropriate.

  Here it is, minus the screen:

  We've added some buttons for development work (they're only about 8 cents on Macrofab's house part list) and greatly reduced the size of the badge. It looks and feels a bit more like a PS Vita, but who's counting. I consolidated the LEDs as well; On the right side of the screen is a full system debug window, showing status of the charger, UART, Radio, and SD card.

  We also won't be populating those test pins. They're going to be empty for the people in the hardware hacking village to experiment with the device.
  

  We also discovered that one of the Macrofab boards was NG. An important SPI line was dead on an inner layer of the board, rendering it useless. Oh well. We learned what we needed to learn from our boards and the screen worked well on the other prototype.

  At this point, our goal is to have a new prototype in hand with all of our issues repaired by March 1st. From there, we can make the go-no-go decision for the main production run.

  Bill is still working on a better audio system for this board, possibly with a real speaker instead of the three-channel piezo hack we've got right now. More when we get that working!

• Prototype: One.

  John Adams • 01/25/2017 at 06:49 • 0 comments

  This morning UPS dropped off our first two prototype boards. they're in good shape!
  

  Initial observation:

  
  

  The badge is way too big.

  If we release these things everyone is going to look like Flava-Flav with huge clocks around their necks. Unacceptable. We will redesign. and reduce the size at least 50% from 6.5" down to a respectable 3.5" or so. I drew this thing on a piece of paper and thought it would be fine, but now that it's in my hands I don't like it at all.

  We got most of the badge right (!)

  Nearly all of our subsystems are working:
  

  • LDO - but maybe we don’t have enough power from USB to run off USB alone. right now the battery has to be there, or we will enter a crazy reboot loop with all LEDs on.
  • JTAG header and programming -- I was really sweating this one but it works.
  • FTDI chip and serial UART console.
  • Power LEDs
  • RESET
  • UART TX RX LEDs
  • UART RF TX OK
  • SD CARD ACT light OK
  • LEDs
  • Buzzer
  • MAYBE: Charging seems to work, but it's very, very hard to tell. I'm going to let it sit for the evening and see if it charges. The status LEDs may actually be backwards.

  What's not working?

  • The external radio clock (Y1) is the wrong part, it should have been a crystal. Instead, we ended up with a powered resonator which will not work at all.
  • The radio circuit is a disaster (but we knew that.)
  • Display, SD, and Touch - Wiring error on PTE19 has caused a massive failure on the display, which is probably not easily fixable. Marking that as not-connected on the board seems to fix our issue for now
  • SCLK - One board is fine, the other board has a dead SCLK line. There is a break, somewhere between the KW01 and the rest of the system. I think Macrofab does continuity testing of their PCBs but I cannot explain this issue at the moment.

  In all, about 18 mistakes to resolve in the next month or so. We'll get there. We even have some audio surprises in store (if we can get them to work.)
  

  Onward!

• Spin #1: Mistakes, and bugs.

  John Adams • 01/19/2017 at 00:43 • 0 comments

  Macrofab has produced our PCBs and pick-and-place has finished up on the boards. They're now completing some hand soldering, and I have a list of the things that didn't work right on our first spin.

  The tale of woe goes a bit like this...

  1. P2 we got a 20 pin instead of a 40 pin (TODO: New part selection)
     
     1. For P2, I misread the data sheet, believing that "2 x 20" meant "40 pins". It actually meant "Two rows of ten pins, totaling 20 pins total."
        The data sheet was mostly in Chinese. Ooops.
  2. P1 orientation - need indicator on board for manufacturing
     1. Less of a failure and more of a "Damn, why didn't I think of that." P1 is a tilt-switch, and will work regardless of orientation. However, since the fab house didn't know what to do, this halted production until I could email them with a proper orientation for the switch.
  3. Forgot path for VR_PA
     1. This effectively made the radio dead. No VR_PA, no output voltage.
  4. All Inductors Wrong footprint (1206 vs 0805)
     
     1. This one, oh, what a massive fail here. All of the radio inductors didn't fit! Since VR_PA was disconnected anyway, I gave up on this and told them to not populate any of the inductors, further ensuring that the radio really would not work when it came back from the fab.
  5. Hashed copper fill problem (on board rev2)

  This last one was probably the most interesting bug of them all. I did the copper pours as hashed-outline fills, tickling a bug in the SVG generation at Macrofab and apparently breaking the PCB preview process. The system "tried to create an SVG file with thousands of lines combined to fill a region." and that's an expensive operation. This was apparently bad enough that Macrofab had to update their KiCad docs.

  I'm glad I could help everyone else with my dumb mistake.

  I expect that I might actually have boards in hand by Friday or Monday. We'll see what else is broken when they show up.

  I also expect that the LED layout will not be to my liking, but I'll verify that when the boards come back.

• State machine hell.

  John Adams • 01/05/2017 at 06:55 • 0 comments

  Our boards are still being manufactured at Macrofab so the last week or so has been an intense pile of work to get the game itself up and running. Our combat system is pretty nice (on paper) and the boards are mostly cooperating.
  

  Unfortunately, synchronizing two devices on a hostile RF network like the one we will encounter at DEFCON isn't that easy.

  At first our game was working but we'd see the occasional signal loss event, where we would drop a critical message and then suddenly lose track of the entire game. It turns out that the radios, even while sitting next to each other are not 100% reliable. After a ton of back and forth in the code I realized that I was going to have to construct a highly reliably radio protocol in very little memory space.

  This code takes concepts from TCP/IP and works well for automatic retransmission of packets.

  While we don't have the memory to implement something like TCP/IP or lwIP, we do have a limited amount of space to roll our own networking code. The code, on the sending side implements a primitive version of ARQ (the same thing TCP/IP does).

  In a sentence, ARQ is "Keep retransmitting the data until the TTL expires on the packet, or, you get an ACK, or, you get a RST."

  My version is a bit ugly, but it looks something like...

  sendPacket(...);
  
  current_fight_state = WAITACK
  
  // loop... 
  while (1) { ... 
  if (current_fight_state == WAITACK) {
        // transmit/retry logic                                                                                                                                                           
        if ( (chVTGetSystemTime() - last_tick_time) > MAX_ACKWAIT ) {
          if (packet.ttl > 0)  {
            resendPacket();
          } else {
            orchardAppTimer(context, 0, false); // shut down the timer                                                                                                                    
            screen_alert_draw("OTHER PLAYER WENT AWAY");
            end_fight();
            playHardFail();
            chThdSleepMilliseconds(ALERT_DELAY);
            orchardAppRun(orchardAppByName("Badge"));
            return;
          }
        }
  }
  }
  /* in the radio loop, if we get an ACK, we advance state */
  case WAITACK:
    if (u->opcode == OP_ACK) {
       // if we are waiting for an ack, advance our state.                                      // however, if we have no next-state to go to, then we will timeout                                                                                                             
       if (next_fight_state != NONE) {
          chprintf(stream, "\r\n%08x --> moving to state %d from state %d due to ACK\n",                 
  u->netid, next_fight_state, current_fight_state, next_fight_state);        
          current_fight_state = next_fight_state;     
       }    
    }
  break;
  
  

  On the recipient's end, we ACK inbound requests and advance our state machine, if we have something to do.

  Needless to say, this sort of thing is ridiculously complex and filled with contention, deadlocks, and all of the things you'd expect in a massively concurrent system. It is severely headache inducing and makes me want to break things.

  I hope to have these issues fixed this week.
  

• Spin, Spun.

  John Adams • 12/31/2016 at 11:09 • 0 comments

  Forgive me Macrofab, it is been a week since my last spin...

  I don't have the rev1 boards back yet, but a careful analysis of our schematic tells me that I blew the entire RF section.

  RF is, well, hard. I misread the reference schematic and didn't realize I'd forgotten to connect VR_PA to the inductors/matching circuit, so I will just assume that any chance of RF output is now lost.

  I also did a few dumb things. I didn't let the autorouter finish correctly and the boards that I manufactured were four layer (2x price) when they could have been a 2-layer board with proper vias and would have been far, far less money.

  Moving a number of our inductors to 0603 and rerouting parts to the back of the boards made 2-layer possible for far less money.
  

  I'd also forgotten one of the ground fills, our logos, and a bunch of other crap.

  The per-board prototype price is down around $140 now, and our bulk price is in the $30-60 a board range before screens and batteries, so that's great.
  

  I might even go ahead and try to repurpose our battery as a phone charger, but we'll see!

  At least our boards are looking better, and much more together now.
  

View all 21 project logs