Rocketometer

https://stkwans.blogspot.com/2013/06/unattended-running.html

The Rocketometer has been running unattended now for several hours. For instance, last I checked it was just over 7 hours, and had generated 77 files of 4MiB each. This represents a data rate of about 12kiB/second. At this rate, the device will fill up in about 350 hours, or 14 straight days.

So, I asked about which direction would be vertical when the Rocketometer is installed in the rocket. It is vertical when accelerometer reads most strongly in the +Z direction. Coincidentally, this means that the interface is on the bottom, and the lights will be hidden.

I put in a piece of code which detects when the Z axis is stronger than the others. Mindful of the Genesis reentry failure, I put in code such that if the board is either +Z or -Z, it will decide that it is vertical. I can't test that Z is correct, but I can observe the board in place in the control section and visually confirm that Z is correct.

Now this piece of code will just reduce the rate of readout from once every 3 ms to once every 30ms, still quite quick, but only 1/10 the data rate. Even if it is wrong, we will still get some data.

I will need a log of when the CCD board is powered in order to match up non-flight measurements with anything, but the flight should be either the last or second-to-last record on the board.

For the unattended run, the system will be set to chunk its data into 1GiB pieces, and use only the last digit of the 4-digit number for chunks in a run, reserving 3 digits for runs. I do not expect the Rocketometer to be power-cycled more than 1000 times.

ttps://stkwans.blogspot.com/2013/06/bugs-crossing-boundaries.html

I am getting the Rocketometer ready for delivery. The hardware appears to be fine, but the software had one particularly weird bug -- the name of the FAT volume on the SD card was changing. This meant that something was poking into the root directory block where it did not belong, which meant that it could be anything.

As it turned out, it was the packet buffer becoming full. Now how in the world did that corrupt the file system buffers? As it turned out, the file system by design wasn't using buffers, in order to save memory.

An SD card is addressed a sector at a time. Data can only be read and written in full sectors. So, if you want to change a single entry in the root directory or file allocation table, you have to read the whole sector into memory, change the bits you want, then write the whole sector back. Now the root directory and file allocation table only needs to be changed when the file is written, and in this case there is already a buffer which has just been used, so I foolishly decided to have the code use this.

In the mean time, this same buffer was part of the circular buffer accumulating packets. If the buffer ever filled up, new bytes would be rejected, and the head pointer would not move. If you start to write a packet to an already full buffer, nothing gets written, and when you finish, the packet length seems to be zero. Now the fun begins.

Upon finishing a CCSDS packet, the code counts how many bytes have been written, subtracts 7 because that's according to the protocol, then goes back and changes the buffer. In the current case, since no data had been written, it actually changed marked data.

Now we combine this with the fact that the interrupt task was filling the buffer. Here then is the final sequence:

1. The buffer is full. No more bytes can be written.
2. The main loop detects this, and starts to drain the buffer and write it to the SD card. It writes the packet, then reads the root directory sector, changes it, then writes it back.
3. During the write, an interrupt fires.
4. The interrupt task takes a measurement and writes another packet. The bytes are dropped on the floor.
5. The interrupt task finishes the packet. The calculated length is zero, since no new bytes were written
6. The length to be written is -7, 0xFFF9
7. The length is written to the buffer, but because of timing, it writes it over the top of the root directory sector in memory. The root directory is now corrupted.
8. The interrupt task finishes
9. The SD writing code writes the corrupted root directory sector back to the card.
10. The card is now named RKT\FF\F9 instead of RKTO3.

It would be pretty easy for this same process to shoot holes anywhere in the filesystem metadata. This bug is hazardous!

So, to fix the bug, we do the following:

1. Give the file system its own scratch buffer, and never let it do scratch work with the user buffer. This doesn't actually fix anything, but it does make the bug, and any similar undiscovered bug, less dangerous.
2. Handle the full buffer case better. When the buffer is full, it throws away the new incoming bytes. Further, it backs the head pointer to the mid pointer, in effect throwing away the fractional packet accumulated before the buffer became full. Even further, since the buffer is now no longer full, to prevent getting the last half of a packet, we keep a flag that says that the buffer was full, and don't accept any more bytes until the flag is cleared by a drain operation.
3. The interrupt task checks if the buffer is already full. If it is, we don't bother to read the sensors, a time-consuming chore which will just discard the result and take time away from the SD writing routines.

This clears a major bug, perhaps one causing all the observed halts. A bug which punches random holes in the metadata can do anything. It only triggered recently when I cranked the data recording rate to 200Hz, which appears to be just beyond the equipment capability. Previous rocket flights must have run at lower rate and not triggered the bug, or not triggered it as hard.

So how did I find it? Lots of print statements and blinklocks. First, I had the SD driver print out the buffer it was going to write. Then I had the direntry driver (the thing which actually handles the root directory) print out its scratch buffer after it had been read, as it changed each byte in turn, and before it wrote it out. From that I found out that the buffer was changing in between lines. That meant that an interrupt task was doing it. Further puzzling out and seeing 0xFFF9, then counting on my fingers to see what number that is in signed decimal, revealed that it was -7, a key number in calculating CCSDS packet sizes. However, it wasn't writing bytes 4 and 5, where it would be if the packet started at byte 0 in the buffer. Instead it wrote one byte before that, implying that the phantom packet started at -1, a key number in the circular buffer pointers. Once the head pointer is one less than the tail pointer, the buffer is full.

For a while I thought that the code was starting a new packet before it finished an old one, which would result in weird numbers. So, I put in some code which blinklocked the device if a packet had not been finished before the next one was started. That never hit. So, I put in some code which blinklocked when the circular buffer filled up, and that finally hit and proved it all. This interacted weirdly with the overlapping packet checker, so I had to fix a bug there.

Now the only remaining known weakness is what we do if the first cluster of the root directory is full. We can make a cluster as big as 64kiB, we can handle multiple sectors in a cluster in the root directory, and we can fit 2ki entries into a cluster that big. But, what if we fill it? The file search code will need to know how to go to the next cluster, and the file creation code will need to know how to allocate a new cluster. Or, the format process can write 10000 files then delete them all.

The flight version of the code will not blinklock, but reset, when blinklock() is called. Between that and gathering enough runtime, I can gain some confidence that the device won't lock up.

I set the circular buffer size back to two sectors, down from the 6 I had given it. Next version will put the circular buffer into USB ram to save space for the stack, and allow me to have up to 16 sectors. USB memory doesn't work for some reason... maybe the USB system needs to be on for it? Then we are exchanging memory for power consumption. I know that the system is dropping packets now, more so than it was at 6 sectors.

https://stkwans.blogspot.com/2013/06/not-another-rocketometer.html

I got an Arduino Leonardo and a Sparkfun Pro Micro as breadboard models of the Arduino-based Rocketometer. Based on my experience with those, I conclude that the ATMega32U4 and the current bootloader are Not Ready for Prime Time, and Not Ready for Spaceflight. Further, I would have had to write my own USB Mass Storage driver anyway.

https://stkwans.blogspot.com/2013/05/yet-another-rocketometer.html

The Arduino Leonardo can effectively be thought of as an Arduino on a chip, except that an Arduino pretty much already is one chip. What makes a Leonardo special is that its core is an ATMega32U4, which has a USB port. The chip can be programmed as normal over ISP, and once it is programmed with the appropriate Arduino bootloader, it can be programmed over the USB port. The Arduino library for Leonardo also supports using the USB as a CDC serial port, and has much other USB support.

The chip only runs at 8MHz. It can run at 16MHz at 5V, but I am planning on running it in the Rocketometer, which only has 3.3V available, which limits the chip to the aforementioned 8MHz.

Now how do I think I can get away with this? Simple: Even with the LPC2148 running at 60MHz, it was spending most of its time waiting for the bits to crawl across the I2C and SPI buses. And how much can we optimize a busy wait loop?

I bought a Leonardo to practice with, and a Sparkfun Pro Micro to practice some more, especially programming the bootloader. I will not be using the 6-pin ISP header on the Rocketometer, but I got the great idea to use the MicroSD sniffer to get at the SPI signals through the Rocketometer MicroSD slot, and I can just hold down the reset button, or solder one wire to the contact of the Reset switch - which reminds me, I should put the solder bridge on the top side of the reset switch. In that case, I can use the solder bridge itself as a temporary terminal, solder a wire to it long enough to program, connect the rest of the SPI signals to the microSD sniffer, then program it with an ArduinoISP.

While I was at it, I also designed a Precision clock around this new chip. It doesn't need a hardware UART to program or talk to the host computer, so no FT232 and no multiplexer is needed. The chip has more pins, so this design doesn't use literally every single possible pin (those two ADC-only pins don't count, since they are not possible to use in this project).

https://stkwans.blogspot.com/2013/04/directtask-and-nested-interrupts.html

One problem with the old rocketometer code is that the sensor readout and SD card writing code were in the same thread, meaning that when the SD card took its endless milliseconds to write the data, the sensors were not being read, leaving an irregular gap in the record. My brilliant idea was to read the sensors in a task at interrupt priority, effectively creating another thread. First effort was with the task manager I described below, which was a dismal failure.

For whatever reason, and perhaps the same reason (see below), the task was not able to read the sensors. I came up with a much simpler task manager with which I am getting incredible accuracy.

I call it the DirectTask manager. Its concept: Rather than using one match channel of the timer and a heap priority queue, we just use all three available match channels (there are four, but the zeroth channel resets the timer every 1 minute). This limits the flexibility enormously, but I only need two tasks. I set up one task to reschedule itself on a regular basis (5ms in my first test) and I use the other task to read the BMP sensor.

However, the sensor readout runs an I2C read, which itself is interrupt driven. The code does not currently support nested interrupts, which means all interrupts are delayed until the current one returns. The I2C state machine was interrupt drive, and its interrupts were getting delayed, including the one which makes the state machine stop waiting forever, so the state machine waited forever.

So, we put in an option to I2C to run without interrupts, instead using a busy loop to check the I2C interrupt status bit, then calling the same state machine driver. We weren't getting anything by being interrupt driven anyway, since we had to wait for the read to finish.

With that, the DirectTask manager worked fine. Maybe the heap task manager would have worked fine too, but this one is simpler.

It takes 641 microseconds to read the sensors. We could probably easily bump the read rate to 500Hz, and maybe to 1000Hz, but this doesn't count anything but reading the MPU and HighAcc sensors.

https://stkwans.blogspot.com/2013/03/every-space-mission-get-patch.html

• NASA is NASA, the sponsor of sounding rocket flight 36.290
• CU/LASP is the University of Colorado/Laboratory for Atmospheric and Space Physics, my employer
• SKHFTTADD is St Kwan's Home for the Terminally ADD (me)

The Rocketometer is the purple board hooked up to measure the rocket, flying over the Earth. In the background, there are three big stars and two little stars on the left, and one big star and seven little stars on the right.

https://stkwans.blogspot.com/2013/03/rocketometer-flight-test-1-and-photos.html

Site: Alpine Elementary field

Weather: Overcast, calm winds during flight, weather at KCOLONGM64 51.5f 29.97 in 6.7mph 49%RH

Hardware: Rocketometer6050 #2, Estes Star Stryker (modified)

Forgot to pull cap from launch rod before flight. Flew straight, seemed to turn level to ground at apogee, I could see a yellow tinge to the tracking smoke. Parachute ejected but never inflated. Rocket fell free from apogee. Rocketometer

was still running after flight. Engine hook unzipped about 1/8 inch to the aft, probably due to ejection. One fin has a cracked glue joint. Minor ding to rocket nose cone due to striking launch rod cap. Rocket could probably be repaired, but not reflown in its current condition.

No video of the flight was captured. I had the camera, forgot to press record.

Rocketometer has one file as expected, RKTO0500.SDS 35,369,984 bytes. Indicates that Rocketometer was running through whole flight.

Switch and card not glued into place. Reset solder bridge in place. Rocketometer ran through flight anyway, indicating that nothing was jarred loose.

Rocketometer on internal power ~11:50am

Nautilus reports weird file sizes for files. fsck.vfat does not report any filesystem errors.

Total packets retrieved: 1,572,103

AD377           STRUCT    = ->  Array[124694]
BMP2            STRUCT    = ->  Array[6234]
DEF             STRUCT    = ->  Array[15]
DUMPDATA        BYTE      = Array[65640]
F               STRING    = 'RKTO0500.SDS'
HMC             STRUCT    = ->  Array[124694]
INF             LONG      =          100
MPU             STRUCT    = ->  Array[1246947]
N_EXTEND        LONG      =       -59604
N_PACKETS       ULONG     = Array[2048]
OUF             LONG      =          100
PKT             STRUCT    = ->  Array[1]
STATUS          INT       =        0
TCC             STRUCT    = ->  Array[68932]
T_PACKETS       ULONG     =      1572103

Greater than 26 minutes of data recorded. Processing took about 14 minutes. Time stamp difference between first MPU record and last: 1601.1205s (26m41.1205s) Range zero is 1058s (last top of second before first acceleration) MPU did saturate during boost

Flight battery is similar to this, not visible in rocket pictures since it is packed in foam behind the board. Photo from Sparkfun Electronics published under CC BY-NC-SA 3.0

Rocketometer rendering, showing sensor side

Rocketometer rendering, showing interface side

This one is from memory - I don't have any log of the actual construction. I did my usual thing, I had a solder stencil made for both sides of the board, just as I do for all of my boards. I carefully used the stencil to apply the solder paste, then carefully used tweezers to place each part on one side of the board. Then, into the toaster oven for five minutes and twenty seconds.

Next, solder paste on the other side. This is more difficult, because the side with components already on it is not flat any more. Part goes back in the oven, this time being careful such that the toaster oven grill does not touch any of the parts on the bottom side.

OSHPark makes three copies of each board that you order. The first board that I made was a failure, but I don't remember exactly why. I could use the standard LPC2148 bootstrap to talk to the controller, but either I couldn't program it or I couldn't get the sensors to work.

The second board worked fine -- this is the one that was ultimately launched into space.

I don't remember if I ever did make the third board. I seem to recall that I did, but that it worked even less well than the first board, like I couldn't even talk to the bootloader.

https://stkwans.blogspot.com/2013/01/gem-of-week-parallax-propeller.html

I write this before ever using one, so consider this a review of the concept rather than the implementation. The Parallax Propeller is a microcontroller with a couple of interesting features, and perhaps more interestingly, a couple of intentionally missing features.

First off, it has no peripherals other than 32 GPIO pins.

Secondly, it has no such concept as interrupt.

Thirdly, it has eight independent processors, called "cogs", each with its own memory. Each one can run on its own resources without interfering with any other processor. Each cog is a 32-bit processor, and gets access to 2kiB of ram, shared between code and data.

Fourth, it has a set of resources that all processors can access, called a "hub". This basically consists of more memory and a round-robin memory controller which each cog can access in turn. The hub has 32kiB of ROM with the bootloader, Spin interpreter, and a couple of tables, and 32kiB of RAM.

The missing features are what make the controller interesting. Want an I2C? Write a program for a cog which can bit-bang it. Same for SPI, UART serial, etc. Presumably it could bit-bang low-speed USB, but high-speed would be difficult due to limited processor speed.

Further -- want an interrupt? Too bad. Instead, assign a cog to sleep-wait for the appropriate signal.

Programming such a beast is clearly a different problem than programming an ARM of any flavor. ARMs are all about peripherals, registers, interrupts, etc. Propeller is about bit-banging. Effectively you can use a cog as a soft-peripheral to do basically any digital process.

As I mentioned above, the processor comes with a Spin interpreter. Spin is a custom language for programming Propeller, which gets compiled into byte code and interpreted with the Spin virtual machine. Of course you give up performance, but they used an interpreter in the Apollo Guidance Computer. There, it was for memory saving - a single interpretive instruction could take the place of many instructions in AGC machine language. They gave up time to gain space. Spin could have similar benefits, but it seems like the main purpose is providing a language which natively handles the very different concepts needed to handle a processor as weird as the Propeller.

A Propeller program then consists of a bunch of Spin and machine language routines, all stored in hub RAM and copied from some external source whenever the chip resets. Aside from self-modification or using a cog to bit-bang a memory bus, this is all the space you get. This is a rather tight restriction, in fact less memory than in the AGC. But, they fit a full-blown Kalman filter into that.

Of course I have already designed it into a Rocketometer. It's what I do. One of the great things about the GPIO and bit-banging style of the Propeller is that I can put the SPI bus on the pins which are closest to their targets outside the chip. This makes routing the board MUCH easier.

So I will say for now that Propeller shows a lot of promise. The design is a gem. We will see about the implementation.

https://stkwans.blogspot.com/2012/12/putting-your-hardware-where-your-mouth.html

I had been keeping this quiet, but it is the natural culmination of all that I have written on this blog.

At my day job, I work with space projects, but I have only been in the same room as flight hardware once, and that was purely as a tourist, to get my picture taken with it. I have written code that has gone into space, but never touched the hardware that carried it.

In October 2013, that will change. I am building space hardware. In a sense, that has already changed, since I have touched the hardware, but it's not space hardware yet. I have arranged for a version of the Rocketometer to fly on a rocket all the way into space.

The people I work for at my day job run an instrument in space that needs to be calibrated every so often. Every year or so, we fly a sounding rocket with a copy of our instrument, pop it up above the atmosphere for a few minutes, then let it fall back down into the atmosphere and descend on a parachute. It goes into space (well over 100km) but not into orbit.

On the campaign building up the rocket for the last flight this past summer, I was tinkering with my rocketometer (actually the 11DoF) and got to talking to my scientists about it. I told them about my daydream to actually get this thing on the rocket, and they said it was a great idea. Naturally it was far too late to get on board that last rocket, but with this next one I have plenty of time, especially considering that the hardware is finished, and I potentially could fly it now. The baseline mission is just to collect the data as fast as possible, and not bother with any on-board processing. That code was demonstrated with the speed test I did a few weeks ago.

My test plan and to-do list then looks like this:

1. Adapt the old code to the new hardware. A couple of the I/O lines were reassigned to simplify the board design.
2. Write an offline Kalman filter to process the data. IDL will work fine for that. Mostly I just need a set of equations of motion that allow the compass, gyro, and accelerometer to calibrate each other.
3. Calibrate the sensors. I have an old record player with no needle which will be perfect for this. I may be able to use some stuff in the labs at work to help with this.
4. Do a test flight in a model rocket. These generate a similar scale of forces and rotation rates, just for much shorter durations, seconds rather than minutes. The Rocketometer was designed to be carried in any rocket with a payload section 1" or larger in diameter.
5. Get USB Bootloader++ working. This is low priority, as I can program the part over serial as I have been doing for a while.
6. Consider on-board processing of the data.

I will be using the Rocketometer2148 with an MPU6050 6DoF sensor, an ADXL377 analog high-g accelerometer, an AD7991 12 bit ADC to read it, an HMC5883 compass, and a BMP180 pressure/temperature sensor.

With a couple of changes to main.cpp and gpio.cpp to tell it where the sensors and lights are on this board, the thing works! It may also be working at my goal rate of 1000Hz, attributable to a faster SD card, reading the compass only 1 of 10 times that the 6DoF is read, and not reading the HighAcc.