Lutetium-
Hack Chat Transcript, Page 2
07/12/2023 at 20:41 • 0 comments![]()
Transcript coming up at the end, btw -- no need to copy and paste links![]()
@Dan Maloney thanks!![]()
I suppose I should hit a couple other things that I mean to mention![]()
Sensors: Your core spacecraft might not need sensors if it's Sputnik, but anything else will want to know where it's pointed and where it's heading!![]()
Jacob, where are you looking to go next? Amazon's project Kuiper is hiring vigorously.![]()
Sensors range from simple gyros to full star trackers that automatically take images of the star field nearby and use an onboard map to find out exactly where they're pointing. When you're close to earth, sun sensors, horizon sensors, and magnetometers can all be used.![]()
@stu I'm not sure yet. Still evaluating my options.![]()
Any lessons from space that can be extra-useful on the ground?![]()
Communications: There's a lot of communication standards in use onboard a spacecraft! At the most basic level, RS-422 and RS-485 are common and that's what I used to talk to most of my payloads! However, MIL-STD-1553 is also popular, although boutique. Spacewire gets used when you want a lot of bandwidth. It's an LVDS serial protocol, although in my experience the signal integrity is a nightmare. Big NASA projects, like the Lunar Gateway, use Time Triggered Ethernet, which I think is the future.![]()
What about using distributed systems to avoid or mitigate SEUs and other faults?![]()
@Thomas Shaddack I think we hackers don't often appreciate that one day our creations might leave our bench! Things have to be able to boot, communicate, and recover on their own.![]()
No CANbus? Seems like it has a lot to offer![]()
@Dan Maloney CAN is definitely used in some systems! I've never had the opportunity myself however. The issue with CAN is the low data rate. It's great for basic sensors, but not when you have lots of data to shuttle![]()
...that's why I am asking. Down here sending a technician to a device is easier than Up There but still can be both costly and annoying. (And sometimes the extra telemetry can be handy even between bench and couch, when the hacker is extra lazy...)![]()
The NASA SCALPSS payload is going to spend _hours_ dumping data back to the spacecraft over a 921600 baud serial connection once Nova-C lands. Lots of other interesting instruments are similar, and spacecraft are always hurting for the extra bandwidth, storage, and processing power!![]()
That said, it's amazing how much terrestrial computing is wasted. Spacecraft always keep available resources in mind!![]()
What about two buses? One for critical realtime control, one for just the payload data that can wait a millisecond?![]()
YES. Compare the computing power needed to get man to the Moon vs how much is needed today to open an email. Something somewhere went wrong.![]()
@Thomas Shaddack yes, absolutely. It's more complicated to have two busses, but it's common to have a control bus where everyone communicates using fixed size messages at a fixed rate for predictability, and a second bus for handling large amounts of data.![]()
Yes, but GUI look better now![]()
@Jacob Killelea - we're an hour in, but wrap time is totally up to you.![]()
@Dan Maloney I've got a few more things I can throw out there, let me get through a few more subsystems!![]()
Absolutely!![]()
Great, ok. Telecom: Your spacecraft needs radios! How else will it phone home! A lot of systems are QPSK modulated and run between 2 and 10 GHz for the high bandwidth communication.![]()
Is there always/often a method of performing firmware updates to the satellites remotely? I'd assume this is a critical component?![]()
I saw cubesats with ISM LoRa downlinks.![]()
The Consultative Consortium for Space Data Systems (CCSDS) creates standards for space data communication. These are fascinating and run in parallel with the terrestrial OSI model![]()
@daniel.fryling Some systems can be reprogrammed, some can't. It's kind of a new development for spacecraft to have that ability. It definitely makes sense for things like core science and guidance applications, but a lot of embedded controllers might even be made with write-once memory to make them more reliable. If programming fails, you've just wasted a very expensive ($1K to $100K) chip![]()
Better waste a chip than a whole probe...![]()
@Thomas Shaddack for sure! A low bitrate, highly available radio is crucial. Your high gain antenna is highly directional as well, so it's usually complimented by omnidirectional, low bit rate radios![]()
However, this means that your spacecraft needs to be able to gracefully go from high bandwidth to low or no bandwidth and back up again. I've written applications with a telemetry budget of 0.2 bits per second. Get used to packing your bitfields tight and think carefully about exactly what data you need to send to the ground.![]()
A lot of times, a simple flag or a few bits for a counter is enough to show that a system is working, or at least cycling correctly.![]()
How do you get an initial pointing with the high-gain antenna(s) after landing?![]()
@Carl We have to control our rotation as we touch down on the moon. There's a certain tolerance for this, and it's important for getting the solar panels pointed correctly, making sure heat is distributed properly, and that our instruments will see what we want them to![]()
Other spacecraft will have gimbaled antenna platforms, but that's a moving part that can break.![]()
Ohh, it is fixed to the structure?![]()
In many cases it is. This can create an issue if you need to point your antenna at earth, your solar panels at the sun, and your instruments at a point of interest.![]()
Computers: Your spacecraft will include at least one computer. It's kinda mandatory these days. This can be as simple as an ARM microcontroller or as complicated as custom silicon implementing a custom architecture. Vorago, AiTech, BAE systems and others all produce space hardare![]()
The lead times for these parts are punishing, the prices are astronomical, and it probably won't have all the support you need!![]()
Software that runs on these systems can range from commercial RTOSes and hypervisors down to arduino and python code.![]()
Even the high end RTOSes like VxWorks are often missing features you might expect, like a disk driver for your special rad-hard SSD. Others are even missing virtual memory, so your processes can overwrite each other's memory!![]()
All this is done in the name of determinism and light-weight computing. You just don't have a lot of resources to go around.![]()
If you all want to take a look at a nicely made RTOS with a lot of spaceflight heritage, check our RTEMS, it's an open source RTOS originally developed for the US military. It supports a POSIX C environment, as well as things like heterogeneous / distributed compute and real time scheduling.![]()
Payloads: Your payloads are probably some kind of special science or communication instrument made by a vendor. They might be late, or over-budget, or just a pain to integrate with your spacecraft. Things are generally made to order, and they will probably all use a different packet structure for their commands and telemetry. It's therefore a lot of work to make sure they're thoroughly tested and properly integrated.![]()
(You didn't hear it here, but I actually worked with a payload that uses Windows Embedded as its base operating system).![]()
Did you have to say that just as I was taking a sip of water?![]()
I hope I didn't cause a spill!![]()
Hot and dry today, no worries![]()
Glad to hear!![]()
Let's see, spacecraft guidance is also an important software area![]()
An area where it is generally unadvisable to mix together imperial and metric units?![]()
@Thomas Shaddack It's kind of inevitable that you'll have to do some conversions, at least here in the US. My school work was mostly metric, but we certainly did imperial as well![]()
Fun fact, the unit of mass in the imperial system is not the pound, but rather the slug![]()
I was referring to that case of the Mars probe that turned into a meteorite.![]()
Just as 1 kg weighs about 9.81 N on the surface of the earth, 1 slug weighs 32.2 lbf![]()
@Thomas Shaddack Yep, Mars Climate Orbiter I think? It wasn't an issue with the spacecraft, but rather with navigation software running on the ground.![]()
GROUND??? I missed that!![]()
Yep, IIRC Lockheed wrote a navigation tool in imperial, and missed converting it to metric before sending it up to the spacecraft, or vice-versa![]()
(facepalm)![]()
That's in the same league as the Hubble's myopia. How could that pass testing?![]()
This is an important point - your commands up to the spacecraft are just as important as the onboard software! A lot of missions will test all their commands against a flat-sat or simulator before sending them up![]()
A flat-sat is an important tool for testing by the way, it's quite literally all the systems of your spacecraft laid out on a bench in a lb![]()
in a lab*![]()
Sometimes the last comm with the sat is a bad command...![]()
Absolutely. Even if your commands are CRC'd and are decoded correctly, is it telling the spacecraft to do the right thing?![]()
Hopefully, your flatsat will catch a lot of the issues, but it's very hard to simulate your gyros and star trackers on a bench. We use a hybrid approach where certain sensors were actually simulated in software with a physics engine, and others could be made with real-world hardware![]()
All the data and signal integrity in the world can't save you from an issue originating between the chair and the keyboard. All it then can do is guarantee proper delivery of an improper command.![]()
This is where your test coverage is important. Test at every level, ideally to 100% test coverage![]()
Yep! That's why the most important part of a space mission is the team of humans! People who know what's going on are invaluable. It's been shown that the success of cubesat missions is directly related to the continuity of the team. Too many students cycling through a university led mission means that a lot of institutional knowledge is lost along the way.![]()
That's part of what amazes me most about the Hackaday community. So many brilliant people who want to stick around and show and teach others cool new things![]()
Also you flatsat can't run arbitrarily fast and provide far-reaching insight into the future. Or can time be accelerate in it?![]()
@Dan Maloney, I think that's a good closing point. Anything else you think I should address?![]()
Long-term stability of the team is important also for terrestrial projects.![]()
@Carl We can certainly propagate physics several years into the future using accelerated simulation. However, you lose some fidelity in the process.![]()
No, I think you've given us a lot to chew on! Great stuff, I was really looking forward to this one and it didn't disappoint!![]()
Good, glad to hear that! I'm available to answer anyone's follow-up questions as well. Send me an email at jacobkillelea@gmail.com any time!![]()
Thanks so much for your time today, I really appreciate it. And thanks to everyone for all the excellent discussion!![]()
Huge thanks for putting this together, Dan! -
Hack Chat Transcript, Page 1
07/12/2023 at 20:40 • 0 comments![]()
Okie doke, top of the hour, let's get going! I'm Dan, I'll be modding today along with Dusan as we welcome Jacob Killelea to the Software for Satellites Hack Chat!Jacob, how about you kick us off with a little about yourself?
![]()
Sure thing!![]()
Hi all, my name is Jacob Killelea! I’m an embedded and aerospace engineer currently living in the San Francisco Bay Area. My background and schooling was in Aerospace Engineering at CU Boulder, and I’ve been interested in just about everything relating to aerospace. I hold a private pilot certificate and a commercial small UAS certificate, and I spend my spare time hacking on microcontrollers, designing PCBs, and avidly watching what the Hackaday community comes up with!My technical background covers aero and thermodynamics, control systems, life support, systems engineering, electrical and RF systems, and lots of software. From 2020 to 2022, I worked for Intuitive Machines in Houston, where I wrote flight software for the Nova-C lunar lander. I worked on payloads from high power laser rangefinders, to missile gyros, to space based camera systems. I got to get hands-on with flight hardware, test instruments in-flight, and contribute some pretty neat code to this mission.
Today, I'm here to tell you just about everything I remember about building, programming, and operating a spacecraft! A lot of stuff is similar to what you might be used to on the ground, but there's also some stark differences, both in how things are done and what the domain forces you to do. It's a fun and inventive domain, with a lot of technology developed for special purposes. Hopefully I can pass on some of what I know and fill in the gaps that people might have in what they know about space systems.
![]()
So please, ask me anything and everything and I'll do my best to answer.![]()
wow!![]()
Quite the resume. 😮![]()
What sort of OSs might one encounter when coding for space systems?![]()
I imagine some sort of real-time OS?![]()
Are you using an off-the-shelf OS os some special "pace grade" OS for such jobs?![]()
Great minds think alike...![]()
hehe![]()
What coding standards, certifications, or guidelines are used for pico satellites? I'm thinking MISRA, DO-178, or similar?![]()
Great question, and the answer is all over the place! There are the systems you would absolutely expect to find, such as Linux and FreeRTOS, but spacecraft have historically had a wide range of solutions. Nova-C uses two computers, with one running Linux and the other running a proprietary RTOS called VxWorks.![]()
Tips and tricks to keeping stuff within power budget? 😅![]()
Which VxWorks? I haven't used them since 6.9?![]()
Other RTOSes include my personal favorite, RTEMS, and lots of specialty systems from aerospace vendors like Green Hills![]()
What are some ways flight software monitors, mitigates, and recovers from errors & failures?![]()
What are the advantages and disadvantages of using linux? Any special variants against "stock"?![]()
What does the UAS certificate enable? Is it easy to obtain? Also, how much (approximately, in total) for private pilot's license. I'm thinking like $5,000-$10,000 all said and done. (practice flights, instructor time, etc).![]()
Which programming language is used most often in space?![]()
C!![]()
@homer.sajonia.ii, Power budgets are really hard in space! If nothing else works, you spacecraft has to be able to keep itself thermally stable and power positive! Those are the core responsibilities of features like a spacecraft's safe mode. When things break, you need to be able to get the system back into a state that will keep it safe.![]()
@stu I think we were still on 6.9! Things move slowly and carefully in space. When things are well proven, we call that flight heritage, and generally that system won't be updated/touched again unless it really needs to be![]()
What's the range of temperatures the spacecraft can encounter, the "healthy" all-ok range, the "probably should survive" range, the "can't survive" range?![]()
They ran a rad-hardened CDP1802 on Galileo, I think.![]()
@Andrew Elbert Wilson That falls under the broad umbrella of FDIR (Fault Detection, Isolation, and Recovery). Every space system should report some amount of telemetry, usually called housekeeping, that indicates if things are working correctly. Faults might be software errors or hardware problems, and spacecraft generally recover by restarting software or power cycling the hardware/.![]()
Can the software be updated in-flight? How is this done securely?![]()
Hardware, but the same conservative prinicple![]()
@Thomas Shaddack Linux don't provide the hard realtime guarantees that other systems do. People don't like to include it in the core spacecraft controls for that reason, but it's really nice and flexible if you have the computer resources to run it![]()
@Jacob Killelea Thanks, I am designing a soft RISC-V in an RTG4 FPGA for work, trying to think of low-level methods for FDIR.![]()
@stu, I just took an online class. Since I already had a PPL, the FAA made it easy for me. The PPL itself was a huge endeavor! I don't reccomend it unless you can commit to flying at least twice a week in order to stay sharp, and twice a month once you have your cert.![]()
@hkurz, yep, C is the most common language. I wrote exclusively C software using the cFS framework from NASA![]()
GitHub - nasa/cFS: The Core Flight System (cFS)
The Core Flight System (cFS). Contribute to nasa/cFS development by creating an account on GitHub.
![]()
This is an amazing framework that runs on a few RTOSes and Linux, and supports a lot of the common tasks that spacecraft computers need, like process management, telemetry, error reporting, command ingest, scheduling, table driven configuration, and so on![]()
It also supports the CCSDS telemetry standards and has add-ons for multi-computer distributed computing![]()
@Thomas Shaddack it really depends on the component and where's it's placed on the spacecraft. A lot of things are rated for -40C/+80C, but actually keeping your components in that range is hard! You need to carefully insulate against both cold and heat, and be mindful of where the sun is hitting your spacecraft![]()
@Dan Maloney We ran AiTech SP0-s computers based around old MCP750s. Similar to the RAD750 from BAE that is running on Mars, IIRC![]()
How do you test your software? Do you have a replica spacecraft, load the softaware there and do end-to-end-tests or is it done mostly with unit tests?![]()
@Andrew Elbert Wilson, that reminds me of a payload I worked on that actually implemented a watchdog message at the FPGA level. It could tell if you if the soft CPU was running at all![]()
What happens if you have a fault in your FDIR housekeeping? Space brick?![]()
@Andrew Elbert Wilson check out the LEON 3 soft CPU design, it's a SPARC architecture written in VHDL that ESA uses, along with RTEMS (version 4.11, which is quite old) on a lot of their systems![]()
Any difference between terrestrial testing environment and the space? What can/did bite due to such differences?![]()
How to recover from a space brick?![]()
Do most of the modern satellites have a way to maintain their orbit indefinitely?![]()
@hkurz the motto is always to "Test like you fly, fly like you test." This is hard though! How do you simulate microgravity for your IMUs? How do you simulate three-axis rotational motion for your star tracker? The answer is to test each section as thoughly as you can, and your whole composite of tests should cover operational cases of your spacecraft![]()
You can probably have a watchdog power system for a reboot or a cold spare as a last resort for FDIR.![]()
@stu, if all else fails, you spacecraft should be able to go into a recovery mode called safe mode. The goal of a safe mode to keep your spacecraft thermally stable so it won't overheat, power positive so it won't drain the batteries, and to be able to phone home so ground operators can try and recover it. This can be tripped by any kind of reboot, watchdog, or software situation that doesn't have another method of recovery.![]()
I do lots of fault injection for Xilinx SRAM-based FPGAs to measure the configuration memory bit sensitivity. Can software fault injection into flight software be helpful as well?![]()
@Thomas Shaddack IIRC, there was a hacker-led recovery effort for an old NASA probe that was abandoned. Its batteries had been destroyed by thermal conditions, but it was spinning slowly in space, and every once and a while, its solar panels would provide enough power for it to boot and begin sending tones. I don't know if it ultimately succeeded, but people have tried!![]()
Mostly written in C, is it a basic "while" loop?![]()
Can be radiation-induced faults simulated on the ground? How are they detected/mitigated?![]()
Radition-induced faults are like bit flips from gamma rays?![]()
You can take your chips to cyclotrons and hit them with strange particles until they stop working.![]()
@daniel.fryling no, basically every spacecraft relies on some kind of expendable propellant. Low earth orbit (LEO) sats have lots of drag from the fringes of the atmosphere, and their orbits will decay back into the planet in a matter of weeks to years without station keeping. Above that, there's not a lot of atmosphere, but perturbations from the earth's non-ideal gravity, the moon, other planets, solar radiation pressure, etc, will all serve to push a satellite off course. Even if it could hover in place, it would need to counteract various torques that develop on the spacecraft. These can be mostly handled by reaction wheels, but eventually those need to be de-saturated and spun down once they reach a limit. The only way to do that is with reaction mass![]()
@Andrew Elbert Wilson exactly, usually a multi-level watchdog with gradually increasing levels of action taken to recover the spacecraft.![]()
@daniel.fryling usually only one![]()
only one while loop*![]()
Jacob, where are you working now? What are you working on these days? I had an offer from capella space back in 2018. Regret not joining them.![]()
You want _everything_ else on the spacecraft to execute in bounded time, so you know that it will eventually finish and hand control back to a main process.![]()
What does safe mode look like. Is it the first to have executive control after POR? Is it VxWorks 6.9, or RTEMS? Does it then wait patiently in the background while operating quiescently?![]()
@Thomas Shaddack and @stu, we tested radiation induced faults with a cyclotron at Texas A&M. It fired a proton beam into a bunch of our chips and produced a variety of faults. Some of these were recoverable and some were not. However, this information needs to be extrapolated into the radiation environment that you're operating in. LEO spacecraft are mostly shielded by the Earth's magnetic field, but beyond that, there's a whole spectrum of charged particles eager to destroy your transistors! These range from gamma rays and electrons to heavy iron nuclei created in supernovas a billion years ago![]()
Super neat!![]()
Its a lot of fun using Cream96 and Spenvis to calculate the possible Radiation TID and SEE effects based on your orbit!![]()
@stu, I'm actually (f)unemployed at the moment. I was working for a Google X project called Everyday Robots, but we got shut down in the layoffs that Google did earlier this year. I should talk to Capella again, I live within walking distance of their office.![]()
Safe mode is hard! It needs to be the most validated thing on your spacecraft. Often times, it's a heritage component, meaning it's something that operated successfully on a previous mission.![]()
This reminds me of that time when LightSail-A got locked up (non-responsive) - and they were hoping for a cosmic-ray hit to force a system reboot. It eventually happened - and the mission continued to give good data/telemetry that helps with the full mission of LightSail-2 in 2019.![]()
@liam![]()
@Liam Kennedy Thats a cool story!![]()
Have you ever encountered flight software written in Forth?![]()
However, that's not always a shortcut around your own engineering. I remember one mission I was taught about in school where a safe mode controller was borrowed from a previous mission and re-flown. This safe mode controller was designed to point the solar panels of the spacecraft at the sun and induce a slight rotation to keep it stable. However, the mission that borrowed this controller put the solar panels around the intermediate moment of inertia axis of the spacecraft. When it spun, this didn't keep it stable, it actually caused it to tumble! The spacecraft lost the ability to communicate since it's antennas were no longer pointed in the right direction and lost its power since its solar panels were no longer pointed at the sun. This was a case where simply borrowing proven hardware was a fatal error.![]()
LightSail-A (or 1) launched with a code error (introduced before spacecraft handover for flight integration) and they discovered it had a fault before launch - but could not fix it on the ground. It was the dreaded "oops I missed that ";"![]()
In short, there is no one size fits all method to do fault recovery or safe mode.![]()
Is it possible (and a good idea) to handle the most basic thermal/attitude control and main system watchdogs with a microcontroller? Something low-complexity, bullet-resistant, radiation-resistant, mistake-resistant, that watches over the high-complexity high-integration "brain"?![]()
@Liam Kennedy Yikes! I had no idea. We are certainly not relying on that for the Nova-C mission. We have seriously dynamic and time critical maneuvers, such as landing on the surface of the moon.![]()
@Liam Kennedy, thankfully cFS actually includes the ability to upload new versions of software from the ground![]()
How much is time-critical? Microseconds, milliseconds, seconds?![]()
Jacob, any insights into the differences between software standards for LEO/MEO/GEO satellites and what's needed for deep-space missions?![]()
Things are generally not microsecond level critical. Physics just doesn't happen that fast. However, being a second late at several thousand m/s can be a big problem...![]()
@Dan Maloney back on track eh? Sure! A lot of it has to do with the mission budget and lifetime. LEO sats are often scientific, mid to low budget systems with modest lifetimes. The focus is on simple systems that collect data and beam it down. You have regular contact with your ground stations, so autonomy isn't as important.![]()
In MEO/GEO, you actually have more communication, since you're loitering over a certain part of the planet, possibly forever! A GEO comm sat is a huge bird, up to the size of a school bus. It has a big power budget, and therefore lots of processing power! They often will co-host a special ARINC certified industrial OS with an application OS to do their data processing![]()
These systems are built by the big players, such as Lockheed and Boeing, and will have accommodations for customers to write their own software payload that is hosted in a virtualized environment onboard.![]()
What was the OS on Everyday Robots?![]()
Thank you for the response, Jacob, however, just a couple of additional question... "orbits will decay back into the planet in a matter of weeks to years without station keeping." What is "station keeping"?![]()
Actually, James Webb is like that! It has a core OS, and then a hosted environment for guest science. Guest science applications are actually written in JavaScript over there, IIRC...![]()
When you get to deep space, everything is harder and your spacecraft has to be able to handle itself. You'll be out of comms for a lot of the time, so data needs to be accumulated onboard and then exchanged in short windows. Bandwidth is constrained, and so is power once you start moving away from the sun! Commands are unlinked in tables that are scheduled in advance. The spacecraft might have to maneuver in a way that points its main antenna away from Earth in order to make its observations.![]()
@daniel.fryling station keeping is the process of keeping a satellite in the orbit you want it to be in! This might be a certain altitude, inclination, or phase. For geostationary satellites, it's probably a certain longitude around the equator![]()
But a Orbit that decays significantly within weeks is more on the low end of LEO.![]()
If people haven't seen them before, I totally recommend checking out cFS and running it yourself (https://github.com/nasa/cFS). It's a pretty friendly C programming framework with a core set of services for spacecraft such as process management, communications, logging, and application hosting. On top of that, it comes with a set of applications that talk over a pub/sub bus to support things like data storage, communications, and housekeeping. It also, of course, has provisions for you to write your own applications!![]()
@Carl totally true! the lowest of LEO sats will fall out of the sky in a short while due to drag, but the time it takes an orbit to decay is exponentially related to altitude. Once you're at MEO/GEO, those things are staying up there for thousands of years past their useful lifetime.![]()
That's why it's become increasingly important to manage and track both all the active satellites in space, but also all the debris! If thing hit each other (and they have), they produce even more debris. This could, hypothetically, cascade into something called the Kessler (?) Sydrome, where our local space environment becomes inaccessible because of the amount of debris whipping around the planet.![]()
Transcript coming up at the e
