Dan Maloney
not long -- a guy by the name of George Silver observed the problem at the Cape while testing LM-3, documented it, and reported it. and he knew instantly what had happened
@Mike Stewart But we understand a lot more of it now. It has not been reported correctly by anyone so far as we can tell. It's a horrendously complicated fault.
The behavior had been observed at both Grumman and during testing, but its impact on the mission wasn't fully realized
@Jeremy Weatherford essentially yes, but exactly how the circuit breaks down and, more importantly, what the dynamic behavior in response to that breakdown was is something we're still working on
@Jeremy Weatherford Yes that's one of the root causes.
In simulated landings, there is a very narrow range of load which generates the alarms but doesn't crash the vehicle. Luck was involved.
The equipment that "failed", as it is not so much a failure as an unfortunate system miswiring, was the digitizers for the Rendez-vous Radar position sensors.
the two power supplies were also at different voltages by the time they reached the CDU -- normal was 28Vrms, but it dropped to 15Vrms when switched to the other supply
@Mike Stewart oh, even more subtleties and details we hadn't noticed before
It's in the CDU, the Coupling Data Unit, which is a big A/D, bigger than the AGC even. Very complicated.
for our landing demonstrations I triggered the alarms by using the test connector to halt the computer temporarily every configurable number of memory cycles, and then tuned that number to be a load that would work well for demonstrations
so things wouldn't get too out of hand :)
Another AGC project I'm currently working on is writing an emulator for the Honeywell 1800 mainframe. This computer ran the assembler that created the binaries for the Apollo Guidance Computer. It also created the tapes that were used to put the code into core rope memory.
Mike's method works equivalently to the real overload because each pulse from the resolver/digitizer would force a hidden instuction thus taking memory cycles
@Mike Stewart Remember how I first made a simple device that produced the pulse stream that was reported in the accident. We connected that to the AGC and that sure ruined your landing.
did any of the tapes/other sw media survive?
haha yep
no, we haven't found any tapes no
pretty much just listings
and all of the mechanical/electrical engineering drawings in the National Archives
and binary in the core rope modules we have been able to read
yep
So it's more subtle than what has been reported. And they were a lot closer to crashing that was also reported. They really lucked out.
May I interrupt with a background question or two? 1. how do you folks find the time for this (those who aren't retired)? And can you please fill us in on your backgrounds that enabled you to do this fascinating work? Thank you.
also, while Ken is working on that Honeywell 1800 emulator, I'm slowly taking all of our listings back to punch-card format to feed into it: https://github.com/thewonderidiot/pyul/blob/master/decks/SOLRUM55.R0.deck
i assume no cores from ground equipment survived?
@Ken Berkun Ken and I are retired, Marc is semi-retired and Mike is a force of nature
I just spend all of my time at home, nights and weekends, working on this stuff. And probably spend too many work hours thinking about it
I just took two weeks off...
we operate at the hairy edge of family resentment
The background that helped me with this project was my experience restoring other old computers, starting at the Computer History Museum and then other projects with Carl and Marc.
I have an embedded software background, but before starting in on the AGC project couldn't even work through simple transistor circuits. I like to pick projects that I don't know how to do, because learning is most of the fun
Do you have education in hardware? EEs? Learn on the go? Just curious and very admiring of your work. (I'm a software guy, hardware amazes the heck out of me. Especially analog.)
Thanks Mike, you answered my question before I asked!
@Ken Berkun mostly learn as we go
Also the part that is most time consuming, the researching and understanding of the documentation, Mike had a 5 years head start on this!
I have half a college education in computer engineering / electrical engineering. Dropped out to accept a job offer and still haven't finished my degree...
I'm a software guy, so I don't have any real electronics qualifications. LTspice helps me figure out a lot of circuits :-)
I agree about taking on projects that you don't know how to do. I've often said that I wouldn't take a job I knew I could do. Of course my wife gets tired of my moaning about having no clue about how to proceed at times...
hehehe
I found the MITX (EdX) EE course sequences very helpful
I am trained as an optical physicist, no formal education in EE either. All picked up.
yeah LTspice is a godsend for poking at and understanding things
Thank you all, keep up the great work!
ken: you also a lot of IC reversing, right? any good resources/chatrooms for that? (besides mcmaster's website and your hackaday talk)
often I will start on a circuit analytically and Ken will get the answer from LTspice before I am done
Also working as a team. This is a project that required more talent that can fit in one person.
1202
100% yeah
hi Dag!
Hi Dag
Houston, give us a reading on the 1202
Complementary talents and shared commitment
Hi guys -- I've been enjoying lurking on thi discussion!
Peter: I don't have a lot of good resources for IC reverse-engineering. I've bought some old (1970s) books on chips that help understand the circuits from that era. Two online sources are Mead & Conway for digital (http://ai.eecs.umich.edu/people/conway/VLSI/VLSIText/PP-V2/V2.pdf) and Designing Analog Chips (http://www.designinganalogchips.com/_count/designinganalogchips.pdf)
Enthusiasm and persistence (and more than a little talent) seem to be essential in this endeavor (as it is in most).
@Dag Spicer Hi Dag. Thanks for letting us poke at the CHM's AGC!
With the help of Dag we were able to recover the core rope that was inCHM's AGC
and have disassembled it here: https://github.com/virtualagc/virtualagc/tree/master/Retread50
Dag is senior curator at the Computer History musesum and a space nuts ;-)
still working on the disassemblies for all of the other modules we've dumped...
Thank YOU guys for recovering and preserving lost knowledge. It's why we exist.
Mike works back from binary code and somehow ends up with commented source listings. Intellectual feat
@Mike Stewart Which makes me think I still didn't make a video of the footage of you going through the original Luminary listing you bought! That was amazing. I'm glad that one ended up in your hands
I've been studying the LVDC recently, the IBM computer that was on the Saturn V rocket. It's interesting because it had a similar role to the AGC, but the two computers were built completely differently. The LVDC used hybrid modules, not integrated circuits. It was triply-redundant, with three copies of the circuitry and voting. It was a 26-bit computer and operated serially, one bit at a time, so it had a 1-bit ALU, bus, etc. It used printed-circuit boards and backplane, not the modules and wire-wrapped backplane of the AGC.
the comments are mostly borrowed from related listings, where I've been able to find precursor/successor code :)
Yes, that triangulation is part of the intellectual feat
@curiousmarc ah yeah, I was wondering if/when that was going to make it into a video, haha
Hi Mike. I read that before you got involved with the real AGC, you built a gate-for-gate implementation of it using FPGAs. Can you describe how you did that and what were some of the challenges (e.g., two phase clocking)? How did that experience lead to working on the real AGC?
@Mike Stewart I just forgot about it :-o ! Guess what I am working on next...
the code is all here: https://github.com/virtualagc/agc_simulation -- be warned that it was the first time I had ever written verilog
the biggest challenge was that the AGC is sort of asynchronous -- there's a main clock but a lot of things are timed with propagation delays
Very common in the 1950s and early 1960s as we know from the IBM 1401, 360 and 1130 systesm
the realization that let me put it into an FPGA was that since it's entirely made of a single type of chip, every gate will have roughly the same propagation delay, barring differences in loading on each net
so you can think of the whole system as having discrete timesteps, where each timestep is one Tpd
so I ran a clock into every NOR gate whose period matched the propagation delay of the gates, and step all of the gates in tandem
and the problem then became initial conditions to keep the flip-flops from getting into bad oscillating states
is Tpd close even when multiple gates are strapped together?
close enough!
apparently :P
we didn't actually measure that on the real thing
but it works fine with that assumption in simulation
helps that you were working with huge feature size RTL logic
That FPGA replica was a Godsend during the restoration. Since it's gate exact, we could look at what each signal should look like and compare with the actual AGC we were trying to repair. It even predicted the glitches accurately.
thats a pretty cool approach to simulating it :)
anyways, it all applied directly to the restoration, because at that point I had worked through all of the schematics and had them all roughly in my head, so it was very easy to pinpoint stuff we were looking for while restoring
not gate for gate but enough that I can normally tell you which page a particular thing is implemented on
Mike was faster than Google Search - almost instant citation of the relevant documents, signals and gates
and yeah, the prediction of glitches was unexpected and very fulfilling, haha
Which you verified existed undetected on the real AGC hardware
@Mike Stewart You should also mention the FPGA replica you made of the ground equipment AGC monitor. And even enhanced it. That was key to the restoration.
that's at https://github.com/thewonderidiot/agc_monitor and the verilog is much better, haha
Plus Marc's company, Samtec, recreated the unobtainium miniWasp connectors without which it would have been much, much more difficult to proceed.
we don't have any schematics for the original AGC monitor, so that was essentially reading through the manual ( http://www.ibiblio.org/apollo/Documents/HSI-208487.pdf ) and figuring out how to make those things happen with the pins available to me on the test connector
and then eventually realizing that the test connector pins gave me far more control over the computer than was really intended, which let us do erasable memory simulation and the interface to NASSP
Mike, how did you get involved with the owner of the real AGC? How did you come to meet Carl, Ken, and Marc?
Ken had to reverse engineer the core rope simulator modules, which were undocumented, in order to provide the core rope data for each AGC fetch. Once he figured it out, he built a device to respond to fetches.
I got into contact with Jimmie very shortly after I found this drawing from Grumman in the National Archives: http://www.ibiblio.org/apollo/Documents/LDW370-54001.pdf
that drawing contained a more-or-less complete pinout of the main connector of the AGC in the LM, which was info I had been missing and trying to find for years
with that and the full/partial schematics available at the time, I became pretty convinced that if I had access to a real AGC for a few days, I would be able to beep out the rest of it and complete our partial schematics
I had also seen this video on youtube by Francois Rautenbach:
I had talked to him a few times on the virtualagc mailing list about the (at the time) mysterious DSKY internals, so I reached out to him to see if he could put me into contact with whoever owned the AGC he was looking at there
and he got me into touch with Jimmie, who brought up the idea of powering the AGC up in his very first email to me :)
so at that point I had access to an AGC, an owner who wanted to power it up. With no experience at all working with old hardware, but I did know the thing inside and out
Mike got in touch with me through John McMaster (siliconpr0n.org). Mike told me about the AGC and asked if I was at all interested. I told him I was ready to book the next flight to Houston :-) Carl and Marc joined me in the restoration, which we did in a hotel room outside Houston.
yeah, so I asked John if he knew anybody that did old computer restorations, and he introduced me to Ken
When Ken asked if I was interested in joining the project, i
Discussions
Become a Hackaday.io Member
Create an account to leave a comment. Already have an account? Log In.