-
Hack Chat Transcript, Part 3
04/22/2020 at 20:22 • 0 commentscarlclaunch511:02 PM
When Ken asked if I was interested in joining the project, it didn't take a microsecond to decide yes
I hate to do this, but we've been at this for an hour and if any of our guests need to get back to work, we should let them go. But this has been really fascinating, and I really want to thank Marc, Ken, Carl, and Mike for coming along today and sharing their insights and experiences. Feel free to keep the discussion going as long as you'd like, of course - the Hack Chat is always open.
Thank you all. Really interesting and inspiring!
Thanks for setting this up, Dan.
Great stuff! Thank you.
Thanks Dan! I'm happy to stick around if anybody has more questions
Really great way to spend lunch!
Thank you for this interesting session.
Thanks Dan for the opportunity. I can stay on a little more but I'll have to go soon.
Thank you all!
was awesome to be able to ask some questions here :) you guys did a great job at the whole thing
I have no work, only hobbies so I will stay on also
Y
Fantastic work AGC restoration team!
Thanks all! I'll wait a while to pull the transcript, in case there are any more interesting bits.
And don't forget next week we'll be sitting down with Ben Krasnow from the Applied Science YouTube channel, to talk about Citizen Science.
Thanks everyone! its amazing to see what you guys did!!! and surprised to find out most of you don't have "formal" EE background... yet do more low level stuff than most EEs I know...
Oh cool. Ben is such a fascinating guy! Is it on the same chat next week?
https://hackaday.io/project/166155/gallery#dcf4b886fedebceec2c7fbdeda92829d (shameless plug... but you all are likely to be interested in this sort of thing!).
I've gone into the rabbit hole with core memory and I am working on a kit for those who are interested. I aim to have some produced this summer (just received another prototype batch of PCBs 20 minutes ago!). Would appreciate your feedback if you are interested in exploring core memory.@Andy Geppert Indeed we are all core memory afficianados
oh cool! what cores are you using?
"target market?" if that's fair to assume!
They are 1mm OD from somewhere in Europe via good ol' eBay. NOS I'm guessing.
yes. Have an Arduino shield core memory board project I bought, but also worked on core more directly. Saw a great hack where someone put lighting to visualize the state of the cores, as an overlay.
It's soooo satisfying to weave them together and have it actually work.
Yes to target market too
@Andy Geppert Nice! How far can you extend this? I have two Russian memory core planes that I'd love to turn into a working demo display. A 4k plane and a less dense and probably better suited 1k plane.
@carlclaunch51 I think you saw my proof of concept then, last year after Maker Faire?
oh, that was you at the after party! Yes, loved it
Oh yes I remember it. That was awesome!
Yes, thanks. Based on the feedback I got, I thought this was worthy to pursue, so this is an extension of that concept to a proper 8x8 bit array. Just think of what you can do with 64 bits!
I have a few minicomputer core memory boards that come with onboard access electronics, much easier than engineering the drive for a naked core board (but less satisfying)
i've only had some real old core planes to play around with and all of them had damaged wiring
My IBM 1130 computer is a 16K word core based machine but can be expanded to 32K. I have the mini core boarrds to provide the memory portion, but alas IBM left off the gates for any address bits not installed on the machine so it requires some modifications to the register boards and other portions of the machine to activate it all.
Marc has some video of our experiments to drive an IBM core plane (successfully) with our own drive circuits and sense detectors. Worth watching if you like core.
Oh, I have definitely watched that video! I learned a lot of fundamental stuff, and it helped me break out of a rut and get my system working.
seems like theres some pretty affordable soviet core planes on ebay
I've been tempted by those. For this project, I'm focusing the interactive/LED part. But it has been suggested that a full LCD behind a larger core plane would be the next logical step.
that makes a lot of sense, Andy
@Andy Geppert Exactly my thought. LCD behind a Russian core plane!
Looks like this would be fairly doable
-
Hack Chat Transcript, Part 2
04/22/2020 at 20:21 • 0 commentsnot long -- a guy by the name of George Silver observed the problem at the Cape while testing LM-3, documented it, and reported it. and he knew instantly what had happened
@Mike Stewart But we understand a lot more of it now. It has not been reported correctly by anyone so far as we can tell. It's a horrendously complicated fault.
The behavior had been observed at both Grumman and during testing, but its impact on the mission wasn't fully realized
@Jeremy Weatherford essentially yes, but exactly how the circuit breaks down and, more importantly, what the dynamic behavior in response to that breakdown was is something we're still working on
@Jeremy Weatherford Yes that's one of the root causes.
In simulated landings, there is a very narrow range of load which generates the alarms but doesn't crash the vehicle. Luck was involved.
The equipment that "failed", as it is not so much a failure as an unfortunate system miswiring, was the digitizers for the Rendez-vous Radar position sensors.
the two power supplies were also at different voltages by the time they reached the CDU -- normal was 28Vrms, but it dropped to 15Vrms when switched to the other supply
@Mike Stewart oh, even more subtleties and details we hadn't noticed before
It's in the CDU, the Coupling Data Unit, which is a big A/D, bigger than the AGC even. Very complicated.
for our landing demonstrations I triggered the alarms by using the test connector to halt the computer temporarily every configurable number of memory cycles, and then tuned that number to be a load that would work well for demonstrations
so things wouldn't get too out of hand :)
Another AGC project I'm currently working on is writing an emulator for the Honeywell 1800 mainframe. This computer ran the assembler that created the binaries for the Apollo Guidance Computer. It also created the tapes that were used to put the code into core rope memory.
Mike's method works equivalently to the real overload because each pulse from the resolver/digitizer would force a hidden instuction thus taking memory cycles
@Mike Stewart Remember how I first made a simple device that produced the pulse stream that was reported in the accident. We connected that to the AGC and that sure ruined your landing.
did any of the tapes/other sw media survive?
haha yep
no, we haven't found any tapes no
pretty much just listings
and all of the mechanical/electrical engineering drawings in the National Archives
and binary in the core rope modules we have been able to read
yep
So it's more subtle than what has been reported. And they were a lot closer to crashing that was also reported. They really lucked out.
May I interrupt with a background question or two? 1. how do you folks find the time for this (those who aren't retired)? And can you please fill us in on your backgrounds that enabled you to do this fascinating work? Thank you.
https://github.com/thewonderidiot/pyul/blob/master/decks/SOLRUM55.R0.deck
also, while Ken is working on that Honeywell 1800 emulator, I'm slowly taking all of our listings back to punch-card format to feed into it:i assume no cores from ground equipment survived?
@Ken Berkun Ken and I are retired, Marc is semi-retired and Mike is a force of nature
I just spend all of my time at home, nights and weekends, working on this stuff. And probably spend too many work hours thinking about it
I just took two weeks off...
we operate at the hairy edge of family resentment
The background that helped me with this project was my experience restoring other old computers, starting at the Computer History Museum and then other projects with Carl and Marc.
I have an embedded software background, but before starting in on the AGC project couldn't even work through simple transistor circuits. I like to pick projects that I don't know how to do, because learning is most of the fun
Do you have education in hardware? EEs? Learn on the go? Just curious and very admiring of your work. (I'm a software guy, hardware amazes the heck out of me. Especially analog.)
Thanks Mike, you answered my question before I asked!
@Ken Berkun mostly learn as we go
Also the part that is most time consuming, the researching and understanding of the documentation, Mike had a 5 years head start on this!
I have half a college education in computer engineering / electrical engineering. Dropped out to accept a job offer and still haven't finished my degree...
I'm a software guy, so I don't have any real electronics qualifications. LTspice helps me figure out a lot of circuits :-)
I agree about taking on projects that you don't know how to do. I've often said that I wouldn't take a job I knew I could do. Of course my wife gets tired of my moaning about having no clue about how to proceed at times...
hehehe
I found the MITX (EdX) EE course sequences very helpful
I am trained as an optical physicist, no formal education in EE either. All picked up.
yeah LTspice is a godsend for poking at and understanding things
Thank you all, keep up the great work!
ken: you also a lot of IC reversing, right? any good resources/chatrooms for that? (besides mcmaster's website and your hackaday talk)
often I will start on a circuit analytically and Ken will get the answer from LTspice before I am done
Also working as a team. This is a project that required more talent that can fit in one person.
1202
100% yeah
hi Dag!
Hi Dag
Houston, give us a reading on the 1202
Complementary talents and shared commitment
Hi guys -- I've been enjoying lurking on thi discussion!
http://ai.eecs.umich.edu/people/conway/VLSI/VLSIText/PP-V2/V2.pdf) and Designing Analog Chips (http://www.designinganalogchips.com/_count/designinganalogchips.pdf)
Peter: I don't have a lot of good resources for IC reverse-engineering. I've bought some old (1970s) books on chips that help understand the circuits from that era. Two online sources are Mead & Conway for digital (Enthusiasm and persistence (and more than a little talent) seem to be essential in this endeavor (as it is in most).
@Dag Spicer Hi Dag. Thanks for letting us poke at the CHM's AGC!
With the help of Dag we were able to recover the core rope that was inCHM's AGC
https://github.com/virtualagc/virtualagc/tree/master/Retread50
and have disassembled it here:Dag is senior curator at the Computer History musesum and a space nuts ;-)
still working on the disassemblies for all of the other modules we've dumped...
Thank YOU guys for recovering and preserving lost knowledge. It's why we exist.
Mike works back from binary code and somehow ends up with commented source listings. Intellectual feat
@Mike Stewart Which makes me think I still didn't make a video of the footage of you going through the original Luminary listing you bought! That was amazing. I'm glad that one ended up in your hands
I've been studying the LVDC recently, the IBM computer that was on the Saturn V rocket. It's interesting because it had a similar role to the AGC, but the two computers were built completely differently. The LVDC used hybrid modules, not integrated circuits. It was triply-redundant, with three copies of the circuitry and voting. It was a 26-bit computer and operated serially, one bit at a time, so it had a 1-bit ALU, bus, etc. It used printed-circuit boards and backplane, not the modules and wire-wrapped backplane of the AGC.
the comments are mostly borrowed from related listings, where I've been able to find precursor/successor code :)
Yes, that triangulation is part of the intellectual feat
@curiousmarc ah yeah, I was wondering if/when that was going to make it into a video, haha
Hi Mike. I read that before you got involved with the real AGC, you built a gate-for-gate implementation of it using FPGAs. Can you describe how you did that and what were some of the challenges (e.g., two phase clocking)? How did that experience lead to working on the real AGC?
@Mike Stewart I just forgot about it :-o ! Guess what I am working on next...
https://github.com/virtualagc/agc_simulation -- be warned that it was the first time I had ever written verilog
the code is all here:the biggest challenge was that the AGC is sort of asynchronous -- there's a main clock but a lot of things are timed with propagation delays
Very common in the 1950s and early 1960s as we know from the IBM 1401, 360 and 1130 systesm
the realization that let me put it into an FPGA was that since it's entirely made of a single type of chip, every gate will have roughly the same propagation delay, barring differences in loading on each net
so you can think of the whole system as having discrete timesteps, where each timestep is one Tpd
so I ran a clock into every NOR gate whose period matched the propagation delay of the gates, and step all of the gates in tandem
and the problem then became initial conditions to keep the flip-flops from getting into bad oscillating states
is Tpd close even when multiple gates are strapped together?
close enough!
apparently :P
we didn't actually measure that on the real thing
but it works fine with that assumption in simulation
helps that you were working with huge feature size RTL logic
That FPGA replica was a Godsend during the restoration. Since it's gate exact, we could look at what each signal should look like and compare with the actual AGC we were trying to repair. It even predicted the glitches accurately.
thats a pretty cool approach to simulating it :)
anyways, it all applied directly to the restoration, because at that point I had worked through all of the schematics and had them all roughly in my head, so it was very easy to pinpoint stuff we were looking for while restoring
not gate for gate but enough that I can normally tell you which page a particular thing is implemented on
Mike was faster than Google Search - almost instant citation of the relevant documents, signals and gates
and yeah, the prediction of glitches was unexpected and very fulfilling, haha
Which you verified existed undetected on the real AGC hardware
@Mike Stewart You should also mention the FPGA replica you made of the ground equipment AGC monitor. And even enhanced it. That was key to the restoration.
https://github.com/thewonderidiot/agc_monitor and the verilog is much better, haha
that's atPlus Marc's company, Samtec, recreated the unobtainium miniWasp connectors without which it would have been much, much more difficult to proceed.
http://www.ibiblio.org/apollo/Documents/HSI-208487.pdf ) and figuring out how to make those things happen with the pins available to me on the test connector
we don't have any schematics for the original AGC monitor, so that was essentially reading through the manual (and then eventually realizing that the test connector pins gave me far more control over the computer than was really intended, which let us do erasable memory simulation and the interface to NASSP
Mike, how did you get involved with the owner of the real AGC? How did you come to meet Carl, Ken, and Marc?
Ken had to reverse engineer the core rope simulator modules, which were undocumented, in order to provide the core rope data for each AGC fetch. Once he figured it out, he built a device to respond to fetches.
http://www.ibiblio.org/apollo/Documents/LDW370-54001.pdf
I got into contact with Jimmie very shortly after I found this drawing from Grumman in the National Archives:that drawing contained a more-or-less complete pinout of the main connector of the AGC in the LM, which was info I had been missing and trying to find for years
with that and the full/partial schematics available at the time, I became pretty convinced that if I had access to a real AGC for a few days, I would be able to beep out the rest of it and complete our partial schematics I had also seen this video on youtube by Francois Rautenbach:
I had talked to him a few times on the virtualagc mailing list about the (at the time) mysterious DSKY internals, so I reached out to him to see if he could put me into contact with whoever owned the AGC he was looking at there
and he got me into touch with Jimmie, who brought up the idea of powering the AGC up in his very first email to me :)
so at that point I had access to an AGC, an owner who wanted to power it up. With no experience at all working with old hardware, but I did know the thing inside and out
siliconpr0n.org). Mike told me about the AGC and asked if I was at all interested. I told him I was ready to book the next flight to Houston :-) Carl and Marc joined me in the restoration, which we did in a hotel room outside Houston.
Mike got in touch with me through John McMaster (yeah, so I asked John if he knew anybody that did old computer restorations, and he introduced me to Ken
When Ken asked if I was interested in joining the project, i
-
Hack Chat Transcript, Part 1
04/22/2020 at 20:20 • 0 commentsHello every, welcome to Hack Chat! Today we've got not one but four hosts - CuriousMarc, Ken Shirriff, Mike Stewart, and Carl Claunch. They've done a lot of cool retro hacks, chief among them getting a real Apollo Guidance Computer working again for the first time in 50 years. Welcome all!
Maybe you can each tell us a little about yourself to kick things off
I am retired and able to devote all my time to restoring old mainframes and other vintage technology
I'm retired and looking for new projects.
@carlclaunch51 - From what I understand, mainframe skills are a growth industry right now. At least for the short term ;-)
I currently write flight software for satellites Capella Space, and devote pretty much all of my free time to Apollo digital archaeology in various forms
Hi. I'm a retired programmer and I've been researching vintage computing, reverse-engineering old microprocessors, mining Bitcoin by hand, and studying old aerospace computers.
Hello CuriousMarc here. In real life work on fiber optics at Samtec. Former Intel Fellow. In fantasy life restore old electronics with Carl and Ken and Mike in my over-equipped basement lab.
@Dan Maloney I ajm focused on the hardware side now, while the current short term interest is in software
Marc, Carl, and I have been working together at the Computer History Museum for a while, helping to keep the old IBM 1401 punch-card mainframe running. We also restored a Xerox Alto in Marc's basement. We joined up with Mike to restore the Apollo Guidance Computer.
Actually that's how we all met (save for Mike), working at the Computer History Museum in Mountain View, CA
We are all volunteers helping with restoration and digital archeology projects
Stop by the museum (one quarantine is lifted) to see a punch-card machine running.
That's still a bucket list trip for me. We had Dag Spicer on last year to talk about CHM and preserving our history. CHM is such a valuable resource.
And Ken and I were introduced through a mutual friend, because I was about to try to restore an AGC, but didn't yet have any experience working with old hardware at all
yeah for sure
And if you have an old 1960's mainframe in your basement please let us know
How about my garage?
I enjoyed watching your Apollo restoration videos and Soyuz clock.
I've always been a bit envious of the CHM and the people around it, here in the Netherlands there is no such pool of retro repair knowledge and resources
It's amazing how much technology progressed from the IBM 1401 at the museum (1959) to the Apollo Guidance Computer (mid-1960s). The 1401 uses transistors (germanium, not silicon) while the Apollo Guidance Computer uses integrated circuits. They both use magnetic core memory for storage, though.
@carlclaunch51 Yes you have quite a few but you won't let me film ;-)
https://blog.samtec.com/post/restoring-agc-part1/ "When powered, the system came to 4.99 +/- 0.01 volts. Think about that: this is 50 plus year old equipment which had been in storage for decades, and it was running within 0.20% of the original specification! "
FromThat is a m a z i n g !
@curiousmarc , if I had a teletype in my garage I'd want to restore it. That last video gave me a nostalgia attack
@curiousmarc yes I have been a bit obstructionist towards filming until my garage looks better.
Can you recommend any books for people who can't get enough Apollo-era technology?
Sunburst and Luminary by Don Eyles is fantastic
Rocket Ranch and related books are excellent for a KSC oriented view
+1 on Don Eyles book
when the series first started i was wondering whether a modern replica of the AGC would be possible, but I couldn't find gates with a similar open collector output and logic function (not in the 74 series at least). are there any chips still in production that could be used one-for-one without significantly altering the design?
especially the bit where they form logic by tying outputs of gates together
Eldon Hall's Journey to the Moon: The History of the Apollo Guidance Computer is another good book.
I have a plan for that for my replica -- one sec
Open collector gates? You can still get them in the 74xx series
Digital Apollo by David A. Mindell
not in the dual three-input NOR variety though
But the tough bit is the memory tray B
not sure about triple input NOR with open collectors
been a while since i looked at it but yep think the 3 input NOR was the problem
Just a note, I'll be pulling a transcript at the end and posting it publically, so don't worry about capturing links or titles of books - it'll all be there for you later.
Dang we have to make a new wafer of it
new project!
https://www.digikey.com/product-detail/en/texas-instruments/SN74LVC06APWR/296-8437-2-ND/377383
i though about spinning small pcbs with real tiny SMD transistors
Peter: you could probably use surface-mount transistors and emulate the integrated circuits directly without increasing the size much. The integrated circuits in the AGC were just 2 transistors + 6 resistors.
those were my plan -- a hex open-drain inverter behaves much like a dual open-drain three-input nor gate with outputs tied together in triples
Sorry, 8 resistors in the AGC's IC.
thats a good one :) couldn't get the layout small enough for a size accurate PCB module, but might just not have found small enough packages
being core memory, was the program still intact when you got the AGC running?
What are you going to do for memory Mike? Wanna use that big IBM FAA core memory I have at home?
Yes, Mike was able to read the memory from the AGC.
quad input 4002b? push-pull output
Any idea of what mission it was planned for?
my plan was always to attempt to weave my own, with those cores I bought, but it's unclear how far into that I would get before giving up
@brianpagephotog we had a dead bit but due to having parity it was possible to rewire and extract the data. One of Marc's videos details the effort
We could tell exactly what the AGC was running and displaying when it was shut off. It was doing an IMU alignment, with coordinates matching the Houston space center. It's in one of Marc's videos.
Nice. I haven't seen the videos. Do you have a quick link?
our AGC was never intended for a mission -- it was a prototype model, but it was installed in the LTA-8 lunar test article, and underwent the full LM thermal vacuum testing in Houston
@Mike Stewart That would be awesome. You might just be able to get some woven ones off 1970's boards like DEC.
have you guys ever thought about wiring up more parts to it? like sensors or a radiomodem?
https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb59FWrLZfdlisi_x7-Ut_-w7
Thanks!
The AGC video series was fantastic -- it made for great viewing, and showcased each of your unique talents beautifully.
That's 31 videos, though - not sure which one Ken was talking about. They're all worth watching, of course ;-)
Peter: the AGC is back in Houston now, so we don't have an opportunity to wire up more things to it.
https://archive.org/download/S68-21511/S68-21511.jpg
@Peter Bosch we did hook up some inputs and outputs including simulating various IMU inputs, but we don't have access to the AGC which stopped any future experiments
^^ there's a picture of our AGC installed in the Lunar Module, taken in the thermal vacuum chamber in 1968
and Ken got a picture of LTA 8 hanging in Johnson Space Center with a gaping hole where the AGC belonged, shot through the open hatch
Dan I'm not sure which video shows the original contents. Probably around number 13. Maybe Marc remembers.
@Peter Bosch I miss that machine so much. It's back with its owner. But between all of us we'd have a lot to attach to it: the radio telemetry, Carl is working on a real DSKY display, I have a gyro, Mike has the ground control box for uploads...
Wow, besides your gyro video, any other videos in the making on those?
I also built a telemetry interface so that I could stream out the data that would have been downlinked to mission control
Give me a moment I'll try to dig the video out.
we can still take trips out to work with it, for what it's worth -- I've flown out with more rope modules from Don Eyles to dump their contents using it. it's just that flying back and forth to Houston to work with it is a bit prohibitive
Plus a frivilous Lego LM model with tiny LEDs that were lit by the AGC as it commanded movement of the spacecraft, while we were running mission simulations
This is a very strange question, but do we know exactly how the telemetry/data from the AGC went into the RTCC and then into mission control on their high resolution monitors? Everything I've researched about RTOS comes up blank.
Thank you guys for putting your energy and talent into all of this restoration work, especially the AGC. It's incredibly inspiring and fun to follow the project. So much to be [re-]learned!
Lego + AGC for the win! :-) Two great things I never imagined would be together.
@cprossu the expert on the RTCC is a guy by the name of Niklas Beug, who is currently the primary maintainer for NASSP -- if you have IRC, we have a freenode channel #nassp, and I'm sure he would be happy to talk about it
Who happens to be here
We know exactly what data was downlinked, but there doesn't seem to be anything on the RTOS on the 360/75 computers. Probably the screen layouts in mission control are known
oh hello
lol
Hi Niklas
hey Niklas!
. That's the first thing we did when we got it fully running for the first time before rewiring the memory.
This video: That is awesomeHey guys. But I don't really know much about telemetry. It was downlinked from the AGC through tracking stations, processed in the RTCC computers and then available for display.
Marc built a bit of hardware to produce the pulses that would have flooded the AGC with spurious movement data from the resolver for the LM rendezvous radar on Apollo 11.
@carlclaunch51 . No pictures though
What amazes me is I found an old philco ford document that has all the numbers and descriptions of how mission control was builtand you got the 1201/1202 alarms?
https://en.wikipedia.org/wiki/Eidophor
As far as the mission control displays, it wasn't easy to project images onto a big screen. They used an early projection technology called Eidophor. It used electrostatic charges to deform an oil surface(!) A scanning electron beam "drew" on the surface and they reflected light off the oil.That's why it would be really cool to get the telemetry stream out of the real thing with Carl's contraption.
https://arstechnica.com/science/2012/10/going-boldly-what-it-was-like-to-be-an-apollo-flight-controller/2/
This Ars article from a while ago has a lot of information on the tech in the MCC -We are all a bit obsessed with getting to the exact conditions and hardware behavior that led to the 1201/1202 errors
yeah, one of the main things I'm working on right now is understanding/simulating *exactly* what happened with those alarms. so far I haven't managed to replicate the described behavior exactly
Marc's pulse generator overloads the computer too much -- we get 1201s and 1202s, but far too many of them
it is more complicated that it seems. Not just a stready stream of pulses at maximum rate.
how long did it take the original team to figure it out?
And just like that managed to find this one.
https://cdn2.hubspot.net/hubfs/413105/Apollo%20Press%20Kits/Philco%20Ford.pdf
^ Still though we need to find out more info on the system 360/75 rtos
Wasn't it something to do with two AC power supplies that could be in different phases?
About Us Contact Hackaday.io Give Feedback Terms of Use Privacy Policy Hackaday API