-
Hack Chat Transcript, Part 3
03/01/2023 at 22:11 • 0 commentshilarious
-1 = 50meters or more underground
160 meters where at point 4 of LHC
And here I was thinking I was being overly enthusiastic with my descriptions of working at CERN
@Dan Maloney Lol.
ok, youtubemy ears popped when going down there
they installed a counter in the new elevators :-)
SPS is different. "only" tens of meters, up to 60 I think. But 24 storeys
ever thought about making it faster so you could experience zero-g?
Awesome.
they are very very fast
At least there's a bathroom right off the elevator
Hehe
@Thomas Shaddack funny enough, exactly the opposite. The elevators are not used very often, when the machine runs, there is no access for months. In this particular one, I got stuck about 15 seconds down every time I was called
first time panic. Second time not again... Third time. ok let's wait 20 seconds, it will unstuck itself
and have to be triple redundant in their operation because huffing up the stairs in case of an emergency isn't really an option
what are the emergencies? helium/nitrogen leak from the magnets?
actually CERN is the only place in the world, where you must use the elevator in case of fire. And you must not use the stairs. 160 meters of staircase is good 50 storey skyscraper
can you run it up in one go?
isnt there a dedicated fire crew for the complex?
What kind of radiation protection do you use?
I'd guess "couple dozen meters of solid rock shielding"...?
@Thomas Shaddack yes, sometimes there is a helium release from the cryogenic system. It is usually collected via the recovery line, valves unfreeze and we carry on. But sometimes there is bigger loss of helium into atmosphere
there's a fob everyone I saw wearing around their neck that measures exposure
@caladan yes, everything is 100 meters underground. You need to shield the machine from the cosmic rays and the human activity too
And the protection for people who have to access the machine?
https://photos.app.goo.gl/92vYVaMHXYeyVYS46
yes, dosimetry is everywhere. But the total absorbed dose for all people working there in a year is a small franction of an annual limit for one person. It is very well organized
that is an image of the observation deck in the CMS cavern, the beam tube is within the concrete blocks below
there are multiple layers of protection. Biometry, mechanical barriers etc. Even if a door would be forced, the LHC will stop and by the time the offender would have a chance to get to the machine, energy from the magnets will be extracted
it is not an easy task if the stored energy in one arc is about a GJ. Hude resistors
Yeah, for those who saw "angels and demons" movie - there are indeed retina/iris scanners at the LHC/SPS access doors. Everything else in this movie is BS, but the scanners are true.
the door interlocks are something... there's an rf fob, iris scanners, etc.
and switches of size of a truck. When you stand next to that switch, and it operates when you don't expect it, you may need to change you brown emergency underwear
do you have a picture of one of the switches?
+
let me find one
those are only the switch elements, it is built into an enclosure of size of a shipping container
and few of the resistors. You dissipate all magnet energy there in 1 minute, then you need to cool it down for more than an hours. If there is a stormy weather in the summer, and there are multiple dumps, it takes time to restart
Ah, I've seen those during open days in UA43
Me too! :-)
the red phenolic boxes are the actuators?
Amazing stuff...
yes, that is typical location, In the power converter gallery parallel to the tunnel in all even points
@dallas those are arc extinguishing chambers. You are opening a circuit with 300 superconducting magnets at 12kA current
not a particularly easy task to do :-)
is that chamber within an additional vacuum chamber or immersed in dielectric oil or some such thing?
for some applications, they managed to replace them by IGBT switches. But mechanical is better, very low resistance, so multiplied by 12 kiloamps squared, it is not producing a lot of heat
no, that is all in air
I have seen it to open, it is impressive
quite a pop when that thing does it's job...
yup... when you think about all the energy you must dispose very quickly in a very controlled way, not an easy job. And you do not want any false positives, it must work only when there is a problem
when energized? or actuation not while electrified?
my experience was a test stand, not energized. The real arc is difficult to see when energized, there is no access
everything is designed to be inherently safe.
yeah. one sector of the LHC has a few henries (H) of inductance. And a 14kA nominal current. I leave the calculation of the energy stored in the magnets as the homework :D
the power converters for dipoles deliver 190V/13kA. But resistive voltage on those coils is exactly zero
fascinating :-)
Agreed.
seriously amazing.
the quench protection system must detect millivolt of resistive voltage in a presence of volts of L*di/dt, and in a very radioactive environment. Collegues from the magnet protection gorup have no simple job
it is mostly digital now
How very ironic that I am this late to the hack chat - I was trying out the CERN Choir!
So, I've got to drop off now, but feel free to keep chatting. Daniel, thanks for the time today, this was great. And thanks to Thomas too -- great insights into the workings of CERN, it was a lot of fun. And thanks to everyone for the great questions. Transcript coming up soon.
https://home.cern/news/news/engineering/engineers-refine-protection-system-lhc-magnets
Engineers refine protection system for LHC magnets
This week, the Large Hadron Collider (LHC) was switched off for its second scheduled technical stop since starting to run at the new high energy of 6.5 teraelectronvolts (TeV) per beam. These regular stops allow engineers and technicians to maintain the machine and ensure that all its components are working well.
I'm a Technical Student, just started this month, so I would've loved to soak up some details here ^^
How is it with these kinds of chats, is the transcript available later?
Yes, I'll be posting a transcript in a few minutes
@dallas guess how many square millimeters of super conductring wire carries those 20kA of current in your photo
hello xasin, welcome to cern :-)
oooh, my host told me that and im now failing the pop quiz. 5?
yes, it is maybe 3-5 mm^2. All the rest is just a resistive filler, like copper, or aluminium. Superconductors are impressive
the situation however dramatically changes, when the superconductor loses its properties and becomes resistive. Then 5kA/mm^2 is not your friend :-)
I remember it being miniscule, especially next to the standard coils used for some of the dipoles.
I heard a story...
Oh, I actually know a little about that! I started in the Quench Detection / Protection systems.
IIRC, when a quench is detected, one of the things you can do is to intentionally make the whole rest of the superconductor ALSO resistive, so that the energy is more evenly dissipated. Was that about right?
Our colleagues are working on high temperature superconducting links to bring current from the converters at the surface down to the tunnel. In a cross section corresponding to your palm they transfer 150 kA in maybe 10 different circuits
@xasin say hello to Jens and company
by high temp, you still mean quite chilly though correct? not high temp as in 30c
@Xasin yes, the coil is warmed up, so the energy is dissipated into a large volume, while rest of the current bypasses the quenched coil by a humungous diode
@dallas just above the critical temperature. 5 Kelvins can be sufficient
5K is hot when you operate at 1.8 :-) everything is relative
ah yes, balmy. Haha, I will do that, he's my supervisor
If in the future the uQDS system works better/worse, it's my fault. I'm learning how to change the VHDL code to add some things to it.
working at so low temperatures can be very dangerous. Everything is perfectly thermally insulated. So you can melt a coaxial cable sending 20 Watts of Rf power through it
@xasin LHC had made its first big bang already. Your FPGA will not be the first. Sorry :-)
I would not want to be the one to cause another event like that, don't worry :P
the dielectric changes properties with temperature. can it be used to diagnose the cable state while running?
yes Thomas. We actually do this quite a lot.
you need to use any observables available. Time domain reflectometry, fibers in the cold bores to measure temperature etc.
there may also be quite some overlap with the field of software-defined radio.
everything is software defined radio these days. You try to digitize as soon as you can and then do things in digital. Regardless if RF, or DC metrology. The software does not drift
I think someone in my dept. is working on a very interesting AC analysis of the superconducting magnets too. Wonder how he's doing it...
Definitely digital though, the frequencies are very low there.
In many machines, we already directly sample RF up to hundreds of MHz and directly generate RF too. A nano-second long pulses from pickups are digitized at 12 Gsps and all the rest os done digitally
12 Gsps. WHOA. at how many bits?
yes
What's the data rate of some of the measurements at CERN? I think it goes into the TB/s...
I heard something like 2Pb for a total event data set in ATLAS
https://hackaday.com/2021/02/26/homebrew-metrology-the-cern-way/
I have had a great privilege to work in the electrical metrology section for the past 14 months. That is where the famous HPM7177 was bornI have used the "software defined radio" to measure 20kA with a sub-ppm accuracy :-)
Yeah, but that's probably just the filtered data, events that were deemed interesting enough...
Tangentially related, the public CERN Grafana panels are really fun to look at :3
@Thomas Shaddack 12 Gsps, you get 8-10 bits ENOB
that's... VERY impressive.
Hack Chat Transcript, Part 2 03/01/2023 at 22:10 • 0 comments
I think it's C++ on CentOS 7
actually, guess what it was Here are the libraries for data analysis:
what was the mean time between failures when LHC started in 2008? The first days?
I love C++ and Python.
any guesses?
What about something where even C won't do? VHDL, Verilog, what are you using for FPGAs?
Two weeks?
1 second?
Three months?
now, 15 years later, when everything is tuned and works well, we can get even 2 weeks
Also, how are you coping with the molasses-like nature of light? That thing is s-l-o-w. How do you compensate for that in the signal speed?
but what it was then?
4 hours?
about 5 minutes. Impressive for such a complex system. You can have MTBF of millions of hours. Plug it in million times and here we go
Woah
5 minutes. By the first week of running, it was about 1 hour. People working days and nights. That was the best times I have ever had. Hacking
hacking such a machine. I still get goose bumps thinking about it even now
very high caliber hacking
Understandably.
now the big lady is running reliably and quitely. We work on many smaller machines and projects
oh, you call it big lady?
Isn't there a Really Big Lady in the works now?
sometimes so reliably, that I start to miss the nigh calls. LHC did not call for 3 months. What is going on :-)
How does the LHC behave when there's an earthquake in Europe?
lol.
it's not afraid of earthquakes. but it shivers in fear when a leap second arrives :-)
we are restarting the complex as we speak. The Linacs are already running, it will take about 1 week for each subsequent machine. LHC is foreseen to have first beam the week before the easter
@simon.dancose yes, LHC can see all Earth movements
How do you compensate for them?
The Moon, water level in the Geneva lake, and all the Earth quakes.
Or are they just seen in the data?
The moon and water is slow. Out of 9000 magnets in LHC, maybe 7000 are correctors. The moon phases are even programmed in the control system as a real time feed forward correction
aperture where beam circulates in LHC at high energy is about 1x1mm. Any movement is visible
cool, I figured so. Moon cycle needed pre-planned compensating.
first the beam is scraped at collimators, for for a bigh earthquake, like the one from Turkey which was visible even by my pendulum clock, that would be an instant beam dump
In case you missed it:
https://hackaday.com/2023/01/28/an-atomic-pendulum-clock-accurate-enough-for-cern/
An Atomic Pendulum Clock Accurate Enough For CERN
That big grandfather clock in the library might be an impressive piece of mechanical ingenuity, and an even better example of fine cabinetry, but we'd expect that the accuracy of a pendulum timepiece would be limited to a sizable fraction of a minute per day.
and indeed, there are sensors and accelerometers all over the place. For example the huge earthquake in Indonesia at the beginning of 2000's was already recorded by the Atlas sensors
https://cds.cern.ch/record/824438?ln=en
THE ASIAN EARTHQUAKES DETECTED IN THE ATLAS CAVERN
At the end of December, mysterious vibrations were picked up by the deformation sensors under the feet that are to support the ATLAS detector. It transpired that they had detected waves produced by the earthquakes responsible for the terrible tsunami in Asia.
But other accelerators in the world can see these too. For example in DESY in Hamburg, they can see when people go to work during the day
when you do an emergency beam dump, do the proton packets just get released into a slab of lead or some other absorbent material? how long till that target is able to be handled?
Previous accelerator in the LHC tunnel (the LEP) showed beam instabilities reflecting the schedule of the Geneva-Lyon TGV line :-)
oh no. The LHC beam can make it through 50 meters of concrete... There is a 900 ton graphite/steel block to absorb it
https://lhc-machine-outreach.web.cern.ch/components/beam-dump.htm
LHC beam dumps
The challenge The nominal LHC beam contains an unprecedented stored energy of 350 MJ, contained in 2808 bunches with a beam sigma of the order of 0.3 mm. The extremely high destructive power of such a beam imposes an external dump, where the beam must be extracted completely from the LHC, diluted to reduce the peak energy density and then absorbed in a dedicated system.
What would happen to the concrete? Molten-through capillary-sized hole?
the beam carries so much energy, that it needs to be spread while dumping. Otherwise it will destroy the target. Some people say there will be plenty of very radioactive diamonds in the beam dump after the life time of the machine :-)
It's like a runaway truck ramp for protons
"The extracted beam is swept in a quasi-circular figure by two sets of orthoganally deflecting dilution kickers"
The beam has a kinetic energy of (now) more than 400 MJ. That is a runaway train. Not a truck. And needs to be disposed in 89 microseconds
Imagine what a train would do when crashing into a barrier. But this is 10^14 protons
same macroscopic effect
Do you have a favorite experiment? Or a finding that you're particularly proud of?
is there any video available of a dump event on the absorber cylinder?
or a swarm of 10^14 mosquitos, each flying at a pretty badass mosquito speed :-)
Daniel, thank you so much for your insight!
@Rosy Schechter I have spent a lot of time with LHC. The stabilizing system is my baby and I have feelings about it. But lately, I started to like very much the other end. My collegues from the operations group say it is a machine at the end of the food chain. The smallest one. But there are many scientists coming from many institutes. And they have say 1 week of time to install their experiment and collect all data
the atmosphere is very different from LHC. Those are big fish. A lot of people involved, a lot of competition. The Isolde is exactly the opposite.
right on. looking it up. thank you!
You help the users to achieve their goals, help them to do their experiments. They are happy, thankful, often bring a good bottle to the control room at the end of their run
As will I.
that's got to be really satisfying!
Agreed.
@Rosy Schechter trust me, it is. A very different feeling. Both are good. But very different. While in LHC I am expected to solve any problem in 1 hour regardless, this is an expert service. You do your best, because you want to help them
and it is very nice to collaborate with different people every time. I like both. With LHC we are like a family. The operators rely on experts and the experts rely on operators.
Is our time up? Thank you, Daniel; this was a most intriguing incite into LHC operation.
don't end it. We have just started :-)
time flies fast. Indeed.
Thank you, Daniel!
Holy cow, I just looked at the clock and saw that it's after 1:00. I'm not going to enforce Swiss precision on this end of the chat, but if Daniel has somewhere else to be, we'll give him the chance to log out. Otherwise, we can keep chatting!
Time flies. There is a rotting clock somewhere here.
Thank you, Daniel. thank you so much
@Dan Maloney I just need to move 5 meter to bed. So no rush
Unless the Big Lady calls ;-)
it is a nice concept here. I recall the times of IRC
https://op-webtools.web.cern.ch/vistar/vistars.php
dromantThe Big Lady is off for now.
Yeah, that's why we keep doing it this way. A little retro vibe is nice now and then
all is cold and tests are running already. It takes weeks to restart
6/8 sectors are at 1.8K, 1-2 and 3-4 is around 4K. Will need few more days
so you work on ISOLDE now, no longer on call for main beam support, is that correct?
https://op-webtools.web.cern.ch/Vistar/vistars.php?usr=LHC2
Vistars
as an expert for systems, you are always on call. There is also a stand-by service, where people must be reachable and come on site within 45 minutes. But those are expected to solve only a limited subset of problems
@daniel valuch and everyone who participated!
Thank youare you on the french or swiss side?
there is always an army of experts, people who designed and built the particular systems, who can be contacted in case the stand-by is not able to resolve the problem
I have the office on the French side, I also live in France. But we commute to the Swiss side multiple times a day. There are labs, cafeteria, other services
btw, the orange pipes at the ceiling are the RF feed coax lines for the SPS. Carrying 1 MW of RF at 200 MHz, each.
coax or waveguides?
these are coax
yes, that is output of the 6x 1MW amplifiers. Two of those are solid state. A transistor amplifier which needs its own building
coax, as the frequency is low. In LHC we run at 400MHz, but there we use waveguide alredy. 58x24cm. Heavies frequencies in the industry
I need to have a crane operating license. I got it one day before becoming a professor. Just a different kind of exam :-)
Why do you need a crane license?
how does the crane recognize if you have a licence? Our forklift didn't ask.
High power RF is a heavy industry. Look at the photos... 1 meter of waveguide weights 50 kilos
@Thomas Shaddack the crane does not recognize, but the safety people do :-)
plus you don't want to smash your multi million worth equipment
I was at a TED talk at CERN a few years ago, got to look at the surroundings, but notthing below ground so to speak.
getting underground is almost impossible for visitors. During shutdowns, you can visit the experiments underground. It is very impressive. But to get to the machine is impossible
@Thomas Shaddack to answer your HDL question: yes, most hard real-time controls are done on FPGAs (hard RT = determinism from 1ms to anywere near a fraction of a nanosecond). CERN is mostly VHDL, but there are some misfits that use SystemVerilog or even things like Migen/Misoc/LiteX.
I recommend to visit, also the surface installations are nice to seeDEFINI
the next long shutdown will be in about 3-4 years time, usually an open days is organized. Then people can visit also the underground facilities. Last time about 120000 people came
my favourite part getting people underground was asking the people in the elevator. Who wants to press a button at CERN? One or two rose their hand. So I proposed to press the -1 button in the elevator. They said they will not wash their finger :-)
Hack Chat Transcript, Part 1
03/01/2023 at 22:09 • 0 commentscheck, 1, 2, 3...
@daniel valuch Welcome aboard
Hello Daniel! Glad you're here, we'll get started in about 20 minutes but feel free to chat while we wait
Hi Dan, thank you for the invitation. I came home earlier, cleaned my desk, prepared the computer, ready for the show. Up to you if we go the Swiss way, or we start already :-)
I like to kick things off right at noon -- maybe I'm Swiss at heart?
how many people usually participate in the hackchats?
It looks like I am in the right place.
It varies, there are about 20 people total logged on right now, I expect that will probably double by the time we're really into it. Could be more though
hehe, yes, that is the Swiss way. I remember being at a conference in Luzern. Everything was prepared, people waiting for registration in an ordered cue. On the other side of the desk were couple of ladies from the institute, everything prepared, ready to go. We were waiting for good 10 minutes, looking at each other, until it was 9:00:00.000 They started to a millisecond
You never know what could happen in that last millisecond, I suppose
well, you can get into a trouble here in nanoseconds...
Hey Daniel :-)
The Fat lady could sing ... or not
hi Tom, I'm glad to have a support here :-)
Looking like an awesome crowd forming :-)
I have no webcam looking at my pendulum clock in the office. Ho how do we know it is time to start?
looks like I just got here in time :)
Waiting on Dan's cue
OK folks, let's get started. I'm Dan, I'll be modding along with Dusan today as we welcome Daniel Valuch to the Hack Chat! Daniel works at CERN -- enough said!
Welcome Daniel, thanks for dropping by. Can you tell us a little about your background?
NTP should be good enough for our timing needs :-)
It is time.
GPS PPS with Chrony FTW.
Daniel: By looking at your e-mail that notifies you when that time is that it starts.
Stratum-1 NTP on LAN.
Sorry, I'm a slow typist ;-(
Hi Dan, welcome everyone!
thank you again for the invitation
I have prepared a bit about the background. I am an electronics engineer, doing the electronics as a passion and as a profession for pretty much 35 years now
I grew up in Slovakia, where I have studied electronics and electrical engineering since the childhood. Actually, I have started with chemistry, thanks to the family background. It was very interesting, but the destiny finally brought me to electronics.
The secondary school was a clear choice. I have contiued my master's studies at Slovak University of Technology in Bratislava.
Since the secondary school, I was working as a sound engineer, first in a local radio, then the national radio. Not a coincidence, my friend's father, who initiated me into electronics was head of R&D in the state owned company, making studio equipment for the whole Eastern block. I started to teach too.
I slowly started to move from audio to high frequencies.
And in the last year of studies, I have applied for a studentship at CERN
https://careers.smartrecruiters.com/CERN/students
At the time, CERN still invited even students for an in person interview. A young guy, who never left the country before, hired a car and was driving to Switzerland for 14 hours. The next day we have started the interview with a coffee in the canteen. My then future supervisor explained me, that all great ideas start right at this place. We were goingh through whole CERN (it is HUGE) for the whole day. We were informally chatting. I have already had some limited RF knowledge from the university, and being a radio-amateur earlier. Chatting with Hans, I felt so uneducated.
The inteview lasted good 9 hours. I have learned more in those few hours than in my whole studies. And that one interview for a master's student position changed my entire life... Thank you Hans.
Life turns on a dime sometimes
It can.
I was lucky, I joined at the time the work on LHC started. I got the PhD. position in the RF group, I was working on high power vector modulators. But already then developing stuff for the mighty LHC. Then Y was again very lucky, many people were retiring, so posts were open. Ang I got the staff position in the low-level RF section. Developing more stuff for LHC
Can you tell us more about how your RF knowledge was applied at CERN? What was the relative blend of electrical engineering physics?
when watching the films and documentaries about the Apollo program, this was it. A young guy in between the giants
*and pjyiscs
Hi Dag!
RF is actually a very important part of any accelerator. You use RF to accelerate and stabilize the beams. You need everything from theory, signals and systems, controls, fine electronics, but also a Megawatt amplifiers to do RF for accelerator. There is no physics without RF and also no RF without physics
everything is unique, specific, working at the limit of what is possible. For example HIE-Isolde. It is the smallest accelerator we have, only ±16 meters long. There are 20 super-conducting RF resonators, creating an accelerating voltage of 20 megavolts
we need to run them with a 1 Hz bandwidth at 100MHz.
Megawatt-class RF will be more and more important industrially too. Electricity can take over many fields where gas burners ruled, in the form of microwaves, from glass melting (see the work of ing. Hajek) to cement production.
Super Proton Synchrotron is a different challenge. We need 6 Megawatts of RF power (continuos wave) to make the beam stable
How do you stabilize the beam? For those who are a bit rusty in their accelerator-fu?
well, a piece of cake. Imagine you have an object, which flies at pretty much the speed of light and it has a kinetic energy of a high speed train. But you can not see it, because it is only 10^14 protons. What can you do?
Thank you Daniel... and hi Dan!
The machine has to be perfectly designed, everything needs to be perfectly made, everything accurate. Then you can start to measure and based on models you can try to control
the speed should control it?
but feedbacks are not very straight forward. It travels at the speed of light. You can not react fast enough. The beam is gone by the time you know what to do
the speed should control the beam in part.
?
In a circular machine, you can wait until it makes a turn and apply the corrections few microseconds later. But properly transformed, because the beam had travelled 27 kilometers in the mean time.
For longitudinal control, we use very strong electromagnetic fields in the accelerating cavities. We need megavolts
Nice.
That's a rather energetic piece of cake.
In the transverse plane, that is cross-section of the vacuum chamber, we use magnets (they are very slow), or electric field deflectrors. Like in the old scope CRTs
yup. CERN draws about 250 MW of power...
How do you pump so much voltage into a capacitor, which the deflector de facto is, fast enough and precise enough?
Since I came to CERN, I have had a dream. I wanted to see the electricity meter.
I don't envy that utility bill.
A prime use case for a modular reactor.
Lol.
Would I be right in comparing this (loosely) to an MRI machine in that huge energies are used as a stimulus while hyper-senitive detectors listen for a tiny signal in response?
many people say that. With reactor. I wonder why :-)
yes, MRI excites by few hundred Watts of RF and measures picowatts of response. In a very short transition time between
Great -- thanks!
You control
This is a problem in our machines as well. You need super strong kickers, or RF systems, but at the same time, you need to measure signals at almost a single charge level. Electromagnetic compatiblity at CERN is a challenge
the LHC extraction kicker needs to ramp from zero to (I think) 40 kA in 100 ns
You control the speed of the beam via magnetization,, in a sense, correct?
how much are you going to perturb your neighbours? :-)
EMP the 'hood!
What are some examples of projects you've worked on? How big are the teams?
@April, kind of. We do not talk about the speed much, as everything travels almost at the speed of light. We rather use term kinetic energy. And yes, in machines, we call synchrotrons, you have a fixed vacuum chamber. So you need to synchronize strenght of the bending magnetic field and the energy
Nice.
i think I understand.
Similar to how cars work but more intricate/detailed/complex?
to produce the kinetic energy and control and direct the beam?
at the beginning, I was working on electronic systems which control the beam. Those are VME format cards, with multiple high speed ADCs, DACs, large FPGA. Later I moved to systems, like the whole accelerating system for the HIE-Isolde. We have started literally on a green field with specs. The building was the first to look after, then start working on the RF. Or the transverse feedback system in LHC. You start with a sensor in the tunnel (many actually), then a lot of RF signal processing *racks), then ADCs, FPGAs, amplifiers, tube amplifiers, large kickers in the tunnel
Hello Daniel, thanks so much for offering your time for this chat. Can you tell us about a typical day for you at CERN?
@Greg it depends, what is your specialization. I work in an equipment group. Our task is to develop, build and run technology for accelerators, and then to operate the accelerators. So depends on time of the year, the day can look very different
in the winter, we have a year end technical stop, the only time you can do something with your hardware in the machine. Otherwise it must run 24/7. So days are long upgrading, installing, testing, but at least predictable
when machines are running, you do your daily work, e.g. working on new projects, upgrades, new methods etc. but you have responsiblity for the machine. If machine calls, that is the highest level interrupt and you have to deal with the problems
Great answers Daniel...thank you.
the simplest problems were fixed when the machines were first started. It is always a challenge to figure out what went wrong and how to fix it. LHC's operating cost is 250 000 Euros/hour. If you have 1 hour downtime, you are not very populr
Daniel, I undestand you just concluded a maintenance window, did it go well?
What's "commuting" through the LHC ring like? Do you go on foot or by bike? Do you bring tools with you or are there repair depots along the way?
the part I like very much is being responsible for the system I have developed, built and now operating. When LHC calls me in the middle of the night, I kind of like it, it is exciting. Weird :-)
What are typical incidents that call for your unplanned attention?
Holy moly, LHC talk.
it is always different. If you have the same problem 3 times, your hardware was not properly designed
Daniel, that’s not weird.
The LHC must be incredibly complex. How would you even begin chasing down errors? Are you more likely to face hardware or software issues?
(to like going at night to fix those issues isn’t weird)
@dvaluch, I just returned from a couple weeks at CERN, curious how the process of beam initiation and acceleration is coordinated. Is there a central master clock that all the various experiments all synchronize to? How is that managed? I understand that the beam tunnel is off limits (ostensibly due to radiation) when the main ring is energized, is all of the hardware along the ring radiation hardened?
Hioften it is software, which controls everything which did not do something as expected. But we have had a case when a DAC was hanging after a thunderstorm hit next to one of the LHC points. Everything was working, and it took me 3 hours to figure out actually the silicon was upset by a lightning strike a kilometer away
Do you think it would be possible to run the LHC without software? Pure HW only?
what is the programming language used?
@Cooper yes, they say it is the most complex machine mankind had ever built. I have had the privilege to be there, and I have a tendency to agree. You need to know all "bolts and nuts", use all the clues, have a ton of diagnostics, and think fast
@caladan no way. The LHC is run by both hardware and software. Hardware does all the heavy work, and it needs to be controlled by the finest software. Everything must be sequenced and in order in microseconds
@April Morone most likely, Excel :-)
Python? Any version of C?
there are hundreds of thousand of systems. The fast beam abort is a giant, about 30000 input and gate, which can get rid of the beam in 200 us. And then there are millions of conditions, which are looked after by software
it is a beast :-)
it boggles the mind that with so many systems running and everything so interdependent that it ever is able to run at all ;)
if I am not mistaken the software runs on C and linux. Then user inetrfaces are Python, Java etc.
Nice!
@dallas when we have started the LHC on September 8th 2008, the mean time between failures was
About Us Contact Hackaday.io Give Feedback Terms of Use Privacy Policy Hackaday API