Close

Hack Chat Transcript, Part 1

A event log for Life at CERN Hack Chat

Big science, big rewards

dan-maloneyDan Maloney 03/01/2023 at 22:090 Comments

daniel valuch11:13 AM
check, 1, 2, 3...

Dr. Cockroach11:34 AM
@daniel valuch Welcome aboard

Dan Maloney11:42 AM
Hello Daniel! Glad you're here, we'll get started in about 20 minutes but feel free to chat while we wait

daniel valuch11:43 AM
Hi Dan, thank you for the invitation. I came home earlier, cleaned my desk, prepared the computer, ready for the show. Up to you if we go the Swiss way, or we start already :-)

Dan Maloney11:44 AM
I like to kick things off right at noon -- maybe I'm Swiss at heart?

daniel valuch11:44 AM
how many people usually participate in the hackchats?

SimonAllen11:44 AM
It looks like I am in the right place.

Nicolas Tremblay joined  the room.11:45 AM

Emanuel joined  the room.11:45 AM

Dan Maloney11:45 AM
It varies, there are about 20 people total logged on right now, I expect that will probably double by the time we're really into it. Could be more though

daniel valuch11:46 AM
hehe, yes, that is the Swiss way. I remember being at a conference in Luzern. Everything was prepared, people waiting for registration in an ordered cue. On the other side of the desk were couple of ladies from the institute, everything prepared, ready to go. We were waiting for good 10 minutes, looking at each other, until it was 9:00:00.000 They started to a millisecond

Dan Maloney11:50 AM
You never know what could happen in that last millisecond, I suppose

twlostow joined  the room.11:50 AM

daniel valuch11:50 AM
well, you can get into a trouble here in nanoseconds...

twlostow11:50 AM
Hey Daniel :-)

Nicolas Tremblay11:50 AM
The Fat lady could sing ... or not

daniel valuch11:51 AM
hi Tom, I'm glad to have a support here :-)

Dr. Cockroach11:53 AM
Looking like an awesome crowd forming :-)

caladan joined  the room.11:57 AM

daniel valuch11:59 AM
I have no webcam looking at my pendulum clock in the office. Ho how do we know it is time to start?

Synthron11:59 AM
looks like I just got here in time :)

Nicolas Tremblay12:00 PM
Waiting on Dan's cue

Dan Maloney12:00 PM
OK folks, let's get started. I'm Dan, I'll be modding along with Dusan today as we welcome Daniel Valuch to the Hack Chat! Daniel works at CERN -- enough said!

Welcome Daniel, thanks for dropping by. Can you tell us a little about your background?

twlostow12:00 PM
NTP should be good enough for our timing needs :-)

SimonAllen12:00 PM
It is time.

Thomas Shaddack12:00 PM
GPS PPS with Chrony FTW.

April Morone12:00 PM
Daniel: By looking at your e-mail that notifies you when that time is that it starts.

Thomas Shaddack12:00 PM
Stratum-1 NTP on LAN.

Dan Maloney12:00 PM
Sorry, I'm a slow typist ;-(

Dusan Petrovic12:00 PM
Hi Dan, welcome everyone!

daniel valuch12:01 PM
thank you again for the invitation

daniel valuch12:02 PM
I have prepared a bit about the background. I am an electronics engineer, doing the electronics as a passion and as a profession for pretty much 35 years now

daniel valuch12:02 PM
I grew up in Slovakia, where I have studied electronics and electrical engineering since the childhood. Actually, I have started with chemistry, thanks to the family background. It was very interesting, but the destiny finally brought me to electronics.

The secondary school was a clear choice. I have contiued my master's studies at Slovak University of Technology in Bratislava.

Since the secondary school, I was working as a sound engineer, first in a local radio, then the national radio. Not a coincidence, my friend's father, who initiated me into electronics was head of R&D in the state owned company, making studio equipment for the whole Eastern block. I started to teach too.

I slowly started to move from audio to high frequencies.

daniel valuch12:02 PM
And in the last year of studies, I have applied for a studentship at CERN

https://careers.smartrecruiters.com/CERN/students

daniel valuch12:03 PM
At the time, CERN still invited even students for an in person interview. A young guy, who never left the country before, hired a car and was driving to Switzerland for 14 hours. The next day we have started the interview with a coffee in the canteen. My then future supervisor explained me, that all great ideas start right at this place. We were goingh through whole CERN (it is HUGE) for the whole day. We were informally chatting. I have already had some limited RF knowledge from the university, and being a radio-amateur earlier. Chatting with Hans, I felt so uneducated.

The inteview lasted good 9 hours. I have learned more in those few hours than in my whole studies. And that one interview for a master's student position changed my entire life... Thank you Hans.

jfranksmith joined  the room.12:04 PM

Dan Maloney12:04 PM
Life turns on a dime sometimes

April Morone12:05 PM
It can.

daniel valuch12:05 PM
I was lucky, I joined at the time the work on LHC started. I got the PhD. position in the RF group, I was working on high power vector modulators. But already then developing stuff for the mighty LHC. Then Y was again very lucky, many people were retiring, so posts were open. Ang I got the staff position in the low-level RF section. Developing more stuff for LHC

Dag Spicer12:05 PM
Can you tell us more about how your RF knowledge was applied at CERN? What was the relative blend of electrical engineering physics?

daniel valuch12:05 PM
when watching the films and documentaries about the Apollo program, this was it. A young guy in between the giants

Dag Spicer12:06 PM
*and pjyiscs

Dan Maloney12:06 PM
Hi Dag!

daniel valuch12:07 PM
RF is actually a very important part of any accelerator. You use RF to accelerate and stabilize the beams. You need everything from theory, signals and systems, controls, fine electronics, but also a Megawatt amplifiers to do RF for accelerator. There is no physics without RF and also no RF without physics

daniel valuch12:08 PM
everything is unique, specific, working at the limit of what is possible. For example HIE-Isolde. It is the smallest accelerator we have, only ±16 meters long. There are 20 super-conducting RF resonators, creating an accelerating voltage of 20 megavolts

daniel valuch12:08 PM
we need to run them with a 1 Hz bandwidth at 100MHz.

Thomas Shaddack12:09 PM
Megawatt-class RF will be more and more important industrially too. Electricity can take over many fields where gas burners ruled, in the form of microwaves, from glass melting (see the work of ing. Hajek) to cement production.

daniel valuch12:09 PM
Super Proton Synchrotron is a different challenge. We need 6 Megawatts of RF power (continuos wave) to make the beam stable

Thomas Shaddack12:10 PM
How do you stabilize the beam? For those who are a bit rusty in their accelerator-fu?

daniel valuch12:11 PM
well, a piece of cake. Imagine you have an object, which flies at pretty much the speed of light and it has a kinetic energy of a high speed train. But you can not see it, because it is only 10^14 protons. What can you do?

Dag Spicer12:12 PM
Thank you Daniel... and hi Dan!

daniel valuch12:12 PM
The machine has to be perfectly designed, everything needs to be perfectly made, everything accurate. Then you can start to measure and based on models you can try to control

April Morone12:13 PM
the speed should control it?

daniel valuch12:13 PM
but feedbacks are not very straight forward. It travels at the speed of light. You can not react fast enough. The beam is gone by the time you know what to do

April Morone12:13 PM
the speed should control the beam in part.

April Morone12:13 PM
?

daniel valuch12:14 PM
In a circular machine, you can wait until it makes a turn and apply the corrections few microseconds later. But properly transformed, because the beam had travelled 27 kilometers in the mean time.

For longitudinal control, we use very strong electromagnetic fields in the accelerating cavities. We need megavolts

April Morone12:15 PM
Nice.

Thomas Shaddack12:15 PM
That's a rather energetic piece of cake.

daniel valuch12:15 PM
In the transverse plane, that is cross-section of the vacuum chamber, we use magnets (they are very slow), or electric field deflectrors. Like in the old scope CRTs

daniel valuch12:15 PM
yup. CERN draws about 250 MW of power...

Thomas Shaddack12:16 PM
How do you pump so much voltage into a capacitor, which the deflector de facto is, fast enough and precise enough?

daniel valuch12:16 PM
Since I came to CERN, I have had a dream. I wanted to see the electricity meter.

Christopher12:16 PM
I don't envy that utility bill.

Thomas Shaddack12:16 PM
A prime use case for a modular reactor.

April Morone12:17 PM
Lol.

Dag Spicer12:17 PM
Would I be right in comparing this (loosely) to an MRI machine in that huge energies are used as a stimulus while hyper-senitive detectors listen for a tiny signal in response?

daniel valuch12:17 PM
many people say that. With reactor. I wonder why :-)

daniel valuch12:18 PM
yes, MRI excites by few hundred Watts of RF and measures picowatts of response. In a very short transition time between

Dag Spicer12:18 PM
Great -- thanks!

April Morone12:19 PM
You control

daniel valuch12:19 PM
This is a problem in our machines as well. You need super strong kickers, or RF systems, but at the same time, you need to measure signals at almost a single charge level. Electromagnetic compatiblity at CERN is a challenge

daniel valuch12:19 PM
the LHC extraction kicker needs to ramp from zero to (I think) 40 kA in 100 ns

April Morone12:19 PM
You control the speed of the beam via magnetization,, in a sense, correct?

daniel valuch12:19 PM
how much are you going to perturb your neighbours? :-)

Thomas Shaddack12:20 PM
EMP the 'hood!

dallas joined  the room.12:21 PM

Christopher12:21 PM
What are some examples of projects you've worked on? How big are the teams?

daniel valuch12:21 PM
@April, kind of. We do not talk about the speed much, as everything travels almost at the speed of light. We rather use term kinetic energy. And yes, in machines, we call synchrotrons, you have a fixed vacuum chamber. So you need to synchronize strenght of the bending magnetic field and the energy

April Morone12:21 PM
Nice.

April Morone12:22 PM
i think I understand.

April Morone12:23 PM
Similar to how cars work but more intricate/detailed/complex?

April Morone12:24 PM
to produce the kinetic energy and control and direct the beam?

daniel valuch12:25 PM
at the beginning, I was working on electronic systems which control the beam. Those are VME format cards, with multiple high speed ADCs, DACs, large FPGA. Later I moved to systems, like the whole accelerating system for the HIE-Isolde. We have started literally on a green field with specs. The building was the first to look after, then start working on the RF. Or the transverse feedback system in LHC. You start with a sensor in the tunnel (many actually), then a lot of RF signal processing *racks), then ADCs, FPGAs, amplifiers, tube amplifiers, large kickers in the tunnel

Greg12:26 PM
Hello Daniel, thanks so much for offering your time for this chat. Can you tell us about a typical day for you at CERN?

daniel valuch12:28 PM
@Greg it depends, what is your specialization. I work in an equipment group. Our task is to develop, build and run technology for accelerators, and then to operate the accelerators. So depends on time of the year, the day can look very different

daniel valuch12:28 PM
in the winter, we have a year end technical stop, the only time you can do something with your hardware in the machine. Otherwise it must run 24/7. So days are long upgrading, installing, testing, but at least predictable

daniel valuch12:29 PM
when machines are running, you do your daily work, e.g. working on new projects, upgrades, new methods etc. but you have responsiblity for the machine. If machine calls, that is the highest level interrupt and you have to deal with the problems

Greg12:30 PM
Great answers Daniel...thank you.

daniel valuch12:30 PM
the simplest problems were fixed when the machines were first started. It is always a challenge to figure out what went wrong and how to fix it. LHC's operating cost is 250 000 Euros/hour. If you have 1 hour downtime, you are not very populr

John Hays12:31 PM
Daniel, I undestand you just concluded a maintenance window, did it go well?

Dan Maloney12:31 PM
What's "commuting" through the LHC ring like? Do you go on foot or by bike? Do you bring tools with you or are there repair depots along the way?

daniel valuch12:31 PM
the part I like very much is being responsible for the system I have developed, built and now operating. When LHC calls me in the middle of the night, I kind of like it, it is exciting. Weird :-)

Thomas Shaddack12:31 PM
What are typical incidents that call for your unplanned attention?

Cooper12:32 PM
Holy moly, LHC talk.

daniel valuch12:32 PM
it is always different. If you have the same problem 3 times, your hardware was not properly designed

April Morone12:32 PM
Daniel, that’s not weird.

Cooper12:33 PM
The LHC must be incredibly complex. How would you even begin chasing down errors? Are you more likely to face hardware or software issues?

April Morone12:33 PM
(to like going at night to fix those issues isn’t weird)

dallas12:33 PM
Hi @dvaluch, I just returned from a couple weeks at CERN, curious how the process of beam initiation and acceleration is coordinated. Is there a central master clock that all the various experiments all synchronize to? How is that managed? I understand that the beam tunnel is off limits (ostensibly due to radiation) when the main ring is energized, is all of the hardware along the ring radiation hardened?

daniel valuch12:34 PM
often it is software, which controls everything which did not do something as expected. But we have had a case when a DAC was hanging after a thunderstorm hit next to one of the LHC points. Everything was working, and it took me 3 hours to figure out actually the silicon was upset by a lightning strike a kilometer away

caladan12:35 PM
Do you think it would be possible to run the LHC without software? Pure HW only?

April Morone12:35 PM
what is the programming language used?

daniel valuch12:35 PM
@Cooper yes, they say it is the most complex machine mankind had ever built. I have had the privilege to be there, and I have a tendency to agree. You need to know all "bolts and nuts", use all the clues, have a ton of diagnostics, and think fast

daniel valuch12:36 PM
@caladan no way. The LHC is run by both hardware and software. Hardware does all the heavy work, and it needs to be controlled by the finest software. Everything must be sequenced and in order in microseconds

twlostow12:37 PM
@April Morone most likely, Excel :-)

April Morone12:37 PM
Python? Any version of C?

daniel valuch12:37 PM
there are hundreds of thousand of systems. The fast beam abort is a giant, about 30000 input and gate, which can get rid of the beam in 200 us. And then there are millions of conditions, which are looked after by software

daniel valuch12:38 PM
it is a beast :-)

dallas12:38 PM
it boggles the mind that with so many systems running and everything so interdependent that it ever is able to run at all ;)

daniel valuch12:38 PM
if I am not mistaken the software runs on C and linux. Then user inetrfaces are Python, Java etc.

April Morone12:39 PM
Nice!

daniel valuch12:39 PM
@dallas when we have started the LHC on September 8th 2008, the mean time between failures was

Discussions