Project Logs

Collapse

• The Vent Gets a Box

  Jake Wachlin • 06/21/2020 at 18:28 • 0 comments

  Revisiting Control Delays
  

  As discussed in the last log, feedback lag is a major concern. Most introductory control system courses focus on single input-single output (SISO) linear systems. For these simple systems, there is a mountain of theory that can be used to analyze a system and predict certain characteristics about performance, observability, controllability, stability, and more. For the Ventilator, we do indeed have a SISO system. We are controlling output commands to the blower motor controller, using pressure feedback as the input. The system is not linear, but may possibly be close enough. For linear SISO systems, Bode plot analysis of the controlled transfer function is typical. To learn more about this process, this page is enlightening. To summarize, the Bode plot can be analyzed to determine stability of a system. A major de-stabilizing factor within these systems is the pure time delay. This makes sense even without  detailed mathematical analysis in the frequency domain. Imagine holding a broom upright on your palm with your eyes closed. A friend will call out which way to move your hand so you stabilize it. The delay added from having your friend tell you what to do will destabilize the control system and you will drop the broom. 

  For the Low Cost Ventilator, I recently adjusted the air circuit so that the pressure sensors are connected to the Wye (just before the ETT). An extra length of hose carries the air from that section to the pressure sensors. That, combined with some apparently slow ramp up on blower speed adds a significant feedback delay. The image below shows one breath cycle at 20 breaths per minute. Even though the command to the motor ramps up relatively quickly, the measured pressure only reaches its peak about 1.25 seconds later. This controller is mainly feedforward, since the delay would destabilize a PID controller.

  Likewise, this delay is visible for a slower breath cycle which allows more time for the dynamics to stabilize.

  Although there are some control approaches that can be done to combat pure delays, such as Model Reference Adaptive Control or the use of a Smith Predictor, the best approach is to adapt the design to reduce this delay. Likely this would mean a stronger blower or a controller that ramps up speed faster, and moving the pressure sensor nearer to the Wye.

  ---

  Vent in a Box
  

  The Ventilator has reached the point that it can be put inside its enclosure. From the beginning, our design was meant to have a single flat front panel that all custom parts would mount to. This panel would replace the panel on a COTS Hammond Industries metal enclosure. This makes assembly easy, fast, and the result looks fairly professional. The below images show it mounted inside the enclosure. Unfortunately there are still a few parts still on order for the air circuit, so it isn't quite right, but it's close enough for testing. Note the smaller hose leading from the Wye back into the vent. This provides pressure feedback back to the sensors inside, likely contributing to the control delay somewhat.

  The system was powered up using its medical grade power supply (port hidden just behind the hoses in the picture below) and two (figuratively and literally) heavy textbooks were used to add some resistance to the "lung" during breathing. This is not a great dummy lung, but is cheap and available.

  Finally, I made a demo video of the ventilator, embedded below. Note a few things in particular:

  • On power up, the alarm buzzer and light turn on. They shut off when the system boots. The hardware that controls those are physically different from the microcontroller, and alarm by default. The microcontroller must periodically toggle a pin to that subsystem, or the alarm will sound. This is an important safety feature.
  • All settings are entered through a pushbutton and knob. The brightness of the screen washes out the settings, but they are shown in real-time on the display. The display refreshes at over 30Hz. The...

  Read more »

• Low Cost Ventilator: First Breaths

  Jake Wachlin • 05/30/2020 at 19:40 • 0 comments

  After waiting for some further parts, we have now begun to assemble and test a simplified version of the breathing circuit. The real breathing circuit will include a hookup to an oxygen supply, check valves to manage the inspiratory and expiratory paths, virus filters, and a humidifier/heater. For this testing, we are mainly focused on a few goals:

  • Testing basic motor control
    • Basic control of speed and interfacing with the motor
    • Implementation of pressure controller
  • Testing pressure sensor feedback effectiveness
  • Implementation and testing of flow sensor for tidal volume measurement

  ---

  INITIAL MOTOR CONTROL
  

  The main challenge to initially implementing motor control was a PCB design mistake related to the mounting of the brushless motor controller module. The design called for female header pins to be soldered onto our PCB and act as a socket for the module. Unfortunately, the drill size for all of the pads was too small, and the header pins would not fit through. We looked for alternative headers with smaller diameter pins, but were not successful. Ultimately, we ended up soldering the header pins "on top" of the pads, using the holes as alignment detents. With a few dozen pins, we figured it would be structurally sound enough for testing, and we could solder enough to be electrically OK. The picture below shows the controller module clamped in place in preparation for soldering it down.

  The design called for Maxon's 380200 OEM brushless motor controller module. During original research, we had purchased an alternative from Anaheim Automation. While mostly pin-compatible, it was somewhat different. In particular, we used Maxon's controller in closed loop speed control mode, but the Anaheim Automation part was open loop only. In addition, the maximum speed of the Anaheim Automation part was not clear. This resulted in some issues.

  During initial testing, things seemed OK. The pressure controller that we had designed was a PIDF controller. A trapezoidal pressure profile was generated based on the ventilator settings, and the controller would attempt to track measured pressure to that profile. The controller source can be seen below.

  /*
   *    \brief Performs PIDF control
   *
   *    Has integral error range and anti-windup
   *    Has derivative filtering
   *    Note: this is a tracking controller, so "derivative" can somewhat abruptly change, need to be careful
   *
   *    \param control Pointer to the control structure defining the pressure profile
   *    \param params Pointer to the structure holding controller tuning parameters
   */
   static float pidf_control(lcv_control_t * control, controller_param_t * params)
   {
      static float error_integral = 0.0;
      static float error_derivative = 0.0;
      static float last_error = 0.0;
  
      float error = control->pressure_set_point_cm_h20 - control->pressure_current_cm_h20;
  
      float alpha = 0.7;
      error_derivative = alpha*(error-last_error) + (1.0 - alpha)*error_derivative;
  
      if(abs(error) < params->integral_enable_error_range)
      {
          error_integral += error;
          if(abs(error_integral * params->ki) > params->integral_antiwindup)
          {
              error_integral    = (error_integral/abs(error_integral)) * (params->integral_antiwindup) / params->ki;
          }
      }
      else
      {
          error_integral = 0.0;
      }
  
      float output = params->kf * control->pressure_set_point_cm_h20 +
                      params->kp * error +
                      params->ki * error_integral +
                      params->kd * error_derivative;
  
      if(output > params->max_output)
      {
          output = params->max_output;
      }
  
      if(output < params->min_output)
      {
          output = params->min_output;
      }
  
      last_error = error;
      return output;
   }

  For testing of the code originally, we had set it up to have only non-zero feedforward gain. The motor controller modules are controlled with an analog signal from the MCU's DAC. In other words, the MCU was initially outputting a voltage linearly proportional to the desired control pressure. I could then look at that signal on the scope to ensure the trapezoidal pressure profile was being properly generated. For initial motor testing, the...

  Read more »

• V1 Build and Testing: Log #1

  Jake Wachlin • 05/09/2020 at 19:18 • 0 comments

  A few weeks back, we made a bunch of orders for parts, including our assembled PCB. This week, the hardware finally arrived and we have been hard at work assembling and testing! So far, the focus has been on testing of the PCB, getting the firmware up and running on it, and addition of firmware features which were not easy to test on the breadboard version since that version did not have all of the necessary components.

  For cost and timeline reasons, we had only the surface mount components assembled for us, and left the through hole components for us. The board  was built by Macrofab, who I have used for many projects and highly recommend. Their boards have always been high-quality, their price is great, and their UI is really easy to use. 

  When it arrived, the board looked like the following:

  There is a lot to note here. First, one can see a lot of safety features that were built into the design. All of the connectors to external parts are ESD protected. Many of the other features of the design are also visible but will be discussed further later. Since the idea of this design is to be low-cost, that doesn't simply cover the cost of components. We also designed it in such a way that the PCB can be built in basically any fab for a low cost. We use generous minimum isolation and width for all traces (none smaller than 10mil, with most much larger than that). The design has all SMD parts on a single side, and it is only a two-layer board.

  ---

  HARDWARE TESTING

  I started assembling the board, going slowly to check all connections for polarity and for any mistakes we might have made in the design. Thankfully, there was nothing impeding the system from being powered up.

   First, here is the board with the 5V regulator, spring pin connectors, programming pins, and motor NTC connector soldered in.

  Here, one of the pressure sensors, the fuse, and the main power connector are also installed. The E-Stop switch is wired in, since the system cannot power up without it. The wires on the top right were connected into 5V and GND so I could examine the 5V rail with my oscilloscope (it looks very clean). The red wire soldered onto a pin near the top left of the board was added so that I could power the system from my power supply. Normally, the design uses a COTS medical-grade power supply, but for testing I wanted to be able to limit and monitor current. Thankfully, that wire can be removed once I am done testing - it is not a bodge wire.

  Once I had confirmed that the single pressure sensor was functional and acting as expected, I mounted the other two. This design uses triple-redundant pressure sensors for reliability. Here, the LCD screen is also connected. Since the LCD interface was something I had previously prototyped with a development board, it worked perfectly the first try with no firmware changes needed.

  I hooked up a short tube to one of the pressure sensors to confirm that it was measuring pressure changes. It was. The nominal operational current is shown at 19mA at 24V, including the backlit LCD screen. The blower motor and its controller are not installed here.

  This image more clearly shows the preliminary display format for the ventilator. This is very much subject to change in the future, but is a useful start,

  In order to simplify manufacturing, the LCV is designed around a COTS enclosure from Hammond. One of the front panels will be replaced by one of our own design. Therefore, the only main custom hardware element needed is a single "2D" part, easily produced on a laser cutter or waterjet. Our prototype panel is laser cut from acrylic, but acrylic should not be used for anything other than a prototype. Acrylic is damaged by alcohol, so it would be difficult to clean this system. The image below shows the PCBA roughly installed in its correct spot. The screws we bought were too short so it isn't fully installed yet, but the placement seems good. The enable toggle switch and the...

  Read more »

• Making the Jump to RTOS

  Jake Wachlin • 04/13/2020 at 04:16 • 0 comments

  While the Arduino platform is great for quick-and-dirty prototypes, everyone who has tried knows it is a huge pain to use it for relatively complex prototypes. We know this, and had been considering for a while moving to a more professional HW and firmware design for the microcontroller on the vent. Feedback from some other engineers, testing of the interface with our LCD, and this article all contributed to a fairly large change in the direction of the HW of this project. Instead of using an Adafruit Feather M4 daughter board and the Arduino platform, we moved to an ATSAMD21G18A microcontroller placed on our main PCB. At first glance, it may appear to be an odd choice. We were worried about capabilities and went backwards? The SAMD21 has a Cortex M0+ and runs at a maximum 48MHz. It is effectively the little brother of the SAMD51 with Cortex M4F that the Feather M4 has. However, it turns out that the extra horsepower is really not needed. What is needed is more careful and efficient control of the MCU peripherals which comes from moving away from Arduino.

  If there was one picture that could explain the challenge with Arduino, it is the following. The LCD interface is I2C and the LCD does not support a frequency higher than 50kHz. The addressing of characters is not sequential on the LCD, so re-writing the entire screen requires 3 separate I2C transactions to set the 80 characters (4x20) on the LCD character display. Because the I2C speed is so limited, this causes a big challenge. To update the screen every 30ms, the I2C bus is heavily used. This image is of our updated hardware and firmware. The Arduino Wire library has a maximum buffer of 32 bytes, so it actually requires even more transactions!

  Some more minor optimization could be done, and was done. I modified the firmware to only write any bytes that had changed, and not the entire buffer every time. However, we must consider the worst case, and in that case roughly 2/3 of the time is spent writing to the LCD. Using a blocking I2C driver like Arduino's Wire library, this would be a major challenge, and likely very dangerous. We cannot allow updates to the display to interfere with pressure control of the ventilator. As the linked article above notes, the answer is to use a Real Time Operating System (RTOS), and intelligently use the microcontroller's peripherals to prevent blocking and free up the CPU for other use. If you have never used an RTOS, give it a try. We are using FreeRTOS, one of the more commonly used RTOSs out there. An RTOS gives the developer access to all sorts of useful features such as multitasking (with defined task priorities), software timers, data queues, semaphores, and others which make developing complex systems far easier, faster, and more reliable.

  In the case of the ventilator, we have screen buffers for various screens (currently a main and an alarm screen). A software timer switches between the two if any errors are detected. A separate software timer generates the needed I2C transactions to write the screen buffer to the LCD, and adds them to the queue. They are asynchronously pulled from the queue, and handled in the SERCOM I2C peripherals with extremely minimal CPU involvement. What used to be 2/3+ of the total loop time is now a negligible amount, and much easier to develop and understand.

  On top of this, FreeRTOS allows easy prioritization. We can ensure that the control loop has higher priority than the HMI task, for example. We don't care if the LCD updates a few milliseconds later, but we want the control profile as perfect as possible.

  At this point, we have most of the code structure up and running, and FreeRTOS running on Microchip's SAMD21 Xplained Pro development board. It updates the LCD at ~33Hz, handles settings inputs from our buttons and potentiometer, and runs an initial control loop depending on the parameters.

  The "control loops" can be separated similar to how GN&C are separated out for flight software/robotics applications....

  Read more »

• LCD and Flowmeter Testing

  Jake Wachlin • 03/29/2020 at 22:07 • 0 comments

  In support of our PCB design, and to test out some of the more uncertain concepts, we set up a breadboarded prototype of the LCD and the flowmeter. The main components in the prototype:

  • Adafruit Feather M4 Express (SAMD51-based microcontroller development board
  • SFM3300 flowmeter (+/-250 SLM full scale range, I2C interface)
  • NHD-0420D3Z-FL-GBW-V3 LCD display (4x20 backlit LCD character display)

  The prototype was set up as shown below. The flowmeter and LCD both require 5V power, but the flowmeter is compatible with 3.3V level I2C communication. The LCD requires logic level shifting up to 5V. The LCD also can only communicate at up to 50kHz SPI. For maximum use of this project, the hope was to set up the firmware to use Arduino API's where possible. Unfortunately, for the Feather M4, the I2C speed cannot be lowered below 100kHz using the Arduino "Wire.setClock()" interface. I had to lower the rate manually. For reference, the snippet of code for setting up the SAMD51 in slower I2C is as follows (and all this on our GitHub as we update it):

  // Setup I2C at <50khz for LCD
  
      Wire.begin();                                     // Set-up the I2C port
      sercom2.disableWIRE();                            // Disable the I2C SERCOM
  
      GCLK->PCHCTRL[23].bit.CHEN = 0; // 23 is SERCOM2
      while(GCLK->PCHCTRL[23].bit.CHEN != 0);
  	GCLK->PCHCTRL[23].bit.GEN = 0x04;			// Generic clock generator 4
  	GCLK->PCHCTRL[23].bit.CHEN = 1;
  	while(GCLK->PCHCTRL[23].bit.CHEN != 1);
  
      // TODO what is GCLK4 actually?
      SERCOM2->I2CM.BAUD.bit.BAUD = 48000000 / (50000) - 1;   // Set the I2C clock rate slow
      sercom2.enableWIRE();                             // Enable the I2C SERCOM  

   For the Feather M4 board, the I2C is set up for SERCOM2. The peripheral index for SERCOM2 is 23. I set up SERCOM2 to be run off of GCLK4. Underneath all the layers of Arduino, I am not quite sure yet what GCLK4's rate is. Nonetheless, I was able to get about 30kHz clock rate, as confirmed by my logic analyzer.

  I set up the firmware to support:

  • Display of current information
  • Settings input mode using pushbutton to page between settings and knob to choose the particular setting's value.
  • Read flowmeter, numerically integrate it and estimate tidal volume

  The picture below shows the prototype setup and connections. The interface to the flowmeter was through a fairly long wire. In our design we will use an I2C driver specifically for long distance communication with high capacitance on the lines. This prototype does not have it, but still seemed reliable. The measurements have checksums and I did not see a checksum fail during testing. The flow measurements seem quite stable, and estimation of tidal volume from integrated flow rate seems reasonable.

• Flow Meter Update

  Erik Wachlin • 03/27/2020 at 02:30 • 0 comments

  We have been working hard to design a connector for the flow meter that can be made anywhere.

  
  The Sensirion 3300-D flow meter communicates via I2C which is typically an on PCB communication protocol.  We are using it at the patient wye which is about 1 meter from the ventilator.  The likelihood of poor data quality was high, so we designed a solution using a Texas Instruments P82B715DR I2C bus extender on the main PCB as well as one on the flow meter PCB.  This bus extender allows us to go off the board to much further distances without sacrificing data quality.  The flow meter PCB is all SMD components.  It has spring pins to connect to the pads on the flow meter, and a molex microfit 3.0 connector on top to connect to the main PCB via and off the shelf cable. 

  The PCB is held onto the flow meter by a 3D printed clip.  Here is a picture of the full assembly.

  Here is a picture of a test print and the clip on the flow meter.

• Information Links

  Erik Wachlin • 03/16/2020 at 02:54 • 0 comments

  Covid-19 Ventilation Strategy - https://emcrit.org/ibcc/covid19/#invasive_mechanical_ventilation

  APRV - http://rc.rcjournal.com/content/respcare/57/2/282.full.pdf

  Spontaneous breathing - http://rc.rcjournal.com/content/respcare/57/2/282.full.pdf

• To dos- rolling list

  Erik Wachlin • 03/16/2020 at 02:28 • 0 comments

  Add 2 modes of ventilation that could be useful for Covid-19

  • Pressure Control mode
    • Inputs
      • PIP- Peak Inspiratory Pressure (cmH2O)
      • PEEP- Positive End Expiratory Pressure (cmH2O
      • BPM - Breaths per minute
      • FiO2 
    • Output
      • Directs user as to flow rate setting for oxygen

  • APRV- Airway Pressure Release Ventilation
    • Inputs
      • P-high (cmH2O)
      • P-Low (cmH2O)
      • T-high (seconds)
      • T-Low (seconds)
      • FiO2
    • Output
      • Directs user as to flow rate setting for oxygen

  Add startup checks for the ventilator:

  • Tightness test
    • Turn on blower to max and ensure that the pressure increases to static pressure and hold for 1 minute to verify that flow does stop.
  • Flow meter offset calibration
    • The tubing and system will have air capacitance that needs to be subtracted to get correct tidal volumes.  Pump up the system to max pressure and create a flows table.  Do this 5 times and take average.

  Add Alarms to ventilator- alarms to be published to cloud and sent out be sent out to hospital staff.

View all 8 project logs