Advanced Photon Source

An Office of Science National User Facility

The LNDS is Now A-OK


Andy Stevens (FMS) and one of the new 9,000-gallon LN2 tanks. Click image to enlarge.The APS liquid nitrogen distribution system (LNDS) was installed in 2000 to supply APS beamlines with a continuous supply of liquid nitrogen (LN2). The LNDS allows beamlines to connect directly to a very large, stable supply of LN2, reducing the need for manually filling portable dewars along with the expense and potential for disruption of the supply.

In 2010, the APS experienced a series of LN2 events that resulted in significant disruption to some beamlines. The original 3,000-gallon supply tanks were undersized relative to the demand for LN2, which had grown over the years. This resulted in some of the tanks requiring refill every day, complicated by changing weather conditions, truck mechanical breakdowns, delivery schedule conflicts and even medical emergencies. This became an operational disruption at the APS when LN2 delivery failures required opening of interconnect valves, which swept accumulated water contamination into the system, changing from water to ice along the way, eventually clogging beamline equipment.

Problems with the LNDS together with feedback from the beamline community made it clear the system was no longer providing the high quality and standard of operations that the APS demands. A task force was commissioned to study the situation and recommend actions to improve the existing system.

The LNDS consists of a main header (1-inch, vacuum jacketed, stainless steel) located on top of the storage ring. This header runs from Sector 1 downstream to Sector 35. The header is segmented into four modules (Mods), Mod A (sectors 1-9), Mod B (sectors 10-17), Mod C (sectors 18-27), and Module D (sectors 28-35). Each the four modules are separated by an “interconnect” valve, and each insertion device beamline has a programmable logic controller (PLC)-controlled “drop” valve.

The PLCs control the drop valves, supply valve, interconnect valves, and electric keep full (EKF) valves that are installed at each end of each module, monitor the header temperatures at three locations in each module, and monitor each supply tank’s level and pressure.

The original programming of the PLC had limited automation: the PLCs only closed the drop valves if the header temperature or tank pressure were too high, or if the LN2 tank level was too low. There was no programming to control the EKF valves, nor the interconnect valves. In the event of the loss of LN2 supply from any of the module’s supply tanks, the interconnect to an adjacent module would be opened by an APS floor coordinator using the PLC and/or a MEDM interface (a remote desktop display of the LN2 controls) to supply that module. This wasn’t automatic; the floor coordinator would have to open the interconnect manually.

Part of the design of the original header included long sections of piping that were between the last mechanical keep full (KF) valve and the adjacent interconnect valve. These long “dead-legs” (one of which is over 100 feet long) would fill up with dry, room-temperature gaseous nitrogen during the long periods between being manually opened, when we had a low tank, or another condition that required opening an interconnect. These dead legs also seemed to be reservoirs for water contamination in the header.

The task force made a number of recommendations:
• Upgrade the PLCs with common, available components. New programming was required to automate the EKF valves and interconnect valves. The storage capacity of the LN2 tanks had to be increased to keep pace with demand. • Upgrade two 3,000 gallon tanks to 9,000 gallon tanks, to keep pace with demand.
• Require the LN2 supplier to install telemetry on all LN2 tanks, monitor the levels, and schedule deliveries to ensure that the tanks are never less than 37.5% full, to ensure continuity of supply to users. The LN2 supplier was also required to communicate any delivery failures/disruptions directly to the APS main control room so that APS beamlines can prepare for problems.
• Increase the reliability of the LNDS system by installing additional mechanical and EKF valves, ensuring that the dead-legs would remain filled with cold LN2. To make absolutely sure that any accumulated water or ice did not make it into beamline equipment, valves were installed to purge the lengths of piping on each side of each of the three interconnect valves.

As a result of requests from the APS beamline community, AES developed a three-phase improvement project for the system, spearheaded by Andy Stevens (FMS), Marty Smith (AES), and Rich Diviero (AES). The improvement work could only be carried out during APS shutdowns and could only progress as fast as funding allowed, so work was prioritized and carried out over a period of 36 months.

Phase one was the replacement of the obsolete PLCs and improved programming. The new PLCS were installed in 2013 and 2014.

Phase two included increasing the installed storage capacity to match present-day LN2 demand. This was achieved by replacing two of the tanks with 9,000-gallon tanks at modules B and C in 2014. At the same time, the LN2 supplier added their own telemetry to the system that allowed the vendor to see the levels of LN2 in the storage tanks in real time and schedule deliveries much more efficiently, taking several manual steps out of the process. These improvements make the system more reliable as it no longer relies on daily deliveries of bulk LN2. The new, larger tanks also had pressure stabilization units installed in order to ensure that the LN2 supply pressure stays close to the nominal 45-psi operating pressure.

Phase three was the installation of new keep-full valves, EKF valves, and purge valves at each of the interconnect valves. Installation of the new piping required the shutdown, lock out and draining, entire LNDS system. Over 14 days of the April-May 2016 Maintenance Shutdown the new piping arrangements were installed.

In addition to the new mechanical keep-full and purge valves, two automatic dewar fill stations were installed in the APS infield at super door A and D, providing ready availability of LN2 to the storage ring during shutdowns. Truck delivery of full dewars are no longer be needed, and the safety issues associated with moving full dewars off lift gate trucks in the infield are largely eliminated.

The new KF valves have already made a dramatic improvement in the operation of the LNDS. Before the piping modifications, one of Module A’s EKF valves would open dozens of times a day due high temperature in the header. With the new piping, those nuisance fills and alarms have dropped to almost zero.

The new EKF valves, purge valves, and interconnect valves will be automated with additional PLCs and programming that will make the system even more reliable. With the new purge valves and EKF valves, the system will be almost completely automated.

The new piping arrangement will also allow for individually shutting down the modules, so they can be periodically warmed up and effectively purged of accumulated contaminants. The system was fully purged before turning the LN2 back on in early June 2016. Starting in 2020, the APS will start shutting down one of the modules every year and all modules every four years, so that each module can be thoroughly warmed, dried, and purged, ensuring consistent quality of the LN2 supply.

It has been a long, steady process, but the upgrade of APS’s liquid nitrogen distribution system is almost complete, making the LNDS more robust than it has ever been and further improving the availability of x-rays to APS users.

By Andy Stevens

Published Date: