Critical services and data infrastructure providers across Texas are racing to keep technology and operations running in the face of a winter storm th
Critical services and data infrastructure providers across Texas are racing to keep technology and operations running in the face of a winter storm that has strained electrical grids and left hundreds of thousands of people without power.
Widespread outages and extreme weather conditions have closed businesses, emptied store shelves and contributed to a number of deaths across the state since the weekend, when temperatures began plunging below freezing.
Investments in cloud services, spurred by the coronavirus pandemic, have helped companies cope with disruption from extreme events such as the Arctic blast, said Rick Holland, the chief information security officer at cybersecurity firm Digital Shadows Ltd., which operates its U.S. intelligence operations facility from Plano.
But data-center operators, which provide the infrastructural backbone for cloud services, have had to react quickly.
Digital Realty Trust Inc., which runs 15 data centers in Texas—including 13 in the greater Dallas area—said some power loads have been transferred to on-site generators to provide uninterrupted uptime for critical services. As of Thursday, none of its plants had gone offline, the company said.
“By running generators at select sites, we have also been able to help reduce the overall power load on the ERCOT grid system,” Erich Sanchack, the company’s executive vice president of operations, said in an email.
He said the company is working with its supply-chain partners to keep generators fueled, while redirecting excess fuel supplies to other data-center operators in the region. Internally, it has juggled work shifts to reduce the amount of travel by workers to and from the site, Mr. Sanchack said.
Jim Hawkins, senior vice president of global data-center operations and engineering at data-center operator Rackspace Technology Inc., said on Monday its Dallas-Fort Worth plant was forced to run on a generator for roughly 20 minutes before switching back to the power grid. The generator is designed to operate for about two days without refueling, he said.
The Dallas plant also implemented business-continuity measures, specifically planned for severe weather, Mr. Hawkins said. This includes keeping workers on-site until relief staff arrive at the data centers, which are stocked with food and other necessities.
Equinix Inc., based in Redwood, Calif., operates nine data centers in Dallas, and one in Houston. All of its sites have remained operational throughout the crisis, running on utility power where available or backup generators, according to a spokeswoman.
The Dallas region acts as a key hub for digital connectivity across the southern U.S., as well as for north-south traffic between the U.S. and Latin America, the company said.
David Cappuccio, distinguished vice president and analyst for information-technology infrastructure strategies at IT research and consulting firm Gartner Inc., said many businesses are running critical workloads in multiple cloud-computing “availability zones,” to safeguard connectivity if one zone becomes unavailable.
The impact of the Arctic blast has largely fallen on staff, rather than technology, said Andrew Walker, chief administrative officer at San Antonio-based insurer USAA. While the company had also shifted its data centers to generator power without issue, Mr. Walker said, its people are facing a challenging situation.
“Our Texas employees are dealing with inconsistent power, lack of water and diminishing gasoline and food resources,” he said. “Of course, power issues directly impact our ability to serve members when our employees are not able to maintain internet connectivity.”
Others describe conditions that make it difficult, or even impossible, for staff to work. Mr. Holland of Digital Shadows said he had long stretches this week where he had no power and no internet connection at his home in Dallas, making his cellphone the only way he could keep up with work—when he had enough signal strength. Ray Greer, chief executive officer of Dallas-based trucking and logistics technology developer Omnitracs LLC said one of his executives was forced to use his car as a makeshift office due to a blackout at his home that had lasted for over 30 hours.
“I cannot remember a moment in the 27 years I’ve lived in Dallas where [an outage] sustained itself for such a long period of time,” he said.
Larger companies, such as USAA, have used employees in other regions to cover those unable to operate in Texas, executives say. Smaller businesses, however, don’t have the ability or head count to do that, and are losing money at a critical time.
“Everything’s frozen, we just had to cancel the full slate of services, and so that’s lost revenue. But, you know, that’s a hard thing to care about when you’re literally in survival mode,” said Denise Hamilton, the Houston-based founder of business consulting firm WatchHerWork LLC, whose home only regained intermittent power Thursday after an outage that began Sunday. She said she was still without water.
Given lengthy estimates in some cases for restoring services, she said, widespread disruption of the workforce is likely to last for weeks.
“People think it’ll be warmer on Sunday or Monday, and we’ll be back to normal,” she said. “I don’t think that’s accurate.”
Copyright ©2020 Dow Jones & Company, Inc. All Rights Reserved. 87990cbe856818d5eddac44c7b1cdeb8