January 1, 2020 - On January 1, a friend was posting about what he was doing 20 years ago today. He was on duty and close to work, ready for the great Y2K meltdown—a meltdown that did not occur in part because of a lot of work done in advance…

Early that morning, 20 years ago, I was standing on a one-lane bridge on a country road at midnight, watching neighbors shoot off fireworks to greet the new century. My phone, not yet a smart phone, was in my pocket, waiting for phone calls from the campus Power Plant and the Electric Distribution group reporting any issues. If I did not get those calls, I planned to call in, as the consultants had warned that the phone switches might fail after midnight as well. I was on that country road so I could be on site in fifteen minutes if any of the phone calls were not good.

At the time, I had the integration of distribution systems for steam and for chilled water in my wheelhouse as well. I felt good, having spent four years remediating system issues from low-level BIOS to inter-system communications. The University did not yet provide complete networking services so that work included patches to routers and switches as well. The parallel work to patch the line-of-business client-server systems had been relatively painless.

My largest issue involved the integration of a hundred buildings with advanced control systems, mostly for energy management. After the oil-price shock of 1973, a few dozen of the most critical high-energy-consuming buildings had been migrated to a single centralized energy management system—one that was still based on RSX running on an ageing PDP-11. Since then, each new building was equipped with digital controls, purchased from and installed by the lowest bidder in accord with State Construction law. To make matters worse, with a couple years to go before Y2K, the local utility had begun giving us each afternoon with 15-minute pricing for power the next day. The need to coordinate building responses had never been greater.

(For the nerdy, the prevailing interaction between control systems in buildings in the 90s was DCOM with even a little DDE. It was a brittle insecurable mess. The interactions were all concrete and low level, in some cases simulating typing on virtual keyboards. Most progress since then has been based on making the interactions more abstract and thereby less fragile.)

For many of the control systems, preparing for Y2K would begin with prying out PROMs from boards and replacing them. Upgrading one system would inevitably break all integration with the next.

The talks I gave after that crisis contributed to the wide-spread adoption of web services to building automation control systems, including a middleware standard in wide use internationally. That work became the roots of the US National Smart Grid roadmap. The common vision at the start that project of direct utility control over building operations was not only bad for tenants, bad for owners, and bad for privacy, but was far more complex than they imagined. The roadmap described distributed autonomous power management systems (microgrids) with high-level abstract communications between them.

Microgrids need only coordinate supply and demand over time. Attempting to manage internal mechanisms and motivations adds complexity, reduces resilience, and creates a cybersecurity nightmare. That work is still percolating as after a decade of false-starts and one-offs, and only now is the are those sites that most value power reliability and resilience, homing in on a standard model for service integration of microgrids.

Hacking critical infrastructure has matured from a loner’s hobby into coordinated incursions by professionals and nation-states. Since 2000, there have been two (or perhaps three) flat-out nation-on-nation cyberwars between Russia and Lithuania. A SCADA worm deployed to take out foreign nuclear weapons facilities is considered a likely contributor to the largest oil spill ever in US waters, friendly fire that is astonishing in its scope. Russian operatives casually took out the entire power grid of Ukraine along with other infrastructure. The attacks on the Ukraine are widely considered to be practice runs for attacks on US infrastructure. Military war games delicately model infrastructure threats, including EMP and physical attacks on substations as zombie outbreaks. Today, military planners will not approve wide deployment of any technology for critical infrastructure until a common model for cyberdefense of the control systems is in place.

There is a growing recognition that Cybersecurity systems for critical infrastructure require integration with those for traditional networking and IT. To prevent technology lock-in and stagnation of innovation, this cybersecurity must be abstract, not reliant on direct controls. These new systems must work in effect as distributed situation awareness, informing highly distributed systems of autonomous components what dangers are present or anticipated, and receiving from those systems added situation awareness in return.

[an error occurred while processing this directive]In 2017, the US promoted USCYBERCOM to unified force command, that is, a top-level inter-service organization able to coordinate responses and technologies across branches of the US military. In the new world, all critical infrastructure systems named above, and more, must fit into common abstract cybersecurity models. Consistent training programs for cybersecurity must prepare personnel to work with them all, even as the accelerating pace of innovation increases the technical diversity among these systems.

The DotCom Boom provided an opportunity to re-write commercial applications, which included removing duplicate code by enabling these applications to communicate with each other. These communications required us to solve the problems of identity and security between applications.

Much of today’s Operational Technology (OT) was also re-written for Y2K, but the work is not done. They share identity with enterprise systems. When OT applications interact with other applications, it is almost always with an application from the same vendor. Today’s systems are highly connected. Lagging best practice, as usual, OT is evolving to lean on cloud-based AI for decision making even as those who want privacy and reliability are leaning toward new techniques bringing even the most sophisticated AI to inexpensive local systems.

The challenge of secure integration of rapidly evolving OT systems has just begun. BACnet/SC (BACnet Secure Connect) is both a necessary and welcome improvement but does not address wider security integration. (Be sure to learn more if you attend the AHR Show this month.)

To me, it feels like I never left working on Y2K integration issues, and will continue to do so for at least another decade.

[an error occurred while processing this directive]
[Click Banner To Learn More]

[Home Page]  [The Automator]  [About]  [Subscribe ]  [Contact Us]