The Need for Open-Software, Open-Hardware What will an Open-Software, Open-Hardware programmable controller look like?

Calvin Slater Member of https://www.project-haystack.org/ Project-Sandstar Working Group -  Sandstar https://www.linkedin.com/in/calvin-slater-15555842/

I'm here to talk about a class of products that are greatly needed in our industry which would address the following outstanding issues:

1)    Proprietary Field Controller Frameworks

2)    Inadequate Enterprise Integration and Site Visibility

3)    Field Controller Obsolescence and Network Security

In general, there are two categories of building automation control devices; Programmable controllers and Configurable controllers. Both have their place in a modern building as long as they are deployed appropriately. Configurable controllers are most well suited for equipment control that is very application specific, whose operational logic is well defined and never needs to be modified. Some good examples of this would be an Electric Meter or Variable Frequency Drive. Because these devices are designed for a very specific task, the internal control logic and discoverable points can be pre-programmed into the device at the factory level. This programming cannot be changed by the user. Usually, only settings and setpoints of the device can be modified.

Programmable controllers are more general-purpose devices. They may in some cases contain application specific hardware such as integral Airflow Sensors and Damper Actuators. However, the internal control logic of the device remains fully flexible when the supplied programming tools are used. Programmable controllers are most appropriate where the controlled equipment relies upon or must interact with, other remote equipment on the site in order to form a complete system. A good example of this would be an Air handler Controller working with VAV Terminal Unit controllers. Increasingly in new construction, the various controllers employed together in a system are not from the same manufacturer/vendor. While this is great news with regard to open choice and competitive pricing, in many cases we are left with a control system whose functionality and maintainability are less than desirable.

Unfortunately, the vast majority of programmable controllers use proprietary programming architectures that are not transparent or fully interoperable with other systems from other vendors. Open networking protocols such as BACnet have helped to a great degree and have moved our industry in the right direction. However, while Open-Networking protocols and services may be completely adequate for configurable devices, programmable controllers are still somewhat lacking. These products need to be further streamlined to offer higher capability and value to the owner. Our discussion here involves programmable controllers. A new generation of Open-Source programmable controllers is beginning to emerge.

What will an Open-Software, Open-Hardware programmable controller look like:

1)    A compact, low-power, fast-booting, hardware device that can be acquired from a multitude of sources; be it from a large or small manufacturer. The most important thing is that you have complete ownership of the device. Basically, a small personal computer that runs on standard 24VAC power or PoE and has an assortment of hardware inputs and outputs as well as industry standard communication ports.

2)    Rather than relying on proprietary platform firmware images, applications and services run on the device much in the same way user software runs on a personal computer using an Operating System. Installed applications can be open-source, readily available, cheap, or even free; such as Sandstar, Haystack Servers, or Sedona. The owner or authorized user can install or uninstall programs and services at will, just like on all other modern smart devices.

3)    Full ownership of the device’s Operating System up to and including choice of operating system (Linux, Android, etc.), much in the same way you can choose your own PC’s OS. The operating system shall also be capable are receiving patches and updates Over the Air (OTA) or locally by a qualified system administrator without significantly affecting the functionality of installed applications and services.

4)    The device incorporates hardware security at boot-time and maintains a chain-of-trust spanning into user space. Apps and middleware communicate to remote stations using secure and open networking protocols which are mostly IP based such as HTTPS and have the ability to directly connect to a building management database server located on site, in the fog, or in the cloud without the need for specialized intermediary network routers or gateways.

Why we need it.

Portable DDC Apps.

Experienced system integrators usually maintain libraries of proven programs or snippets of programs from previous successful jobs. This is because it is not efficient or cost-effective to re-write a new program from scratch for every new job. Ideally, a controls programmer should be able to take a proven program or snippet from one type of hardware device, load it into a different platform, and then simply re-number the assigned physical I/O ports. This is not always the case. In some product lines, it is possible to do this across the whole family of hardware devices offered by that particular vendor. On other lines, DDC application formats are specific to only certain groups of products, and multiple separate libraries of programs and code need to be maintained by a single vendor-dealer for each group. This situation is not ideal for the programmer or other programmers who may someday service the site. A common DDC architecture that is abstracted from the hardware is the limiting factor in the ability to reuse portions of a DDC from one device to another. This was never done intentionally, but rather is an outgrowth of the fact that legacy controllers have been previously built using low-cost microcontrollers. Because they are limited in resources and have proprietary hardware architectures, the underlying firmware that lies below the DDC control application must be compact and tailored to that specific device. This is no longer the case with the wide availability of lower-cost higher-performance controller components.  These newer devices have the hardware resources to support the necessary software abstraction layers for a common architecture DDC framework.

Enhanced site integration.

Another benefit to using a common open-source software framework for direct digital control is increased site visibility from the front-end building management system. As individual enterprises continue to move away from single source manufacturers for building controls. It becomes critical to increase the level of interoperability between these various devices from different manufacturers. Open communication protocols such as BACnet are sufficient for configurable devices, where shared networks and point discovery services are all that is needed from the device. The problem with programmable controls and complex custom program logic is that while the third-parties can see the values of data points for the controller, they cannot see these software points in any kind of logical programming context. In mixed systems, owners and operators are mostly unable to view the logical program flow in a third-party field device without accessing it using vendor-specific tools. They would need to do this for the purposes of maintenance, troubleshooting, and system expansion. Viewing only points and their values can be ambiguous in nature, especially with individually programmed DDC controls.

Another issue with proprietary software architectures on mixed sites is overall enterprise database management. In single vendor systems, site backup and management for all controllers as well as the required hierarchical semantic data relating to locations, equipment, and networks are usually all stored in a centralized location. On sites with controllers from different vendors, the only option is to use some form of Enterprise Integration Software for the automation system front-end. While this type of software has the ability to discover networks and controller point data, there must also be a mechanism to discover semantic data about the site which would be necessary to support the generation of topological visualizations such as geographic site locations and associated equipment. This would otherwise have to be done manually if at all. Metadata for the purpose of Site Discovery and Analytics can be served directly from the edge devices themselves. Unitary field controllers are the best location to deploy metadata server schemes such as Haystack. The point-level data can be had directly from the hardware device that has the physical sensors and outputs attached to it. The best opportunity to input this semantic data is when the device is being programmed and commissioned.

Long-term maintainability.

[an error occurred while processing this directive]Unitary field controllers that use underlying proprietary firmware for DDC apps to run on top of can be a long-term problem for building owners and operators. Often these embedded devices continue to function properly long past the time where firmware running on them is current or even supported. Many building owners routinely find themselves in the position of removing and replacing large quantities of controllers that are still operating correctly, yet are no longer supported or compatible with newer systems. Responsible manufacturers are stuck with updating and supporting controller firmware packages for products that they have discontinued for years. This will soon become a major problem. Network connected devices that rely on unsupported or unmanageable proprietary firmware are increasingly becoming a very large security risk. For example, the 2016 Mirai Botnet denial of service attack was based on a collection of internet-connected devices whose firmware could not be remotely updated. This is especially true for newer IP based controllers which will be expected to directly and securely connect to cloud services across the open internet.  

Future controller platforms must be capable of being updated. This must be done by the vendor, and if not possible, the devices must be capable of being updated and patched by the owner or authorized user. As mentioned previously, devices based on microcontrollers must have very architecture specific firmware packages created that are tailored to a very small and limited code footprint. Patches and updates are much easier to accomplish with full open-source embedded operating systems. Many of the utilities required to perform updates are already built into these systems and would require little to no product development.

 

How can we do it?

The Open-Software Open-Hardware edge controller is not an imaginary item. It would seem from the description above that such a device would be too costly and complicated to produce and employ. This is not the case. There are already many devices from several manufacturers in production right now that can fulfill some or all of these requirements. For example, the EAC controller from Anka Labs is an Open-Hardware, Open-Software Programmable Edge Controller. The device is based on a very popular 1GHz Cortex-A8 processor intended for industrial control applications. The controller ships with a completely open-source Linux operating system and a variety of pre-installed building automation control applications and services. The EAC is a fully programmable DDC controller that uses Sedona for graphical-control applications as well as serving point level Haystack metadata which has been combined into a fully open-source application known as Sandstar. In addition to Sandstar, the device is capable of running other applications and services concurrently such as Skyspark or Anka’s Visualytik framework.

Sedona framework applications offer the possibility of being portable as long as a few required conditions are observed and agreed upon up front by various parties. Project Sandstar has proven this concept to be a reality. In order to use a Sedona DDC app (known as a .sab file) running on one device and load it on another completely different hardware device, both devices must support the same Sedona kits. A kit is analogous to a module in the Niagara framework which is where Sedona originated. The kits that a particular Sedona app are reliant upon are declared in a section of the application file known as the schema. If two Sedona DDC applications have a common schema, then they are cross-compatible and portable across devices regardless of physical hardware. Of the kits enumerated in the app-schema, there are two that are of critical importance to platform independence. The first one is the platform kit. Sandstar uses the generic default “platUnix” kit. This kit can be built and is compatible with, many Unix based operating systems of which Linux is a member. This offers an extremely wide variety of choices for embedded SOC based hardware that Linux has been ported to, especially with respect to Arm processors from various manufacturers who have participated in the Linaro Consortium.

The second kit of more critical importance is the one for the controller’s physical I/O (i.e., the UIs, AOs, and BOs, etc.). There is no default template in the Sedona framework for this. This kit must be fully implemented by the controls designer(s) for each device. This kit is the most critical for hardware independence as it models the controller’s actual physical points. Project Sandstar has demonstrated a solution to this issue by the implementation of a fully open source I/O kit known as “EacIo.” The kit contains DDC components for physical I/O that are commonly found on most controllers. Every controller has different quantities of these of these I/O ports as well as differing types of signals they accept. The EacIo kit does not dictate what ports will exist on a device, but rather it abstracts them through the channel property. By modeling the I/O points as channels, a particular input or output point can exist if the channel exists. This layer of abstraction not only allows hardware independence with regard to the Sedona world but also creates a mechanism to support point-level Haystack metadata to be seamlessly integrated into the application.  Haystack tags are generated on the device at the time of device programming and commissioning. This is the best time to complete this work as it avoids performing repetitive tasks. The Haystack data is served from the actual controller rather than a separate remote proxy device. Sandstar rolls all of this functionality into a single application that can be easily built and run on multiple Linux based platforms. The Sandstar software is an application much in the same way any application is a separate executable entity on any personal computer or smart device. It can be installed, updated, or uninstalled without affecting the underlying operating system.

Because the EAC uses Linux for an OS, and the system is completely open to authorized users, the device can be managed by a qualified system administrator much in the same way an IT department manages its machines. Authorized individuals can apply patches and updates to the system without greatly affecting existing installed BMS applications. The system administrator does not necessarily have to have a vast knowledge of building automation system software. Likewise building automation system technicians do have to have great knowledge about Linux System Administration. This is because Sandstar and other programs on EAC run as separate applications in user space. This creates a distinct boundary between the system platform and user applications.
All of these characteristics combine to together as an example of what the edge controller to streamline building automation interoperability might look like. The EAC controller is just one example. The real key to meeting all of these challenges is for everyone in our industry from both small and large manufacturers to come forward join in and offer similar devices.

About the Author

Calvin Slater is a U.S. Navy Veteran and Graduate of UCLA with a BS in Mechanical Engineering. He has spent eight years in the Building Automation Controls Industry, and is highly interested in Embedded Hardware as well as Open-Source building automation software frameworks.