Intelligent, Small and Simple BAS BACnet Routers at Large Sites The initial hardware, software and planning costs may have been higher, but the results in simplified ongoing maintenance are outstanding.

Albert Putnam VP Tech Ops Cimetrics

Recent Cimetrics projects at very large sites (airports and universities in Asia) with many BACnet MSTP networks routed by our B6000 BACnet Router have sharpened our sense of the way one should go about approaching such situations.

B6000 Cimetrics BACnet/IP to BACnet MS/TP Router

Cimetrics' Analytika business involves collecting large amounts of data. Many types of equipment and many instances of the same type of equipment are collected, and collected quite quickly as compared to standard clipboard or monthly methods. Analytics performed on these data are in service of economization of energy (or other commodity), and, as importantly, in service of maintaining the reliability of the purpose that said commodity serves.

Cimetrics thus often consults about the reliability of data flow in service of the above goals. One needs to "see" the data to perform analytics. Sometimes the analytics are even about "data flow" itself - data might be the commodity of interest. That gets us into IT issues surrounding the reliability of data flow.

Getting back to our Analytika core: The main data we deal with is building automation data, primarily that available on a building’s BACnet network. Cimetrics participated in the writing of the BACnet standard; we also created our own tools and BACnet protocol stacks (BACstac). We therefore have significant experience and expertise in the examination of issues surrounding the reliability of energy and automation data flows.

BACnet has some very useful features. One of the most useful is that devices and objects are discoverable. Other features are methods for reading change of value, trending, security, and many other properties. The network topology is largely a static entity when running well. However, some useful features can work against stability - especially when devices and routers are operating in their startup or discovery modes. There can be conflicts. There can be loops. There can be table overflows. There can be “flapping”.

Given our role in analytics we often deal with very large automation deployments, with thousands to millions of devices and hundreds to thousands of routers. Often such networks have differing data link layers with speed and technology mismatches, from RS485 MSTP or other serial direct or bus to TCP/IP variants of the protocols (which are often related to firewalls and security).

A frequent issue is device ID conflict for so many devices, but there are easy strategies of mandated (or unique) defaults which can get most systems right. Luckily, the network layer is not affected by the device mapping, and most end applications have large tables for keeping track of any large number of devices.

Another frequent issue is network numbering. Again there are easy strategies of mandated (or unique) defaults which can get most systems right. But the network layer depends on routers. And unfortunately most protocol specific routers (BACnet, Modbus, etc) never envisioned the sizes of systems with which we routinely deal. The routers tend to have tables for about a hundred networks. BACnet, for example, allows one to have tens of thousands of networks. Further there are issues of conformity with IT topology. Many application-specific routers have to deal with mismatches in the sub-segmentation for pure IT topology (think VLANs and broadcast subnets) and the automation protocol topology (global broadcast domains and physical LANs). There are also mismatches with respect to guaranteed (or setup/turn-down) and best effort services. For BACnet this leads to the concept of BACnet Broadcast Management Devices and Foreign Devices. But for broadcast management to work well the application-specific routers need to have large tables with fast access. And sometimes large fast tables are not possible, given the economics surrounding the price of building automation systems devices (such as routers).

Intelligent compromises need to be made...

Like intelligent choices of which entries to pin in tables (and whether to block updates during operation), whether to use FIFOs or LIFOs, and timescales for persistence of state (think like issues regarding a DHCP and DNS and LDAP). BACnet routers are not easy for large topologies. To restate, they would be easy if they had fast tables approaching the maximum allowed tens of thousands of BACnet networks. In the non-ideal world, small routers have about a hundred router table entries. We use a carefully crafted LRU algorithm (least recently used) to selectively purge less useful entries. We also make intelligent assumptions about noisy traffic. Slower and older network types will tend not to be re-routed (or have connections to peer or subordinate high traffic networks). There are sometimes bridge applications, but they are unusual. So we essentially use a persistence algorithm regarding MSTP (or any slower) networks.

And on the fast and noisy networks, we give some priority to discovery packets, to allow setup to complete at the expense of losing a few normal operation application layer data transfer packets (which generally have their own retry mechanism - note carefully what was said initially about applications being able to handle tables of millions of devices.)

It is as common sense as that.

So let us get to a real world scenario of a large international airport in India:

Systems are being (re)constructed using BACnet for all HVAC devices. Many air handling units and thermostats and sensors are distributed locally. There are on the order of a thousand MSTP LANs. Local tight loop control exists within each MSTP LAN, but then upper level supervisory control and monitoring is from central controllers and collectors (like data bases and workstations and mobile applications AND proxies for such). The central controllers and proxies are all on TCPIP LANs.

The key features drawn from above:

Thousands of MSTP networks (each with unique network number). Little to no MSTP intercommunicaton globally in any routine way. Central controllers are on a handful of big LANs (maybe with about ten airport wings), and they might even be aggregated by BBMDs (often most efficiently connecting to MSTP by the router itself being a Foreign Device). There is a lot of traffic on the main LAN(s), but this can often be filtered for the myriad MSTP networks. Likewise the MSTP networks can be considered very important pinned networks for the main central controllers.

Any required global exchanges can be proxied by variables collected or sent from a handful of central points to subsidiary MSTP controllers. This can be achieved through application design bearing in mind the topology. For example the airport weather station might have a key control parameter (overall regional temperature and humidity) which is useful for every air handler and thermostat to have in its control loop. Instead of getting key parameters directly from the MSTP sensor, it is collected centrally and published for all to see or pushed from one location (high capacity, high bandwidth).

Consider another real world scenario of a major healthcare campus.  Systems are in place for BACnet and there is an ongoing mandate for BACnet.  BACnet is being slowly added by new contractors over time. All the existing low level loops are local on MSTP.  New systems get added as small local MSTP LANs connected via a router OR directly into BACnet/IP (these include: meters, new AHUs in building expansions, pumps, central plant additions and retrofits). The central controllers exist on MORE than a handful of big LANs (usually subnets associated with each of maybe a hundred buildings). And they are ALWAYS integrated by BBMD or are put into a massive VLAN - if IT could not deal with the massive subnetting and organizational problems associated with the traffic, the VLAN moves the IT traffic problem into the hands of the HVAC systems designers - which is often not  a great idea... because such “designers” might not even exist. In any case there will be high amounts of central network traffic, and it can be hard (though not impossible) to filter this rationally for local MSTP networks.

[an error occurred while processing this directive] Cimetrics BACnet routers endeavour to exactly address the central traffic problem head on while at the same time maintain the viability of the local MSTP networks through intelligent pinning. The statements made about proxies and global exchanges for the airport example above also carry through. The key ends up being a way of maintaining control and visibility of the architecture (by fiat) while the overall system grows and changes. This is not so much an issue of technicalities of the routers (though visibility of their service APIs can certainly help), as much as it about understanding the BACnet rules and maintaining a good master architecture and visibility of which networks numbers and device IDs are where.

Once running smoothly the sites above and their kin are joys to behold (and they are worthy of being beheld). They are massive. They have visibility to keep issues under control simply and locally. The initial hardware, software and planning costs may have been higher, but the results in simplified ongoing maintenance are outstanding. They run trouble free for years.

If you have a big BACnet site, come talk to us about our experiences. We probably have value to add.