Collaboration Driven Development and Testing Unique requirements drove an incremental development effort and, of course, the corresponding integration and test activities.

Steve Jones Managing Partner, The S4 Group, Inc

Lately, it seems like we are practically living in our test lab. We recently completed a major upgrade to our test lab, and were inspired to consider what was driving this activity. This particular update was driven by a collaborative experience with one of our largest partners. Their unique requirements drove an incremental development effort and, of course, the corresponding integration and test activities that go along with such developments. In order to get our lab closer to replicating what this partner and all partners encounter out in the real world, the test lab is a continual work in progress.  Certainly a large part of our lab activities involve testing product enhancements, as well as confirming interoperability with other manufacturers’ equipment. This is all a part of the collaboratory in which we co-exist with our partners and customers. We frequently receive compliments on the quality of our customer support and credit the investment in the test lab as a large component of this capability. Many situations that our partners run into out in the field can be reproduced here in the test lab.

When The S4 Group was formed, we knew we needed to invest in a test environment. In fact, we wrote an article back in October of 2004 for Automatedbuildings.com Extreme N2 Networking discussing the importance of establishing the test environment and including the flexibility to reconfigure it as needed.

That first test bed was one side of a single wire rack with Metasys® N2 devices and a supervisory controller. The primary focus in testing our initial products was to successfully communicate to the N2 bus and the N2 devices. This was no small feat since the N2 protocol is actually three sub-protocols that co-exist on the N2 bus. This meant that we needed to have N2 devices from each Metasys® N2 product family, along with a sample of N2 Vendor (VND) devices. Taking the sensitive timing characteristics of the N2 technology into account, we also needed to include enough devices to generate an adequate transaction volume to exercise the N2 protocol, make sure that everything worked even when the bus was saturated, and ensure the integrity of all transactions. We made heavy use of the documentation that we received as a result of becoming a member of the Metasys® Compatibility Program.

We listened to the feedback from our first product release (we were already participating in the collaboratory but didn’t realize it) and one of the first enhancements to the S4 Open Appliances was the addition of our Upstream N2 Interface which enables co-existence with the Metasys® supervisory controller. This enhancement called for an expansion in the test lab to include examples of each generation of Metasys® supervisory controller and the requisite tools to monitor both the upstream and downstream interfaces. We had to confirm the integrity of all N2 transactions independently of whether they originated from the Metasys® supervisory controller or from an OPC HMI or SCADA system, as would be the case with our flagship product, The S4 Open: OPC-N2 Router. The complexity of the product increased dramatically with the Upstream enhancement: integration to three downstream N2 sub-protocols, the upstream N2 interface to the supervisory controller, and the upstream OPC interface. Testing became correspondingly complex and added to the demands on our test environment. The OPC-N2 Router continues to be an important offering, especially in high availability environments, pharmaceuticals, and other applications where industrial or process control systems needed input from environmental control systems.

The S4 Open: BACnet-N2 Router came to fruition next. The introduction of this product meant that interoperability testing became a very important function of the test lab. After attending our first BACnet International PlugFest, we quickly learned that interoperability testing needed to be a standard part of our environment, not just a once a year event. Testing in a sterile environment was necessary, but not adequate, to deliver the quality that we were committed to. Even though BACnet is the most widely used communications standard in the BAS world, we still found it necessary to test each enhancement with a selection of different manufacturers’ global controllers and BACnet Operator Workstations. We found that once a year participation in PlugFest was just not enough to accomplish this task. So, another panel in the test bed was dedicated to hosting BACnet global controllers from various BAS manufacturers. A high end workstation in the test lab hosts virtual machines for every major BACnet OWS manufacturer in the industry. In a few cases, we found that the versions we were testing with did not perform well in a virtual machine environment, so we added dedicated machines to host these OWS environments.

We have put the necessary investments into the test environment to keep up with the ever increasing complexity of the product offerings. Along with this growth, our partners have grown with us.  There has been a steady increase in the number if S4 Integration Partners and Distributors, as well as a number of strategic partnerships. This growth brought with it the requirement to have multiple tests running in parallel to make sure that everyone who needs support gets it as quickly as is practical, and so we can keep product enhancements and new product development moving forward at the same time. To accommodate these needs, we split our available N2 devices into multiple color-coded N2 buses.

The need to support multiple S4 Open Appliance product families, and multiple generations of hardware, including a brand labeled version for one of the major BAS manufacturers, meant that we needed to have one panel dedicated to hosting an array of S4 Open Appliances. Again, flexibility is the key to successful utilization of the test environment. Any of the S4 Open appliances can be running the developmental firmware, or be rolled back to exactly match the firmware installed at a customer site we are providing support for. When necessary, we can have our partners export the configuration of their installed system so that we can load it onto one of the S4 Open appliances in the test lab and duplicate their installation to the extent possible in our test lab. Any of the appliances in our lab can be connected to any of the available N2 buses as needed for each test.

A flat Ethernet network was no longer adequate to test BACnet BBMD and Foreign Device Registration capabilities, so we expanded the test lab into multiple IP subnets. Support for testing cloud-based applications and applications hosted at remote locations, where the BACnet-N2 Router is acting as an on-site agent for those applications is defining the next round of enhancements to the test environment.  We already frequently utilize desktop sharing capabilities to share our lab facilities while working with partners. These new applications will require secure access from external locations.

[an error occurred while processing this directive] Testing doesn’t start here at our S4 Test lab. Our development partner, Obermeier Software, of Verl, Germany puts every firmware build through an automated test sequence before it is released to S4. This sequence includes tests to verify all core system functionality. Then the automated tests go a step further to emulate the situation that exposed the root cause for all previous trouble tickets. This automated testing is a very thorough, but robotic, process. When we perform our Q/A testing here at S4, we attempt to simulate real world building conditions that the product will be working in when installed by our Integration Partners. We try to interject as much randomness as possible into the test process to simulate what might happen when different installation engineers approach the installation and integration process a bit differently. Of course, the final test is when the product is installed in a live project and performs as it is designed!

S4 Open Appliances are not just for Metasys® anymore. The lab has physically expanded to four wire racks with both sides of each rack able to support various test environments. From the outset, the architecture of the system anticipated many more integrations than just Metasys® N2. We’ve been chomping at the bit to get going on these projects, so we’ve been building separate sections of the test lab in anticipation of the next two integrations to be released. What about those empty spaces on the racks? There is a long list of potential new integrations that we may offer in the future. Whenever the opportunity presents itself to acquire equipment that we might need for future testing, that’s where it ends up.

Is the test lab worth the investment in time and expense? Absolutely! It facilitates the collaboration between S4 and our partners, helping to build strong relationships and our reputation for providing high end service and support. We’ve found an equation that works and we will keep evolving and improving it to make sure that the success continues into the future.

[an error occurred while processing this directive]
[Click Banner To Learn More]

[Home Page]  [The Automator]  [About]  [Subscribe ]  [Contact Us]