Industrial IoT and the Edge

Contributing Analysts

David Floyer

Peter Burris

Premise

Industrial IoT, the largest segment of the Internet of things (IoT) with the highest potential value, will require a deep integration between modern IT (Information Technology) and OT (Operations Technology), but that won’t happen by trying to centralize all processing in public or private clouds. The reality is that modern IT technologies, to be truly extensible to OT, will need a hybrid cloud approach, with by far the majority of data and processing residing at the so-called “Edge”. Architectures and software written by industrialists for industrialists, such as GE Predix, are showing how that can work. Managing Edge projects to extract the most value will come by ensuring primary value is achieved at the edge itself, and that data and processing for secondary value is paid for by the external parties. This research will also conclude that 95-99% of Industrial IoT data will be created, live, be processed and be discarded at the edge.

Internet of Things Classification

Figure 1 describes Wikibon’s IoT segmentation by types of uses.

This segmentation includes seven general sectors covering many different use cases:

Personal IoT

Smart phones, smart cars, entertainment, location, travel, wearables, etc.

Home IoT

Smart meters, security and access control, smart appliances, health & safety, etc.

Government IoT

Federal, state & local government: criminal justice, internal security, traffic management, emergency response, etc.

Health IoT

Smart equipment, home health, electronic medical records, provider and payor applications,  etc.

Small Business IoT

Asset management, smart retail, customer management, inventory, security, health & safety, etc.

Office IoT

Building management: security and access control, equipment management, utilities management, maintenance, etc.

Industrial IoT

Plant operations, equipment maintenance (predictive and preventive), vehicle and fleet management (aircraft, trucks, shipping, trains, etc.), warehouse operations and inventory management, site health & safety, security and access control, etc.

The Industrial Internet can be subdivided into:

• Static equipment: Stationary plants and buildings characterized by expensive equipment and machinery with long depreciation cycles.
• Mobile equipment: vehicles (trains, planes, ships and trucks) used in commerce.  All have the potential to become highly automated, with extreme amounts of data gathered both internally and externally. There are very significant restrictions on how much data and the cost of data transmission that can be transmitted while mobile.

Wikibon will be updating its IoT projections later in 2016. The Industrial Internet is the largest sector of IoT, and has the highest potential for added value.

Fundamental Drivers for Industrial IoT

The Industrial Internet of Things (IIoT) includes two general application patterns:

1. The connection of industrial machine sensors and actuators for local processing for individual equipment performance (i..e, a wind turbine);
2. The onward connection to both internal and external industrial systems that can independently generate value across an entire operational system (i.e. a wind farm or power grid).

Both of these general patterns are moving forward based on sensor technology, network and compute advances, as well as the growing awareness of the business value potential. These advances are discussed further:

Sensor technology advances

The advances in the consumer space from smart phones, digital watches, Fitbits and other wearables have accelerated sensor technologies.  Sensor design is rapidly evolving towards smart sensors that perform sophisticated analytical processes, using (for example) nano-structured materials and nano-materials such as conducting polymers and composites. This rapid sensor improvement gives rise to additional types of sensors that are smaller, cheaper, and faster, with lower power consumption, and less overhead impact on the equipment being sensed. Video sensor technology and has also dramatically improved in cost and quality, with the iPhone 7 now being better than all but a few traditional cameras. The ecosystem round these sensors and their software is build almost exclusively around low cost low power ARM processors.

This consumer sensor technology is being industrialized and brought to market, and applied in industrial settings. When an expensive piece of industrial equipment is thought of as a system, the sensors, networking and processing components are additional parts that can fail, and can add significant cost to the equipment. Very high availability and low maintenance versions of a sensor ecosystem are essential in this environment to reduce the overall impact on industrial equipment cost.

The Industrial Internet has begun to connect industrial machines together, and extend them into other industrial networks. This can drive efficiencies in industrial machine operations but also has potential value in contributing to the efficiencies of second-order industrial networks. For example, avoiding the cost of a single power-generation turbine failure can be significant. In addition, the data collected can contribute to upstream processes, such as billing, optimizing wind farm processing or as input into regional grid management.

Network advances

The advances and standardization in local and remote Internet technologies allow easier connection and low-cost transmission of information from physical sensors to other locations. However, as has been shown in earlier Wikibon research, data is heavy and expensive to move. This cost and how to mitigate it will be the subject of further discussion later in this research. Network speeds are just keeping pace with data generation and storage capacities, so faster future networks will not be a solution to the relative cost of data transfer. Most of consumer and enterprise network technology has used TCP/IP protocols, and network firewalls have been used as “moat” security. These protocols and firewalls are woefully inadequate for protecting high-value Industrial equipment!

Compute advances

Computer technologies have been rapidly advancing both in cost and functionality. Traditional mechanical disk storage is being replace by electronic NAND flash storage which is orders of magnitude faster. ARM processors are only 10% to 20% the cost of Intel processors, and for many application give similar throughput; ARM processors are deployed in a very high percentage of cases in the sensor and storage ecosystems. Nvidia is driving volume GPUs and providing new ways of linking to high-speed processors with NVLink, building off consumer volumes of screen and gaming GPUs in modern mobile devices. New processor technologies using GPUs and FPGAs, with memory protocol such as RoCE (RDMA over Converged Ethernet) on ultra high-speed interconnects are introducing new ways of processing complex analytics and machine learning, at a fraction of the cost.

The bottom line is that the cost and density of computing that previously would have taken a football field are being reduced to closet dimensions. Theses systems can do in real-time what would have previously taken months to achieve. To achieve the potential value of real-time systems (see Figure 2 below) means that the processing has to be close to the data source.

Experience and understanding of potential benefits

Businesses have realized that sensor data can be used to optimize the value extracted from high cost industrial investments. Businesses are also beginning to see opportunities for disruptive changes in industrial business models.  Governments are supporting greater use of sensor data to avoid catastrophic failure of key infrastructure networks such as power, water, and transport.

Edge Processing

In order to meet operational goals, much of the processing and analysis for action must be accomplished close to the equipment – or at the “Edge”.  To an IT professional, “edge processing” can connote functions that are tangential or less important.   However, from the perspective of business management and OT, the latter are responsible for delivering the product – which requires keeping very expensive equipment up and running at very high levels of availability while also running at high levels of efficiency. As such, the effective application of real-time data to achieve these goals is of the highest priority.  Figure 2 below illustrates this by showing that the highest value of industrial IoT data will come from information systems that operate in real-time – by definition close to the operations themselves.  On the other hand, historic data and reports will have use cases with relatively lower potential value.

The “Edge” processing environment will generally be managing extremely expensive and high value equipment.  This will ensure that IT vendors will continue to develop new technology for modernizing and creating traditional and new IT architectures, to serve these use cases including applications, cloud, software-defined infrastructure, container-based development, big data analytics, and tooling for cross-technology automation.

Moreover, the drive to digital business, made possible in part by dramatic advances in sensor and actuator technologies, is fueling great interest in IoT as a strategic capability across all industries and company types. Many are presuming that these two trends will merge, effectively upending how technology is deployed in traditional operational technology (OT) use cases, like power plants, industrial automation, and other supervisory control and data acquisition (SCADA) domains. However, while many of the advances in modern IT can be translated into OT, the truth is that these advances will have to be specifically engineered to support OT realities, including:

Real-Time OT Won’t Wait

In many OT applications, physics (i.e., the speed of light) dictates how resources must operate at the edge, i.e., as close as possible to the data sources. To boost performance, OT has often had to hardwire the assets and data together, leading to long design, deployment, and change cycle times. This is a drag on competitiveness, so the edge processing environment must be both fast and flexible.

A device that must take action in milliseconds to avoid a safety disaster can’t put a routine request to the cloud in the critical path for action. Similarly, a temperature trace comprised of millions of writes in an instant — continuously — can’t be supported by traditional RDBMS technology. And the mass of data from OT curves, loads, and other types of OT data streams can’t cost-effectively be moved into the cloud for storage, processing, and analysis. These edicts aren’t managerial; they’re physics. And these physics have dictated highly specialized technology, architecture, development methods, and administrative practices that won’t easily bend to modern IT capabilities. Simply put, the real-time processing requirements of a vast swath of OT use cases create an impedance mismatch with the traditional IT world. Any effort to modernize OT will require technologies that can bridge edge realities and modern IT advances.

GE’s IIoT platform, Predix, shows how that bridge will work. Predix is the product of GE’s industrial heritage, SCADA leadership, and commitment to digital transformation. Predix embodies a modern, architected approach to both support specialized edge needs and exploit the scale, connectivity, and analytics of the cloud. Especially notable are Predix’s:

• Flexible deployment options that spans OT and IT requirements. At the heart of Predix is a connectivity architecture that spans an array of assets, from the sensors, control systems, gateways, and equipment management needs typical of OT, to the appliance, server, and storage needs to process that data. A multitude of OT and IT protocols are supported, but tuned and hardened, minimizing the need to employ specialized communications technologies.
• Data flows that serve edge and overall system operation requirements. Predix provides a robust messaging model to commonly handle alarms and events, time series data, secure distribution of software packages, and commands. Baked into the architecture are data management technologies capable of handling edge, time series, and analytic data needs.
• Micro-services engineered to serve IIoT applications. Predix provides a set of “Industrial Micro-services” for asset, analytics, data management, security, and operations functions. These microservices are employed to handle the UI and many of the management resources, like a catalog to track asset and data attributes.
• Edge software components that make physicist and developers happy. Predix offers a set of components for handling the extreme needs of edge data processing. These include a engines for rules, expression, and aggregation, as well as data handling components, local database services, and securely and cost-effectively synching data to the cloud.

OT Environmentals Can Be Harsh

IT pros may not refer to their personal work environments as “plush,” but OT pros would regard the IT environment in glowing terms compared to the harsh conditions they and their equipment must operate within. OT frequently operates dangerous environments or in remote areas where humans infrequently tread safely. As such, modern edge processing must take advantage of predictive analytics and cloud-based management of distributed assets to accomplish its reliability and performance goals.

OT professionals typically plan, build, and operate OT systems based on environmental constraints that would make heads spin in the IT world. Why? Because OT design constraints include physical, security, and service realities that are far more demanding than typical IT environments.

On the physical front, for example, OT deploys systems in domains that feature excessive heat, corrosive forces, shaking, and other physical features that require significant hardening of equipment; a $1,000 server in an OT environment can quickly become a $10,000 investment, not because of any traditional technological issues, but because the server must be braced with bracketing and control technology that dampens the effects of local conditions.

On the security front, ensuring the security of data and OT operations in a nuclear power plant just is not as simple as installing a firewall. It requires total separation — an air gap — of traffic and execution from any and all other systems. With other industrial equipment, networks that must be connected but must satisfy strict security requirements often are hierarchically organized according to the direction data can pass.

On the service front, replacing a failed OT storage device may not be as simple as pulling and replacing a blade in rack housed an air conditioned warehouse located in a familiar zip code; if the storage is located at the edge, it may require servicing a device in a mine dug into the side of a mountain located hundreds of miles from an airport. The design of Edge systems must include sufficient redundancy to minimize replacement costs.

Another problem: limiting the physics effects on state management and data movement can’t be solved by using data approximations that are easier to handle with traditional IT technologies. OT typically requires a high degree of fidelity between things and digital representations. Badly designed digital representations and architectures can unexpectedly generate error propagations that can take down “thoroughly tested” systems.

Asset optimization is the heart of industrial applications. Complex machines and systems must be kept up and running, or revenue is lost and costs accelerate. The Predix platform is built to ensure that the best asset decisions that can be made, will be made, either by the system or human management. To do this, Predix combines a number of advanced technologies for managing OT assets, including:

• Predictive analytics. A central benefit of bringing the edge and the cloud together is to take advantage of emerging big data technologies for advanced analytics in a way that doesn’t compromise the environmental and real-time processing realities of the edge. Predix has pre-engineered an array of data science pipelines for monitoring asset conditions and health, prognostics, and running simulations. Predix doesn’t presume all data will go back to a central location, which would be both physically impossible and prohibitively expensive. Instead, Predix supports rational approaches, like allowing a wind turbine to query another proximate wind turbine regarding settings and enacting those settings if they seem superior.
• Software-defined machines. Representing assets in software is essential to modernizing the edge. The Predix architecture supports the development, deployment, and maintenance of software-defined machines (SDMs) that autonomously connect to the Industrial Internet, execute native or cloud-based machine apps, analyze collected data and securely respond to changes in those data. Built by GE, partners, or customers using common software tools, the Predix architecture includes services for ensuring that SDM models can be thoroughly tested, simply deployed, and enhanced to reflect the actual conditions of the asset.
• Security. OT data is highly sensitive, heavily regulated, and has high intellectual property value. Industrial machines have very specific attack surfaces and threats – unique to those devices. This threat extends to IT systems connected to these devices, both locally and in the cloud: cyber security is a key priority. Aggressive processes to secure and certify operational infrastructure are required. Advanced governance is required to maintain provenance for data, copies of data and usage of data both within and outside the Edge. GE Predix and similar systems must have to have end-to-end security build in. Again, much of the processing for security, e.g. surveillance face-recognition, detection of anomalies and many, many others require large amounts of data from the site and large amounts of processing power. The results are needed in realtime, and most security processing has to be done at the edge; the use of the hybrid cloud can include for example, sharing the signature for new friendly faces with corporate, and downloading additional friendly and unfriendly signatures.

Institutional Structures Often Are Regulated

If the physical attributes of OT weren’t stringent enough, the governance structures regulating IT can be especially complex. The “edge” in the IT-oriented IoT lexicon actually is the “center” in the OT lexicon. While the physics and environmentals served by OT are enough to justify its evolution, many OT domains are further complicated by regulations governing safety, environmental, or competitive behavior. OT did not emerge because industrial businesses wanted a separate technology group, but because it needed a separate technology group. Moreover, the OT ecosystem evolved to fit needs and niches generated by the technical and institutional realities of IIoT. The emergence of modern IT technology does not obviate the need for OT practices, disciplines, reporting, and partnerships. Any effort to modernize the “edge” must include practical approaches to serving the institutional requirements of OT.

GE is taking a chapter out of the IT industry to ensure that the Predix platform works for both IT and OT — and the industrial businesses they serve. By using modern technologies, hardened to handle the specialized needs of OT, GE Predix can deliver:

• Edge-to-cloud business support. GE is providing Predix utilizing a pay-as-you-go, subscription pricing and packaging structure, in a similar way to other Cloud SaaS providers.
• Technology and partnership structures that fuel growth of a new ecosystem. By choosing to utilize the standard communication, language, data management, and security technologies that make up cloud computing, GE ensures that Predix not only is modern, but can remain modern, as technology evolves. Moreover, the company is investing heavily in an ecosystem approach to attract critical partners, which can exploit Predix’s facilities for building SDMs, handling analytics, and ensuring asset optimization.
• Business case-to-business model services. Acknowledging the transformative nature inherent to IIoT, GE is flipping the go-to market playbook with Predix. Instead of leading with products, they focus engagement on what they call the “Total Addressable Problem” (TAP). The breadth of the question is intentional: GE knows that IIoT can lead to innovation in multiple domains, including technology, product, service, institutional structure, and business model. Once again, they are applying home-grown insight to their business approach, such as their transformation from a seller of jet engines to a provider of “thrust services,” whereby GE uses IIoT to offer an engine service intended to reduce aircraft downtime, fuel costs, and emissions. Wikibon believes that this “business design” approach will become standard in complex B2B domains, including industrial.

Managing Edge Projects

In general the best way to kill an Industrial IoT project at the Edge is to let it be managed by external IT. The primary initial focus of an Edge project is to ensure that the data extracted from local smart sensors will enable the Edge processes to be more efficient, create greater throughput, and reduce risk.

There is a significant danger that other parts of an organization see (or think they see) huge potential value from mining the data, and put forward great reasons why their department should have sole responsibility to collect, process and use of all or some of the data. Wikibon recommends that decisions about processing of edge data should be taken by Edge management. This should extend to data transmitted to outside parties, and the Edge should ensure that those outside parties (inside or outside the enterprise) should pay for the costs of processing and distributing data.

Managing IoT Data

The instinct of IT is to capture and own all the data, and the instincts of data scientists are similar. Figure 2 shows that in the majority of cases the vast majority of value will be extracted close to where the data is generated. Data is extremely heavy and expensive to move, and previous Wikibon research entitled “The Vital Role of Edge Computing in the Internet of Things” has shown that it is more cost effective to keep and process the data at the edge, rather than move all the data to the center. And physics dictates that to extract the maximum amount of value from the data comping from the sensors, video streams or from local processing (see Figure 2 above), the data must be processed at the edge. Data for down-stream processing can and will be extracted from the edge by distributed analytics as required. The management process for setting up and funding this is important.

• If there is a direct benefit to the edge management and another party for data, and agreement can be made between the edge and the other party to deliver that data. Hybrid cloud with the necessary security is the likely to be the IT topology of choice, though other technologies such as Blockchain could be used.
• If there is little or no benefit to the Edge management, other parties, both internal and external to the business, may be able to add or change data filters on the sensor data, and extract data using local resources.
• It is also vital that these agreements are reviewed for risk of misuse. For example, it may be useful to sell log data back to a manufacturer; however, if the log data can be used to establish business demand for the products produced at the edge, this data could be misused for insider trading.

It will be very difficult to justify Big Data projects if all the data is extracted and sent to clouds. The cost and elapsed time for data transmission is likely to be prohibitive, and the results delayed and reduced in value. The optimum strategy is to use distributed data analytics at Edge center(s), and extract small amounts of data for further analysis in the cloud. Individual projects can also turn on specific filters to save additional data at the edge for processing and onward transmission of results.

Action Item

Digital business goals require greater cooperation and coordination of modern IT and OT technologies and organizations. To extract the maximum value from Edge data it will need to be streamed, reduced and processed locally. Edge projects should focus primarily on giving value to the edge business processes. Extracts of data can be processed and delivered to upstream parties who pay for it, subject to careful review. Big data projects will be designed with distributed Edge analytics combined with hybrid cloud services.