These are exciting times in the evolution of stream-processing. As we have seen in previous blogs, the digital twin model offers a breakthrough approach to structuring stateful stream-processing applications. This model organizes key information about each data source (for example, an IoT device, e-commerce shopper, or medical patient) in a software component that tracks the data source’s evolving state and encapsulates algorithms, such as predictive analytics, for interpreting that state and generating real-time feedback. Using digital twins offers three key benefits over more traditional, pipelined stream-processing techniques (such as Apache Storm and Apache Flink): automatic event correlation by data source, deeper introspection with enhanced state information, and data-parallel analytics to discover aggregate trends for all data sources in real time. It represents a big step forward for building stream-processing applications.

When using the digital twin model, each data source in a physical system has a corresponding software object in the stream-processing platform as depicted here:

This object encapsulates both state information and code. State information includes a time-ordered list of the device’s incoming event messages along with key state information about the dynamic state of the data source. This information could include parameters, service history, known issues, and much more. Application code handles the management of the event list and the real-time analysis of incoming events, along with APIs for performing device commands. This code benefits from the rich context provided by dynamic state information, enabling deeper introspection than analyzing the event stream alone.

The secret to keeping event analysis times low when handling events from thousands of data sources is to host these digital twin objects in an in-memory data grid (IMDG) with an integrated compute engine, such as ScaleOut StreamServer. IMDGs harness the memory and computing power of multiple commodity (or cloud-based) servers to scale computing resources, and they minimize network bottlenecks by analyzing events within the grid. Their NoSQL, object-oriented storage precisely fits the requirements for digital twin objects, making it straightforward to deploy and host these objects with both scalable performance and high availability.

The following diagram depicts an IMDG hosting a large set of digital twin objects in an IoT application. The IMDG transparently distributes the digital twin objects across a cluster of commodity servers for scalable processing. These objects receive telemetry from various devices and perform real-time analysis:

Let’s take a look at how object-oriented techniques can simplify the design of digital twins. Because a digital twin encapsulates state information and associated analysis code, it naturally can be represented as a user-defined data type (often called a class) within an object-oriented language, such as Java or C#. The use of an object class to represent the controller conveniently encapsulates the data and code as a single unit and allows us to create many instances of this type to manage different devices. For example, consider the digital twin for a basic controller with class properties (status and event collection) describing the controller’s status and class methods for analyzing events and performing device commands. This class can be depicted graphically as follows:

Here’s how a basic controller class could be written in Java:

public class BasicController {
    private List<Event> eventCollection;
    private DeviceStatus status;

    public void start() {…}
    public void stop() {…}
    public void handleEvent() {…}
}

We also can make use of the class definition to construct various special purpose digital twins as subclasses, taking advantage of the object-oriented technique called inheritance. For example, we can define the digital twin for a hot water valve as a subclass of a basic controller that adds new properties, such as temperature and flow rate, with associated methods for managing them:

This subclass inherits all of the properties of a basic controller while adding new capabilities to manage specialized controller types. Using this object-oriented approach maximizes code reuse and saves development time.

Here’s a Java example that illustrates how inheritance could be used to create the hot water valve class. It also shows how the hot water valve class can override the implementation of the Start and Stop methods defined by the basic controller:

public abstract class BasicController {
    protected List<Event> eventCollection;
    protected DeviceStatus status;

    public abstract void start();
    public abstract void stop();
    public abstract void handleEvent();
}

public class HotWaterValve extends BasicController {
    private double temperature;
    private double flowRate;

    public double readTemperature() {
        return temperature;
    }

    public double readFlowRate() {
        return flowRate;
    }

    @Override
    public void start() {…}

    @Override
    public void stop() {…}

    @Override
    public void handleEvent() {…}
}

As discussed in an earlier blog, we also can build a hierarchy of digital twins that represent successively higher levels of analysis and management for complex systems, and this hierarchy can further leverage object-oriented techniques. Consider the following set of interconnected digital twin instances used in managing a hypothetical pump room:

In this example, the pump room has two digital twins connected directly to devices, one for a hot water valve and another for a circuit breaker. These twins are both implemented as subclasses of a basic controller and add properties and methods specific to their devices. They feed telemetry to a higher-level digital twin instance which manages overall operations for the pump room. This digital twin also can be implemented as a subclass of a basic controller even though it is not connected directly to a device. What’s important to observe about this example is how both object inheritance and hierarchy play separate roles in defining the digital twin objects which work together to analyze event streams. Inheritance lets us refine the behavior of digital twin models to customize their actions, and hierarchy lets us build systems of interconnected digital twins which process events at successively higher levels of abstraction.

Digital twin models for stateful stream-processing have evolved from concepts largely unrelated to object-oriented programming, in particular, product life-cycle management and industrial IoT (where they are often called device twins).  Object-oriented techniques give software developers powerful tools for applying digital twins to stateful stream-processing and streaming analytics. Applications now can benefit from automatic event correlation, stateful event analysis for deeper introspection, and the scalable computing power of IMDGs.