gmoffett

edge data store - first look!

Blog Post created by gmoffett on Oct 11, 2019

Hello!

 

Following on the recent-ish release of OSIsoft Cloud Services (ocs) there is also the soon to be released Edge Data Store (eds) which in part is "related" to ocs - more on that later. This blog post provides a light overview - a look under the bonnet/hood - by running the edge data store for the purpose of understanding components and general functionality.

 

The objective is to collect Modbus TCP data, eds health and diagnostic data and forward a subset of this data set to a PI Server and OSIsoft Cloud Services.Worth noting that eds is also an OPC UA client, accepts OSIsoft Message Format (omf) messages and includes a REST API.

 

Environment

 

here is a picture 

 

edge data store lab environment

 

Deployment

 

Docker on Linux was the platform of choice for this environment - a number of platforms are supported - and edge data store documentation details how to build and run a Docker image including examples to persist configuration/data to the Linux host and optionally enable local host access to the REST API.

 

Configuration options

 

Configuration is via a REST API and while ultimately stored in json files, the recommendation is to use the REST API only due to how certain properties are handled. Options to configure include:

  1. edgecmd program - introduced with the release candidate
  2. REST API calls using command line/tool/language of choice.

 

Modbus configuration

 

In  the eds world there are three configuration steps to start data collection from a Modbus TCP device (this would also apply to OPC UA):

  1. define a component
    • i.e.: Modbus and a name for this "component instance"
      • (There is already one component configured: Storage)
  2. configure the datasource
    • IP Address, port, query parameters for the Modbus TCP device
  3. specify the datasource measurements
    • for Modbus the registers that will be retrieved and stored as streams (more on streams later) in eds.

 

This is fairly straightforward process, documentation provides a quick start guide including example json files that once customized can be used with example commands:

 

curl -v -d "@component.json" -H "Content-Type: application/json" -X POST http://localhost:5590/api/v1/configuration/system/components
curl -v -d "@emer.json" -H "Content-Type: application/json" -X POST "http://localhost:5590/api/v1/configuration/Modbus1/DataSource"
curl -v -d "@emer-data.json" -H "Content-Type: application/json" -X POST http://localhost:5590/api/v1/configuration/Modbus1/DataSelection

component.json configuration file, tell eds we want to add a Modbus collection source:

{
"ComponentId": "Modbus1",
"ComponentType": "Modbus"
}

 

emer-json configuration file for general Modbus configuration properties:

{
"IpAddress": "10.1.1.2",
"Port": 502,
"ConnectTimeout": 15000,
"ReconnectInterval": 5000,
"RequestTimeout": 9000,
"DelayBetweenRequests": 0,
"MaxResponseDataLength": 250
}

 

If you are familiar with the PI Interface for Modbus (or most interfaces or that matter) and have configured tags (known as streams in eds) using your favorite editor (e.g.: Excel!), the process is similar for eds with different columns and saving as/converting to json format i.e.: the emer-data.json file used in the last command above to configure registers to query looks like:

[
{
"Selected": true,
"UnitId": 1,
"RegisterType": "Holding16",
"RegisterOffset": 0,
"DataTypeCode": 20,
"BitMap": "",
"ScanRate": 1000,
"Name": "holding0"
},
{
"Selected": true,
"UnitId": 1,
"RegisterType": "Holding16",
"RegisterOffset": 1,
"DataTypeCode": 20,
"BitMap": "",
"ScanRate": 1000,
"Name": "holding1"
},
...

 

 

Storage and data access

 

eds uses the Sequential Data Store (sds) which happens to be the same store used by OSIsoft Cloud Services (ocs) to 

"store, retrieve and analyze data". This means that certain storage REST API calls will be the same for both environments - noice!  Three components of sds are:

  • namespaces - eds is configured with two: default and diagnostics
    • namespaces contain objects that include types and streams
    • types
      • describe a data shape and include properties
        • property examples include: timestamp, temperature, event rate, memory size, ...
    • streams
      • based upon a type definition and stores stream events
        • for example: the modbus data collection configured previously.

To be fair this is a simplification, but will do for this discussion, for further information, eds and ocs documentation.

 

As mentioned the same REST API queries/library can be used to access both ocs and eds!  So we can use the OCS Python sample library to query eds, albeit with one customization to not get a token as required by ocs. If you are familiar with the Python library a sample config.ini is pasted below with the required properties.

[Configurations]
; specify the tenant namespace
Namespace = default
[Access]
Resource = http://10.1.1.1:5590
; specify the tenant id
Tenant = default
ApiVersion = v1

In the configuration above one parameter not yet mentioned is the tenant where data will be stored and for eds there is one tenant called default.

 

namespace: default - user data

 

Accessing the REST API using the OCS Python library we can query to determine what types and streams have been created in the default namespace:

for ocsType in ocsClient.Types.getTypes(namespace_id,query='*'):
print(ocsType.Id,ocsType.Name)
for stream in ocsClient.Streams.getStreams(namespace_id,query='*'):
print(stream.Id,stream.Name)

These requests return:

  • 1 type
    • created by the Modbus Adapter based upon the Modbus configuration: Modbus.UInt16 
  • 100 streams based on that type
    • corresponding to the Modbus configuration including the two properties:
      • stream id: Modbus1.1.Holding16.0 through Modbus1.1.Holding16.99
        • id is defined by the Modbus Adapter
      • stream name: holding0 through holding99
        • Yes you can customize the stream name!

 

So how many stream events have been collected and stored in eds? The Python library supports querying the sds REST API GetSummaries call that along with some Python code can produce the table below for stream holding1.

Viewing the table, the count is close to 86400 stream events per day and with a configured scan rate of 1 second leaves some room for exploration to determine the missing scans or ...

 

 

What won't be obvious to the reader is that data collection was configured in August and consequently there *should* be stream event counts for the first three days and more events on the fourth rows in the table above. This is by design. 

For each stream there is a global total storage and target limit. When a stream reaches the limit value, storage will be reduced to the target value for that stream. The values displayed below are the defaults and can be customized.

{
"StreamStorageLimitMb": 2,
"StreamStorageTargetMb": 1,
"IngressDebugExpiration": "0001-01-01T00:00:00"
}

 

 

namespace: diagnostics - health and performance data

 

Running a similar query on the diagnostics namespace, the following streams are returned:

  • Modbus1.StreamCount
  • Modbus1.ErrorRate
  • Modbus1.IORate
  • Storage.default.default.Counts
  • Storage.default.diagnostics.Counts
  • Storage.Total.Counts
  • Storage.IngressRate
  • System.Diagnostics.

 

Egress

 

While in production most likely the majority or all streams collected would be forwarded. For the lab just two of the Modbus streams are sent to both a PI Server and OCS. Egress was configured in a similar way to Modbus, copying a configuration sample from the documentation, updating and sending to the REST API. A few of the configurable parameters of interest are listed below and the following configuration snippet:

  • period/frequency to send data from eds
  • filtering which streams are sent
    • for the lab a stream id and stream name were specified to match the two required streams.
  • prefix to add to the objects that will be created on the remote endpoint and uniquely identify these objects from this instance of eds.
"ExecutionPeriod": "00:01:00",
"EgressFilter": "Modbus1.1.Holding16.50 holding0",
"StreamPrefix": "oak.i.eds2.modbus.",
"TypePrefix": "eds2.modbus."

 

Similarly health and diagnostic data is streamed to PI Server and ocs. A selection of that data is displayed in an OSIsoft Cloud Services trend:

 

Streams are based upon types which can have a variable number of properties (and other configuration parameters). The System.Diagnostics stream for example includes a date-time property and several other properties, two of which are included in the trend above.

 

Wrapping it up

 

The Edge Data Store documentation does a nice job of making it easy to get started - appreciating this is just a lab environment and not production! - and providing information on design considerations (aka hardware/performance/sizing guidance). The bonus is that the Sequential Data Store is also used by OSIsoft Cloud Services so the REST API is the same for each platform.

 

For more information check out the documentation and recent presentations from OSIsoft. Eds is not yet released with a lighthouse program currently available for approved scenarios.

 

Cheers,

Outcomes