Skip navigation
All People > gmoffett > Glenn Moffett's Blog > 2018 > June
2018

Glenn Moffett's Blog

June 2018 Previous month Next month

Hello,

 

PI World SF 2018 included a lab on Industrial IoT Deployment – What and How that provided six examples of data collection from sensors. When a sensor or data source can only be queried or deliver data one way that can simplify (or possibly complicate!) data collection, however when there are options, which one should you select? An understanding of the data collection "tools" in the toolbox along with considerations and caveats could be useful! There are of course a number of methods to ingress data to the PI Server and over the past few years there have been two REST based methods introduced:

     • PI Web API - which uses a REST endpoint

     • PI Connector for UFL - REST endpoint feature (referred to as PI Connector for UFL - REST in this document)

          Side note: The Connector also supports reading from files and requesting data via a REST call to external system(s) - these features will not be covered in this post.

and now (as of this week) we have a third:

  • PI Connector Relay with OMF (OSIsoft Message Format) support which also has  REST endpoint.

 

The purpose of this blog entry is to explore REST based data ingress options.

 

Terminology and general approach to data ingress

First some terminology, since there are several components involved with ingress of data to the PI System:

• data source - generates the data - Modbus slave, OPC Server, IIoT sensor, ...

• "custom code" - script or code- that enables data flow from the data source to the data collector or directly to the PI Server

• data collector - the PI System component or custom code that enables data flow from the data source to the PI Server

• PI Server - the PI Data Archive and PI Asset Server

• message format - the definition of how the data is formatted

• configuration information - the name of the PI Data Archive, PI Asset Framework (AF) Server, tag and AF object

 

Typically and historically - and there are always exceptions - the "data collector" selection process generally follows this progression/list of options:

1. Is there a PI Connector or PI Interface that is appropriate?

2  Is the data provided in files or could a file be created and then processed using the PI Interface for UFL?

3. Utilize one of the PI Developer Technologies (includes PI Web API) or PI Connector Relay with OMF support

4. PI Manual Logger

5. Insert other options here, including PowerShell, ...

Note: If there is not a product solution, check in with OSIsoft in case one is in progress or to see about having your request added. Maybe others are asking for the same requirement and your request could potentially drive priorities.

Also, check out and feel free to add requests via Customer Feedback for OSIsoft & the PI System

 

Observations about option 1 above

• Zero coding, configuration only

• Software versions may be released with enhancements and/or bug fixes

• Limitations may exist with current functionality - let us know if this is the case! See customer feedback note above

• Support for buffering if the connection between the data collector and PI Server is unavailable

• In general, can be the most straight forward approach (as long as the data source cooperates).

 

Factors that may impact the option selected:

• Operating System or platform that is available for the data collector

• e.g.: Windows, Linux, worker role in Microsoft Azure, ...

• The network between the data source and the data collector, and between the data collector and the PI System

• LAN, WAN, Internet connections must factor in adequate security. For example: TLS, IPSEC, ...

• Satellite or cellular connections - think bandwidth considerations

• Dodgy (unreliable) connections, retry to send data along with buffering

• Application requirements - i.e.: read/write, supported technologies.

• Performance requirements

• Connection and data rates.

 

PI WEB API and PI Connector for UFL - REST endpoint considerations?

 

Observations about the PI Web API and PI Connector for UFL - REST

• Support for points, Asset Framework objects (elements, attributes, ...) and event frames

• Flexibility!

• Unrestricted by operating system, platform and language used for scripting or code - e.g.: Windows, Linux or Worker role in Azure and C, C#, Python, Java, ...

• The fact that custom code is required and needs to be maintained

• Code must deal with buffering of data if/when necessary.

 

Observations about the PI Web API

• In the past had heard this product discussed as being designed and targeted for egress of data from the PI System. However if the recent update is any indication this was either misinformation or no longer the case. See the PI World SF talk 2018 What’s New & Upcoming in Developer Technologies for details about performance for ingress and egress

• Since PI Server 2017 the PI Web API can be installed as part of the PI Asset Framework Services.

• Read and write support

• Rich set of methods to send and receive data

• Configuration information must be supplied to to the PI Web API - Requires knowledge of the PI Server, AF Server, AF database and location in the hierarchy and/or WebID. Searching can help here!

• WebID feature is perfect when you know where you want to send the data! See information on Web ID 2.0 in  2018 What’s New & Upcoming in Developer Technologies  that simplifies the use of WebID.

• Credentials used to authenticate with the PI Web API (or indirectly via claims) are used to authenticate to the PI System (assumes WEB API anonymous mode is not used).

 

Observations about the PI Connector for UFL - REST

• No configuration information required to be supplied to the PI Connector for UFL (REST) - i.e.: do not need to know about destination PI Server

• Requires configuration of a script to process data received by the Connector.

UFL scripting samples @ GitHub - osisoft/PI-Connector-for-UFL-Samples: Documentation and supporting files to demonstrate usage of the PI Connect…

• Credentials used to authenticate are not related to the back-end PI System(s) or Operating System.

 

Hmm, that was a lot of words, how about a table version:

 

Requirement/Consideration
PI Web APIPI Connector for UFL - REST
PerformancePI World SF talk:
2018 What’s New & Upcoming in Developer Technologies
includes ingress and egress performance data
Have heard of performance in the 10s of thousands
events/sec. Have reached out for more details
ComplexityRequires knowledge of where to send dataScript to parse UFL payloads
Edge Data Store and
OSIsoft Cloud Service support
NoNo*
PI Server object supportPoints, Elements, Attributes, Event Frames
KB article provides list of data objects  supported
Points, Elements, Attributes, Event Frames
Read and  write dataRead and writeWrite only
data collection component - script or
program - requirement
for information about the PI Server
(e.g.: Server name, AF database name)
Path - including server and object/WebIDNone
data collection configurationURL and token

URL and username/password

Script to parse received payloads

Data formatcontroller (API call) specificUser defined (as long as the UFL script engine can parse)
Securitytransport: https
authentication: Kerberos, Basic, Anonymous, Claims

transport: https

authentication: username/password

Software

PI Web API

Note: as of PI Server 2017 included in AF Service installation kit

PI Connector for UFL
Co$tPI System Access licenseSoftware license

* At least not today, stay tuned, Engineering are planning for Connectors (possibly not all) to support egress to these platforms.

 

 

PI Connector Relay with OMF support (REST endpoint). What, another REST endpoint?

 

Yes, but first some context. The "OSIsoft platform" now consists of:

  • OSIsoft Cloud Services (OCS) - public beta
  • PI Server
  • Edge Data Store (EDS) - private beta
  • Open Edge Module/FogLAMP - available on GitHub!

As a developer if there is a requirement to support sending data to one or the first three platforms, it could require multiple transformations of the data to support the specific endpoint.

This is one reason the OSIsoft Message Format specification - that consists of headers and messages to describe data - has been created. The PI Server, EDS and OCS all support ingress of OMF*, while FogLAMP and EDS support egress of OMF.

 

This week OSIsoft released the PI Connector Relay with OMF support, so that a user created application that formats a payload using the OMF specification can send data to the PI System, via the Relay.

The Relay exposes an endpoint that once configured using the PI Data Collection Manager can receive OMF messages from an application. The configuration process registers the OMF application with a PI Server, enabling configuration of location and prefixes for the Asset Framework and PI Data Archive respectively, and creates a token that is used by the application to authenticate with the Relay.

 

Observations about OSIsoft Message Format:

• (as mentioned) applications that use OMF can also send data to PI Server, EDS and OCS*

* Support for OMF may/will vary between PI Server, EDS and OCS because

- different platforms with different data type support, consult the relevant guides for more information (see reference below).

- as the OMF specification evolves a platform may support a newer version of the specification.

 

Observations about the PI Connector Relay with OMF support:

• no configuration information required to be supplied to the Relay, i.e.: does not need to know about the destination.

• a token is used to authenticate with the Relay from the client application, so the client application does not need to have any credentials that could access the PI Server

• client application must deal with buffering of data if/when necessary

• client needs to format messages as per the OMF specification

• while we do not yet have published numbers for Relay performance, lets just say it's looking good for certain scenarios - awaiting more details.

 

Want to know more about the OSIsoft Message Format? There is now a group OSIsoft Message Format (OMF) that includes links to guides for each of the endpoint platforms, relevant software links and a place to discuss and ask questions.

 

Which REST endpoint should I use if both PI Connector for UFL and PI Connector Relay are candidates?

So, a partner asked me this question and one way to answer is to consider where each method shines and what to be cautious about.

 

So where do each of these shine (and partially as a recap. of earlier points)?

 

PI Connector Relay with OMF support:

  • Minimal information required about the destination - Relay URL and token
    • perfect for a sensor/IoT device/application that you want to plug-in, configure and start data collection via the Relay
  • Support for OMF by EDS and OCS

 

PI Connector for UFL

  • Minimal information required about the destination for the sending application
  • Data is already in a file(s)/simple format that can easily be sent to the UFL REST endpoint
  • Data can be easily parsed by the UFL scripting language ("Martin Freitag scripting language(tm)")
  • PI Server object support - tags, assets, elements and event frames.

 

So what would be items to be cautious about each of these?

 

PI Connector Relay with OMF support

  • Formatting messages using the OMF specification may be more difficult than formatting for the PI Connector for UFL
  • Data objects supported by OMF, for example there is no type (yet!) that could create an event frame.

 

PI Connector for UFL

  • Requirement for a script/program to send data to the REST endpoint and a UFL script to parse the content.
  • Need to consider the complexity of the UFL script required to parse the file

 

Let's try the table format again:

Requirement/Consideration
PI Connector Relay with OMF supportPI Connector for UFL - REST
PerformanceEngineering currently performing testing

Have heard of performance in the 10s of thousands

events/sec. Have reached out for more details

ComplexityFormatting data using the OMF specificationScript to parse UFL payloads
Edge Data Store and
OSIsoft Cloud Service support
Yes*No***
PI Server object supportPoints, Elements, Attributes**Points, Elements, Attributes, Event Frames
Read and  write dataWrite onlyWrite only
data collection component - script or
program - requirement
for information about the PI Server
(e.g.: Server name, AF database name)
NoneNone
data collection configurationURL and token

URL and username/password

Script to parse received payloads

Data formatOMF specificationUser defined (as long as the UFL script engine can parse)
Securitytransport: https
authentication: token

transport: https

authentication: username/password

SoftwarePI Connector Relay (with OMF support)PI Connector for UFL
Co$tIncluded with a PI Server licenseSoftware license

* Check documentation for each platform as certain OMF types may not be supported across platforms

** Note: the OMF specification will evolve over time to include additional functionality.

*** At least not today, stay tuned, Engineering are planning for Connectors (possibly not all) to support egress to these platforms.

 

 

This article is not intended to definitively tell you how you should get data into the OSIsoft platform, instead the purpose is to show there are a number of tools in the toolbox and provide information about several considerations for each.

For further information about data ingress to the PI System, check out the documentation, conference presentations, speak with your local PI expert, partner or OSIsoft representative or ask questions right here in PI Square.

 

Cheers!

Hello,


Recently a new OSIsoft partner asked about SQL access to the PI System, specifically JDBC. This post lists a variety of information relating to JDBC that could be useful to the partner and maybe other folks too!

 

First off, a confession, I have been biased (ok, so a little bit of click bait, as there is also some wariness too!) against JDBC because:

1. different connection types required for points (if not referenced through Asset Framework (AF) and Asset Framework.

2, requires middleware

3. queries can be complicated

4. not good for retrieving *lots* of data

5. and I had little data on the performance.

So in general, instead would generally steer folks towards PI Web API or AF SDK.

 

That all stated, it was time to take a closer look and also get up to speed on new developments with the SQL family of products.

 

PI SQL 2018!

 

The good news is that a number of these items are being addressed in 2018! At PI World SF 2018 Bodo Bachmann presented on the next generation of SQL products and a number of the items above are being addressed! Relating back to the previous list, here are some updates:

1. and 2.. There is a new PI SQL engine that supports the PI Server and is part of AF Server 2018

3. Simplified query support

5. Improved performance.

 

Now these are not available today for JDBC (May 2018). OLEDB support will be available as part of PI Server 2018 with support for ODBC and JDBC later this year - see the link to the presentation for more details.

 

So, good to know for the future, what about the request from the partner who is looking to implement integration now?

 

What are the requirements?

 

Always good to step back and take a look at the big picture in terms of requirements and then options.

Requirements

- Linux based

- Using SQL technologies to access other on premise systems

Integration options:

KB00276 - What methods can be used to access PI data from Linux and UNIX systems?

JDBC is looking good, what about feature supported?

JDBC connections can make use of two possible middleware components:

- PI OLEDB Enterprise

- PI OLEDB Provider

that are covered in the PI Data Access feature comparison

OK, so the comparison is an eye chart, however does a fairly good job of comparing the functionality of the developer technologies.
(and I'm not just saying that because I'm one of the document contributors)

 

To Asset Framework or not to AF?

 

At this point it might be tempting to go with the OLEDB Provider because it keeps it simple by accessing only points (tags).

Not so fast Wonderwoman or Batman or <insert-reference-of-choice>, why not utilize the Asset Framework even if the customer is not using it because:

- Worth noting that points are accessible via OLEDB Enterprise too as long as they are referenced in Asset Framework (i.e: an attribute with a PI Point data reference).

- enable additional context to be made available to the partner application - units of measure, non-time series artifacts

- provide a standard view to be created in Asset Framework that can be applied across multiple customer sites

- utilize templates to simplify implementation.

While this will be some work up front, benefits can be realized longer term.

 

More JDBC please


Ok, so our new partner would like to get more information about JDBC and try it out. Where to go?

 

JDBC Requirements

Windows system to run the PI SQL DAS components (OLEDB software below)

Supported Linux system to run JDBC driver (the driver is also supported on Windows)

 

Software

There will be three software packages to download and install from the Tech support website > Products > PI Developer Technologies:
- JDBC driver

- PI OLEDB Enterprise

- PI OLEDB Provider (for testing purposes! See AF section above)

 

Once installed, a suggested place to start is on the Windows system and run PI SQL Commander Lite that includes a set of sample queries.

(In case the link fails https://livelibrary.osisoft.com  > PI Developer Technologies > PI SQL Commander Lite > Object Explorer > Run sample SQL queries)

 

References

JDBC driver manual

Virtual Learning Environment (labs)

If you are a PI Developer Club member (labs are included) or have purchased access to the learning labs you can access PI World event labs using a pre-configured Virtual Machine environment (once started a lab can be accessed for 24 hours).

There is a lab that includes the PI SQL Commander Lite and related PI SQL software (note the JDBC software is *not* installed): PI World 2018: Develop a PI JDBC Project to Exchange Data with Raspberry Pi Gadgets.

 

Cheers,