We’ve been working on a new version of PI Connector for UFL for the last couple of months. The new version brings better performance, more efficient resource management, and couple of highly requested features (native JSON and FOREACH construct support). Feel free to download and try it out!

 

What’s new in PI Connector for UFL 1.2?

  • More efficient processor consumption & faster creation of AF structure
  • REST Client Data Source supports bulk calls by URL parameters in a pipe separated list
  • ForEach() construct for stepping through collections in JSON and CSV formats
    • Native support for parsing JSON
  • Bulk function for sending events – StoreEvents()
  • New ToString() function

 

Bellow I would like to show & explain the main new features. For the full reference please read the User Guide and take a look to the GitHub examples (https://github.com/osisoft/PI-Connector-for-UFL-Samples).

 

How to process JSON data with PI Connector for UFL?

The following JSON contains information like “timestamp”, “rowType” etc. on the root level. These keys value can be easily parsed by using the function JsonGetValue([JSON_AS_STRING], [KEY]). But tagnames and values are inner JSONs in “channels” array[], so we need to iterate through it by using FOREACH(JSONGetItem([JSON_AS_STRING], [KEY[]])). New “__ITEM” field is then populated by the inner JSONs (ex: {"tag":"controller.stats.idleTime_hours","value":"194.806808"}) one by one.

 

Data:

{"timestamp":"1433409463","rowType":"data","rigName":"ericg03","toolName":"Power_System3","channels":[{"tag":"controller.stats.cpuUsage_percent","value":"8.657751"},{"tag":"controller.stats.idleTime_hours","value":"194.806808"},{"tag":"controller.stats.logSizeKB","value":"6.48123"},{"tag":"controller.stats.upTime_hours","value":"213.45"3796]}

 

INI:

[FIELD]
FIELD(1).NAME="TagName"
FIELD(2).NAME="Value"
FIELD(2).TYPE="Number"
FIELD(3).NAME="TimeStamp"
FIELD(3).TYPE="DateTime"
FIELD(3).FORMAT="SECONDS_GMT"
FIELD(4).NAME="Counter"
FIELD(4).TYPE="Int32"

[MSG]
MSG(1).NAME="Data"

[Data]
Data.FILTER=C1=="*"

TimeStamp = JsonGetValue(__MESSAGE, "timestamp")

FOREACH (JsonGetItem(__MESSAGE, "channels[]")) DO

     TagName = JsonGetValue(__ITEM, "tag")
     Value = JsonGetValue(__ITEM, "value")

     StoreEvent(TagName, ,TimeStamp, Value)
ENDFOR

 

How to process CSV data with PI Connector for UFL?

Parsing CSV with the current UFL can be quite difficult, especially when there's one line of tag names and several lines of values - I've seen INI files with thousands of lines. Since we want to make configuration as easy as possible, we have added FOREACH and Collection support to the INI capabilities. Take a look to the data bellow, the easiest way to process such a CSV is to have a collection of tagnames, collection of values and one field for a timestamp. That's exactly how it works now. With those collections it's possible to call StoreEvents method where strings in tagnames collection match the order and number of items in the value collections. Thanks to this UFL Connector can pair it together with timestamp to create an event.

 

DATA:

TimeStamp,TagCol1,TagCol2,TagCol3,TagCol4,TagCol5,TagCol6,TagCol7,TagCol8,TagCol9,TagCol10,TagCol11,TagCol12,TagCol13,TagCol14,TagCol15,TagCol16,TagCol17,TagCol18,TagCol19,TagCol20,
7/12/2017 6:10,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
7/12/2017 6:20,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
7/12/2017 6:30,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
7/12/2017 6:40,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20

 

INI:

[FIELD]
FIELD(1).NAME="TagNames"
FIELD(1).TYPE="Collection"
FIELD(2).NAME="Values"
FIELD(2).TYPE="Collection"
FIELD(3).NAME="TimeStamp"
FIELD(3).TYPE="DateTime"
FIELD(3).FORMAT="M/dd/yyyy h:mm"
FIELD(4).NAME="Counter"
FIELD(4).TYPE="Int32"
FIELD(5).NAME="Value"
FIELD(5).TYPE="Number"

[MSG]
MSG(1).NAME="Tags"
MSG(2).NAME="Data"

[Tags]
Tags.FILTER = C1=="T*"

TagNames = Clear()

FOREACH (CsvGetItem(__MESSAGE, ",")) DO

     IF(Counter > 0) THEN
          TagNames = Add(__ITEM)
     ENDIF
ENDFOR

[Data]
Data.FILTER = C1=="*"

Counter = 0
Values = Clear()

FOREACH (CsvGetItem(__MESSAGE, ",")) DO

     IF(Counter == 0) THEN
          TimeStamp = __ITEM
     ELSE
          Value = __ITEM
          Values = Add(Value)
     ENDIF

     Counter = Counter + 1
ENDFOR

StoreEvents(TagNames, ,Timestamp, Values)

 

REST Client Data Source BULK calls

With REST Client, HTTP GET method can be executed to bring in content from a remote server. It can be a REST Server or for example an IOT device. But what if there's multiple IOT devices? How to get data from all of them with just one datasource configured in PI Connector for UFL? New support of “UFL_PLACEHOLDER” keyword makes it easy. The following string could be used as a DS address:

 

     UFL_PLACEHOLDER:6894/RESTServiceImpl.svc/json/ |10.105.0.1|10.105.0.2|10.105.0.3

 

As a result 3 GET requests are executed each time. The responses are processed by your INI file, subsequently tags and AF structure get created for all three listed devices.

 

There is much more, just try it and let us know

------------------------------------------------------------------------------------------------------------------------------------------

FULL LIST OF CHANGES IN CURRENT BETA VERSION

Enhancements:

    WI# 147909 - Stepping through collections in Json and Csv formats via the ForEach() construct.

    WI# 149795 - Implementation of the ToString() function.

    WI# 149933 - Implementation of the StoreEvents() function.

    WI# 153597 - REST Server Data Source Type performance optimization for frequent inputs coming from many clients (IoT scenario).

    WI# 164127 - REST Client Data Source Type now supports replacing parts of the end point address by items in a pipe separated list.

    WI# 164148 - The structured input (json) can now be read as one line when Word Wrap = -1.

Bug Fixes:

    WI# 162882 - The CHAR(#) function converted numbers to ASCII for numbers between 0 and 127. This has been changed and now the CHAR(#) function recognizes every whole number from 0 to 255.

    WI# 162883 - The PRINT() function now correctly handles sub-milliseconds.

    WI# 176099 - For performance reasons, data files larger than 256 MB will not be zipped after processing.