AnsweredAssumed Answered

Best Practice for PI Web API StreamSets Update?

Question asked by nDimensional on Feb 1, 2019
Latest reply on Feb 4, 2019 by msingh

I'm try to get PI Point streaming working with PI Web API StreamSets Update. I'm using Java client generated from the OSISoft OpenAPI spec. After a few small glitches, I am able to get the basic streaming to work following this simple C# example from Malvika Singh (Stream Updates in PI Web API ). Here is my prototype in Scala & program output from a sample run (sorry, if this is too long). As you can see, I'm query updates for two demo tags (SINUSOID & SINUSOIDU) against a dev/test PI Server installation. There are a few questions in the code. I would like to get some advices on how to best use the feature as well as the generated Java Client API. Note, I'm quite new to PI system. I beg your pardon if some of the questions may not be specific to the StreamSets Update and it might be have already been answered some where else.

 

1) First, for this PoC, I only have 2 tags (streams). What if I have large number of tags? The API call will result in URL definitely exceeds the URL limit? In that case, should I be using BatchApi rather than StreamSetsApi? Or I could have multiple parallel queries each handle a subset of streams using StreamSetsApi and I manage the multiple queries in my application logic?

2) As you can see in my code, I passed a null webIdType. My understanding is that we maybe able to use a shorter version of webId which can significantly reduce the length of resulted URL. How do I get the shorter version of webId? Can I get it from the longer full version of the webId? (this definitely is not a new question I think)

3) You can also see from the output of the program, it contains lots of information (quite verbose). Not all of them are useful to my application. My applications maybe only case about:

the latestMarkers, status, event timestamps, event value. Maybe I will need PaginationLinks if we have too many updates (leave it out if the updates fit in a single page). I don't think I even need the source field since it is just echo the passed in list of webIds in the same order. I noticed that there is a selectedFields, how should I set this in my example?

4) It looks like event with Stream Update, we still need to periodically pull the PI Web AI Server using the latestMarkers. Besides we will have similar issues with large number of marks which will cause URL too long (see question #1), we also need to specify a pull period. How do we determine the pull period? What is general rule of thumb when we pick this parameter?

 

Thanks for any advices/guidances?

 

----------------------------------------------------------Code & Program output below---------------------------------------

 

object PIWebAPITest extends App {

//PI Web API WIN-R9EI8TJ50S0
val url = "https://x.x.x.x/piwebapi"
val user = "xxxx"
val password = "xxxxx"

val client = new ApiClient()

client.setBasePath(url)

client.setUsername(user)

client.setPassword(password)

client.setVerifyingSsl(false) // We use a self signed certificate at the moment turn off the SSL verification.

val systemApi = new SystemApi(client)

 

println("Trying get PI system version!")

Try(systemApi.systemVersions()) match {

case Success(v) =>

println(v)

case Failure(e) =>

println(e.getMessage)

println(e.getStackTrace.mkString("\n"))

}

 

//This is required for bypassing CSRF defense for POST requests. Could we add this header for POST only?
client.addDefaultHeader("X-Requested-With", "foo")

 

val streamSetApi = new StreamSetApi(client)

 

//PI Points for \\\\WIN-R9EI8TJ50S0\\SINUSOID & \\\\WIN-R9EI8TJ50S0\\SINUSOIDU
val webIds = List("F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE",
"F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ")

 

//How do we use the selectedFields? If I only want timestamp and value?

//How do we handle large number of PI Points which definitely will cause URL exceeds the maximum length? Using BatchApi controller?
//How do we use webIdType for shorter webId to keep the URL short?
val itemsStreamUpdatesRegister = streamSetApi.streamSetRegisterStreamSetUpdates(webIds.asJava, null, null)

 

val items = itemsStreamUpdatesRegister.getItems().asScala.toList

var marks = items.map(_.getLatestMarker).asJava

 

println(itemsStreamUpdatesRegister)

 

val delay = 0L
//How do we determine an optimal pull period?
val period = 100000L

val t = new java.util.Timer()

val task = new java.util.TimerTask {

def run() = ReceiveUpdates
}

t.scheduleAtFixedRate(task, delay, period)

 

scala.io.StdIn.readLine()

 

def ReceiveUpdates() = {

//Closure on marks which will be updated here.
val itemsStreamUpdatesRetrieve = streamSetApi.streamSetRetrieveStreamSetUpdates(marks, null, null)

println(itemsStreamUpdatesRetrieve)

println("Press any key to exit")

marks = itemsStreamUpdatesRetrieve.getItems().asScala.toList.map(_.getLatestMarker).asJava

}

}

 

Here is the program output looks like:

 

Trying get PI system version!

....

 

class ItemsStreamUpdatesRegister {

items: [class StreamUpdatesRegister {

status: Succeeded

source: F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE

sourceName: SINUSOID

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOID

latestMarker: 21d7e5ece38946f6a3a663c79f7bd356_0

exception: null

}, class StreamUpdatesRegister {

status: Succeeded

source: F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ

sourceName: SINUSOIDU

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOIDU

latestMarker: fe7354e878604bd49dfbd431e8907494_0

exception: null

}]

links: class PaginationLinks {

first: null

previous: null

next: null

last: null

}

}

...

class ItemsStreamUpdatesRetrieve {

items: [class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE

sourceName: SINUSOID

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOID

requestedMarker: 21d7e5ece38946f6a3a663c79f7bd356_0

latestMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

status: Updated

events: [class DataPipeEvent {

action: Add

timestamp: 2019-02-01T20:30:45Z

unitsAbbreviation:

good: true

questionable: false

substituted: false

annotated: null

value: 1.620164

errors: null

webException: null

}, class DataPipeEvent {

action: Add

timestamp: 2019-02-01T20:31:15Z

unitsAbbreviation:

good: true

questionable: false

substituted: false

annotated: null

value: 1.56553769

errors: null

webException: null

}]

exception: null

}, class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ

sourceName: SINUSOIDU

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOIDU

requestedMarker: fe7354e878604bd49dfbd431e8907494_0

latestMarker: fe7354e878604bd49dfbd431e8907494_0

status: NoChange

events: []

exception: null

}]

links: class PaginationLinks {

first: null

previous: null

next: null

last: null

}

}

Press any key to exit

class ItemsStreamUpdatesRetrieve {

items: [class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE

sourceName: SINUSOID

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOID

requestedMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

latestMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

status: NoChange

events: []

exception: null

}, class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ

sourceName: SINUSOIDU

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOIDU

requestedMarker: fe7354e878604bd49dfbd431e8907494_0

latestMarker: fe7354e878604bd49dfbd431e8907494_0

status: NoChange

events: []

exception: null

}]

links: class PaginationLinks {

first: null

previous: null

next: null

last: null

}

}

Press any key to exit

class ItemsStreamUpdatesRetrieve {

items: [class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE

sourceName: SINUSOID

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOID

requestedMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

latestMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

status: NoChange

events: []

exception: null

}, class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ

sourceName: SINUSOIDU

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOIDU

requestedMarker: fe7354e878604bd49dfbd431e8907494_0

latestMarker: fe7354e878604bd49dfbd431e8907494_0

status: NoChange

events: []

exception: null

}]

links: class PaginationLinks {

first: null

previous: null

next: null

last: null

}

}

...

class ItemsStreamUpdatesRetrieve {

items: [class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAQAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lE

sourceName: SINUSOID

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOID

requestedMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

latestMarker: 21d7e5ece38946f6a3a663c79f7bd356_2

status: NoChange

events: []

exception: null

}, class StreamUpdatesRetrieve {

source: F1DPIU3z49a1xk2APeTtzCy29gAgAAAAV0lOLVI5RUk4VEo1MFMwXFNJTlVTT0lEVQ

sourceName: SINUSOIDU

sourcePath: \\WIN-R9EI8TJ50S0\SINUSOIDU

requestedMarker: fe7354e878604bd49dfbd431e8907494_0

latestMarker: fe7354e878604bd49dfbd431e8907494_2

status: Updated

events: [class DataPipeEvent {

action: Add

timestamp: 2019-02-01T20:38:45Z

unitsAbbreviation:

good: true

questionable: false

substituted: false

annotated: null

value: 0.8573376

errors: null

webException: null

}, class DataPipeEvent {

action: Add

timestamp: 2019-02-01T20:39:15Z

unitsAbbreviation:

good: true

questionable: false

substituted: false

annotated: null

value: 0.817578

errors: null

webException: null

}]

exception: null

}]

links: class PaginationLinks {

first: null

previous: null

next: null

last: null

}

}

Press any key to exit

...

Outcomes