10 Replies Latest reply on Mar 31, 2017 6:57 AM by abrodskiy

    Set URL parameters in UFL Connector

    abrodskiy

      I'm trying to make the new PI UFL Connector read data from a RESTful web service from a system which holds 3rd party time series data.

      The web service requires URL parameters to be set to retrieve time series data for each stream in the format like the one below.

      Essentially, streamid there is a sort of a "Tag", and there is a need to specify stard and end time for each call.

      https://server/Data.ashx/GetPMACData?{"streamId":"1001_1","start":1388549700000,"end":1388556000000}

       

      Reading all documentation and examples on GitHub i've not seen any cases where the URL parameters would be manipulated.

      So how do I set these parameters for each call of the PI UFL Connector?

        • Re: Set URL parameters in UFL Connector
          Marcos Vainer Loeff

          Hi Alexander,

           

          I am not very familiar with PI Connector for UFL to be honest. But I have installed it today and by looking at the Data Source configuration on the administrative web site, it is pretty clear to me that the URL for the REST client is fixed and not dynamic. As a result, you need to create or modify a data source for each request you want to do it, which doesn't make many sense to me.

           

          Do you have any experience with programming? By creating a C# application using PI Web API instead of PI UFL Connection, you can generate a different URL for each request.

           

          Hope this helps!

          1 of 1 people found this helpful
            • Re: Set URL parameters in UFL Connector
              rsun

              I agreed with Marcos Vainer Loeff. I don't think the url can be dynamic in the PI Connector for UFL. Would you be able to make one URL link with all the data in it?

                • Re: Set URL parameters in UFL Connector
                  abrodskiy

                  No, it's 3rd party web service, not something we have control on.

                  Imagine you would have to get data out of PI Web API. This would be very similar in approach - in order to get any data out, you would have to specify URL parameters like web ID, start/end time, etc.

                    • Re: Set URL parameters in UFL Connector
                      rsun

                      PI Interface for HTML supports dynamic generate URLs.

                      https://techsupport.osisoft.com/Downloads/File/ef53fc60-9cc4-4ea6-8b0f-9e13e5b2be76

                      Could this be a solution for you?

                       

                      Dynamic URL Generation

                      Dynamic URL generation is useful when there is a page you are trying to read on a regular basis, whose URL changes every so often. For example, if there is a page that has today’s weather, and the date is part of the URL, a dynamic URL will need to be generated. So, if the URL for that page looks like: http://www.yourfavoriteweathersite.com/OAK_08_12_2002.html this will obviously be different for each day the weather needs to be read from the site. With dynamic URL generation, the plug-in is given a “dummy” URL that the user specifies in the HTML locator script, which is configured using the PI ICU or the simpler PI Interface for HTML configuration tool. Continuing the weather example, this URL could be: http://www.yourfavoriteweathersite.com/OAK_[month]_[day]_[year].html. The plug-in would then be responsible for making any text substitutions in this URL. So the plug-in could look for [month] and replace it with the current month, and so on.

                       

                      Ricky

                        • Re: Set URL parameters in UFL Connector
                          abrodskiy

                          where would I get the tag names from for substitution and how would I use these two interfaces together?

                          Ideally, I would like UFL connector to work similar to PI RDBMS interface - PI Tags should drive the data retrieval.

                          E.g. aliases in PI tags would tell what data to call for. And TS parameter would tell the connector last Timestamp of retrieval for each tag.

                          Then the connector would build URLs as required. Pretty much like the RDBMS interface does with SQL queries.

                            • Re: Set URL parameters in UFL Connector
                              rsun

                              PI Interface for HTML and PI Connector for UFL don't work together. In this case, I think none of them works for you.

                              What is your data source name? we may be able to find a PI Interface that can take data from it.

                              Thanks.

                                • Re: Set URL parameters in UFL Connector
                                  abrodskiy

                                  Our client has a few 3rd party data sources, living on the clouds, exposing only RESTful webservices, that we need to consume to get data into PI.

                                  To be honest, i'm quite surprised that the UFL Connector doesn't support what seems to be the very basic functionality for every RESTful webservice - manipulating URL parameters to make requests.

                                  Imaging that you would have to interrogate PI Web API from PI UFL Connector. This would be very similar in nature.

                                  Is there any way to do it out of the box, without going wild programming in python or elsewhere (and thus defeating the purpose of the UFL connector all together in that case)?

                                  1 of 1 people found this helpful
                            • Re: Set URL parameters in UFL Connector
                              mtippett

                              Hi Alexander,

                               

                              I've had the same issue as you in the past and I what I've done is write a script (say in powershell or python) which executes the query to the web service and then either writes the response to a file for the UFL connector to read or posts the data to the UFL connector acting as a REST endpoint.  The script itself is triggered by the windows task scheduler to run as frequently as desired.

                               

                              It's not as neat as using the connector alone as you need to maintain the script as well as the connector, but it gets the job done.  Below is a very simple example which gets data from a public REST endpoint where the query parameters are in the URL as in your case and writes the response to a file.  If you were going to do something like this in a production environment I would recommend that you add some error handling to your script as it is a key part of the data flow.

                               

                              Function Get-CountryData($location)
                              {
                                  $destinationFolder = ".\"
                                  $baseUrl = "https://restcountries.eu/rest/v1/alpha/"
                                  $query = ($baseUrl + $location)
                              
                              
                                $response = Invoke-WebRequest -Uri $query
                                Out-File -FilePath ($destinationFolder + $location + ".json") -InputObject $response.Content
                              }
                              
                              
                              $countries = "LIE","JPN","AUS","NZ","FRA","USA","CAN","SG","ARG","IND"
                              Foreach($country in $countries)
                              {
                                  Get-CountryData($country)
                              }
                              
                              4 of 4 people found this helpful
                                • Re: Set URL parameters in UFL Connector
                                  abrodskiy

                                  Hi Michael,

                                   

                                  Thanks, we also resorted to follow this approach with the exception that we used PI Web API instead of UFL Connector - as we need to read tags from PI first, get the Site IDs for the 3rd party service (stored them in instrument tag), also check last snapshot timestamp of each tag and based on that prepare a request to 3rd party web service, providing Site ID, start time (last snapshot timestamp - to properly sync the data sync last scan), current time as end time.

                                  Pushing data to PI also appears to be easier via PI Web API in this case, as the WebIDs are already pre-cached from the above steps, so pushing via streamsets arrays of data is just naturally easier. Done this all in python in the end...

                                   

                                  Hoping this functionality will soon be available in the PI UFL Connector, so that we can use it to pull the data from any 3rd party web service into PI.

                                   

                                  Thanks and regards,

                                  Alex Brodskiy

                                  2 of 2 people found this helpful