16 Replies Latest reply on Sep 26, 2012 5:25 PM by bklimaszewski

    Loading Online XML Data into PI

    bklimaszewski

      Hi all,

       

      I am trying to pull some data into PI from an online XML data feed (e.g., see here) and was informed that this would be possible with the help of PI Web Services and some coding in VB or C#. The intent is either to backfill the full set of data displayed in the XML file into the archive, or at least pull the latest data value and timestamp. Having never coded for Web Services before, could anybody give me a sense for how involved this would be and/or where I may find some sample codes (preferably in VB, since I have not coded in C# before)?

       

      I appreciate any help you can offer !

       

      -Vincent

       

      P.S.: I have PI Web Services 2010 R3 installed on a Windows 2008 R2 box with PI Server 380 and IIS7.5 and everything appears to be operational.

        • Re: Loading Online XML Data into PI
          Bhess

          Vincent,

           

          PI Web Services does support inserting and updating time series data, but I am wondering if an interface might be a better fit for your use case.

           

          Does your data feed update regularly, and will you want to capture those updates on an ongoing basis?  If so, I'd recommend using the OSIsoft XML Interface, which can be configured to grab the XML data feed from a URL at a set interval.  The only coding you'd need to do is an XSL transform to translate from the feed's format into the OPC XML-DA format, which is pretty straightforward even if you've never done it before.  I can provide some samples to get you started if you're interested.

           

          -Brad

            • Re: Loading Online XML Data into PI
              Ahmad Fattahi

              On top of Brad's suggestions, which I totally agree with, I'd like to give you a sense of programming with PI Web Services. It is not complicated and it provides access to PI Server data using simple and intuitive methods. There is documentation in vCampus Library. Also, I suggest you to take a look at the PI Application Development Course in vCampus Training Center to get familiar with the technology should you decide or need to go beyond the interface.

                • Re: Loading Online XML Data into PI
                  bklimaszewski

                  @Brad - That's great! The feed does update regularly and we will want to capture those updates on an on-going basis. As you might have seen in the URL I linked to in my original post, the time frame of data is actually part of the URL. Would this be an issue in using the OSIsoft XML Interface? If not, if you could provide some samples that would be much appreciated :).

                   

                  @Ahmad - thanks! I took a look in the vCampus Library and retrieved the "Programming with PI WEB Services" PDF. Is that the one you were referring to?

                    • Re: Loading Online XML Data into PI
                      Bhess

                      @Vincent--  Well, actually the interface does expect that the URL stays constant.  You could technically set some variables in the interface configuration batch file to add those timestamps in, but you'd have to work around two problems.  First, since the batch file is loaded once, and the variables would be computed only once, when the interface is started, you'd have to plan around the fact that your timestamps would probably get "old" the longer the service is running.  Second, you'd no longer be able to edit/update the batch file in the Interface Configuration Utility, which would complicate any configuration changes down the road.

                       

                      However, for the URL you provided, it looks like if you don't supply a start and end time, that it defaults to the values for the current day.  Further, it appears that if you omit the end time but supply a start time, then you'll get all the values between the supplied start time and the current date.  So the XML interface might still be able to meet your needs.

                        • Re: Loading Online XML Data into PI
                          bklimaszewski

                          Brad,

                           

                          That should work indeed, assuming the interface continuously checks for new XML data at some pre-defined interval and will skip / overwrite existing values in PI.

                           

                          I downloaded the latest XML ICU installer from here as well as the HTTP reader plug-in. Running them both on my server, I appear to be missing the executable necessary to configure the interface in ICU. Any thoughts? And could you post the sample XSL transforms you had alluded to earlier?

                           

                          Thanks :),

                           

                          -Vincent

                            • Re: Loading Online XML Data into PI
                              Bhess

                              Vincent,

                               

                              Looks like the links there are actually missing the installer for the interface itself.  You can find the full setup kit on the downloads page here (search for "XML").  Sorry for the confusion, I dropped a note to the responsible party to get that cleaned up.

                               

                              Here's a zip file containing a sample xml and xslt file that I'm using in an XML interface today.  It's a use case that's a little bit simpler than yours; I'm extracting a single value from an xml document in real time and storing it in the PI Data Archive.

                               

                              There's two key changes that you'll need to make:

                              • First, since you have multiple values to extract from a single document, you'll need to use either an xsl:for-each or an xsl:template to ensure that you have multiple values in the result
                              • Second, you'll need to include the timestamp in the result.  My data is real-time, so I'm not including any timestamps from the source in the result.  

                              Ultimately, you're going to be looking for output that looks like this:

                               

                              <ReadResponse xmlns="http://opcfoundation.org/webservices/XMLDA/1.0/">

                               

                                <RItemList>

                               

                                  <Items ItemName="Flow" Timestamp="2012-04-20T00:00:00">

                               

                                    <Value>2890.000</Value>

                               

                                  </Items>

                               

                                  <Items ItemName="Flow" Timestamp="2012-04-20T02:00:00">

                               

                                    <Value>2687.400</Value>

                               

                                  </Items>

                               

                                </RItemList>

                               

                              </ReadResponse>

                               

                              Finally, since xsl tends to be an interative/trial-and-error process, I would highly recommend ironing out any kinks with your transform to match it with the sample above before you configure your interface.  Microsoft provides a small tool to do it at the command line here, or you can use the capabilities built into Visual Studio.

                               

                              Brad

                              • Re: Loading Online XML Data into PI
                                Ahmad Fattahi

                                Vincent (Brian?),

                                 

                                Yes, that is the documentation for PI Web Services with multiple examples. There is also many threads on vCampus as well if you are looking for specific answers; you can search for them in here. Obviously you can post your programming-related questions here as well if the answer could not be found elsewhere.

                                  • Re: Loading Online XML Data into PI
                                    bklimaszewski

                                    @Brad - Thank you for posting the sample and providing a lot of background, that was very helpful! I tried my luck at writing the XSL code and am running into some issues in navigating through the different XML / node levels. The date and times I need appear to be at the /timeSeriesResponse/timeSeries/values/value level, but I am unsure how to navigate there. I saw that in your sample case you use the path expression  //namespace:QName (//yweather:condition), but I wasn't sure how to replicate this.

                                     

                                    I tried two different approaches to navigate to / access the right level: i) using the <xsl:template> and ii) specifying the absolute path in the value statement.

                                     

                                    Approach i) My understanding is that <xsl:template> uses XPath statement, correct? I used this website to develop my XPath and write my code as shown below. As you can see, it unfortunately did not produce the desired output.

                                     

                                    4454.01.PNG

                                     

                                     

                                     

                                    Approach ii) I kept the <xsl:template> at the root and instead wrote the absolute path of the property into the <xsl:value-of> statement. This time, no value was returned.

                                     

                                    6557.02.PNG

                                     

                                    Appreciate any additional help you can provide!

                                     

                                    -Vincent

                                      • Re: Loading Online XML Data into PI
                                        Bhess

                                        You're close with the first approach.  Here's a few changes I suggest:

                                        • First off, in a simple xslt, you'll always want the xsl:template to match the root node.  That's why you're getting the funky unstructured output; if an element is hit in the source document but doesn't match a template, its value is going to be written straight to the output.
                                        • Second, you'll need to get your namespaces in order. The "xmlns" in your source document refers to the waterML namespace.  But your transform doesn't have a default namespace.   This means you'll need to set a namespace prefix for waterML in your transform and use it in your XPath expressions.  (NB:  It's a best practice to avoid using a default namespace like xmlns="http://sample" in the xsl:stylesheet element of your transform.  In a small transform, it might be easy to work with, but in a large transform that's thousands of lines long and includes multiple namespaces, it can quickly get out of control.)
                                        • Finally, I'm guessing you'll be interested in iterating through all of the data values, not just the first in the list.  xsl:for-each is the most intuitive way to do this, though it can be done using multiple xsl:templates.

                                        Here's a sample that I think gets you close, if not final:

                                         

                                        0820.SimpleTransform.png

                                          • Re: Loading Online XML Data into PI
                                            bklimaszewski

                                            Very cool! Thank you very much, especially for giving so much background around each item! You don't know how much I appreciate your help and time!

                                             

                                            Two brief follow-up questions if you don't mind: it appears that contrary to my original assumption of needing to specify an absolute path to the XML element in the overall tree structure, you can use the //step command to jump directly to the desired element. With the code you provided in mind,

                                            • What would happen if the XML source had two or more <values> elements at different levels within the overall tree structure? Would the code default to the first one in absence of an absolute reference?
                                            • Could the <xsl:for-each select="//wml:values/wml:value"> statement not be trimmed down even further to <xsl:for-each select="//wml:value">? Or is it perhaps just good practice to provide at least one level of parent/child in referencing values?

                                            By the way, when I followed the link to the Downloads page you provided, it defaulted to the "PI OPC Interface" product group. Was I supposed to change this? I tried searching for XML within it as well as within the overall "PI Interfaces" product group and unfortunately was not returned any results (see below). Any idea why this may be?

                                             

                                            0167.Download-Center.PNG

                                             

                                            Thank you again!

                                             

                                            -Vincent

                                              • Re: Loading Online XML Data into PI
                                                Bhess

                                                The best way to think about XPath is as a "matcher".  //wml:values/wml:value will match *all* the elements in your document that have that structure, even if there are multiple instances of that structure.  This actually drives the answer to your second question.  I have a habit of always providing a couple of levels so that the XPath doesn't inadvertently match something I didn't intend it to.  But in your case, a simple match on //wml:value would work.

                                                 

                                                Sorry about the link to the install kit, though if you chose product "PI Interfaces" from the product dropdown, you should have seen an entry called "XML (xml) Interface -- Install Kit".  If you're not seeing it, I'd recommend a call in to our Tech Support line.  As a Developer, I don't know all the nuances as to how the download site works in relation to product licensing, though my understanding is that with a vCampus subscription, you should have access to everything.

                                                  • Re: Loading Online XML Data into PI
                                                    bklimaszewski

                                                    That makes sense, thanks Brad! I'll follow-up once I have the XML interface. Thanks again!

                                                      • Re: Loading Online XML Data into PI
                                                        bklimaszewski

                                                        Brad,

                                                         

                                                        I have the XML interface up and running, thanks again for your help!

                                                         

                                                        One brief follow-up question: are the XML interface and associated plug-ins still actively being supported / updated by OSI? I understand, for example, that the PI HTML interface is no longer being updated having been replaced by PI Web Services.

                                                          • Re: Loading Online XML Data into PI

                                                            Hello Brian,

                                                             

                                                            The interface search page at OSIsoft TechSupport (intsearch.osisoft.com/interfaceSearch.aspx) helps you finding the answer to your question for a lot connection standards.

                                                             

                                                            We differ between "Standard" and "Non-Standard" interfaces. Those interfaces showing in "Maintenance" status are no longer sold but maintained for customers that have licensed it already. Maintenance mainly refers to fixing bugs.

                                                             

                                                            Please also pay attention to the comments for Interfaces showing in "Maintenance" status since there are usually other standards as alternatives mentioned. If you are unsure, please do not hesitate contacting OSIsoft TechSupport for clarification.

                                                             

                                                            PI Interface for XML (part # PI-IN-OS-XML-NTI) and PI Interface for HTML (PI-IN-OS-HTML-NTI) are currently showing as "Standard" interfaces.

                                                             

                                                            Gregor

                                                            • Re: Loading Online XML Data into PI
                                                              hanyong

                                                              Brian Klimaszewski

                                                              I understand, for example, that the PI HTML interface is no longer being updated having been replaced by PI Web Services.

                                                               

                                                              Just like to further clarify on the above statement, PI Web Services could not have been a replacement product for PI HTML interface. PI Interfaces are designed to collect data from different data sources into the PI System, whereas PI Web Services as part of the PI Data Access suite is meant to expose data in the PI System to other applications or system.