4 Replies Latest reply on Mar 25, 2015 6:35 PM by dng

    Can PI Web Services be used a a near-realtime data feed?

    fdeschenes

      Hello. I'm trying to get data from a set of PI servers into a Teradata data warehouse in a near-realtime mode. Specifically, I have 7 geographically placed (in the US) PI servers collecting a lot of data. From each of those 7 I need to push around 18,000 measurement values each minute (approx 300 tags from each of the PI servers at 1 second per tag rate) up to a Teradata data warehouse so it can maintain a rolling 24 hour set of 1 sec data. I've looked at several methods including PI-to-PI, OLEDB with Linked Servers, and the SAS PI connector as an intermediary. PI Web Services might be an elegant and fast-to-implement solution if it can handle the throughput. Web Services would be connecting to each of the 7 services every minute, pulling around 18,000 measurements, and building the XML messge structure to be consumed by Teradata. That's about 126,000 data values being generated per minute across the combined 7services. I don't have any concerns on the Teradata side. Is this reasonable for PI Web Services?
      Thank you!

      Frederick

        • Re: Can PI Web Services be used a a near-realtime data feed?
          dng

          Hi Frederick,

           

          Performance metrics will greatly depend on factors such as network latency, web server load, security overhead, etc. I encourage you to check out previous discussions related to the topic here:

           

          With that said, I believe your data rate is achievable. However, let me check internally whether we have any performance metrics, and whether alternative architecture will be more appropriate in this case. Please allow me get back to you for more information.

          • Re: Can PI Web Services be used a a near-realtime data feed?
            dng

            Hi Frederick,

             

            Thanks for letting me get back to you. I have discussed your case with the product manager and some developers internally. Since performance greatly varies based on the above-mentioned factors, we would like to find out more information about your environment and application. If you'd like, we are happy to arrange a phone call with you. Just let me know by commenting on this forum or messaging me directly. Thanks!

            • Re: Can PI Web Services be used a a near-realtime data feed?
              Roger Palmen

              Some other ideas to consider:

              • PI Web API. The bulk calls in the latest version should allow the required performance. And the REST interface is easy to integrate. I would prefer this over the heavy SOAP of the PI Web Services.
              • Long term option: Project CAST is designed to push data from PI to analytics platforms (SAP HANA, Microsoft Machine Learning, etc.). Not sure if this is sufficient for near-realtime though.
              • Bad option, although most powerful: create a custom solution using AFSDK.

               

              PI OLEDB Enterprise, while it should be able to reach the throughput, my experience is that it is difficult to keep latency down to make it near-realtime.

              • Re: Can PI Web Services be used a a near-realtime data feed?
                dng

                Hi Frederick,

                 

                Does Teradata only have a SOAP adapter but not a REST one? If not, take a look at PI Web API as Roger has pointed out. Our current development process are focused on PI Web API rather than PI Web Services.

                 

                I looked at some performance metrics on PI Web Services, the data rate you specified is definitely achievable across local network. However, your environment involves remote PI Data Archives across WAN, and the performance is a little more difficult to predict. One thing you could do is to put a PI Web Services installation on each site and make SOAP calls from the central office. This would distribute the work among all sites. The alternative is to put the PI Web Services installation in the central location. This would mean PI SDK communication between the central warehouse and the various sites. We tend to be “chatty” over the wire at that level which has been a problem in wide-area deployments before.