8 Replies Latest reply on Apr 7, 2015 2:25 PM by dng

    How to send data from AF to the cloud.

    jose_dai

      Hi,

       

      I'll create  a new demo where I'm going to create a new AF structure receiving tag data from Random values, after that I wish put this data into the cloud(not using PI Cloud), and a third party could take these values from the cloud for their own proposes.

       

      What is the recommendation or examples for developing an application with af sdk. I hear suggestions.

       

      Regards.

        • Re: How to send data from AF to the cloud.
          dng

          Hi Jose,

           

          There could be multiple ways to accomplish what you wanted. First, please help me understand a little more about your application.

          • Is your application primarily responsible to publish the data into some services in the cloud? Is there other things that the application are responsible to achieve?
          • Do you need to maintain historical data in the cloud? Or do you just need a way to access on premise PI data through the web?
          • What data format does the third party tool take?

           

          The reason I am asking the above questions is because we have different developer tools that serve different purposes. Depending on your use case, we can look at which tool best suits your application.

          • PI AF SDK is a .NET library that can be used to access PI and AF data. It is useful if you want to build a .NET application. There are a lot of examples regarding how to build applications using PI AF SDK. A good start would be the tutorials offered in Master PI (Developing Applications with PI AF SDK) and the programming references that get installed with the PI AF Developer Tools (PI AF SDK Programming References).
          • PI Web API is a REST-based data access product, targeted at providing cross-platform, multi-user programmatic access to PI System Data. You can build an application that publish PI data into your cloud service using PI Web API. Alternatively, if your third party product can directly interact with a REST endpoint, you can directly get PI/AF data through PI Web API, without having to publish it on a separate service.


          FYI, I have moved this thread over to PI Developers Club > PI System Architecture for continued discussion.

          • Re: How to send data from AF to the cloud.
            bshang

            Hi Jose,

             

            Could you describe in more detail what your cloud infrastructure is like? Are you using Azure? Is it a publisher-subscriber model (message queues) or are you sending to another historian in the cloud or both? As Daphne mentioned, the most natural approach may be to use PI Web API, which exposes an HTTP/S endpoint for reading/writing data to/from PI.

             

            Although you mention sharing it with third-parties, I should let you know about the new PI Cloud Connect, which allows you to publish and subscribe to data among PI Systems via the cloud (Azure). For more information, see https://www.picloudservices.com/

             

             

            Barry

             

            --

            hackathon_sig.png

            Click here to register for the TechCon Programming and Security hackathon on April 29, 2015!

            • Re: How to send data from AF to the cloud.
              jose_dai

              Hi Daphne/Barry,

               

              I was talking with my partners and their approach in this moment is the following: "Our idea is push the data from PI to our private Cloud, convert the data into a structure database, so we are thinking create an agent that pushing  that data, we are no interesting in this moment has a PI Db receiving data from different PI Systems"

               

              With this information could you give us your advice?

               

              Regards.

                • Re: How to send data from AF to the cloud.
                  bshang

                  Hi Jose,

                   

                  My understanding is that you'd like to push PI data periodically to a structured database in the cloud (e.g. SQL DB). In some regards, this SQL DB should be in sync with the PI data. Let me know if I have this correct. Also, do you have an AF Server or only a PI Server (Data Server)?

                   

                  Then, there are several options, some using out-of-box tools, others with custom coding.

                   

                  1) Use SQL Server Integration Services and PI OLEDB Enterprise to periodically push and/or sync data from PI to the cloud db. We have a White Paper on this

                  Utilizing MS SQL Server Tools (DTS and SSIS) for Data Exchange between PI and RDBMS

                   

                  I will admit it is an older example, but the principle is there. It shows an example of pushing to a flat file, but the steps are similar to pushing to another SQL db.

                   

                  2) Creating your own agent service to push the data. If the end data store is a structure database, it may be easier to use our SQL Developer Technologies, such as PI OLEDB Enterprise, PI JDBC, and/or PI ODBC to read the data from PI in relational format, transform it if necessary, then load into the cloud destination. An alternative is AF SDK. The tradeoff is that AF SDK may involve more advanced coding to shape the data into the structured format you desire, but will offer higher performance if coded correctly as it's a more native layer.

                   

                  3) Inquire about our future PI Integrators with your Account Manager. Although still in the development pipeline, these products will soon allow you to configure services that periodically push data from PI to external systems, such as Azure and cloud/on-premise SQL. You may find a lot of the functionality desired here that may be satisfied by this future offering.

                    • Re: How to send data from AF to the cloud.
                      jose_dai

                      Hi Barry/Daphne ,

                       

                      I have more details about the client requirement, they want to develop an application with java for getting values from the snapshot, archive file or AF structure and later  locate these values in the private cloud for posterior analysis. I have a question, For this requirement is better to use JDBC connector?, How to work the JDBC connector?

                       

                      Do you have any presentation or video where explain how to work this connector?

                       

                      Regards.

                        • Re: How to send data from AF to the cloud.
                          jose_dai

                          Barry,

                           

                          I forgot asking something is better to use PI web API instead of JDBC connector for this requirement?.

                           

                          Cheers.

                            • Re: How to send data from AF to the cloud.
                              dng

                              Hi Jose,

                               

                              The PI JDBC Driver is a JDBC 4.0 compliant driver that provides data access to the PI System through SQL queries. Since the JDBC API is built into every JRE, a Java application can use JDBC to gain access to relational database. Note that even though the PI Data Archive is not a relational database, the data can be structured in a relational datasets format to be consumed by "SQL" clients using the JDBC, ODBC, OLEDB standards. If you are familiar with PI OLEDB and PI OLEDB Enterprise, JDBC uses them in the backend to communicate with PI and AF respectively.

                               

                              To find out more about JDBC, I encourage you to check out the following resources:

                              • PI JDBC product page (link)
                              • PI JDBC user guide in Live Library (link)
                              • PI JDBC basics lab (link) - note that this lab is using an older version of PI JDBC
                              • PI JDBC Basics: a video that walks you through the process of setting up your PI JDBC Driver
                              • KB00632 - Accessing PI data from Java applications (link)


                              If you are pushing data into a SQL-like structured database in the cloud, using PI JDBC would require less manipulation and transformation of those data because the SQL query should return datasets that are ready to be pushed to the cloud.

                               

                              On the other hand, PI Web API is more light-weight than PI JDBC driver (JDBC driver requires SQL DAS, PI OLEDB and PI OLEDB Enterprise to translate the queries), and is currently undergoing active development. It can be advantageous if your cloud service database doesn't store structured data in SQL format.

                               

                              As Barry has mentioned, do check out the new PI Integrators that has the pre-built ability push data to external systems. You can schedule these integrators to push data periodically to your cloud database.  If you are interested, check out the following presentations in last year's UC:

                              • Big Data and the PI System (link)
                              • The PI System and Hadoop (link)