6 Replies Latest reply on Nov 16, 2018 3:50 PM by gmichaud-verreault

    What is the record format within the Pi Data Archive?

    DRS10

      What is the record format within the Pi Data Archive?

        • Re: What is the record format within the Pi Data Archive?
          vkaufmann

          Hi Daymond,

           

          Can I ask what you are trying to accomplish and why you need to know the structure of the archive files?

           

          --Vince

          1 of 1 people found this helpful
            • Re: What is the record format within the Pi Data Archive?
              DRS10

              I'm trying to figure out the best method for efficiently and inexpensively, externalizing Pi data into an on prem database for some advanced analytic purposes.    We collect roughly 300,000 Pi Points every second into our Pi Data Archive via many Pi OPC / DA interfaces.   So a few questions come to mind.

               

              1) How do I get that much data out of Pi and persist it into an external datastore and keep up?

              2) Should I try and store that to an SQLServer2016 database or should I use something like Cassandra?

              3) How many bytes do I need to pull out of Pi each second (that's why I was asking about the data structure)

              4) How best should I add the AF structure to the data as I store it in the external datastore?

              5) I have looked at the Pi Business Analytics integrator (basic and advanced).   It looks like I can use the basic version that collects data in batches, then stores it in an external database, but am I limited to a SQL db or can it store data to a noSQL database?    Can that integrator keep up with that kind of data rate?   Any other way for me to get that much data out of PI efficiently and keep up?

              6) Before I persist the data in the external data store, I may need to uncompress the data to make it usable for analytic tools....don't know if the basic integrator has that capability?

              7) Trying to calculate how much compressed / exception filtered data has to come out of Pi and go into this uncompressed / unfiltered external database

            • Re: What is the record format within the Pi Data Archive?
              afink

              I agree with Vincent Kaufmann, what are you looking to accomplish? If you are asking about general archive/record structure.  I would point you to the following LiveLibrary documentation.

               

              Archive Records

               

              - Adam

              2 of 2 people found this helpful