9 Replies Latest reply on Apr 4, 2016 4:03 PM by Charles Henze

    PI compressing algorithm questions

    martin.mertens

      Hi,

       

      following OSIsoft: Exception and Compression Full Details - YouTube, I have some questions:

       

      • is there any code or pseudo-code snippet available for the "real", slope-based compression?
      • if I reprocess old archive values by using exactly the same compdev value as used when creating the archive, could this lead to a further reduction of archive values? This is exactly what I experience when simulating this algorithm using my own code.

       

      Thank you!

       

      Martin

        • Re: PI compressing algorithm questions
          gregor

          Hello Martin,

           

          I wouldn't know we share any pseudo-code but here are some additional resources on the subject:

          Compression applies when a new Snapshot event arrives and the until then current Snapshot becomes pushed out of Snapshot. Compression does not become applied when reprocessing archives.

          Can you please explain about your experience with archive reprocessing and "simulating this algorithm using your own code"?

           

            • Re: PI compressing algorithm questions
              martin.mertens

              Hello Gregor,

               

              thank you for the additional ressources. After reading KB00699 - Compression Explained (which is quite similar to the YouTube illustration), I am pretty sure a have implemented this algorithm.

               

              I should clarify the word "reprocessing": I did not mean any PI Tool to manipulate archives. Instead, due to existing requirements, I wrote a tool that is able to reproduce the illustrated algorithm on existing archive values. Some of our archives are much too dense and need to be reprocessed with less compression (i.e. higher compdev). The basic idea is to simulate the compression and delete the archive values that would not be archived by the algorithm. The first experiences with that proceeding are good.

               

              In one test case, I just tried to process archive values with the original tag's compdev. My expectation was that there will be no archive value filtered out by the algorithm, but instead I experienced a reduction of about 25% which I cannot explain. Detailled outputs show that there are sequences of archive values that fit into the narrowing max/min slopes, and these values are filtered by the algorithm. Thus my question whether this result can be expected.

               

               

                • Re: PI compressing algorithm questions
                  Rhys Kirk

                  It depends whether any of those values we sent out of order. OOO data will bypass compression and be written to the archive regardless of compression settings. So reprocessing would mean those values "could" be removed, as during reprocessing they would be classed as in order data.

                  1 of 1 people found this helpful
                  • Re: PI compressing algorithm questions
                    gregor

                    Hello Martin,

                     

                    Thank you for explaining what you like to do. I see 2 general approaches.

                    1. Process history from tag A to tag B on the same PI Data Archive
                    2. Process history from Server A to Server B

                    As mentioned by Rhys, Out Of Order (OOO) events will bypass compression. Therefore you need to ensure you've configured B with the correct compression settings and you read from A in the correct historical order and write in the same order to the snapshot at B.

                    This way you can apply compression without recreating the algorithm and it doesn't matter if you use e.g. piconfig.exe or AF SDK.

                      • Re: PI compressing algorithm questions
                        martin.mertens

                        Hello Gregor,

                         

                         

                        thank for for the explanation. Since there is an almost completed application, I would like to try out approach 1), i.e. copying values from tag A to B by using AF SDK. This functionality is already there, but I do not understand how to enable the compression. As far as I tested, the AF method calls

                         

                         

                        pointCopy.UpdateValues(values, AFUpdateOption.InsertNoCompression, AFBufferOption.Buffer)

                        as well as

                        pointCopy.UpdateValues(values, AFUpdateOption.Insert, AFBufferOption.Buffer)

                         

                         

                        used for a newly created tag "pointCopy" with the same compdev than the "original" tag do not compress any value, but simply copy all values.

                         

                         

                        Which is the preferred method in AF SDK to force the compression when writing to the archive of "pointCopy"?

                      • Re: PI compressing algorithm questions
                        bshang

                        I believe repeated compression of the data will not preserve the result from the initial pass (i.e. it is not an idempotent operation).

                         

                        One reason is the constraint that the slopes must constantly narrow. See last figure in Step 4 in KB00699. This implies we keep track of the prior state (slopes) inferred from snapshot-only events (e.g. SMin old in that same figure).

                         

                        When you compress again, we no longer have the information to build that state because the snapshot-only event was not archived. Therefore, the new slope window could be wider than that of the initial pass.

                         

                        For example, in last figure in Step 4, imagine the new event comes in just under the line "SMin old". Its previous event will get archived. Now, we compress again. This time, the window will be larger (it is the "SMax" to "SMin" window). The event that was just under "SMin old" will be just above "SMin" and cause its prior event to be compressed out during the second pass.

                        1 of 1 people found this helpful
                          • Re: PI compressing algorithm questions
                            martin.mertens

                            Thank you, you are absolutely right! Looking at archive values retrospectively, we can not see the values that lead to the slope narrowing. Thus, there can be additional values that might be compressed. In particular, the snapshots that are discarded during compression can be local maxima/minima in the current slope. Thus, when compressing the compressed values the next time, these extremas do not contribute to the slope anymore, thus making the slope different.

                      • Re: PI compressing algorithm questions
                        ernstamort

                        Hi,

                         

                        The OSIsoft swinging door compression - as described above - requires buffering all values between the last archived and the current value and calculating the deviations to the interpolation. This is not very efficient.

                         

                        The following shows the calculation of the GE Archive compression with the steps to reproduce:

                             www.evsystems.net/files/GE_Historian_Compression_Overview.ppt

                         

                        I tested both and they lead to identical results.

                         

                        Holger