26 Replies Latest reply on Jun 28, 2012 2:37 PM by charlie@osisoft.com

    Sending data to PI using PISDK with compression

    paroda

      I have developed one windows service to query data using HttpWebRequest and send to a PI Collective through PISDK. The PI buffer subsystem is enabled on the client system.

       

      The PISDK method i am using is PIData.UpdateValue(value,timestamp). The service is configured to send data at 1 min interval.

       

      The PI tag is configured for compression with following parameters: compdev: 60, compmin: 0, compmax: 10min

       

      I checked and found that the PI system is getting the values properly, but there is no compression. The value is constant for more than 20 min, but it has recorded each point at 1 min interval.

       

      I am not sure what else need to be configured. Could someone please advice how to send data to PI through PISDK and enable compression.

        • Re: Sending data to PI using PISDK with compression
          MvanderVeeken

          Compression is only applied by the server when dealing with values that have a timestamp newer than the snapshot.

           

          If you are updating values with timestamps before the timestamp of the snapshot, no compression is applied.

            • Re: Sending data to PI using PISDK with compression
              paroda

              This service is the only source and it sends new data with current time as timestamp. Hence it is sequential in time. Each update has a newer timestamp.

                • Re: Sending data to PI using PISDK with compression

                  Is the "Compressing" Point Attribute set to 1?

                    • Re: Sending data to PI using PISDK with compression
                      paroda

                      yes the compressing is on.

                        • Re: Sending data to PI using PISDK with compression
                          andreas

                          Please verify that you are sending data with a timestamp newer than the snapshot. What tool are you using to verify that the data is every minute in the archive? Do you have any out-of-order data in your archive?

                            • Re: Sending data to PI using PISDK with compression
                              paroda

                              This program is the only source that sends the data for this tag. It uses the DateTime.Now as the timestamp each time it calls the UpdateValue method.

                               

                              I used the Archive Editor view in PI SMT to view the data for this tag. It shows all the values that the program has sent.

                               

                              I am not sure how to check out-of-order data. could you advice please?

                                • Re: Sending data to PI using PISDK with compression
                                  mhamel

                                  @Pradyumna: One way to see if you have had out-of-order data is to use the PI System Management Tools (PI SMT) with the Snapshot And Archive Statistics plug-in (located under the operation section). You will see a table exposing different metrics and statistics about the PI Data Archive. One counter is called Out of Order Snapshot Events. This counter has two fields: Value and Change. The former one shows you the number of out-of-order event you had since the PI Data Archive is up, the latter one shows how many new out-of-order events were added since the last reading. Readings are occurring every 5 seconds in general.

                                   

                                  If you see this fields increasing, this signifies an application or a PI Interface is writing data which are older than the current snapshot.

                                   

                                  I invite you to take a look at these 2 videos (video 1, video 2) from our YouTube OSIsoft Learning Channel. It explains how compression works on the PI Data Archive (PI Server).

                                   

                                  Let me know if that helped.

                                    • Re: Sending data to PI using PISDK with compression
                                      paroda

                                      Thanks Mathieu for the tutorials. They are useful. Though i want to clarify that my PI tag is not populated  by any Osisoft interface or any other application. The tag pointsource attribute is L. The only way it gets data is through my application. One thing is for sure that, the update is always with newer timestamp as current time is used.

                                       

                                      Also i found the out-of-order count is not increasing.

                                        • Re: Sending data to PI using PISDK with compression

                                          Change the point source to something other than L.  For example, "MyApplication".

                                            • Re: Sending data to PI using PISDK with compression
                                              andreas

                                              Out of curiosity, what is the PI PointType? Is it a string, or is it Integer, Float, Digital?

                                                • Re: Sending data to PI using PISDK with compression
                                                  paroda

                                                  The point type is float32.

                                                    • Re: Sending data to PI using PISDK with compression
                                                      andreas

                                                      Pradymna, I must confess that I can reproduce your problem - usually this should be handled by techsupport as this seems to be an issue with the PI SDK Buffering. As I have already reproduced the issue with a simple code sample I will forward this to techsupport and will post the results here.

                                                       

                                                      For those interested in code snippets, here is a little code snippet creating a point and writing data to it:

                                                       
                                                      using System;
                                                      using System.Collections.Generic;
                                                      using System.Text;
                                                      
                                                      namespace UpdateValueTest
                                                      {
                                                          class Program
                                                          {
                                                              static void Main(string[] args)
                                                              {
                                                                  try
                                                                  {
                                                                      // the PISDK
                                                                      PISDK.PISDK MyPISDK = new PISDK.PISDK();
                                                      
                                                                      string strServer = "";
                                                                      if (args.Length > 0)
                                                                      {
                                                                          // one argument - the PI server
                                                                          strServer = args[0];
                                                                      }
                                                                      else
                                                                      {
                                                                          // no argument - use the default
                                                                          strServer = MyPISDK.Servers.DefaultServer.Name;
                                                                      }
                                                      
                                                                      // get the server
                                                                      PISDK.Server myPIServer = MyPISDK.Servers[strServer];
                                                      
                                                                      // construct a unique tagname
                                                                      string MyTagName = "MyTag_"
                                                                          + System.Environment.MachineName
                                                                          + "_"
                                                                          + System.Diagnostics.Process.GetCurrentProcess().ProcessName
                                                                          + "_"
                                                                          + System.Environment.TickCount;
                                                      
                                                                      // set compression and exception
                                                                      PISDKCommon.NamedValues myNV = new PISDKCommon.NamedValues();
                                                                      myNV.Add("CompMin", 0);          // CompMin 0
                                                                      myNV.Add("CompMax", 600);        // CompMax 10 min
                                                                      myNV.Add("CompDev",60);          // CompDev 60
                                                                      myNV.Add("Compressing", 1);      // Compressing on
                                                                      myNV.Add("ExcMin", 0);           // ExceptionMin 0
                                                                      myNV.Add("ExcMax", 0);           // ExceptionMax 0
                                                                      myNV.Add("ExcDev", 0);           // ExceptionDeviation 0
                                                      
                                                                      // create the PI Point
                                                                      Console.WriteLine("Adding Tag {0} to PI Server {1}", MyTagName, strServer);
                                                                      PISDK.PIPoint myPIPoint = myPIServer.PIPoints.Add(MyTagName, 
                                                                                                                        "classic", 
                                                                                                                        PISDK.PointTypeConstants.pttypFloat32, 
                                                                                                                        myNV);
                                                      
                                                                      // add a value
                                                                      myPIPoint.Data.UpdateValue(1, DateTime.Now);
                                                                      System.Threading.Thread.Sleep(1000);
                                                                      for (int ii = 0; ii < 10; ii++)
                                                                      {
                                                                          // add more values
                                                                          Console.WriteLine(DateTime.Now.ToLocalTime().ToString("HH:mm:ss"));
                                                                          myPIPoint.Data.UpdateValue(1, DateTime.Now);
                                                                          System.Threading.Thread.Sleep(1000);
                                                                      }
                                                                  }
                                                                  catch (Exception ex)
                                                                  {
                                                                      Console.WriteLine(ex.ToString());
                                                                  }
                                                                  Console.WriteLine("");
                                                                  Console.ReadKey();
                                                              }
                                                          }
                                                      }
                                                      

                                                       

                                                        • Re: Sending data to PI using PISDK with compression
                                                          andreas

                                                          Here is what happens - when not specifiyng the MergeType - PI SDK defaults to dmReplaceDuplicates. In this case the pibufss will ignore compression and the event is sent to the archive. As a workaround, you can specify dmInsertDuplicates:

                                                           
                                                          myPIPoint.Data.UpdateValue(1, 
                                                              DateTime.Now,
                                                              PISDK.DataMergeConstants.dmInsertDuplicates);
                                                          

                                                           

                                                            • Re: Sending data to PI using PISDK with compression
                                                              paroda

                                                              Sorry for late reply. I have been testing it for last couple of days. Still couldn't get it to enable compression. I tried the dmInsertDuplicates option too.

                                                               

                                                              However, another peculiar thing i found. On the PI server itself i used the piartool -aw to look directly into the archive, and found that it has only 8 events till date, roughly one per day. Where as i can see all the events in SMT/Archive Editor. One change though i have done, increased the compmax to 8 hours (28800).

                                                               

                                                              the results of piartool is shown below

                                                               

                                                                       Point ID: 3401  Record Number: 3255

                                                               

                                                                       Chain Record Number - Next: 0 Prev: 0 Index: 0

                                                               

                                                                       Record Version: 3  Data type: 8

                                                               

                                                                       Flags - Index:1 Step:0 Del:0 Dirty:0

                                                               

                                                                       Sizes - Record 1024  Data: 998

                                                               

                                                                       Parent Archive 0000000006B3D9C0  Data Head: 26

                                                               

                                                                       Event Count: 8

                                                               

                                                                       Storage (bytes) - Available: 998 Used: 77

                                                               

                                                                       8 Event(s):

                                                               

                                                                       0: 7-May-12 06:11:21, S,O,A,S,Q [0,0,0,0,0]: 148383

                                                               

                                                                       1: 18-May-12 15:13:18.93901, S,O,A,S,Q [0,0,0,0,0]: 149282

                                                               

                                                                       2: 18-May-12 17:17:20.68102, S,O,A,S,Q [0,0,0,0,0]: 148382

                                                               

                                                                       3: 21-May-12 11:35:05.29701, S,O,A,S,Q [0,0,0,0,0]: 130951

                                                               

                                                                       4: 22-May-12 00:26:10.39201, S,O,A,S,Q [0,0,0,0,0]: 125892

                                                               

                                                                       5: 22-May-12 18:46:25.35901, S,O,A,S,Q [0,0,0,0,0]: 119202

                                                               

                                                                       6: 23-May-12 13:06:40.35201, S,O,A,S,Q [0,0,0,0,0]: 112995

                                                               

                                                                       7: 24-May-12 07:29:55.38400, S,O,A,S,Q [0,0,0,0,0]: 106656

                                                               

                                                              Also given below is dump from pibufss -sd pointid

                                                               

                                                              *** Point RecNo: 3255 (1 / 1)

                                                               

                                                              * Dump of Snapshot Record, Point ID: 3401

                                                               

                                                               -- Configuration:

                                                               

                                                                 type: Float32 (12), zero: 0, span: 10000

                                                               

                                                                 archiving: 1, compressing: 1, step: 0, filter code: 0

                                                               

                                                                 compression min: 0, max: 28800, deviation: 2

                                                               

                                                               -- Security:

                                                               

                                                                 ACL ID: 9 [  ]

                                                               

                                                               -- Snapshot Event:

                                                               

                                                                 time: 24-May-12 15:58:18.89900 (utc: 1337855298.89900)

                                                               

                                                                 type: 12, value: S,O,A,S,Q [0,0,0,0,0]: -50, -50

                                                               

                                                               -- Compressed Event:

                                                               

                                                                 time: 24-May-12 15:17:52.40501 (utc: 1337852872.40501)

                                                               

                                                                 type: 12, value: S,O,A,S,Q [0,0,0,0,0]: -49, -49

                                                               

                                                               -- Compression State:

                                                               

                                                                 out-of-order: 0, compression slopes: -0.00123635, 0.000412117

                                                               

                                                                 updates: 0, 3ph: 0, buffered: 0, bufdirty: 0

                                                               

                                                                 dirty: 0, bits: 0x60c, delete pending snap/comp: 0/0

                                                               

                                                              * End of Dump

                                                                • Re: Sending data to PI using PISDK with compression
                                                                  paroda

                                                                  Finally! it's working. Thanks to Andreas. Your suggestion did worked. Adding dmInsertDuplicates to UpdateValue method did the trick. my earlier post was too early. Now i see the data in the SMT/Archive Editor to be nicely placed at 8 hours intervals.

                                                                   

                                                                  Thanks Andreas, for finding out the real issue and resolving it.

                                                                    • Re: Sending data to PI using PISDK with compression
                                                                      ktram

                                                                      Pradyumna, a KB Article that describes the behavior you observed has been published: KB00662.  Thanks for reporting it.

                                                                       

                                                                      By the way, regarding the archive walk output in your previous post, the archive record that had 8 events was actually an index record. To see the actual events, navigate to the data records (the values of an index record are record numbers).

                                                                        • Re: Sending data to PI using PISDK with compression
                                                                          alexandre.bouffard

                                                                          According to my experiments, using dmErrorDuplicates or dmInsertDuplicates does indeed work as a workaround but only for present events and not for past events. In other words:

                                                                          • Buffering disabled: I can insert past events with compression.
                                                                          • Buffering enabled: compression does not work for past events even if I replace dmReplaceDuplicates with dmErrorDuplicates or dmInsertDuplicates.

                                                                          Is this to be expected ?

                                                                           

                                                                          PS: PtCreated events are deleted before each experiment to allow for compression.

                                                                            • Re: Sending data to PI using PISDK with compression

                                                                              The PI Server should not compress out of order PIValues.  An out of order PIValue is any PIValue with a timestamp older than the snapshot value.  

                                                                               

                                                                              If the mode is dmErrorDuplicates, and a PIValue does exist, without buffering an error will be returned to the caller, but with buffering the error will be logged on the PI Server.  Did you examine the log file?

                                                                                • Re: Sending data to PI using PISDK with compression
                                                                                  alexandre.bouffard

                                                                                  I refactored my experiment in order to clarify my question.

                                                                                   

                                                                                  The following code snippet inserts 20 events: 10 one hour in the past followed by 10 at the current timestamp. Before running the snippet, the PtCreated snapshot event is first deleted to allow for compression.

                                                                                   

                                                                                  All events are correctly inserted, but Is this to be expected that the first 10 events are not compressed but the last 10 are compressed ?

                                                                                   

                                                                                  using System;
                                                                                  
                                                                                  namespace UpdateValueTest
                                                                                  {
                                                                                      class Program
                                                                                      {
                                                                                          static void Main(string[] args)
                                                                                          {
                                                                                              try
                                                                                              {
                                                                                                  PISDK.PISDK myPISDK = new PISDK.PISDK();
                                                                                                  PISDK.Server myPIServer = myPISDK.Servers.DefaultServer;
                                                                                                  PISDK.PIPoint myPIPoint = myPIServer.PIPoints["MyTestTag"];
                                                                                  
                                                                                                  for (int ii = 0; ii < 10; ii++)
                                                                                                  {
                                                                                                      myPIPoint.Data.UpdateValue(1, DateTime.Now - TimeSpan.FromHours(1), PISDK.DataMergeConstants.dmInsertDuplicates);
                                                                                                      System.Threading.Thread.Sleep(1000);
                                                                                                  }
                                                                                                  for (int ii = 0; ii < 10; ii++)
                                                                                                  {
                                                                                                      myPIPoint.Data.UpdateValue(1, DateTime.Now, PISDK.DataMergeConstants.dmInsertDuplicates);
                                                                                                      System.Threading.Thread.Sleep(1000);
                                                                                                  }
                                                                                              }
                                                                                              catch (Exception ex)
                                                                                              {
                                                                                                  Console.WriteLine(ex.ToString());
                                                                                              }
                                                                                          }
                                                                                      }
                                                                                  }
                                                                                  

                                                                                   

                                                                                   

                                                                                  Thanks,

                                                                                   

                                                                                  Alexandre

                                                                                    • Re: Sending data to PI using PISDK with compression
                                                                                      mhamel

                                                                                      Alexandre: Yes it is expected. As Micheal pointed already from a previous post:

                                                                                       

                                                                                      Michael @ OSIsoft

                                                                                      Compression is only applied by the server when dealing with values that have a timestamp newer than the snapshot.

                                                                                       

                                                                                      If you are updating values with timestamps before the timestamp of the snapshot, no compression is applied.

                                                                                       

                                                                                      I invite you to take a look at these 2 videos (video 1, video 2) from our YouTube OSIsoft Learning Channel. It explains how compression works on the PI Data Archive (PI Server).

                                                                                        • Re: Sending data to PI using PISDK with compression
                                                                                          alexandre.bouffard

                                                                                          Mathieu, Hi from Montréal ;-)

                                                                                           

                                                                                          As I mentioned, I first delete the Pt Created event. After that the snapshot event is "No Data" at timestamp "12/31/1969 7:00:00 PM". Therefore past events are not out of order as long as they are inserted in chronological order. In other words I am backfilling with compression.

                                                                                           

                                                                                          As I mentioned in my first post, but maybe was not clear enough: executing the same code snippet results in the compression of all the 20 events when buffering is disabled, but only results in the compression of the present events (last 10) when buffering is enabled.