3 Replies Latest reply on Aug 8, 2016 3:44 PM by gregor

    PI data is missing/overwritten

    pachalad

      Hello,

       

      I'm having issue with pushing PI Data into the historian using the SDK. The issue I'm having is that the only data being saved is the first point and last point. For example, if I'm pushing points (1,2,3,4,5), only points 1 and 5 are saved. I've created a small application to test this behavior (below), I've run it on a colleagues machine and it successfully added all the points. So I've narrowed the issue down to my machine. I've reinstalled and updated all of the Pi software.

       

      I'm using version 4 of the SDK and I've tried running it using .net 4.5 to 4.6 and got the same problem. An interesting behavior I found is that if I add a breakpoint after the UpdateValues and wait a couple of seconds before continuing, it successfully adds the point. Although, using a Thread.Sleep won't add the point. I've tried using the PiPoint.UpdateValue/s method and I get the same issue. While steeping through the application, without waiting, I can see the data being added via the Pi System Management Tools but then they either get deleted or overwritten on the next iteration. There are no errors being returned.

       

      Does anyone have any suggestions on what I can do to solve my problem?

       

      Code:

        server = new PIServers()[PiServer_AF];
        var pipoints =  PIPoint.FindPIPoints(server, "DominiksPiPoint");
        Console.WriteLine("Deleting pi points {0} points",pipoints.Count());
        var error = server.DeletePIPoints(pipoints.Select(point => point.Name));
        Console.WriteLine("Deleted.");
        
        PIPoint myPoint=server.CreatePIPoint("DominiksPiPoint");
        Console.WriteLine("Starting to write PI points");
        for (int i=0; i < 20; i++)
        {
            List<AFValue> updates = new List<AFValue>();
            AFTime myTime = new AFTime(DateTimeOffset.Now.UtcTicks);
            AFValue value = new AFValue();
            value.Timestamp = myTime;
            value.Value = i;
            value.PIPoint = myPoint;
            updates.Add(value);
            var errors = server.UpdateValues(updates, OSIsoft.AF.Data.AFUpdateOption.NoReplace);
            if (errors != null)
            {
               foreach (var err in errors.Errors)
               {
                    Console.WriteLine(err.ToString());
               }
            }
            Thread.Sleep(1000);
         }
         var points = PIPoint.FindPIPoint(server, "DominiksPiPoint");
         AFValues values = points.RecordedValues(new AFTimeRange("*-2h", "*"), AFBoundaryType.Inside, "", false);
         foreach (var value in values)
         {
              Console.WriteLine("My Value is {0} at time {1}",value.Value,value.Timestamp);
         }
      
      
        • Re: PI data is missing/overwritten
          bshang

          I see you are setting the time to current

          AFTime myTime = new AFTime(DateTimeOffset.Now.UtcTicks); 

           

          So without the Thread.Sleep, the timestamps will all be very close to each other. If you have compression enabled for the tag, then intermediate values may have gotten compressed out. You can try using AFUpdateOption.InsertNoCompression to check if compression is the culprit.

           

          If it is, you can create a point turning compression off by using an overload of CreatePIPoint that takes in the dictionary of point attributes. One of the key value pairs should be "compressing"=0

          PIServer.CreatePIPoint Method

          • Re: PI data is missing/overwritten
            pachalad

            I was able to solve this issue by just restarting the services on the server.

              • Re: PI data is missing/overwritten
                gregor

                Hello Dominik,

                 

                Please accept my apologies for replying pretty late and for not agreeing to your conclusion. The answer you marked as Correct Answer is indeed wrong. The reason for correcting you is a) to educate you and b) to avoid other users become mislead when reading this discussion. Please be ensured that I don't like to play the mean guy against you nor do I like to be misunderstood as a the nitpicker. It's part of my job to share insights about how the PI System works and believe me, it happens from time to time that an answer provided by me turns out to be wrong. The only excuse that I have is that I am a human being and human beings make mistakes.

                 

                I have tried your code and the behavior you describe is what I would consider standard behavior. Your program connects against the PI Data Archive node, searches for PI Point DominiksPiPoint, deletes it and creates it again without specifying any point attributes. This means that the newly created DominiksPiPoint will be created with default values for point attributes. The relevant attributes are:

                 

                • compressing = 1 (on)
                • compdev = 0.2
                • span = 100
                • zero = 0

                There sleep(1000) within the loop creating the events will cause that timestamps are ~ 1 second apart. For the value you count from 0 to 19. So you have evenly spaced events. When plotting a trend, the result will be a straight line which means, no change in slope. Even with comdev = 0, all the events between the first (0) and the last (19) would become compressed.

                When looking at e.g. a ProcessBook trend, you will see the events first as they are send to PI Snapshot Subsystem. The compression is applied when a new, more recent event (Timestamp, Value, Status) arrives and the 'old' snapshot is pushed out. When compression is violated (see settings above) the 'old' event becomes written to PI Archives by PI Archive subsystem but without the violation, the event does not become archived. When after compression was applied, you refresh the trend, only a value of 0 and one of 19 is found in the archive. They will receive markers and a straight line is drawn between them. This video at our Youtube Learning Channel is likely easier to consume compared to my attempts to explain how compression works.

                 

                Please allow me some additional comments to your code.

                 

                • Point deletion and re-creation

                Deleting and re-creating a PI Point each time you launch your program is not good practice. I suggest seeing if your point was found and in case it wasn't, create it.

                 

                            PIServer server = new PIServers()[PIServer_AF];
                            PIPointList ptList = new PIPointList();
                            ptList.AddRange(PIPoint.FindPIPoints(server, "DominiksPIPoint"));
                            PIPoint myPoint;
                            if (ptList.Count == 0)
                            {
                                myPoint = server.CreatePIPoint("DominiksPIPoint");
                            }
                            else
                            {
                                myPoint = ptList[0];
                            }
                

                 

                • Insert Random values rather than values along a straight line.

                Please see further below for an example

                 

                 

                • Use Server.UpdateValues only once for each collection of AFValues.

                UpdateValues is a bulk call meaning that it is used to send multiple updates in one shot. Continuously adding to the collection while looping will result into the same events being repeatedly send. If you like to send an update one-by-one, please use UpdateValue instead of UpdateValues. Otherwise, see below .

                 

                • Use sub second timestamps only when necessary.

                Because of how Timestamps are stored in PI Archives, one should only use sub second timestamps when there is additional value in the sub second portion

                 

                            List<AFValue> updates = new List<AFValue>();
                            Random rnd = new Random();
                            for (int i = 0; i < 20; i++)
                            {
                                AFValue value = new AFValue();
                                value.PIPoint = myPoint;
                                value.Timestamp = AFTime.NowInWholeSeconds;
                                value.Value = rnd.NextDouble() * 100;
                                Console.WriteLine("New value: {0} at {1}", value.Value, value.Timestamp);
                                updates.Add(value);
                                Thread.Sleep(1000);
                            }
                            AFErrors<AFValue> updateErrors = server.UpdateValues(updates, OSIsoft.AF.Data.AFUpdateOption.NoReplace);
                            if (updateErrors != null)
                            {
                                foreach (KeyValuePair<PIServer, Exception> err in updateErrors.PIServerErrors)
                                {
                                    Console.WriteLine("{0}: {1}", err.Key, err.Value);
                                }
                            }
                

                 

                I believe marking Barry's answer correct is just fair because he was the first one identifying your issue as one originating from compression.