5 Replies Latest reply on Sep 14, 2012 3:48 PM by Gregor

    Memory Leak using PIValues

    sulebek

      Hello,

       

      i have a big problem with a service that i wrote myself.

       

      This service should write data read from some files to a PI Server Collective.

       

      Right now i have the problem, that the serivce is getting bigger and bigger within some seconds.

       

      Using ANT Memory Profiler it looks like the PI SDK isn't cleaning up the PIValues i write to the server.

       

      If I comment out the part where I create and fill the PIValue objects then everything works fine, i can parse hundreds of files without memory increasing.

       

       

       

      Can anyone tell me how to tell the SDK to free memory right after writing the data?

       

      I did try the ISlimFast for every point i write data to and also for the PIPoints collection of every server but this did not help, is there a way have to directly free the PIValues.

       

      Hopefully someone can help me

       

       

       

      I use the PI SDK 1.4.0 Build 416 with a Windows 7 x86

       

      The service is programmed with Studio 2010, .Net 3.5, C#

       

       

       

      Thanks

        • Re: Memory Leak using PIValues

          What if you force GC after updating and your PIValues objects go out of scope, do you see the same behaviour?

           

          Any chance of a code snippet of the offending code segment?

            • Re: Memory Leak using PIValues
              sulebek

              Hello,

               

              sorry for the late response!

               

               

               

              The GC can't really help with this. Thats why i don't know how to free this memory.

               

              Having a look at the ANTS Memory Profiler (see the screenshot below) the memory is reserved by an unmanaged library.

               

              So the GC can't take care of that.

               

              Any help on this would be very appreciated, i can't really see why this doesn't work.

               

               

               

              This are the snapshots of the memory before and after writing 5 files into the server, if I simply parse them and only comment out the part where

               

              the PISDK.PIValues object gets filled then the memory consumption stays constant at 60MB even after parsing thousands of files.

               

              (RED is the Memory reserved from unmanaged elements (46 MB at start, after 5 files 198MB)

               

              5076.SnapshotBeforeUpdateValues.png 

               

              2553.SnapshotAfterUpdateValues.png

               

               

               

              This is the class for the PI SDK Object.

               

              Usually i simply used the SDK object but after I ran into this problem i thought that a threadSafe approach would be helpful somehow.

               

               

               
              using System;
              using System.Collections.Generic;
              using System.Linq;
              using System.Text;
              
              namespace PI.DLL.Core
              {
                  // implementing the Singelton Pattern for the PI SDK access, also making it thread safe
                  public sealed class PI
                  {
                      private static volatile PI instance;
                      private readonly static object syncRoot = new Object();
              
                      private readonly PISDK.PISDK gPISDK;
              
                      private PI()
                      {
                          gPISDK = new PISDK.PISDKClass();
                      }
              
                      public PISDK.PISDK SDK
                      {
                          get { return gPISDK; }
                      }
              
              
                      public static PI Instance
                      {
                          get
                          {
                              if (instance == null)
                              {
                                  lock (syncRoot)
                                  {
                                      if (instance == null)
                                          instance = new PI();
                                  }
                              }
                              return instance;
                          }
                      }
                  }
              }
              

               

               

              To seperate Values and PI Tags from the SDK i created two small classes that represent tags and pivales in my code  

               
              namespace PI.DLL.Core
              {
                  public struct PIValue
                  {
                      public DateTime timeStamp;
                      public object value;
                      public string annotation;
                  }
              
                  public enum DataMergeConstant
                  {
                      dmDelete,
                      dmErrorDuplicates,
                      dmInsertDuplicates,
                      dmReplaceDuplicates,
                      dmReplaceOnlyDuplicates
                  }
              
                  public class PITag : IEqualityComparer , IDisposable
                  {
                      public string TagName { get; set; }
                      public string TagQuery { get; set; }
                              
                      public DataMergeConstant DataMerge { get; set; }
                      public IList Values { get; set; }
                      
                      public PITag(string TagName, string TagQuery, DataMergeConstant DataMerge)
                      {
                          this.TagName = TagName;
                          this.TagQuery = TagQuery;
                          this.DataMerge = DataMerge;
                          Values = new List ();
                      }
              
                      public bool Equals(PITag x, PITag y)
                      {
                          if (string.IsNullOrEmpty(x.TagName) || string.IsNullOrEmpty(y.TagName))
                          {
                              if (string.IsNullOrEmpty(x.TagQuery) || string.IsNullOrEmpty(y.TagQuery))
                                  return false;
                              else
                                  return x.TagQuery == y.TagQuery;
                          }
                          else
                              return x.TagName == y.TagName;
                      }
              
                      public int GetHashCode(PITag obj)
                      {
                          return obj.TagName.GetHashCode();
                      }
              
                      public void Dispose()
                      {
                          Values.Clear();
                          Values = null;
                      }
                  }
              
              }
              

               

               

              This is the part responsible for writing to PI

               
               private void WriteTags(IList Tags, bool doWriteStatusTag)
                      {
                          if (Tags != null)
                          {
                              try
                              {
                                  ////if (logger.IsDebugEnabled) logger.DebugFormat("WriteTags -> checking connection");
                                  CheckConnections(); // check if the connections to the servers are open
              
                                  int valuesWritten = 0;
                                  ////if (logger.IsInfoEnabled) logger.InfoFormat("{0} tag(s) received", Tags.Count);
              
                                  foreach (PITag tag in Tags)
                                  {       
                                      DataMergeConstants mergeType;
                                      switch (tag.DataMerge)
                                      {
                                          case DataMergeConstant.dmErrorDuplicates: mergeType = DataMergeConstants.dmErrorDuplicates;
                                              break;
                                          case DataMergeConstant.dmInsertDuplicates: mergeType = DataMergeConstants.dmInsertDuplicates;
                                              break;
                                          case DataMergeConstant.dmReplaceDuplicates: mergeType = DataMergeConstants.dmReplaceDuplicates;
                                              break;
                                          case DataMergeConstant.dmReplaceOnlyDuplicates: mergeType = DataMergeConstants.dmReplaceOnlyDuplicates;
                                              break;
                                          default: mergeType = DataMergeConstants.dmReplaceDuplicates;
                                              break;
                                      }
              
                                      if (tag.Values != null)
                                      {
                                         PISDK.PIValues piVals = new PISDK.PIValues() { };
              
              // loop through all values for the tag and add them to the PIValues
                                          foreach (PIValue val in tag.Values)
                                          {
                                              PITimeServer.PITimeFormat piTime = new PITimeServer.PITimeFormat();
                                              piTime.LocalDate = val.timeStamp;
                                              object piVal = val.value;
                                              piVals.Add(piTime, piVal, null);
                                          }
                                                                      
                                          piVals.ReadOnly = true;
              
                                          ////if (logger.IsDebugEnabled) logger.DebugFormat("{0} values to be written for tag '{1}' on '{2}'", piVals.Count, point.Name, server.Name);
                                          valuesWritten += piVals.Count; // counter for values writte for all tags
              
                                          try
                                          {
                                              // since fanning doesn't work properly, write to every member in our collective
              foreach (Server server in collectiveMembers)
                                              {
                                                  PIPoint point = !String.IsNullOrEmpty(tag.TagName) ? point = server.PIPoints[tag.TagName] : null;
                                                  if (point != null)
                                                  {
                                                      point.Data.UpdateValues(piVals, mergeType);
                                                      ((PISDKCommon.ISlimFast)point).SlimFast(); // use slimfast for the used point
                                                      point = null;
                                                  }
                                              }                                    
                                          }
                                          catch (Exception ex)
                                          {
                                              ////if (logger.IsErrorEnabled) logger.ErrorFormat("'UpdateValues' failed for tag '{0}' on '{1}'-> '{2}'", point.Name, server.Name, ex.Message);
                                              valuesWritten -= piVals.Count;
                                          }
                                          
                                          //((PISDKCommon.ISlimFast)piVals).SlimFast(); // this isn't possible, is there another way??
                                          piVals = null;
              
                                          foreach (Server server in collectiveMembers)
                                              ((PISDKCommon.ISlimFast)server.PIPoints).SlimFast(); // call slimfast for every member int he collective
                                      }
                                  }
                                  if (doWriteStatusTag) WriteStatusTag(valuesWritten);
                              }
                              catch (Exception ex)
                              {
                                  ////if (logger.IsErrorEnabled) logger.ErrorFormat("WriteTags Error -> '{0}'", ex.Message);
                              }
                          }
                          else
                          {
                              ////if (logger.IsDebugEnabled) logger.DebugFormat("Taglist to write is null");
                          }
                      }
              

               

                • Re: Memory Leak using PIValues

                  Some comments:

                   

                  - In your comments you say fanning doesn't work properly, what isn't working?  PIBufss would be a better approach if you can get it working.

                   

                  - In my opinion your update of the Collective is the wrong way round.  You are treating a PI tag first then looping through a collective.  I would first establish a connection to a member server and then loop through the tags, then repeat for the next member.  In the past I always found switching between members was expensive.

                   

                  - I don't see the IPICollective interface or the SwitchMember methods.  Is the data updating on all Collective members?

                    • Re: Memory Leak using PIValues
                      sulebek

                      Thank you for your comments:

                       

                      about your first comment:

                       

                      Fanning doesn't work properly, this was of course the first approach. The problem is, that i have to delete all old values that are saved for this tag, this is a requirement. This can only be done member wise, after connecting to every member, the new connection to the collective is not fanned. this is why i used this workaround.

                       

                       

                       

                      about the second comment:

                       

                      Your are right about this, i did it this way in the first place, the problem is, that i have to create more pivalue objects this way and the memory usage raises even more, so i changed it the other way. switching between the  servers is not a problem because i open all members at the beginning (this is done by the CheckConnection function) so switching is not a problem.

                       

                       

                       

                      about the third comment:

                       

                      maybe now this is clear, I use the IPICollective (i just didn't post this code) to get all Members of the collective and then use the MemberOpen-method to connect to the members right at the beginng.

                        • Re: Memory Leak using PIValues

                          Hello Bekim,

                           

                          Bekim Sulejmani

                          The problem is, that i have to delete all old values that are saved for this tag, this is a requirement.

                           

                          What is your application doing? Replacing a history or substituting existent events? What's the motivation to delete all events?

                           

                          Bekim Sulejmani

                          This can only be done member wise, after connecting to every member, the new connection to the collective is not fanned. this is why i used this workaround.

                           

                          Please be sure not using the IPICollective interface when you intend using PI Buffer Subsystem for fanning. Instead connect against the Primary PI Server and let Buffer Subsystem take care for sending events to all Collective members. Connections through the IPICollective interface are by design not fanned.

                           

                          Gregor