13 Replies Latest reply on May 18, 2012 1:11 PM by RyanBrown

    SDK 101 - Archive Time

    RyanBrown

      Hi Everyone,

       


      Following on from my previous post here is the next in my PI SDK series. In my last post I showed a simple console application that dealt with snapshot values. This post takes a small jump forward and looks at how to deal with archived values. I’ll again wrap this in a simple console app to keep the code as simple as possible.

       

      To make it a little more interesting I’ll also explain a small problem that was given to me by a colleague and show you how I went about solving this using the SDK. You might have your own ideas on how you would solve it so let me know.

       

      Okay so here is the first application

       

      Description:
      This is a simple console application where we are going to look at the sinusoid test tag and read some archived values for the tag. Told you it was going to be simple!

       
      using System;
      using PISDK;
      
      namespace SimpleArchiveRead
      {
          class Program
          {
              static void Main(string[] args)
              {
                  // All values hard coded for this example but I'm keeping it simple
                  PISDK.PISDK sdkObj = new PISDK.PISDK();
                  string tag = "sinusoid";
                  string start = "*-1h";
                  string end = "*";
                  Server myServer;
      
                  try
                  {
                      myServer = sdkObj.Servers["localhost"];
                  }
                  catch (Exception ex)
                  {
                      Console.WriteLine(ex.Message);
                      Console.ReadKey();
                      return;
                  }
      
                  if (!myServer.Connected)
                  {
                      try
                      {
                          myServer.Open();
                      }
                      catch (Exception ex)
                      {
                          Console.WriteLine(ex.Message);
                          Console.ReadKey();
                          return;
                      }
                  }
      
                  // This is where we get the archived values for the point
                  PIValues archiveValues = myServer.PIPoints[tag].Data.RecordedValues(start, end);
      
                  // For every archived value retrieved just going to output the timestamp and value
                  foreach (PIValue value in archiveValues)
                  {
                      string valString;
      
                      // Want to handle the value being a COMObject ( Digital State )
                      if (value.Value.GetType().IsCOMObject)
                      {
                          valString = ((DigitalState)value.Value).Name;
                      }
                      else
                      {
                          valString = value.Value.ToString();
                      }
      
                      Console.WriteLine("{0}: {1}", value.TimeStamp.LocalDate.ToString("dd/MM/yyyy H:mm:ss"), valString);
                  }
      
                  Console.WriteLine();
                  Console.WriteLine("Press any key to exit");
                  Console.ReadKey();
              }
          }
      }
      

       

       



      For running the above application you will need to add three PI references: OSISoft.PISDK, OSISoft.PISDKCommon, OSISoft.PITimeServer
      Okay so I’m sure the above example makes sense – what I’d like to do now is to introduce a problem that I was set to solve. If you are just starting out with the SDK then be worth trying to write your own application to solve it before looking at what I did.

       


      So the problem given was that there is a PI server with thousands of tags on it (typical scenario). When administering and testing certain things it is valuable to be able to identify the tags that are changing the most and being archived in a specified time frame. There might be some slick way of doing this but we didn’t find one so using the knowledge of reading archive values I set about creating another console app to solve this issue.

       


      The same references are required that was needed for the previous application. Bit more code in this as I wanted to have some kind of error handling. You cold strip all that out if you wanted just to see the relevant PI calls.

       
      using System;
      using System.Collections.Generic;
      using PISDK;
      using PISDKCommon;
      
      namespace TagUpdates
      {
          // Going to build a generic list of structs later
          struct TagCount:IComparable
          {
              private string tagName;
              private int count;
      
              public string TagName
              {
                  get { return tagName; }
              }
      
              public int Count
              {
                  get { return count; }
              }
      
              public TagCount(string _tagName, int _count)
              {
                  tagName = _tagName;
                  count = _count;
              }
              
              // Override the CompareTo so that I can sort my list later by count
              public int CompareTo(TagCount other)
              {
                  if (other.count == count)
                  {
                      return 0;
                  }
                  else if (other.count < count)
                  {
                      return -1;
                  }
                  else
                  {
                      return 1;
                  }
              }
          }
      
          class Program
          {
              static void Main(string[] args)
              {
                  // All these variables will be provided by the user
                  string server;
                  string start;
                  string end;
                  int count;
                  string tagFilter;
      
                  // If no arguments are specified then going to have to ask some questions
                  if (args.Length == 0)
                  {
                      Console.WriteLine();
                      
                      Console.Write("Specify server name (default: localhost): ");
                      server = Console.ReadLine();
                      if (server == "")
                          server = "localhost";
                      
                      Console.Write("Specify start time (default: *-1d): ");
                      start = Console.ReadLine();
                      if (start == "")
                          start = "*-1d";
      
                      Console.Write("Specify end time (default: *): ");
                      end = Console.ReadLine();
                      if (end == "")
                          end = "*";
      
                      Console.Write("Specify tag filter (default *): ");
                      tagFilter = Console.ReadLine();
                      if (tagFilter == "")
                          tagFilter = "*";
      
                      Console.Write("Specify number of results to return: (default 10) ");
                      try
                      {
                          string cnt = Console.ReadLine();
      
                          if (cnt != "")
                          {
                              count = Convert.ToInt32(cnt);
                          }
                          else
                          {
                              count = 10;
                          }
                      }
                      catch (Exception)
                      {
                          PrintHelp("Valid number required for the count");
                          return;
                      }
      
                      // So going to output the full command so that they can copy later for scripting if needed
                      Console.WriteLine();
                      Console.WriteLine("Full command is: TagUpdates {0} {1} {2} {3} {4}", server,start, end, count, tagFilter);
                      Console.WriteLine();
      
                  }
                  else if (args.Length < 5)
                  {
                      PrintHelp("Not provided enough arguments.");
                      Console.ReadKey();
                      return;
                  }
                  else
                  {
                      // Provided command with arguments already specified
                      server = args[0];
                      start = args[1];
                      end = args[2];
                      tagFilter = args[4];
      
                      try
                      {
                          count = Convert.ToInt32(args[3]);
                      }
                      catch (Exception)
                      {
                          PrintHelp("Valid number required for the count");
                          Console.ReadKey();
                          return;
                      }
                  }
      
                  // Got all the information needed. Lets go get the counts
                  OutputTags(server, start, end, count, tagFilter);
      
                  Console.WriteLine();
                  Console.WriteLine("Press any key to exit");
                  Console.ReadKey();
      
              }
              
              // If any errors are encountered then just output some help along with the error message
              private static void PrintHelp(string _messsage)
              {
                  Console.WriteLine();
                  Console.WriteLine("************************************************");
                  Console.WriteLine(_messsage);
                  Console.WriteLine("************************************************");
                  Console.WriteLine();
                  Console.WriteLine("Format of the command is as follows:");
                  Console.WriteLine("\"TagUpdates [serverName] [startTime] [endTime] [resultCount] [tagFilter]\"");
                  Console.WriteLine();
                  Console.WriteLine("Example: TagUpdates localhost *-1d * 10 sin*");
                  Console.WriteLine("Command above will return the top 10 changing tags on local host over one day");
                  Console.WriteLine();
                  Console.WriteLine("You can also run the command without specifying any arguments.");
                  Console.WriteLine("This will prompt you for a series of parameters.");
                  Console.WriteLine("Once specified the application will run and also output the full command");
              }
      
              private static void OutputTags(string _server, string _start, string _end, int _count, string _tagFilter)
              {
                  // define top level sdk object and variable for piServer
                  PISDK.PISDK pi_sdk = new PISDK.PISDK();
                  Server piServer;
      
                  try
                  {
                      // Exception could be thrown if the server is not known
                      piServer = pi_sdk.Servers[_server];
                      
                      if (!piServer.Connected)
                      {
                          piServer.Open();
                      }
                  }
                  catch (Exception ex)
                  {
                      PrintHelp("Problem connecting to PI Server:" + ex.Message);
                      return;
                  }
      
                  Console.WriteLine("PI Server is now connected.");
                  Console.WriteLine();
      
                  // Using the struct here for generic list
                  List topCount = new List();
      
                  Console.WriteLine("Tags to process: {0}", piServer.GetPoints("tag = '" + _tagFilter + "'").Count);
                  Console.WriteLine();
      
                  DateTime start = DateTime.Now;
                  
                  // using where clause parameter to limit results gathered from server based on filter provided
                  foreach(PIPoint point in piServer.GetPoints("tag = '" + _tagFilter + "'" ))
                  {
                      TimeSpan timePassed = DateTime.Now - start;
      
                      // Keep user updated every 5 mins of progress if it's taking a while to run
                      if (timePassed.Minutes > 4)
                      {
                          Console.WriteLine("Processed: {0}", topCount.Count);
                          start = DateTime.Now;
                      }
      
                      int count;
      
                      try
                      {
                          // returning the count of archived values for this point
                          // How can I use the Async object parameter to make this faster??
                          count = (point.Data.RecordedValues(_start, _end, BoundaryTypeConstants.btAuto)).Count;
                      }
                      catch (Exception ex)
                      {
                          PrintHelp(ex.Message);
                          Console.ReadKey();
                          return;
                      }
      
                      topCount.Add(new TagCount(point.Name, count));
                  }
      
                  // Calling the custom sort method for the structs stored
                  topCount.Sort();
      
                  // Just incase user specified number of results to return which is greater than results gathered
                  if (topCount.Count < _count)
                  {
                      _count = topCount.Count;
                  }
      
                  for (int i = 0; i < _count; i++)
                  {
                      try
                      {
                          // Okay dokay... output the results at the end
                          Console.WriteLine("Tag: {0}, Count:{1}", topCount
      .TagName, topCount
      .Count);
                      }
                      catch (IndexOutOfRangeException)
                      {
                          return;
                      }
                  }
              }
          }
      }
      

       

       


      So that’s it. Above solution worked fine for us and gave us a quick way of getting tags that are changing often. Let me know what you think. I’m particularly interested in ideas on how to make the solution faster. I believe there should have been a way to take advantage of the PIAsynch object so any examples of that would be great.

       


      Not decided yet where I’m heading with next posting so if you have any ideas feel free to put forward a suggestion. Failing that I’ll think of a logical step forward for next one.

       


      Ryan

        • Re: SDK 101 - Archive Time
          Rick Davin

          Ryan,

           

          I salute your efforts to help broaden the knowledge of the community.  Two remarks:

           

          1) You have comment asking how to use PIAsyncStatus to make things faster.  Bottom line: PIAsynchStatus does not make anything faster on the server.  If used correctly it can make your UI appear more responsive.

           

          2) Your assumption that every COM object is a DigitalState can be wrong.  The PIValue.Value property could be a COM object and also not be a DigitalState.

           
          // Suggest Change this
          if (value.Value.GetType().IsCOMObject)
              {
                  valString = ((DigitalState)value.Value).Name;
              }
          

           

           
          // Suggest replace with this
          if ((value.Value) is DigitalState) 
              {
                  valString = ((DigitalState)value.Value).Name;
              }
          

           

            • Re: SDK 101 - Archive Time
              RyanBrown

              Thanks Rick,

               

              Yeah that was a bit of a school boy error with the COM assumption considering that was commented on in my last post!

               

              I'm going to pretend that I did that on purpose to make sure people were paying attention. Yes... actually from now on any mistakes in the code are done on purpose to test the reader

                • Re: SDK 101 - Archive Time

                  Ryan...nice post on PI SDK, keep it up.  I think it was mentioned in your last 101 posting that a natural progression once you have familiarity with the PI SDK and understanding of the PI Server structure/behaviour with data is to then move your 101 series to the AF SDK, with emphasis on the new PI and Data namespaces introduced by v2.5.

                   

                  @Rick: not sure I totally agree with your #1 (I may have misread it).  If you send 2 requests in parallel to a PI Server then the PI Server can process both requests in parallel and return the data.  This doesn't make the UI appear to be more responsive, it probably is more responsive because what would have been processed sequentially has been processed is parallel.  

                    • Re: SDK 101 - Archive Time
                      RyanBrown

                      Hey Rhys, Thanks for the comment. I'm hoping that I can do one of these a week. It was mentioned that moving onto AF SDK is a good move and I promise I'll be doing that. On starting with the PI SDK though I'll stick with it for a little bit as there are more things to explore and post examples about.

                       

                      Interested to see the outcome of the parallel chat. I did read some where ( I think ) that it does help the UI appear more responsive but wasn't exactly how it did that. I certainly didn't find any Console Applications using it but just Forms apps. Don't think there is anyway for me to take advantage of it here.

                       

                      Out of interest is there anyway that others can see to speed up the second application? When running on a server with say 100,000 tags it can take a long time. Any tricks to get this working more quickly? Is there a way to send some sort of bulk request to the server? I did try a parallel foreach loop but since I was actually adding to a list it failed since the list isn't thread safe.

                        • Re: SDK 101 - Archive Time
                          andreas

                          Ryan

                          Out of interest is there anyway that others can see to speed up the second application? When running on a server with say 100,000 tags it can take a long time. Any tricks to get this working more quickly? Is there a way to send some sort of bulk request to the server? I did try a parallel foreach loop but since I was actually adding to a list it failed since the list isn't thread safe.

                           

                           

                          Yes, there is. That is the place for the async calls. The async call will allow you to throw a couple of calls to the server and wait for the result, taking advantage of the multithreaded server and the latency.

                            • Re: SDK 101 - Archive Time

                              You only seem to be needing the count of events in your code, rather than using RecordedValues switch to PIData.Summary method and set the SummaryType as Count with an EventWeighted Calculation Basis.

                                • Re: SDK 101 - Archive Time
                                  Rick Davin

                                  @Rhys, while one could issue multiple async calls to possibly speed up an application, the way the loop was being done (per point, fetch values, extract count, then write that count to the console), I wanted to state that merely adding the async parameter in the code does not make it faster.  To speed things up there would have to be a radical redesign of the code logic, which goes beyond a simple conversion of VBA examples to C#.  Even still, I am normally a big proponent of adding async for any form-based UI application, except again in this case I don't see much benefit for a console-based application.  I do find your suggestion of adding a Summary call to be quite excellent.  My testing shows this at least 4 times faster if all you want is counts.

                                   
                                  public int GetRecordedEventsCount(PISDK.PIPoint tag, PITime StartTime, PITime EndTime)
                                  {
                                      PISDK.PIValue sumval = tag.Data.Summary(StartTime, EndTime, ArchiveSummaryTypeConstants.astCount, CalculationBasisConstants.cbEventWeighted);
                                      return (int) sumval.Value;
                                  }
                                  

                                   

                                   

                                   

                                   

                                   

                                    • Re: SDK 101 - Archive Time
                                      RyanBrown

                                      Sounds like the summary call is the best option for this case. Great learning some new things.

                                       

                                      What I'll do is try the application using the summary call and see what comparisons I get with the timing. Obviously I'll let you know how that goes but four times faster looks sounds good to me.

                                        • Re: SDK 101 - Archive Time
                                          Rick Davin

                                          Feel free to read more regarding the differences in the methods, but in a nutshell, the RecordedValues returns a PIValues collecton whereas the Summary method returns a solitary PIValue object whose .Value property contains the summary result.  The problem with the PIValues collection - again if all you want is counts - is that every PIValue item must make the trip back from the server to the client's memory.  If a PI Point has 250,000 values in the time range, the network will be busy sending all 250K values across the wire, and the client gets hit with 250K PIValue objects in memory.  With Summary, one value is sent.

                                           

                                          If you have obscenely dense data, the PIValues collection may be capped around 1 million values.  This would throw an exception.  However, the summary count method can be used to return a count > 1 million.  Don't expect it to be blistering speed with that many values.

                                            • Re: SDK 101 - Archive Time
                                              RyanBrown

                                              For this challenge it was simply the counts that were needed so they could get the top most changing tags. The network is fairly slow from what I understand so getting only the counts would be the best bet.

                                               

                                              The counts are probably what I would deem as normal and if there is anything over 1 million then that probably tells us there is a problem that needs looked at.

                                                • Re: SDK 101 - Archive Time
                                                  Rick Davin

                                                  @Ryan, I'd be interested to see if the recommended changes improved performance by a significant amount.

                                                    • Re: SDK 101 - Archive Time
                                                      RyanBrown

                                                      Been a busy week but will hopefully manage to post the performance comparison on Friday at the latest.

                                                        • Re: SDK 101 - Archive Time
                                                          RyanBrown

                                                          Okay finally got a chance to run the two programs and have the results below.

                                                           

                                                          The system it was running on had a lot of tags and I could not realistically run it to completion. Instead what I did was every 5 minutes output the count to the console of how many ( in total ) has been processed.

                                                           

                                                          This is the result from the code NOT using summary call

                                                           

                                                          Processed: 457
                                                          Processed: 1414
                                                          Processed: 2332
                                                          Processed: 3251

                                                           

                                                          and here is the result from the code using the summary call.

                                                           

                                                          Processed: 1080
                                                          Processed: 3151
                                                          Processed: 5197
                                                          Processed: 7204

                                                           

                                                          Looks here like the summary call is a good bet when just wanting the count.