12 Replies Latest reply on Dec 1, 2015 12:05 PM by PraveenCTS

    How to retrieve time range data for each Attribute in bulk?

    PraveenCTS

      Hi all,

      I am trying to retrieve bulk data using AF SDK every five seconds.

       

      Since i need to retreive data corresponding to each attribute, What i am doing is..

      First i am getting all the elements then, i tried to get attribute list corresponding to each element, then i tried to get 5 seconds data for each attribute as below:

       

      foreach (var element in elements)
                      {
                          AFAttributeList attributes = new AFAttributeList(element.Attributes);
                          var afvalues = attributes.GetValue(myTimeRange);
      Parallel.ForEach(afvalues, afValue =>
                          {
      Console.WriteLine("TagName" + afValue.Attribute.Name);
                                  Console.WriteLine("Tagvalue" + afValue.Value); 
                                  Console.WriteLine("Timestamp" + afValue.Timestamp.ToString());
      }
      

       

      Now i am passing timerange of five seconds to get AFValues from my Client

      var afvalues = attributes.GetValue(myTimeRange);

      .. That means for each attribute i should get 5 records but instead i am getting single record only..

       

      Also in System Explorer there is setting button from where you can set the Data Reference configuration to get bulk data based on time range..

       

      What should be my configuration so that when client passes the timerange, he should get the pi point data based on that time range..

       

      Looking forward to your solutions. Thanks in advance.

       

      Message was edited by: Praveen Odughat

        • Re: How to retrieve time range data for each Attribute in bulk?
          Rick Davin

          Though you may pass a time range to the GetValue method, GetValue will still only return 1 value per attribute.  It's left up to each data reference in how to implement what to do when a time range is passed as the time context.  Some DR's may return a value at the time range's StartTime.  Others may return the EndTme.  Nonetheless, only 1 value per attribute.

           

          You probably are aware that AFAttributeList does not have a GetValues method. 

           

          You may use the AFAttributeList.Data.RecordedValues method.  This requires a rich data access (RDA) call.  Again its up to whether each data reference supports the RDA call.  You must use AF 2.5 or higher.  And your code must use the 4.0 version of the AFSDK.

          1 of 1 people found this helpful
          • Re: How to retrieve time range data for each Attribute in bulk?
            pthivierge

            Hello Praveen,

             

            Could you tell a bit more about what you need to do with the data?

             

            If you need data every 5 seconds, that would be best to use a DataPipe, this is by far the best way to gather realtime data as it comes, there is an implementation example in the clues project: osisoft/PI-AF-SDK-Clues · GitHub

            However, I might have understood wrongly!

             

            It would be good to know how you are gathering your elements, do you get the same elements each time or do you keep them in memory?

             

            Gather Elements

            What I'd recommend is that you get your elements only once in your program initialization and keep a reference to those in your application when it is running so you don't need to query elements over and over.

             

            If you have a lot of elements to retrieve, you should consider using one of the method that returns a paged collection, this will reduce the amount of data that is sent into smaller chunks and let your application consume it by iterating over the elements collection.

             

            Load Attributes

            You should force the attributes to be loaded all at once, partially or all of them.  If you are not doing that, the AF SDK will lazy load them when the element attribute is accessed, and this is slow.  AF SDK is designed for clients applications such as PSE, so this is why default behavior is this way.

             

            To load the attributes, once you have your elements list, you can either load:

             

            Fully:

            AFElement.LoadElements(elementsList);
            

             

            Partially:

            AFElement.LoadAttributes(elementsList, new[] { attributeTemplate });
            

             

             

            Retrive the data

            If you need realtime data updates, you should really consider using the AFDataPipe.

            Otherwise, using the AFAttributeList should be efficient enough.

             

            Hope this helps,

            • Re: How to retrieve time range data for each Attribute in bulk?
              PraveenCTS

              Thanks for the reply RICK,PATRICE AND MARCOS. Loading the elementsList did help me in improving my code performance. Though it is not much but still it is acceptable. Also i used RecordedValues suggested by RICK. Thank you all for the help..

                • Re: How to retrieve time range data for each Attribute in bulk?
                  pthivierge

                  You are welcome Praveen,

                  In the case you need to optimize further, we'll need to see your full code.

                  --

                  Also, I'd encourage you to mark the correct answer so it closes the question and let others know what was the answer that helped you the most for this issue.

                  Its not always easy as there can be many answers,  in this case you can flag those other answers as Helpful.

                   

                  Have a nice day!

                    • Re: How to retrieve time range data for each Attribute in bulk?
                      PraveenCTS

                      Hi Patrice. I am attaching the full code of mine:

                       

                      using OSIsoft.AF;
                      using OSIsoft.AF.Asset;
                      using OSIsoft.AF.PI;
                      using OSIsoft.AF.Time;
                      using System;
                      using System.Collections;
                      using System.Collections.Generic;
                      using System.Diagnostics;
                      using System.IO;
                      using System.Linq;
                      using System.Net;
                      using System.Text;
                      using System.Threading.Tasks;
                      using System.Timers;
                      using System.Collections.Concurrent;
                      using OSIsoft.AF.Data;
                      
                      
                      namespace AFReadConsoleApp
                      {
                          class Program
                          {
                              private static PIServer _piServer;
                              private static PISystem _afServer;
                              private static List<AFElement> elementList;
                      
                      
                              static void Main(string[] args)
                              {
                                  Properties.Settings.Default.intervalTime = "";
                                  Properties.Settings.Default.Save();
                      
                      
                                  PIServers piServers = new PIServers();
                                  _piServer = piServers[Properties.Settings.Default.serverName];
                                  NetworkCredential netCred = new NetworkCredential(Properties.Settings.Default.userName, Properties.Settings.Default.password);
                                  _piServer.Connect(netCred);
                                  Console.WriteLine("PI Data Archive Connection Done.... ");
                      
                      
                                  var afServers = new PISystems();
                                  _afServer = afServers[Properties.Settings.Default.serverName];
                                  _afServer.Connect(netCred);
                                  Console.WriteLine("Connected to AF server : " + Properties.Settings.Default.serverName);
                      
                      
                                  var database = _afServer.Databases[Properties.Settings.Default.databaseName];
                                  var elements = database.Elements;  // we select the element we want to monitor
                                  elementList = new List<AFElement>();
                                  foreach (var element in elements)
                                  {
                                      elementList.Add(element);
                                  }
                      
                      
                                  AFElement.LoadElements(elementList);
                                  Timer aTimer = new Timer();
                                  aTimer.Elapsed += new ElapsedEventHandler(new Program().OnTimer);
                                  aTimer.Interval = 5000;
                                  aTimer.Enabled = true;
                                  Console.ReadLine();
                              }
                      
                      
                              private void OnTimer(object source, ElapsedEventArgs e)
                              {
                                  Program program = new Program();
                                  program.GetHsitoryData(_piServer, _afServer);
                      
                      
                                  System.GC.Collect();
                              }
                      
                      
                              public async void GetOSIPISampleDataAsync()
                              {
                                  var result = await Task.Run(() => GetHsitoryData(_piServer, _afServer));
                              }
                              private List<PIResponse> GetHsitoryData(OSIsoft.AF.PI.PIServer _piServer, OSIsoft.AF.PISystem _afServer)
                              {
                                  Console.WriteLine("Start of Execution : " + DateTime.Now + " Thread ID: " + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString());
                                  List<PIResponse> piResponseList = new List<PIResponse>();
                                  int recordCount = 0;
                      
                      
                                  try
                                  {
                      
                      
                                      AFTimeRange myTimeRange = new AFTimeRange();
                                      String propertyDate = null;
                                      if (string.IsNullOrEmpty(Properties.Settings.Default.intervalTime))
                                      {
                                          myTimeRange.StartTime = Convert.ToDateTime(DateTime.Now.AddSeconds(-5).ToString("yyyy-MM-dd HH:mm:ss"));
                                      }
                                      else
                                      {
                                          myTimeRange.StartTime = Convert.ToDateTime(Properties.Settings.Default.intervalTime);
                                      }
                      
                      
                                      propertyDate = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss");
                                      Properties.Settings.Default.intervalTime = propertyDate;
                                      Properties.Settings.Default.Save();
                                      myTimeRange.EndTime = Convert.ToDateTime(propertyDate).AddSeconds(-1);
                      
                      
                                      Console.WriteLine("Data Time Reange : " + myTimeRange);
                      
                      
                                      PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 100);
                                      PIResponse piResponse = new PIResponse();
                      
                      
                                      foreach (var element in elementList)
                                      {
                                          AFAttributeList attributes = new AFAttributeList(element.Attributes);
                      
                      
                                         
                      
                      
                                          piResponse.DeviceName = element.Name;
                                          ConcurrentQueue<PIData> piDataList = new ConcurrentQueue<PIData>();
                                          var afValuesList = attributes.Data.RecordedValues(myTimeRange, OSIsoft.AF.Data.AFBoundaryType.Inside, null, false, config);
                                          foreach (var afValues in afValuesList)
                                          {
                                              //Console.WriteLine("afValues count : " + afValues.Count());
                                              //Parallel.ForEach(afValues, afValue =>
                                              foreach (var afValue in afValues)
                                              {
                                                  PIData piData = new PIData();
                                                  piData.TagName = afValue.Attribute.Name;
                                                  piData.TagValue = afValue.Value.ToString();
                                                  piData.TimeStamp = afValue.Timestamp.UtcTime.ToString();
                                                  piDataList.Enqueue(piData);
                                                  piData = null;
                                              }//);
                                          }
                      
                      
                                          //Console.WriteLine("piDataList.Count : " + piDataList.Count);
                                          recordCount = recordCount + piDataList.Count;
                                          piResponse.PIDataList = piDataList.ToList();
                                          piDataList = null;
                                          piResponseList.Add(piResponse);
                      
                      
                                      }
                                      Console.WriteLine("Total records : " + recordCount);
                                      recordCount = 0;
                                      Console.WriteLine("End of Execution : " + DateTime.Now + " Thread ID: " + System.Threading.Thread.CurrentThread.ManagedThreadId.ToString());
                                  }
                                  catch (Exception ex)
                                  {
                                      Console.WriteLine("Error @ " + " is : " + ex.Message + "  " + ex.StackTrace);
                                  }
                                 
                                  return piResponseList;
                              }
                          }
                      }
                      

                      Now the problem is its taking more than 15 seconds to fetch 30000 records from osi pi historian which includes getting element then its attributeList and then the values..

                      How can i optimize the above code to run faster.

                      Sorry for the trouble and thanks in advance..

                        • Re: How to retrieve time range data for each Attribute in bulk?
                          pthivierge

                          Hello Praveen,

                           

                          Your application already uses a timer logic, so to me it would be event easier to use a data pipe. I don't want write all the code here, but let me highlight parts of your code and point you to appropriate resources.

                          --

                           

                          First, I'd recommend you to download the Clues examples, here. ( If you are familiar with GitHub, you might want to fork and clone instead).

                          Then look at the AFDataPipeLister example.

                          2015-11-30 14_41_52-PI-AF-SDK-Clues - Microsoft Visual Studio.png

                          If you compile it, you also test the datapipe in action on the command line, and this can give you an idea on how datapipes works:

                          clues AFDataPipeListener -s SRV-PI -d AFDatabase -a "Motors\Motor1|Temperature"
                          

                           

                          --

                           

                          Introducing the AFDataPipe will improve performances a lot. The PI Data Archive Server, through the PI update manager, will be keeping all the changes for you.  And you application just need to ask to get the latest updates.  This means that the system does not need to get all the data for all tags every time, it just keeps new values for you as they come in the system.

                           

                          To introduce the AFDataPipe in your code, I'd recommend that you use the very good DataPipeHandler class that was kindly provided by  Ian Gore.

                          • Add a static variable:  private static DataPipeHandler afDataPipeHandler
                          • You'll need to create new observer Class (YourDataObserverClass) , to receive the data and take the appropriate actions (i.e. insert the AFValue you are receiving into the concurrent queue), you should start with the AFConsoleDataObserverClass as a model that will be easier.  This is probably the part that changes the most to what you have now.
                          • In your main, right after you have loaded you elements and created an attributes list you then just need the following code and the data collection will start:

                           

                                          afDataPipeHandler = new DataPipeHandler(new YourDataObserverClass());
                                          afDataPipeHandler.AddSignupsWithInitEvents(attributes);
                                          afDataPipeHandler.StartListening(TimeSpan.FromSeconds(5));
                          

                           

                          You also may need to adjust the interval ( I have seen you already have settings for that ).

                          In this this case, interval is the frequency at which the DataPipeHandler will go and retrieve the data that the PI Data Archive (pi update manager) kept.

                          A good fast interval could be somewhere between .250 and 1 second.  And this could go up to 5-20 seconds  if you don't need fast updated.

                           

                          Using the update manager requires care, so be sure to call afDataPipeHandler.Dispose(); when your code terminates to avoid leaving unecesary subscriptions on the PI Data Archive. ( they would eventually timeout, but still this could cause issues if too many until the timeout comes.)

                           

                          I'll let you look into this first information, let me know if you have questions.