5 Replies Latest reply on Apr 6, 2016 3:31 PM by pthivierge

    What is the best way to get hourly data using AFAttribute?

    eradwan

      Hi There,

       

      I'm trying the following code where I need hourly data for a time range, however, I noticed that the returned values aren't stamped exactly at the beginning of the hour although the time range begins at mid-night! Is this the best way to navigate from AF attribute to its pi data?

       

      AFDatabase eMDatabase = connection.GetDatabase();
      AFCategory nOxWeeklyEmissions = eMDatabase.AttributeCategories["NOxWeeklyEmissions"];
      
      
      AFAttributeList nOxAttribList = AFAttribute.FindElementAttributes(eMDatabase,null, "*",null, null, AFElementType.Any, "*", nOxWeeklyEmissions, TypeCode.Empty, true, AFSortField.Name, AFSortOrder.Ascending, 100);
      
      
                      var strDate = fromDate.Substring(0, 24);
                      DateTime startSearchDate = DateTime.ParseExact(strDate, "ddd MMM d yyyy HH:mm:ss", CultureInfo.InvariantCulture);
                      string sDate = startSearchDate.ToString();
                      string eDate = (startSearchDate.AddDays(7)).ToString();
      
      
                      timeRange = new AFTimeRange(new AFTime(sDate), new AFTime(eDate));
      
      
                      foreach (AFAttribute attribute in nOxAttribList)
                      {
                          AFValues values = attribute.GetValues( timeRange, -186, null);
                      }
      
        • Re: What is the best way to get hourly data using AFAttribute?
          Kenji Hashimoto

          You can use Attribute.Data.InterpolatedValues method to get hourly interpolated values. For example, try following.

          AFValues values = attribute.Data.InterpolatedValues(timeRange, AFTimeSpan.Parse("1h"), null,"",true);
          

          For more information,

          AFData.InterpolatedValues Method

          • Re: What is the best way to get hourly data using AFAttribute?
            pthivierge

            Hello Emad,

             

            I can see from the code you provided that you have many attributes and that you are performing a for-each loop to get the data for each attribute, one after the other.  I don't know how many attributes you have, if the number is small your current logic may be ok, otherwise you should aim for the usage of Bulk calls: this is another type of call where you get/write the data for all the attributes/tags at once. 

            It has the following advantages:

            • Latency over the network is encountered only once and will not sum-up as for each call in a for loop. This can be a huge performance gain if you have a lot of attributes to get data from.
            • There is internal optimization when receiving the results made with paging collections that limits the number of events loaded in memory, events will be loaded as you iterate over the result set returned by the bulk call.
            • Your code will be ready to support more attribute over time

             

            There are several other factors that can impact performances, I'd recommend you have a look at AF SDK performance, serial vs. parallel vs. bulk, this is a very interesting discussion.

             

            To make use of bulk calls you need to start with a list of Tags or Attributes, those lists are exposing methods to make that bulk calls:

            • AFAttributeList for attributes. ( Bulk methods are under the Data property, which is an AFListData object.)
            • PIPointList if you are manipulating PIPoints directly. (AFAttribute object has a PIPoint object set to not null when the attribute is a PI Point)

             

            Example:

            DateTime st = DateTime.Today - TimeSpan.FromDays(7);
            DateTime et = DateTime.Today;
            
            
            var connection = new AfConnectionHelper(Server, Database);
            connection.Connect();
            
            
            AFDatabase database = connection.GetDatabase();
            
            
            AFCategory attrCategory = database.AttributeCategories["energy"];
            AFAttributeList afAttributeList = AFAttribute.FindElementAttributes(database, null, "*", null, null, AFElementType.Any, "*", attrCategory, TypeCode.Empty, true, AFSortField.Name, AFSortOrder.Ascending, 100);
            
            
            // this is the bulk call
            // it returns one list (AFValues) per attibutes
            IEnumerable<AFValues> interpolatedValues = afAttributeList.Data.InterpolatedValues(new AFTimeRange(st, et), AFTimeSpan.Parse("1h"), string.Empty,
                false, new PIPagingConfiguration(PIPageType.TagCount, 5000));
            
            
            var listSeparator = CultureInfo.CurrentCulture.TextInfo.ListSeparator;
            
            
            // goes over each attribute and get its results
            foreach (var afValues in interpolatedValues)
            {
                // prints the results
                afValues.ForEach((afvalue) => Console.WriteLine("{0}{1}{2}{3}{4}", afvalue.Value.ToString(), listSeparator, afvalue.Timestamp, listSeparator, afvalue.Attribute.Name));
            }
            

             

            I hope this answers your questions, and you will see that the methods (Attribute, PIPoint) to get data out of the PI System are really numerous and rich, if you can't find what you need just let us know J

             

             

             

             

            • Re: What is the best way to get hourly data using AFAttribute?
              eradwan

              Many Thanks Patrice.

               

              One clarification! How do you calculate the page size parameter? Assume that I have 10 attributes and I aiming for hourly averages over a week or month.

               

              IEnumerable<AFValues> interpolatedValues = afAttributeList.Data.InterpolatedValues(new AFTimeRange(st, et), AFTimeSpan.Parse("1h"), string.Empty,

                  false, new PIPagingConfiguration(PIPageType.TagCount, 5000));

                • Re: What is the best way to get hourly data using AFAttribute?
                  pthivierge

                  Hello Emad,

                   

                  For example, is you leave it to 5000 ( something that works generally well), this just means that you should get all the data in one page.  Because the partitioning here is made by tag/attribute.

                  Functionally in the code that changes nothing.

                   

                  In the case you get a lot more attributes, e.g. 10 000 attribute.  And your interpolate values call gets expensive for the PI Data Archive, you could lower the page size to 1000 for example, to make smaller calls at the time.

                  Another thing is the page size needs to be balanced between:

                  • the load put on the PI System to return one page => the bigger the page size and time range, the more load is put on the PI System at one point in time to serve this request
                  • the number of total network calls made => the smaller the page size, the more network calls are made
                  • the time it takes to your application to process one page => if your time range is big and the data density you are retrieving is high, you will get a lot of data.  You client application will need to process it and this will take time.  In these situations it make sense to make your page size smaller so you save memory on the client application, and you are also saving server-side resources by balancing the data requests into several smaller pages rather than a big one.

                   

                   

                  I am not aware of a way to calculate this, the number I provided are from experimentation, let's see roughly that somewhere between 1000 and 10 000 works well. 

                  If your data density is high your page size should be closer to 1000, if not, somewhere between 5000 and 10 000. 

                  The size of the system and network latency may also impact the choice of these numbers.  I usually make it as a parameter so I can change it and make several runs to determine what works better for a specific PI System installation.

                   

                  For your specific case, I would not worry too much and start with 1000.

                   

                  Hope this helps J

                  1 of 1 people found this helpful