19 Replies Latest reply on Oct 7, 2013 8:47 AM by Paurav Joshi

    Retrieval of snapshot values from AF-Performance issue

    Paurav Joshi

       

       

      Hi,

       

      I have one Query regarding retrieval of AF values for Element attributes in Asset Framework(AFSDK).It is like I am able to retrieve the values for my attributes but it is taking too much of time to do so.I have like 319 sub elements and there are around 20 attributes for each element those that I am interested in.I need to fetch the current values of each of these in every 5 minute interval(using getvalue function).This is taking too long(for around 25 elements around 2.5 minutes).Is there an alternative approach or am I missing out on something.Please suggest.

       


      Thanks

        • Re: Retrieval of snapshot values from AF-Performance issue
          lZheng

          Hi, Paurav

           

          There is other way to retrieve data from AF – using PI SQL Commander with PI OLEDB enterprise, but I doubt it would be faster.  

           

          May I ask which version of AF SDK you are using? AF 2.5 SDK with rich data access would be significantly faster than previous version.

           

          There can be different reasons why it took long time to get the data, could you share your code snippets so we can have a look?  Also is your AF server on a different machine far away from your client machine where you make the request?

            • Re: Retrieval of snapshot values from AF-Performance issue
              skwan

              Are you retrieving values one-at-a-time?  If so, you need to use bulk calls.  It will dramatically cut down on your data retrieval time.

                • Re: Retrieval of snapshot values from AF-Performance issue
                  Marcos Vainer Loeff

                  Hello Paurav,

                   

                  Please refer to the third exercise from the vCampus Live! 2012 workbook called “Accessing PI AF Server and PI Server Data using the PI AF SDK”. You can have access to this document by downloading all vCampus Live! 2012 material from the Download Center under Extras category. The third exercise is about retrieving data in bulk.

                   

                  Hope this helps!!

                    • Re: Retrieval of snapshot values from AF-Performance issue
                      Paurav Joshi

                      Hi All,

                       

                      Thanks for your responses.

                       

                      @ Steve,

                       

                      I am retrieving bulk data in a sense that Getvalue function is extracting data for all the attributes listed in my attribute list,one element at a time(All 17 attribute values for a single element).Is there a way out wherein I can retrieve values for all my elements in one shot ,considering that they are based on a common template?

                       

                      @Lingli,

                       

                      My AFSDK version is 2.0.And the AF client version happens to be 2.5.1.5159.Also Working with Visual studio Framework 3.5 .Please let me know as to how do I switch to 2.5 Rich Data Access and will that be helpful enough considering the fact that my Server machine is far from my client machine as rightly pointed out by out.

                       

                      Thanks

                        • Re: Retrieval of snapshot values from AF-Performance issue
                          lZheng

                          I believe you already have the AF client version with Rich Data Access installed, you can double check the AFSDK reference in your solution,

                           

                          It should be referenced from PIPC\AF\PublicAssemblies\4.0\OSIsoft.AFSDK.dll, with Runtime Version 4.0.

                           

                          Since your Server machine is far from your client machine, it would be very helpful if you could reduce the round trip when requesting data.  I see the question you had for Steve, you may want to check out AFAttribute.FindElementAttributes method,  you can retrieve all the element attributes that belong to a certain AFElementTemplate. You can find the example in the workbook from vCampus Live! 2012 suggested by Marcos.

                            • Re: Retrieval of snapshot values from AF-Performance issue
                              Paurav Joshi

                              Hi Lingli,

                               

                              I tried checking for the AFSDK version,it showed 2.0 and also I could not add the reference pointed as suggested,it gives me the message"Could not find the reference specified" although it is there in path mentioned.I am still trying for this,if any clue please let me know.

                               

                              Regarding the exercise content pointed by Marcos,it doesn't serve the purpose as I am intrested in attributes which have formula data reference and not PI Points for which it is given.

                               

                              Lastly,AFAttribute.FindElementAttributes  would give me all attributes whereas I am interested in only one-third of them,so what is the point getting all of them .Also as per the hierarchy that I have,I cannot filter it via attribute category or attribute name(specifying wildcards)etc.

                               

                              So I am just interested in getting snapshot values of these 1/3 rd of my attributes for all my elements without much delay.

                               

                              Thanks

                                • Re: Retrieval of snapshot values from AF-Performance issue

                                  Your project has to be set to compile against .Net Framework >= 4.0 to use RDA. You said earlier your compiling against 3.5.

                                   

                                  Formula Data Reference is known for its performance overhead, more so based on the complexity of the calculation. Can't the calculations be pre-calculated so you're only fetching the data and not fetching data & calculating on request? (Abacus would help here shortly.) Are the formulas based on other attributes that are using PI Point DRs, or some other Data References? How complex are the calculations (can they be done as on-the-fly performance equations)? You need to understand where the latency is being introduced - network, formula DR, source DRs, ...

                                   

                                  FindElementAttributes will do what you originally asked, getting Attributes based on an Element Template. If you're only interested in 1/3 of them then you can easily filter client side, then check the AF Database for changes (FindChangedItems) rather than re-querying for the complete Attribute List each time.

                                   

                                  You may need to manage expectations if you can't optimise your calculation method; the Formula DR doesn't scale well.

                                    • Re: Retrieval of snapshot values from AF-Performance issue
                                      Paurav Joshi

                                      Thanks Rhys,

                                       

                                      So I cannot use RDA as I am restricted towards using .Net Framework 3.5.

                                       

                                      As far as formula DR is concerned,it is just for avoiding and handling of bad data.As in no complex calculation is going on.To be specific the attribute has a child attribute linked to a PI Tag and if the value is good it goes into the parent else a -ve no. indicating a bad value.Also calculations are not in my hand and scope as part of the project,so I can't do anything with them.It's moreover to do with data retrieval only,that too in an optimized manner.

                                       

                                      So, FindElementAttributes seems to be the way out as of now.Can you please possibly share a piece of code as an example,that would indeed be helpful.

                                        • Re: Retrieval of snapshot values from AF-Performance issue
                                          lZheng

                                          Hi Paurav, 

                                           

                                          Following is a piece of code retrieving all the attributes that belong to an element Template called "Transformers" within the database, since you mentioned most of your elements are based on a common template.  

                                           
                                                      PISystems Systems = new PISystems();
                                                      AFDatabase database = Systems.DefaultPISystem.Databases["TransDistribuCo"];
                                                      AFElementTemplate myTemplate = new AFElementTemplate();
                                                      myTemplate = AFElementTemplate.FindElementTemplate(Systems.DefaultPISystem, new Guid ("f7749c8f-c89b-4476-a396-fcb51fc5144e"));                //Retrieve the element template using the Guid.     
                                          
                                                      AFAttributeList allAttributes = AFAttribute.FindElementAttributes(database, null,
                                                         "*", null, myTemplate , AFElementType.Any,
                                                         "*", null, TypeCode.Empty, true,
                                                         AFSortField.Name, AFSortOrder.Ascending, 100000);
                                          
                                                      Console.WriteLine("There are " + allAttributes.Count + "attributes using Transformer Element Template.");
                                          

                                           

                                            • Re: Retrieval of snapshot values from AF-Performance issue
                                              cmanhard

                                              You can create your own AFAttributeList (new AFAttributeList), and use the AFAttributeList Add and AddRange methods to create your custom list of attributes for which you wish to retrieve data.  This works in either the 2.0 or 4.0 version of the AF SDK.

                                                • Re: Retrieval of snapshot values from AF-Performance issue
                                                  Paurav Joshi

                                                  Hi Chris,

                                                   

                                                  This is what I am actually interseted in-"creating my custom list of attributes for retrieving data" and I am also able to do so.Just one thing here-When I loop through all my elements I have to add the attributes to my Attribute List for every element one at a time.Isn't there a way out to create the list on one shot for all my elements?If yes,please let me know.

                                                   

                                                  Thanks

                                                    • Re: Retrieval of snapshot values from AF-Performance issue
                                                      Marcos Vainer Loeff

                                                      Hello Paurav,

                                                       

                                                      Please try the code snippet below:

                                                       

                                                       

                                                       
                                                      using System;
                                                      using System.Collections.Generic;
                                                      using System.Linq;
                                                      using System.Text;
                                                      using OSIsoft.AF;
                                                      using OSIsoft.AF.Asset;
                                                      
                                                      namespace CreateAFAttributeLists
                                                      {
                                                          class Program
                                                          {
                                                              static void Main(string[] args)
                                                              {
                                                                  PISystems myPISystems = new PISystems();
                                                                  PISystem myPISystem =  myPISystems["MARC-PI2012"];
                                                                  AFDatabase myDb = myPISystem.Databases["Training Sample"];
                                                                  Console.WriteLine(myDb.Name);
                                                                  Application myApp = new Application();
                                                                  AFAttributeList myAttributeList = myApp.GetAllAttributes(myDb);
                                                                  Console.WriteLine("There are " + myAttributeList.Count + " attributes on the database.");
                                                                  Console.ReadKey();
                                                      
                                                              }            
                                                          }
                                                      
                                                          public class Application
                                                          {
                                                              private AFAttributeList myAttributeList;
                                                              public AFAttributeList GetAllAttributes(AFDatabase myDb)
                                                              {
                                                                  myAttributeList = new AFAttributeList();
                                                                  foreach (AFElement myElement in myDb.Elements)
                                                                  {
                                                                      GetAttributesFromElement(myElement);
                                                                  }
                                                                  return myAttributeList;
                                                      
                                                              }
                                                      
                                                              private void GetAttributesFromElement(AFElement myElement)
                                                              {
                                                                  myAttributeList.AddRange(myElement.Attributes);
                                                                  //Console.WriteLine("Element path:" + myElement.GetPath());
                                                                  if (myElement.HasChildren)
                                                                  {
                                                                      foreach (AFElement SubElement in myElement.Elements)
                                                                      {
                                                                          GetAttributesFromElement(SubElement);
                                                      
                                                                      }
                                                                  }
                                                      
                                                      
                                                              }
                                                      
                                                          }
                                                      }
                                                      

                                                      You can also use the AFAttribute.FindElementAttributes() method but I am not sure which one is more efficient.

                                                       


                                                      Hope this helps you!

                                                      • Re: Retrieval of snapshot values from AF-Performance issue
                                                        cmanhard

                                                        You have said previously that you are only interested in 1/3 of your attributes, and that this third cannot be identified by a search (category or name).  While you can add multiple attributes to the list at one time (for example myAttributeList.AddRange(AFElements.Atttributes) ), and you can apply filtering here as well (example, myAttributeList.AddRange(AFElements.Attributes.FilterBy(myAttributeCategory))  ), can you explain how you would expect to inform the AFAttributeList which attributes it should add in your scenario?

                                                          • Re: Retrieval of snapshot values from AF-Performance issue
                                                            cmanhard

                                                            Marco's example is "walking the hierarchy".  In general, unless processing in hierarchy order is important to your algorithm, it is generally faster to:

                                                             

                                                            1) Execute one of the AFElement.Find methods find all elements you are interested (effectively, a flattened list of elements)

                                                             

                                                            2) Execute AFElement.LoadElements on the list returned (loads the attributes to those elements in bulk)

                                                             

                                                            3) Then build up your attribute list from these loaded elements.

                                                             

                                                            This significantly reduces the number of RPC's to the AF server to 2.  In Marcus' example, there will be either 2 RPC's for each element with children, 1 RPC for leaf elements).  

                                                              • Re: Retrieval of snapshot values from AF-Performance issue
                                                                Marcos Vainer Loeff

                                                                Hello Paurav,

                                                                 

                                                                Following Chris' suggestions, the new code snippet would be:

                                                                 

                                                                 

                                                                 
                                                                    PISystems myPISystems = new PISystems();
                                                                            PISystem myPISystem =  myPISystems["MARC-PI2012"];
                                                                            AFDatabase myDb = myPISystem.Databases["Training Sample"];
                                                                            AFAttributeList myAttributeList = new AFAttributeList();
                                                                            AFNamedCollectionList<AFElement> foundElements = AFElement.FindElements(
                                                                                myDb,                   //AF Database
                                                                                null,                     //The root object to start the search. If null reference (Nothing in Visual Basic), then the search starts at the AFDatabase. 
                                                                                "*",                  //Query
                                                                                AFSearchField.Name,       //Specifies which of the object's properties are searched. 
                                                                                true,                     //Specifies if the search should include objects nested further than the immediate children of the searchRoot. 
                                                                                AFSortField.Name,         //The field or property of the object used to sort the returned collection. 
                                                                                AFSortOrder.Ascending,    //The order that the returned collection is sorted. 
                                                                                100000);                    //The maximum number of objects to be returned. 
                                                                
                                                                            AFElement.LoadElements(foundElements);
                                                                            foreach (AFElement myElement in foundElements)
                                                                            {
                                                                                myAttributeList.AddRange(myElement.Attributes);
                                                                            }
                                                                

                                                                 

                                                                 
                                                                There is another example from the AF SDK programming reference called "Loading Top Level Objects". It is slightly more complicated due to the fact that it deals with "paging" through the collection in case there is a large number of elements involved.

                                                                • Re: Retrieval of snapshot values from AF-Performance issue
                                                                  Paurav Joshi

                                                                  Hi Chris,

                                                                   

                                                                  As per your last post,I am reaching out to my desired elements and loading them in one go using AFElement.LoadElements .But when it comes to getting values for the desired attributes(one-third of them),I am hard coding their names as of now,but have thought of picking them from the configuration file.But I am not very much convinced with this approach.Kindly see the code snippet below for reference.

                                                                   

                                                                  For Each W_element As AFElement In myW_Elements

                                                                   

                                                                                    Dim attributes As AFAttributeList = New AFAttributeList

                                                                   

                                                                                     attributes.Add(W_element .Attributes("A"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("B"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("C"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("D"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("E"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("G"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("F"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("P"))

                                                                   

                                                                                     attributes.Add(W_element .Attributes("S"))

                                                                   

                                                                                      afvalues = attributes.GetValue()

                                                                   

                                                                                      For Each W_attribute As AFAttribute In attributes

                                                                   

                                                                                               //Some code processing etc.

                                                                   

                                                                                     Next

                                                                   

                                                                  Next

                                                                   

                                                                  Thanks

                                                                    • Re: Retrieval of snapshot values from AF-Performance issue

                                                                      You'd want to do it outside of the Element loop so that the values for the 1/3 attributes of all Elements are retrieved in one go, then you can regroup the returned AFValues collection per Elements to parse. Something like: 

                                                                       
                                                                                  OSIsoft.AF.Asset.AFElement.LoadElements(elements);
                                                                      
                                                                                  string[] attributeNames = new[] { "A", "B", "C", "D", "E", "G", "F", "P", "S" }; // Or get from configuration file.
                                                                                  OSIsoft.AF.Asset.AFAttributeList list = new OSIsoft.AF.Asset.AFAttributeList();
                                                                                  foreach (OSIsoft.AF.Asset.AFElement element in elements)
                                                                                  {
                                                                                      list.AddRange(element.Attributes.Where<OSIsoft.AF.Asset.AFAttribute>(e => attributeNames.Contains(e.Name)));
                                                                                  }
                                                                      

                                                                       

                                                                        • Re: Retrieval of snapshot values from AF-Performance issue
                                                                          cmanhard

                                                                          As Rhys mentions, you are building and retrieving your AFValues within the for each loop.  This means if you have 1000 elements, you will make 1000 RPC's to the PI Server to retrieve the data.  You can improve the performance by moving the creation of the attributelist above the foreach, and the invoking GetValue and processing the results below the for each, combining all elements and attributes into one call to the PI Server.

                                                                           

                                                                          Whether you add attributes to the list one by one or with AddRange as Rhys demonstrates will not make a significant difference to the performance since these are all in memory client side operations.  The performance is achieved by reducing RPCs to the AF and PI Server.