6 Replies Latest reply on Aug 7, 2017 9:15 PM by dmoler

    AF SDK and dealing with a large number of elements/attributes

    neilg

      Hi All, This is a general question seeking guidance on the dealing with a large number of elements/attributes when using AFSDK.

       

      I am developing an engine which needs to run in near real time.

       

      It needs to retrieve all elements in an extensive AF structure (~3000 parent elements with each having an average of 8 children = ~24000 elements).

       

      I then need to monitor for attribute value changes across attributes on an average of 20 attributes for each child and trigger calculations which are done by another system/engine). This makes it an approximate total of 480,000 attributes for monitoring of data in near real-time.

       

      Some specific questions based on the above:

      1. What is the best/fastest method to BULK load the 24000 elements we need to load. This will be refreshed on a once a day cycle only as I am expecting the initial load to be slow and hence trying to determine what the fastest method will be.
      2. For monitoring attribute value changes we are thinking of using AF Data Pipes. Is there some performance and scalability metric available for AF Data Pipes and will it be able to handle monitoring of ~480 000 attributes for value changes?

      3. Also, in order to add signups to the AFDataPipe, we would need to loop through all the attributes across all the elements. Are there any recommended approaches to traversing through this large number of AF Attributes?

       

      Looking forward to the discussions around this.

        • Re: AF SDK and dealing with a large number of elements/attributes
          Rick Davin

          Hi Neil,

           

          Interesting questions.  Got a few of my own for you:

           

          • Are all the elements based on template(s)?  How many templates?
          • Are all 480K attributes based on attribute templates?
          • What data references are used by the attributes?  Can you give a rough breakdown, e.g. 70% PI points, 10% table lookups, 20% formula?
          • Are any of the attributes mapped as outputs from analyses?
            • Re: AF SDK and dealing with a large number of elements/attributes
              neilg

              Hi Rick, See my responses in red.

               

              • Are all the elements based on template(s)?  How many templates?

                Yes they are based on templates (2-4) at most.


              • Are all 480K attributes based on attribute templates?

                Yes they are.

              • What data references are used by the attributes?  Can you give a rough breakdown, e.g. 70% PI points, 10% table lookups, 20% formula?

                ~80% on PI points, rest are on Table lookups (but they need to be monitored separately as there issues with AFDataPipes monitoring AF Table lookup value changes). I guess which reduces the 480K by 20% but still a high number.

              • Are any of the attributes mapped as outputs from analyses?

                None (or potentially a very small number).

              • Re: AF SDK and dealing with a large number of elements/attributes
                neilg

                I have been playing with the various retrieval methods in AFSDK and have come to the conclusion that AFElementSearch (non-cached as we will refresh once a day only) will allow for the fastest retrieval when compared to other methods available in AF SDK (FindElementsByTemplate, FindInstantiatedElements, simple traversing across all attributes, etc). This answers the first of my questions.

                 

                Next is around how much AFDataPipe will be able to handle considering the large number of attributes we will be dealing with. I am researching this area as well but would be good if someone has some pointers as well.

                  • Re: AF SDK and dealing with a large number of elements/attributes
                    dmoler

                    The AFDataPipe is bound more by data throughput rate than attribute count.  We recommend planning steady-state operation of no more than ~50,000 PI events/s for each AFDataPipe (they can handle >150,000 but room should be left in case events get buffered so there is capacity to catch up). Each AFDataPipe is independent so you can create multiple instances and poll them in parallel if you have higher throughput requirements.

                     

                    As you mentioned, there can be more overhead for other types of events so if there are many formula or table lookups, the load should be spread further. What is the nature of the table lookups configuration that makes the data pipe unsuitable?

                      • Re: AF SDK and dealing with a large number of elements/attributes
                        neilg

                        Thanks for the pointer David. Is there some published resource which has the data throughput information somewhere? What you have put in is exactly what I wanted to be aware of to cater for, in the component we are building.

                         

                        What I can see from tests I have carried out and from the documentation is that if an AF Attribute is referencing a static AF Table lookup, changes done on the underlying data will not be picked up by the AF Datapipe. I have had this confirmed by OSIsoft techsupport some time back as well.

                          • Re: AF SDK and dealing with a large number of elements/attributes
                            dmoler

                            Hi Neil,

                             

                            I'm not sure if there is a published resource with data throughput - it is tricky because it can depend on so many different factors.

                             

                            Regarding subscribing a table lookup to an AFDataPipe - how it behaves will differ depending on the configuration of the Table Lookup data reference. If it is configured with a time column (if the table is providing the timestamps) then this will be polled periodically based on the tables cache time and it will reflect changes to the table.  If the table lookup is not configured with a time column but has attribute inputs in its WHERE clause (I assume this is your situation), then the data pipe will trigger evaluations when new input events arrive.  If the table lookup uses a static attribute and the changes are to the table itself, then the data pipe doesn't currently know to relate those changes to the attribute so you won't get update events.  It sounds like that is the situation you are in.

                             

                            The AFDataCache was enhanced in 2017 (2.9) to recognize cases where events cannot be generated for some changes. In these cases the inputs rather than the outputs are cached and the table lookup is evaluated on-demand using cached inputs. We plan to generate change events in more situations in future releases.