Hi All, This is a general question seeking guidance on the dealing with a large number of elements/attributes when using AFSDK.
I am developing an engine which needs to run in near real time.
It needs to retrieve all elements in an extensive AF structure (~3000 parent elements with each having an average of 8 children = ~24000 elements).
I then need to monitor for attribute value changes across attributes on an average of 20 attributes for each child and trigger calculations which are done by another system/engine). This makes it an approximate total of 480,000 attributes for monitoring of data in near real-time.
Some specific questions based on the above:
- What is the best/fastest method to BULK load the 24000 elements we need to load. This will be refreshed on a once a day cycle only as I am expecting the initial load to be slow and hence trying to determine what the fastest method will be.
- For monitoring attribute value changes we are thinking of using AF Data Pipes. Is there some performance and scalability metric available for AF Data Pipes and will it be able to handle monitoring of ~480 000 attributes for value changes?
- Also, in order to add signups to the AFDataPipe, we would need to loop through all the attributes across all the elements. Are there any recommended approaches to traversing through this large number of AF Attributes?
Looking forward to the discussions around this.