8 Replies Latest reply on Aug 9, 2018 9:06 AM by Roger Palmen

    AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?

    erikjlee

      Hi,

       

      I have a small winform app that will be subscribing to > 100,000 (perhaps ~300k) AFAttribute events via an AFDataPipe instance. Right now it subscribes to ~1500 AFAttributes which takes about 15 seconds for AFAttribute.FindElementAttributes() to query. The process memory goes up to about 400mb. Knowing the .NET process limit is 2GB, will this be feasable? Will the querying of the AFAttributes take minutes to load?

       

      My other thought was to subscribe to the  PIPoint events instead, and then fetch the associated AFElement when I do receive an event (my app displays static AFAttrributes when an event is fired).

       

      Thanks!

        • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
          Rick Davin

          Hi Erik,

           

          Right now the performance issue you have isn't with AFDataPipe but rather in finding the attributes you wish to subscribe to.  You are using an older search method with AFAttribute.FindElementAttributes.  You may want to consider using the new AFAttributeSearch class.  Since attributes are required, this does mean a full load is needed for the elements, and that does consume extra memory and take extra time.  You would have the benefit of better paging and using server-side caching, which is a performance boost.  Will querying of 300K attributes take minutes to load?  Probably.

           

          Should you subscribe to PIPoints instead?  How do you plan on finding 300K PIPoints?  Is there a clean naming pattern, or do you need to query the attributes first and then get their PIPoints?  This seems like more work on your end, as well as more time, but really it sounds like you need an asset-based application since you must later fetch the associated AFElement.  I would therefore stick to it being asset-based.

           

          The real question I have regarding performance isn't how many attributes, but rather how many elements are used?  Is there 1 attribute per element, or do you have many attributes per element?  Can the attributes be filtered by category or name?  The attribute search would require a nested element search, so how those elements can be filtered (template, category, name, etc.) would also affect the search.

          3 of 3 people found this helpful
            • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
              erikjlee

              Hi Rick,

               

              Thanks for the tip about the AFAttributeSearch class, I'll try that out.

               

              Also, good point on filtering out the right PIPoints. There is no unique pattern I can use; hence the AF structure must be used.

               

              This is for an electric AMI installation, so there will be 1 element per meter, and 1 "power status alarm" attribute per meter/element (this is what I'm subscribing to). These elements will have many other attributes, both PIPoints and static values (although I am only subscribing to one of them). I am hoping to eventually filter the attributes by an "office area" or "region" static attribute to coordinate the operational area to that in which a dispatcher monitors (this should redice the element/attribute count to 100k max).

               

              Erik

            • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
              dmoler

              Hi Erik,

               

              A single AFDataPipe instance is is constrained by the data rate more than the attribute count. If all your attributes are configured to use PIPoints, 100k should not be a problem. We recommend keeping the typical rate of snapshot events to PIPoint attributes in a data pipe to 50k. If you need more than this, you can create multiple AFDataPipe instances and poll them in parallel.

               

              As to the memory question - this is mostly a function of the elements and attributes loaded. Without knowing the specifics of what is loaded in your 1,500 case and what would be loaded in the 100k case, it's hard to say without doing the testing. If you think 1 pipe is okay for the data rate, I would give it a try and address problems if they arise.

              3 of 3 people found this helpful
                • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
                  erikjlee

                  Hi David,

                   

                  The data rate is very small. It will be idle most of the time - it is a "power status" of an electric AMI meter. So only when an outage occurs or power is restored will an attribute fire an event.

                   

                  The attributes I am subscribing to are 1:1 with a parent element. That parent element contains ~25 attributes that are PIPoints, and there's another 15 static attributes.

                   

                  Erik

                • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
                  Roger Palmen

                  Not sure if this helps, but if AF datapipe scales as well as the PI datapipe, up to 1M subscriptions i can confirm to work properly in a custom .NET application

                  1 of 1 people found this helpful
                    • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
                      dmoler

                      The AFDataPipe and PIDataPipe share the same infrastructure under the hood. There is a small overhead added in the PIPoint to AFAttribute translation but it is pretty insignificant. The one place where the overhead is nontrivial is on adding signups. AFAttributes backed by PIPoints need to have their points resolved before signing the points up. We do this in bulk for all attributes in an AddSignups call so it is as fast as it can be, but this step is much slower than registering the signup with the Data Archive (which needs to be done for both AFDataPipe and PIDataPipe).

                      1 of 1 people found this helpful
                        • Re: AFDataPipe Scalablility - 100,000 AFAttribute subscriptions?
                          Roger Palmen

                          Thanks for the additional information! Good to hear it should be similar and only the (one-time) signups can cause some differences. But one-time performance impacts should not be a major deal when dealing the the datapipe scaleability.

                           

                          I am a but curious if and how the AFDataPipe works for non-PIPoint attributes, as it cannot use the existing infrastructure of the PI Update Manager, but that's a different subject.