41 Replies Latest reply on Sep 24, 2015 5:47 PM by SvenBatalla

    What are the optimal settings for Filtered Summaries?

    SvenBatalla

      I have several calls to the FilteredSummaries method.  By "several" I mean upwards of 46,200 distinct calls.  Each individual call (which is usually about 8h worth of pretty normal data) seems pretty fast.  However, when they are placed in a loop, it takes a long, long time.  I'd like to know what the most efficient parameters are for collecting results from the FilteredSummaries call.

       

      Here's some background:

       

      The calls I am making utilize an expression that evaluates a tag against several other tags (e.g. limits & masks).  The "other" tags all have values in PI at a frequency of 1 minute.  The "source" tag is variable in frequency.  (I'm not sure if this plays a role, but better to explain it than leave it out.)  As alluded to earlier, the time range is typically 8-12h.  The goal of the expression is to count the number of MINUTES the value exceeded a limit.  So based on that, I have the following call to FilteredSummaries:

       

      IDictionary<AFSummaryTypes, AFValues> values = myElement.Attributes["My Source Tag"].Data.FilteredSummaries(
         timeRange,
         new AFTimeSpan(timeRange.Span),
         myFilterExpression,
         AFSummaryTypes.Count,
         AFCalculationBasis.TimeWeighted,
         AFSampleType.Interval,
         new AFTimeSpan(minutes:1),
         AFTimestampCalculation.Auto);
      

       

      The 2 parts I imagine need tweaking are the 2 parameter (summaryDuration) and the 2nd-last parameter (sampleInterval).  However, even when reading the document, I'm not sure how best to tweak these values to get the best performance while not sacrificing the clarity of the result (which is technically the # of seconds the value was outside the limits).

       

      Alternatively, maybe someone can tell me if FilteredSummaries can be run in bulk like other data calls can?  Maybe the answer is to perform many calculations at once rather than all of them individually.  This would be akin to getting a bunch of snapshot values at once rather than tag-by-tag.  Is this possible with FilteredSummaries?

       

      Thanks!

        • Re: What are the optimal settings for Filtered Summaries?
          SvenBatalla

          I should mention the performance.  Individually, a call takes about 00:00:00.0520727.  Placing all 46k calls in a straight loop takes ~00:45:00.0000000.  Placing the calls in a parallel loop takes about 00:01:43.5347576.  I need this to complete in less than 30s at a minimum.

            • Re: What are the optimal settings for Filtered Summaries?
              Rhys Kirk

              Indeed, much like your post yesterday the PIPointList will likely help you again here. It supports FilteredSummaries.

              So from your code yesterday you've already pre-loaded the Points & their Point Attributes in the PIPointList, you can now just requested the FilteredSummaries from the same object.

               

              Are you going to be performing repetitive, continuous filtered summaries? You might be better to pre-calculate.

               

              You should always consider that each RPC will be bloated by network latency, so the more RPCs you need to make the larger the overall timing. Bulk eliminates the number of RPCs need, but of course you could still request something that needs the PI Data Archive to work a bit harder to calculate - especially if multiple users of the same system (using your code) will make the same calls.

              1 of 1 people found this helpful
                • Re: What are the optimal settings for Filtered Summaries?
                  SvenBatalla

                  Thanks.  Overnight, I did exactly this.

                   

                  Strangely, it did not work as well as I had hoped.  I am assuming therefore, that the AFSDK must do the same thing under the hood that I am doing.  I will need to do more investigation.  Doing a bulk call on the FilteredSummaries resulted in a wait-time of 1:54 for the data to return.  That's the same as if I had paralleled individual calls.  I could be doing something wrong, obviously, so I'll be checking on that, but maybe that function simply doesn't perform when there is lots to do.  So here's a couple questions:

                   

                  1. Should I batch the calls?  Rather than asking FilteredSummaries to calculate on 20,000 values (for example), should I make 4 calls to calculate 5,000 (again, as an example)?
                  2. Am I using the wrong list type?  In your response you suggested using the PIPointList.  I already have an AFAttributeList.  Will either suffice, or is PIPointList more efficient?

                   

                  Thanks!

                    • Re: What are the optimal settings for Filtered Summaries?
                      SvenBatalla

                      I am going to amend my comment based on logged evidence.

                       

                      I have several types of FilteredSummaries calls that I need to make.  Basically it's just that the expression is different between them.  So I have 2 choices.  I could perform them back-to-back, or I could run them parallel.  I chose the latter assuming that if I could ask PI for 2 FilteredSummaries values at the same time, I would be in a good position.  The thinking is that if each call takes (let's say) 10s, then a parallel call would result in all calcs being done in ~10s rather than a linear addition of each run.  However, what I found was that there must be some locking going on somewhere because each FilteredSummaries call runs back-to-back despite the parallel request.  Here's a sample from my log file:

                       

                      • DONE Bulk calculating filtered summaries - Took 00:01:37.0673126
                        • DONE Calculating time masked - Took 00:00:10.7362182
                        • DONE Calculating values high - Took 00:00:25.6024475
                        • DONE Calculating total values - Took 00:00:38.9991603
                        • DONE Calculating time high - Took 00:00:54.4961672
                        • DONE Calculating values low - Took 00:01:07.8820053
                        • DONE Calculating time low - Took 00:01:22.5754060
                        • DONE Calculating values abnormal - Took 00:01:36.7509939

                       

                      As you can see, each FilteredSummaries call seems to add on top of the previous time.  There is simply no way the abnormal call took as long as it did.  To test this, I ran them without parallel.

                       

                      • DONE Bulk calculating filtered summaries - Took 00:02:06.1555235
                        • DONE Calculating time masked - Took 00:00:09.6489459
                        • DONE Calculating time high - Took 00:00:22.3676794
                        • DONE Calculating time low - Took 00:00:19.8447267
                        • DONE Calculating values high - Took 00:00:23.8330824
                        • DONE Calculating values low - Took 00:00:18.5579727
                        • DONE Calculating values abnormal - Took 00:00:17.9673887
                        • DONE Calculating total values - Took 00:00:13.8617262

                       

                      Thoughts?

                        • Re: What are the optimal settings for Filtered Summaries?
                          Rhys Kirk

                          I don't have my old data in front of me, but here are some things I remember from previous testing...

                          - AFData FilteredSummaries actually creates a PIPointList object under the hood.

                          - AFData FilteredSummaries is checking that the DataReference has a method called "GetListFilteredSummaries" (or something like that), which the PI Point DR does (4.0 version onward).

                          - There was some funny business on whether the PI Points were already resolved or not. I think that if PI Points weren't already resolved then AF SDK would perform the lookup in batches of 10,000.

                           

                          Also, bulk summary RPC was introduced in 3.4.390.19. You need to be on that or later to make use of it. If you're not on the right version of the PI Data Archive it will use the iterative approach of querying per Attribute rather than in bulk.

                           

                          Chris Manhard thoughts?

                          1 of 1 people found this helpful
                  • Re: What are the optimal settings for Filtered Summaries?
                    Eugene Lee

                    Hi Sven,

                     

                    Yes, this is of course possible with Filtered Summaries from AF2.6 onwards

                     

                    Please have a look at the AFListData.FilteredSummaries Method.

                     

                    Do let us know the performance improvements that you are able to achieve so that the community can know.

                    1 of 1 people found this helpful
                      • Re: What are the optimal settings for Filtered Summaries?
                        cmanhard

                        Performance of the bulk summary will depend on the AF Client and PI Data Archive versions.  The ideal combination is AF 2015 and PI Data Archive 2015.  Versions less than that are unlikely to make bulk summary calls.  Also note, that the attributes can only be summed in bulk if their filter is the same AFTER they have been resolved into PI Point specific syntax.  For example, a filter of "'.' >= 'lowlimit'", would be resolved into the specific tags referenced in the attributes.  Unless all attributes resolve to the same two tags, the bulk call to the server will not be achieved, and the AF SDK will make multiple individual calls.

                        1 of 1 people found this helpful
                          • Re: What are the optimal settings for Filtered Summaries?
                            SvenBatalla

                            Chris,

                             

                            Thanks for the feedback.  Let me parrot so that I know I understand...

                             

                            If I have an expression like this:  "'Is Masked' = 0 AND 'Source Tag' >= 'High Limit'", that breaks down into 3 distinct tags PER element.  That also means that for each element, the expression is fundamentally different.  And because of the difference, that means that there is no capability to bulk and thus the AFSDK will actually perform each call individually.  So if I have a bulk call with 20,000 elements, It will actually be 20,000 distinct FilteredSummaries calls and not 1.

                             

                            Do I have that right?

                             

                            If that is correct, is there some other method that I can improve the performance of my application?  I have tried lots of parallel activities, batching, and so on, but I can't seem to make it any faster than what I've already documented in the ticket.

                            • Re: What are the optimal settings for Filtered Summaries?
                              SvenBatalla

                              Chris,

                               

                              Further to my other question, I'm not sure I get it logically anyways.  I can perform the FilteredSummaries call and have it come back between 10-20s.  But if I perform another FilteredSummaries call asynchronously (i.e. another thread) and it comes back in 20-40s.  If I run them back to back, they are each 10-20s in timing.  The implication is that the FilteredSummaries call in the AFSDK 2014 version (which I am using) has a lock in it that prevents other calls from happening at the same time.

                               

                              Can you confirm?

                               

                              Thanks,

                              - Sven

                                • Re: What are the optimal settings for Filtered Summaries?
                                  cmanhard

                                  Regarding your expression, you are correct, that call would not be bulked.

                                  Also, the Summaries/FilteredSummaries bulk calls require PI 3.4.390.19 (2014 SP1) or PI 3.4.395.  Anything less than this version does not support the bulk summary call (other bulk calls are supported in PI 3.4.390.16 and .18, just not summaries).  That won't affect you here in this particular case, but could if you were not doing the filter. 
                                  Regarding your performance observations that indicate a lock contention, I have not been able to duplicate that behavior.  There may be something more specific about your particular configuration or query.

                                  1 of 1 people found this helpful
                                    • Re: What are the optimal settings for Filtered Summaries?
                                      SvenBatalla

                                      Thanks for that information.  I am talking to the guys about applying the .19 patch since we're at .18.

                                       

                                      As for the lock contention, I see no other possibility other than perhaps the version of PI I am talking to is the culprit?  Or at least I will find out after the upgrade (if we upgrade).  In any case, you see my code from the original post and the expression I use (or at least close enough) in one of the replies, so I don't think I'm doing anything crazy.  There is no overhead where the timing is gathered.  For clarity, here it is all put together:

                                       

                                      using(new ScopeTimer(_logger, "Calculating time high"))
                                      {
                                        results = attributeList.Data.FilteredSummaries(
                                            timeRange,
                                            new AFTimeSpan(timeRange.Span),
                                            "'High Limit' <> -999 AND (BadVal('Is Masked') OR 'Is Masked' = 0) AND 'Source Tag' >= 'High Limit'",
                                            AFSummaryTypes.Count,
                                            AFCalculationBasis.TimeWeighted,
                                            AFSampleType.Interval,
                                            new AFTimeSpan(minutes:5),
                                            AFTimestampCalculation.Auto,
                                            new PIPagingConfiguration(PIPageType.TagCount, attributeList.Count));
                                      }
                                      

                                       

                                      I've played with a few configurations in the call (namely the various time spans and paging), but to no avail.  In any case, if you spot something, I'm open to anything.

                                       

                                      Thanks!

                                        • Re: What are the optimal settings for Filtered Summaries?
                                          ekuwana

                                          Hi Sven,

                                          Could you confirm a couple of things:
                                          1.  In one of your posts you mentioned that you're using AF Client 2012 R2, and in another you're using AFSDK 2014.
                                          Could you confirm if you're seeing the performance/contention problem with (single/non-bulk) FilteredSummaries in both versions?
                                          2. There is a known issue, which is fixed in AFSDK 2014, related to FilterSummaries and CalculateSummaries that could generate server error -11110 when sampleType is Interval and sampleInterval does not divide evenly into the requested timerange.
                                          This could potentially impact performance.
                                          http://cssapps.osisoft.int/KnownIssues/KnownIssues.aspx?id=100917&s=TFS%202013%20-%20OSI3

                                            • Re: What are the optimal settings for Filtered Summaries?
                                              ekuwana

                                              Just to clarify my previous post, the issue I mentioned was regarding single FilteredSummaries call, since the bulk Summaries calls were not yet available in AF Client 2012. If you're looking at performance with this call in AF 2012 then it may not be reliable if the issue is encountered (with certain input parameters).

                                              In AF 2014 the issue is fixed hence should not be observed in (single or bulk) FilteredSummaries call.

                                                • Re: What are the optimal settings for Filtered Summaries?
                                                  SvenBatalla

                                                  I am using AF 2014, so that won't be the issue.  I believe the issue is that we're using PI v3.4.390.18.  We're one patch back.  Ideally we'd upgrade to PI 2015, but we have a custom interface that is blocking us at the moment.  I imagine we can at least to the patch.  I am hoping to do that today.  I'll keep you all posted.

                                                    • Re: What are the optimal settings for Filtered Summaries?
                                                      ekuwana

                                                      It may also be useful to look at some performance counters (on both client and server).

                                                        • Re: What are the optimal settings for Filtered Summaries?
                                                          cmanhard

                                                          As Rhys mentioned - you may want to consider what portions of this you can pre-calculate - perhaps using asset analytics to output the pre-filtered and/or pre-summarized output to a tag.  Consider that you are asking for 1 minute filter calculations over an 8 to 12 hour period for 46,200 tags - each filter referencing 3 tags.  That multiplies out to 2-3 million filter calculations and many more millions of tag interpolation requests for the source tags.  To achieve your 30 second calculation time, it is going to require a process rate of 100,000 filters calculations per second.  Parallelizing this activity on the server will not completely solve the I/O bottleneck of getting this data off of disk, nor the relatively heavy CPU utilization for performing the filter calculations  .

                                                            • Re: What are the optimal settings for Filtered Summaries?
                                                              SvenBatalla

                                                              Chris,

                                                               

                                                              You are absolutely correct.  However, pre-calculating will be difficult because doing so will require additional tags (which the client is hesitant on).  It would also raise the concern about what to do with data that arrives late (or out of order) and how that would affect calculations (either directly or indirectly).  I certainly understand the ramifications of what I am asking, and that's why I asked what sort of resources we would require in PI and PI AF (client & server) to accommodate.

                                                               

                                                              I am told we are doing the patch update on my dev system later today.  In the meantime, any further advise would be most welcome.

                                                               

                                                              Thanks!

                                                                • Re: What are the optimal settings for Filtered Summaries?
                                                                  Rhys Kirk

                                                                  Use a PI Collective with multiple members, and then self-partition your filter queries to other nodes to each query a PI Collective member. The cost though is probably more expensive than the extra tags.

                                                                  You could look at SSD for archive storage, or forcing the archive data to be in the File System Cache before running queries (which would mean you need to consider a large RAM installation).

                                                                   

                                                                  Out of interest, how is the Event Count per attribute being used downstream?

                                                                    • Re: What are the optimal settings for Filtered Summaries?
                                                                      SvenBatalla

                                                                      Having the application be aware of the components of a PI Collective is not really conducive to a generic product.  I'd rather that was all taken care of automatically by PI.  Plus, I have to assume that some of my clients will not have a PI Collective and will have this issue anyway.  In any case, it still doesn't solve the problem of late or out-of-order data.

                                                                       

                                                                      As for how this is being used, the results of these calculations are applied against each other and used to determine conformance against user-defined limits.  Those that do not conform are presented to the user in reports.  To prevent the user from having to take the burden of the calculation performance, I have created a set of services whose job it is to calculate the conformances based on the relevant periods of time.  The one I am most concerned about at the moment is a real-time shift-based calculation.  It takes 20,000 elements and performs these calculations.  It takes so much time, that it is not really "real-time" any more.  Or at least not close enough to be considered such.

                                                                • Re: What are the optimal settings for Filtered Summaries?
                                                                  SvenBatalla

                                                                  Eddy,

                                                                   

                                                                  Can you tell me which performance counters?  I am monitoring CPU and memory usage on each machine, but if there are specific PI counters, which ones should I look at and what should I be looking for?

                                                                   

                                                                  Thanks,

                                                                  - Sven

                                                                    • Re: What are the optimal settings for Filtered Summaries?
                                                                      ekuwana

                                                                      Hi Sven,

                                                                       

                                                                      PI performance counters such as:

                                                                      - events read / sec, archived events / sec , getsnapshot calls / sec, and snapshots/sec.

                                                                      - Archive SS process e.g. % processor time, etc.

                                                                      It may also be useful to get a "piartool -thread piarchss -info" output when the performance test is running so that we can see what calls / duration in terms of RPC.

                                                                      • Re: What are the optimal settings for Filtered Summaries?
                                                                        atang

                                                                        hi Sven,

                                                                         

                                                                        If there are multiple parallel bulk queries going on (NOTE: internally, the bulk query thread pool can have up to 16 threads to accommodate paralleled calls), then there could be possibilities of parallel calls taking more time than performing back to back individual calls. The PI Data Archive server doesn't know if there are one or more users doing queries, so it tries to serve all clients equally -- and if all bulk query threads are being used, it means that the PI Archive Subsystem will respond to all clients slower. In addition, bulk queries require some setup time.

                                                                         

                                                                        If you are doing large queries for bulk data with many tags, it'll definitely be much faster than using the bulk query calls for non-bulk query type operations (for instance, getting a list of filter summaries for a single tag for a range of time).  However, because of that initial setup, if you are doing relatively small queries with small amounts of data, you might actually get better performance doing a non-bulk call.

                                                                         

                                                                        Performance changes with respect to bulk query hasn't been touched in PI Data Archive 2015, so I wouldn't expect much performance gain by upgrading.

                                                                         

                                                                        What do your bulk calls look like in terms of the amount of data you get back and the amount of tags you specify? Are they querying for data for the same tags?

                                                                         

                                                                        It could be the read cache is thrashing, could be that all RPC threads are busy -- all these are speculations and it'll be good to confirm what is actually happening on the PI Data Archive side.

                                                                         

                                                                        Just to echo / add on to what Eddy mentioned, some additional counter data to collect:

                                                                         

                                                                        PI Archive Subsystem \ Archived Events/sec

                                                                        PI Archive Subsystem \ Bulk Queries/sec

                                                                        PI Archive Subsystem \ Bulk Query Ongoing Count

                                                                        PI Archive Subsystem \ Events Read/sec

                                                                        PI Archive Subsystem \ GetSnapshot Calls/sec

                                                                        PI Archive Subsystem \ Read Operations/sec

                                                                        PI Archive Subsystem \ Summary Calls/sec

                                                                        PI Archive Subsystem \ Write Contentions/sec

                                                                         

                                                                        Running the piartool utility with "piartool -thread piarchss -info" while the test is going on will give us a picture of what threads are busy in the PI Archive Subsystem and how long each RPC took to execute.

                                                                        1 of 1 people found this helpful
                                                    • Re: What are the optimal settings for Filtered Summaries?
                                                      SvenBatalla

                                                      To all those that have been following this thread, I thought I'd give an update in a easier to spot area rather than responding to the various sub-conversations.

                                                       

                                                      Yesterday we applied the update to our PI system.  It is now running PI v3.4.390.28 (2012 SP1).  I continue to run AF SDK v2.6.1.6238.  I let my application run overnight so I'd have a series of performance numbers and an obvious trend up or down.  Sadly, the performance did not seem to change.  In fact, if anything, it is averaging out to slightly worse.  The total time to calculate is:

                                                       

                                                      • DONE Bulk calculating filtered summaries - Took 00:02:01.7633666
                                                        • DONE Calculating values high - Took 00:00:19.7442100
                                                        • DONE Calculating time masked - Took 00:00:28.4863551
                                                        • DONE Calculating total values - Took 00:00:46.4261349
                                                        • DONE Calculating time high - Took 00:01:06.2847751
                                                        • DONE Calculating values low - Took 00:01:24.6352463
                                                        • DONE Calculating values abnormal - Took 00:01:43.1990315
                                                        • DONE Calculating time low - Took 00:02:01.7619114

                                                       

                                                      Remember that my strategy is currently to call the FilteredSummaries method via the AFAttributeList class.  We have already established that underneath the hood, this will still result in many calls to the PI server because of the fact that each expression will essentially be a different expression.  However, the PI server update would hopefully allow for multiple concurrent connections from the AF SDK and thus result in better performance due to threading.  Obviously we're not seeing that.  You will also note from the times above that there is a distinct pattern of locking happening (still).  The time to calculate the time low is not 00:02:01, but more likely, ~00:00:18.  Each FilteredSummaries call appears to wait for the previous one to be completed.  I won't speculate as to why (though I have some ideas), suffice it to say that this is the result.

                                                       

                                                      One of the things we tried late last week was to increase the RPC count on the PI server.  We increased it from the default 8 to 32.  This provided some performance improvement, but all-in-all it was marginal.  We will likely revert back to the default 8 as I have to assume that's where my customers will be set.

                                                       

                                                      The server my application runs on (which is not the PI server or the AF server) has 2 cores and 4GB of RAM.  The CPU is not overly taxed and the RAM seems to sit at about 2GB being used by the application (3GB by the server).  So from what I can see resource-wise, I'm OK.  Any thoughts?

                                                       

                                                      The final test I can run is to use AF and PI 2015.  The problem with that is that I cannot know if my customers will be that up to date.  I know for sure one of my clients is still running 2012.  So I'd really like to get this sorted with PI 2012.  Does anyone have any ideas?

                                                        • Re: What are the optimal settings for Filtered Summaries?
                                                          dmoler

                                                          Hi Sven,

                                                           

                                                          If I understand the "locking" issue you are seeing, it is due to AFSDK ensuring that the returned values in the IEnumerable correspond to the ordering in the AFAttributeList.  If you don't care about the order that things come back, then you can do the parallel requests on your end and consume them as the individual calls complete.

                                                           

                                                          I think it might be worth taking a step back to think about the best way to tackle this problem.  As Chris identified, this is a lot of data to be drawing on.  Are there things that could be done to pre-filter using inexpensive calls?  This will be data dependent, but with some domain knowledge about what the 46k attributes are likely to look like it could be possible.  So the process might look like this:

                                                          1) do a bulk (simple, single) summary for the min value for all 'high limit' attributes

                                                          2) do a bulk (simple, single) summary for the max value for all 'source tag' attributes

                                                          3) check the lowest 'high limit' against the highest 'source tag' - these are attributes that might have an excursions (hopefully this is a small subset of the 46k)

                                                          4) do the full filtered summaries call any candidates that might have an excursion

                                                           

                                                          Of course, some of these can be done in parallel, too.  The details will be data dependent (e.g. maybe a better approach will be to try to narrow the range to search) but as long as the excursions are rare, it should be possible to narrow the search first.

                                                           

                                                          One thing to note is that bulk summary calls are not supported by the PI Server natively until 2012 SP1.

                                                            • Re: What are the optimal settings for Filtered Summaries?
                                                              SvenBatalla

                                                              David,

                                                               

                                                              That is very sound advice.  However, for the purposes of this post, I was keeping my expression a little simple.  There are actually 7 different expressions and I felt that they were conceptually the same for the purposes of this post, but here's an example of a real one:

                                                               

                                                              'High Limit' <> -999 AND (BadVal('Is Masked') OR 'Is Masked' = 0) AND 'Source Tag' >= 'High Limit'

                                                               

                                                              For each evaluation interval, the expression:

                                                              • Checks that the limit is set (-999 means it isn't and that can toggle at any time back-and-forth).
                                                              • Checks that the attribute "Is Masked" is not a bad value and if it isn't, that it isn't actually set.  The reason being that the user can choose to mask the calculation for a period of time and we cannot consider those times in our calculation.
                                                              • Check that the value >= limit.

                                                               

                                                              Before knowing about FilteredSummaries, I was collecting data manually and performing calculations.  Those can get complex and slightly more inaccurate due to interpolation or data culling.  I doubt the performance would have been much better.

                                                               

                                                              The interesting thing you mentioned was ordering of the response.  And no, I could not care less about the order of results since I convert the results into a dictionary lookup anyway.  So is there a way to tell the FilteredSummaries call to forget about the ordering?  Obviously one such way is to do the calculations in my own Parallel loop.  This is where this all started.  Right now the calculations take ~02:09 to run.  When I was looping the data myself, it was taking ~01:55.  Admittedly that was prior to the various changes we've made that I've documented in this post.  So the question is:  is this better in PI 2015, should I do individual FilteredSummaries, or both?

                                                               

                                                              Thanks,

                                                              - Sven

                                                                • Re: What are the optimal settings for Filtered Summaries?
                                                                  dmoler

                                                                  Hi Sven,

                                                                   

                                                                  There is no way to turn off the ordering and I don't expect the performance for this call will be improved in PI Server 2015.

                                                                   

                                                                  I understand that the actual logic you need is more complicated, what I was suggesting is to try to find an inexpensive way to exclude calculations that never have excursions.  So if you do a simple summary for the max value of the source tag over the entire range and it returns 80 and a simple summary for the min value of the high limit over the entire range and it returns 100, then you'd know that it never exceeds the limit over the entire range - then the filtered summaries at 1-min intervals would be unnecessary.

                                                                  • Re: What are the optimal settings for Filtered Summaries?
                                                                    Rhys Kirk

                                                                    If you're really stuck on getting the performance down then I still think that an element of pre-calculating is the only way you'll get that.

                                                                    Instead of continually performing the same queries on all PI Points you should perform it by exception. In AF Analyses terms you could just build this as an Event Triggered calculation and only execute the filter as new events flow in (limit change, Is Masked change, or Source Tag change), rather than repeating a query on a data set that potentially hasn't changed. Then you just ask for an Event Count summary on the resultant PI Points without the need for a filter.

                                                                     

                                                                    If there is no possibility of creating PI Points at clients then you could build a custom Data Reference to write out to a SQL Table or equivalent so that you get the output directly into your report as data is streaming in, which is very much real-time.

                                                              • Re: What are the optimal settings for Filtered Summaries?
                                                                SvenBatalla

                                                                Lots of good tips and tricks buried in this post.  Here's where I intend to go:

                                                                 

                                                                • I am adjusting my attribute-based AF calculations to be event-driven.  This should reduce the amount of data in PI and hopefully improve performance of the FilteredSummaries calls.
                                                                • I must continue to use the FilteredSummaries calls due to the complexity involved in doing it any other way.  I've explored using raw or interpolated data and it's simply not efficient to gather all of that data.
                                                                • I cannot pre-calculate anything because this service IS the pre-calculator and data can come in out-of-order or late.
                                                                • I have gained access to a PI 2015 system and will connect to it using AF 2015.  There is a helpful comment from Anthony Tang that suggests this will make no difference, but since it is as yet untested, I will let you all know.

                                                                 

                                                                I expect further posts as the day moves on.

                                                                  • Re: What are the optimal settings for Filtered Summaries?
                                                                    SvenBatalla

                                                                    Update...

                                                                     

                                                                    • Event-driven won't work.  This won't work for 2 reasons...  First, signing up for as many events as I have takes so long and uses so much memory, that the IT folks noticed and shut it all down.  We're talking hours and 12GB of memory.  We even got on the phone with some AF experts on this one.  Second, some of the items that should trigger the calcs are static attributes which cannot trigger a calc.  And since that happens on root calculations, having those be calculated periodically essentially means everything has to be.
                                                                    • All my FilteredSummaries alternatives have so far failed.
                                                                    • I am converting as much as I can to formulas (where tags aren't required) so as to minimize load on AF.
                                                                    • I have the 2015 system, but there were some issues with it and it was sent back to the PI team to resolve.  I expect I will have access today.  I also expect there will be no difference in performance.
                                                                      • Re: What are the optimal settings for Filtered Summaries?
                                                                        Rhys Kirk

                                                                        How frequently were you collecting events via the Data Pipe? How frequent are the incoming events?

                                                                        12GB of memory on the client? I assume you were running the calculations yourself rather than the PI Analysis Processor?

                                                                         

                                                                        You have to create your own static attribute pipe by encapsulating the AFDatabase.FindChangedItems method. That way you can feed in changes to static attributes.

                                                                          • Re: What are the optimal settings for Filtered Summaries?
                                                                            SvenBatalla

                                                                            Rhys,

                                                                             

                                                                            We are not collecting data very frequently.  I'd say my test tags are updating once every 15 minutes or so (on average).  The 12GB memory usage is not the client, but rather the analysis service.  OSIsoft told us yesterday that 60GB is not unheard of and should be no cause for concern (obviously the IT folks in the room nearly feel off their chairs).  The calculations I refer to in this case are just running on their own.  Feeding changes to static attributes causes one of the PIAF tables to grow out of control (that's what sparked to call to OSIsoft), so we can't really do that.

                                                                             

                                                                            Hope that helps,

                                                                            - Sven

                                                                              • Re: What are the optimal settings for Filtered Summaries?
                                                                                Rhys Kirk

                                                                                Ah, ok. Yeah that will be the AF Data Cache then. 12GB could well be in proportion based on the large number of PI Points & Attributes in your Analyses.

                                                                                 

                                                                                Thing is, static attributes are given that name because generally they are referring to information that seldom changes & is not mastered in another system. Your tests are likely simulating that static data changing more frequently than it would - or perhaps I'm wrong. Anyway, maybe that data should be in a SQL Table and surfaced into the AF Analyses via the Table Lookup DR. This will prevent the FindChangedItems SQL table from becoming bloated. Your application would then update the static information directly in the SQL Table (or an AFTable object).

                                                                                  • Re: What are the optimal settings for Filtered Summaries?
                                                                                    SvenBatalla

                                                                                    This is promising...  Not let's see if I can understand how to do this...

                                                                                     

                                                                                    • I create a table in AF.  It am thinking it is very simple and only has 2 columns; the element ID and the value I want.
                                                                                    • I update my element template such that the attribute (upon which I was trying to add the formula) is now using a Table Lookup DR and with a simple query:
                                                                                      • SELECT DataColumn FROM [MyTable] WHERE ElementID = '%ElementID%';RWN=No Data
                                                                                    • I create an AF analysis on the same attribute to output a value to the table periodically.
                                                                                      • It is this part that I'm not clear on.  I'm not sure how it knows where to put the value in that table.  Maybe this isn't where it goes?