9 Replies Latest reply on Mar 2, 2016 1:20 PM by pthivierge

    AFAnalysis error: Values are not in ascending time order

    acote

      Sweet, I found a new error and I can't figure out a way to fix it. Any idea?

       

       

       

      Here is the error in About-SDK:

       

      Here is the error as logged within the AFAnalysis server.

      ANRecalculationAnalysis.ProcessRecalculationRequest[]: Exception encountered during backfilling of Analysis: E3APP - SILO - LIVRAISON - TOTAL C2A, request ID 2e0c2db4-1653-4408-b86c-a56e243dddd4, Start Time 1/1/2016 5:00:00 AM, End Time 2/4/2016 8:04:20 PM

      System.ArgumentException: Values are not in ascending time order: current timestamp 1/14/2016 2:27:47 AM, previous time 1/17/2016 12:00:46 PM at value index 5

      Parameter name: initialValues

         at OSIsoft.AF.Data.AFDataCache.WriteToCache(AFAttribute attribute, IList`1 initialValues)

         at OSIsoft.AN.Analysis.Calculation.ANDataCacheWithArchiveData.AddInputs()

         at OSIsoft.AN.Analysis.Calculation.ANDataCacheWithArchiveData.UpdateData(AFTimeRange timeRange)

         at OSIsoft.AN.Analysis.Calculation.ANDataCacheWithArchiveData..ctor(AFTimeRange timeRange, IEnumerable`1 inputs, IEnumerable`1 outputsUsedAsInput)

         at OSIsoft.AN.Analysis.Calculation.ANCalculationExecutor.InitializeBase()

         at OSIsoft.AN.Analysis.Calculation.ANRecalculationExecutor.Initialize()

         at OSIsoft.AN.Analysis.Calculation.ANRecalculationExecutor..ctor(ANStateCalculation calculation, ANRecalculationRequest request, IANExecutionProgressTracker executionTracker)

         at OSIsoft.AN.Analysis.Calculation.ANRecalculationAnalysis.BackFillAnalysis(ANRecalculationRequest request, CancellationToken cancelToken)

         at OSIsoft.AN.Analysis.Calculation.ANRecalculationAnalysis.ProcessRecalculationRequest(ANRecalculationRequest request, CancellationToken cancelToken, Object throttleContext)

       

      Thanks

      Alex

        • Re: AFAnalysis error: Values are not in ascending time order
          acote

          For info:
          - AF 2.7.0.6937 (from AF Client 2015)

           

          The Analysis error occurs on a backfill from 2016/01/01... however the error is not raised when backfilling 2016/02/01

          • Re: AFAnalysis error: Values are not in ascending time order
            acote

            Having less than 200 analysis, I thought the RAM usage would be way lower... is the a 1.5GB RAM Limit for this process?

              • Re: AFAnalysis error: Values are not in ascending time order
                Mike Zboray

                Typically, caching of time series data uses a lot of memory, so its not just about number of analyses. It's about how many inputs and how much data those analyses are using. Rollup type analyses will cache data for every attribute matching the criteria, so you can easily get large amounts of data in the cache. I'm not sure which version it was first exposed in, but AF 2.7.5 version of analysis service has InputAttributesCount and EventsCachedCount statistics.

                  • Re: AFAnalysis error: Values are not in ascending time order
                    skwan

                    Mike is correct.  PI Analysis Service utilizes a cache heavily in order to provide best performance to the user.  It's a trade off, higher RAM usage for better performance.

                     

                    Please contact tech support for your backfill error.

                     

                    --

                    Steve Kwan

                      • Re: AFAnalysis error: Values are not in ascending time order
                        acote

                        This issue does point out to a RAM usage limitation which on a Windows Server 2012 x64, is limited (I think) by 1/2 allocated Paging File Size. This change will require a reboot, therefore restarting the PIAnalysis service and clearing its cache. If I had to bet, all my money would go on the service coming back alive with very few RAM consumption and never growing as large as it is right now. I have done various tests in the pas few weeks but the current analysis setup (only Expression calculation) is quite optimal and parses very few inputs at low event rates... Have any one pointed out a memory leak snag the AFAnalysis service?

                          • Re: AFAnalysis error: Values are not in ascending time order
                            pthivierge

                            Hello Alexandre,

                             

                            Did you contact technical support as proposed by Stephen Kwan?

                             

                            About your question: Have any one pointed out a memory leak snag the AFAnalysis service?

                            There is no memory leak known as of today for the PI Analysis Service, however its caching mechanism may consume a lot of memory if your analyses retrieve a lot of data in the past, or in your case,  if you are running a backfilling.

                             

                            This is controlled by cache settings of the analysis service (to open the configuration window you need to do this):

                            • Data goes in the cache only if it is within the time period covered by CacheTimeSpanInMinutes.
                            • When the service starts, it loads values in the cache according to the MinCacheEventsPerAttribute
                            • Then,  values are kept in cache until they reach CacheTimeSpanInMinutes or exceed the count of MaxCacheEventsPerAttribute.

                             

                             

                            I do not recommend that you change these settings on a production server unless you fully understand consequences, technical support may help you to determine what are the best settings for your system.  And in your situation probably adding more physical RAM would be the best option.  You provided a picture that shows settings of the page file, if the OS starts paging, the entire system may become really slow and that may not provide you good performances.  Nowadays RAM is quite cheap and you should consider increasing it if you see the system is using the page file . 

                             

                            Hope this clarifies a little more,

                              • Re: AFAnalysis error: Values are not in ascending time order
                                acote

                                Thanks everyone for your replies,

                                 

                                Regarding the backfilling error, I found the root cause and the solution. In a few words, I create tags today and back fill data since 2016/01/01, therefore using PI Server 2012 capability of pushing data to shifted archives without having to reprocess them... For each archive shift of type "autodyn", the PI Archive introduce an "Arc Off-Line" status within the archive events indicating an archive gap (Please note below that there is no archive gaps). I assumed that when backfilling data, AF Analysis encounter this "Arc Off-Line" event and fails to proceed with its calculation.

                                I did not contacted the tech support because of the amount of work on my desk and also because the solution was very simple: Set the Tuning Parameter > Archive > MarkArchiveGaps to 0 and restart piarcss - therefore avoiding these annoying "arc off-line" status .

                                 

                                Huge thanks to Patrice Thivierge for pointing out the configuration page of the Analysis service and the cache configuration behavior. Here is my actual config which I assume is by default:

                                      <CacheTimeSpanInMinutes>5</CacheTimeSpanInMinutes>

                                      <MaxCacheEventsPerAttribute>1024</MaxCacheEventsPerAttribute>

                                      <MinCacheEventsPerAttribute>1</MinCacheEventsPerAttribute>

                                      <NumberParallelDataPipes>5</NumberParallelDataPipes>

                                 

                                Understanding I had 210 very simple Analysis and giving an insanely large assumption of 50 attributes per analysis, 1024 event cached per attributes and 8 bytes per events, this only explains 84MB of cache! (Am I missing something?)

                                 

                                Also, because I increased the Paged Pool Size to from 3GB to 8GB, I rebooted my machine running the Analysis Service 2 weeks ago and since then introduced another 150 Analysis with backfilling since 2016/01/01 and max inputs frequency of 2 seconds and has it's development calculations, I have started over and backfilled half a dozen time. Here is my current RAM and cache consumption:

                                You can note that RAM consumption is 0.6GB where is was 1.3GB before reboot 2 weeks ago. Not saying it is a memory leak as many things are involved beneath the hood, but it looks like it and it's definitely something that will require to be monitored as this system is used in production environment.

                                 

                                Anyway, I will increase RAM just in case.

                                 

                                Thanks again for your help and good comments.

                                 

                                Alex