4 Replies Latest reply on Jun 27, 2018 7:49 AM by biker2000on

    Python Recalculation Script drops data


      I have written a Python script based on this post: Automatic Backfill using SDK (Python)


      I am able to find all of the correct analyses and looking in the logs it fires every 4 hours as configured.

      # -*- coding: utf-8 -*-

      #Allows us to import the AFSDK
      import sys
      import time
      #pythonnet module let's us import CLR for AFSDK use
      import clr
      from System.Collections.Generic import List
      from OSIsoft.AF import *
      from OSIsoft.AF.PI import *
      from OSIsoft.AF.Asset import *
      from OSIsoft.AF.Analysis import *
      from OSIsoft.AF.Data import *
      from OSIsoft.AF.Time import *
      from OSIsoft.AF.Search import *
      afservers = PISystems()
      afserver = afservers.DefaultPISystem
      DB = afserver.Databases.DefaultDatabase
      myAnalysisService = afserver.AnalysisService
      timerange = AFTimeRange('*-1d','*')
      #%% queue individual plow mixer recalculations
      analysislist = List[AFAnalysis]()   #initializes a C# list
      search = AFAnalysisSearch(DB, 'All recalced analyses',"Category:'PMRecalc'").FindAnalyses()
      for analysis in search:
          print('Target Element: {0} \tAnalysis: {1}'.format(analysis.Target, analysis.Name))
      myAnalysisService.QueueCalculation(analysislist, timerange, myAnalysisService.CalculationMode.DeleteExistingData)
      print('all analyses queued for recalculation')


      The problem that I am having is that when it recalculates the data through Python and the analysis service queue calculation method, it drops data as shown below:




      However, you can see in the next screenshot that when recalculated with PI System Explorer manually, all data is written correctly:




      The purpose of this recalculation is to update the event frame calculation once new lab data is entered. It is entered using a VBA script in a ProcessBook display so I cannot trigger the AF recalc at insertion time.


      Any ideas?

        • Re: Python Recalculation Script drops data

          After looking into it further. Recalc over the same time interval that Python is recalculating over also leaves the data incorrect.


          It seems the issue may be in the script recalculating *-1d intervals every 4 hours.


          When I recalculated from 6/24 to now instead of just 6/24 13:25 to now, it fixed the missing data.


          Is there an optimal recalculation range for 2-3hour event frames so it does not drop data as I am seeing?

            • Re: Python Recalculation Script drops data

              Hi Justin,


              If '6/24' works for you, 't-1d' instead of '*-1d' may work too but I believe it will not be this easy.


              According to Scheduled Analysis Execution DeleteExistingData is the only valid CalculationMode for Event Frames. Existing data is deleted. I can imagine that it is problematic if the timerange you refer with QueueCalculation is causing problems if it overlaps with existing Event Frames. It may be necessary to verify if there is an overlap and adjust the starttime of the timerange to be right before the starttime of the overlapping Event Frame. The adjusted timerange may cause overlaps with Event Frames created with other Analysis meaning that you may have to queue recalculation Analysis by Analysis. Ideally you can identify a time where all Event Frames are complete, meaning there is no Event Frame without endtime. You could than stick with the pretty simple code and define the start of your timerange as this specific time of the current or previous day.

              1 of 1 people found this helpful
                • Re: Python Recalculation Script drops data

                  Thanks for the reply. These event frames are all back to back looking at batches made on each piece of equipment.


                  Looks like I will have to search for the end of a valid event frame for each and then recalculate based on that.


                  Thanks for the insight.

                    • Re: Python Recalculation Script drops data

                      For those that come later, here is the code I got to t find the last End Time of a relevant event frame: The key was the search mode that conveniently orders the list so that [0] is the most recent before the "starttime".


                      def ef_recalc(element, starttime='*-1d', endtime='*-2d', recalctime='*'):
                          This queues event frame recalculations to the AF Server
                          with the following parameters.
                          element: string of element wanted
                          starttime: valid PI time of starting time for analysis
                          endtime: valid PI time (before the start time to find event frames)
                          recalctime: valid PI time (recalc ends at this time)
                          #build query string and then event frame list
                          query = "Start:>='" + endtime + "' ElementName:" + element
                          searchmode = AFEventFrameSearchMode.BackwardFromEndTime
                          eflist = [EF for EF in AFEventFrameSearch(DB,"search",
                          #adds first result to list and queues analysis
                          if eflist:
                              analysislist = List[AFAnalysis]()
                              elem = eflist[0].PrimaryReferencedElement
                              analysis = eflist[0].Analysis
                              timerange = AFTimeRange(eflist[0].EndTime, AFTime(recalctime))
                              calc_mode = myAnalysisService.CalculationMode.DeleteExistingData
                              myAnalysisService.QueueCalculation(analysislist, timerange, calc_mode)
                              print(elem.Name, analysis.Name, eflist[0].EndTime, recalctime)
                              print('Event Frame Search Yielded no results for {}'.format(
                      1 of 1 people found this helpful