8 Replies Latest reply on Dec 14, 2017 6:23 PM by Rick Davin

    Is there a way to find the total event frames in a selected time interval ?

    ibrahimnoureddine

      Hello ,

       

      I am trying to delete event frames within a time interval using AFEventFrameSearch , however finding event frames is too slow (i have up to 130000 frames per hour) , below is the code  :

       

         AFEventFrameSearch search = new AFEventFrameSearch(_dataBase, "FindEvents", AFSearchMode.Inclusive, startDate, searchEndDate);

                          foreach (AFEventFrame item in search.FindEventFrames(0, false, 10000))

                          {   

                                  item.Delete();

                          }

       

      So i am trying to make it multi-threaded by using AFElement.getEventFrames() , and specifying start index and count for each thread , however i can't find a way to get the total number of event frames per hour to split it into multi-thread.

      how can i get the total number of event frames per hour , or what is the fastest way to delete event frames ?

        • Re: Is there a way to find the total event frames in a selected time interval ?
          rschmitz

          Hi Ibrahim,

          I've personally been a fan of using the built-in System Library for this and letting it handle those details, rather than chunking the search. For Example:

           

          AFEventFrameSearch search = new AFEventFrameSearch(_dataBase, "FindEvents", AFSearchMode.Inclusive, startDate, searchEndDate);
          Parallel.ForEach(search, EventFrame =>
               {
                    // Try Deleting the Event Frame
                    try
                    {
                          EventFrame.Delete();
                          EventFrame.CheckIn();
                          Console.WriteLine("Sucessfully Deleted Event Frame: {0}", EventFrame.Name);
                    }
                    catch
                    {
                          Console.WriteLine("Error, unable to Delete Event Frame: {0}", EventFrame.Name);
                    }
              });
          
          
          
          

           

          Cheers,

          Rob

          1 of 1 people found this helpful
            • Re: Is there a way to find the total event frames in a selected time interval ?
              Dan Fishman

              Since you are looking for Event Frames within a timerange, the AFDiag command will not work.  Yes, deleting EF is rather slow and there is not too much you can do.  There are a few tricks that will make your deletions a little faster, such as multiple threads (like Robert Schmitz suggested) and bulk checkins.

               

              You might want to avoid calling EventFrame.CheckIn() every EventFrame within a parallel.ForEach loop.  I've used a partitioner class with range size of 500 to allow myself a quick way to perform check-ins every 500 EF.  After the end of the delete, you could checkin the objects using AFDatabase.Checkin(AFChecedOutMOde.ObjectsCheckedOutThisThread) if you work on partitions.  Also, try playing with the ParaellelOptions MaxDegreeOfParallelism as too many threads could be spawned and bog down the SQL server.

               

               

              You might want to reindex your EF SQL tables before and after the deletions to give a slight performance boost if they indexes are fragmented.

              2 of 2 people found this helpful
              • Re: Is there a way to find the total event frames in a selected time interval ?
                David Hearn

                When using the search and you are going to return multiple pages of items as in this case, you should enable the server-side caching to improve performance and to ensure you don't skip results because of the deletes which changes the index into the search results.

                 

                using (AFEventFrameSearch search = new AFEventFrameSearch(_dataBase, "FindEvents", AFSearchMode.Inclusive, startDate, searchEndDate))

                {

                    search.CacheTimeout = TimeSpan.FromMinutes(10);

                    Parallel.ForEach(search, EventFrame => 

                    { 

                        // Try Deleting the Event Frame 

                        try 

                        { 

                            EventFrame.Delete(); 

                            EventFrame.CheckIn(); 

                            Console.WriteLine("Sucessfully Deleted Event Frame: {0}", EventFrame.Name); 

                        } 

                        catch 

                        { 

                            Console.WriteLine("Error, unable to Delete Event Frame: {0}", EventFrame.Name); 

                        } 

                    });

                }

                2 of 2 people found this helpful
                • Re: Is there a way to find the total event frames in a selected time interval ?
                  Rick Davin

                  Hi Robert,

                   

                  If a routine is already slow, adding in a check-in and Console.WriteLine for each iteration will only add to the execution time.

                   

                  I agree with you in general that its best to use the built-in Libraries, but with the added caveat of only if they perform well.  Each delete call takes about the same amount of time for each event frame.  Spinning up each parallel task takes a small bit of time and uses some resources.  That's not too bad normally but a little bit can add up to a big bunch.  For 130K event frames in an hour, if you have 4 hours that's over a half a million little bits of processing just to spin up a million parallel tasks.  And this is a prime example of bad use of Task Parallel Library (TPL) because such attempts to use parallel over serial will be slower, not faster.

                   

                  Thus I respectfully disagree with you in that regard.  When such processing within the inner loops is consistently the same, its absolutely best to chunk.  There are many ways to do this.  If you already have the collection at your fingertips, you can just keep a counter.  Or there is the Partitioner class for ranged partitions.  For the OP's use, I have chunked the dates into hours.  Granted this isn't perfect.  It works best if there are a fairly homogeneous distribution of EF's over each hour.  If one hour has 5K and another hour has 300K, then this would not be the optimal solution.

                   

                  All of which is a way to say that with parallel calls, there is no one-size-fits-all solution.  Sometimes a simple Parallel.ForEach is the right use.  Sometimes its not.

                • Re: Is there a way to find the total event frames in a selected time interval ?
                  skwan

                  Are you trying to delete event frames within a specified time range, i.e. from DateTime1 to DateTime2?  Or are you trying to delete event frames older than a specific time, i.e. delete all event frames older than Dec-01?

                  --

                  Steve Kwan

                  • Re: Is there a way to find the total event frames in a selected time interval ?
                    Rick Davin

                    Hi Ibrahim,

                     

                    The problem with your design is you want to use one big Find or Get, and then chunk by hour.  Based on how EF's are retrieved in pages, it's very likely that EF's for a given hour may span over many pages.

                     

                    An alternative would be to partition your date range into hourly segments first, and then make parallel calls to find and delete the EF's for a given hour.

                     

                    Frequent check-ins will degrade performance.  Checking-in too many changes also is sluggish.

                      • Re: Is there a way to find the total event frames in a selected time interval ?
                        Rick Davin

                        Taking a stab at it.  Note I have not tested this.

                         

                        You will need to have these usings:

                         

                        using System.Threading.Tasks;
                        using System.Threading;
                        

                         

                         

                        I wrote a separate method to chop the times into hourly segments. 

                         

                        private List<AFTime> GetHourlyTimes(AFTime searchStartDate, AFTime searchEndDate)
                        {
                            List<AFTime> times = new List<AFTime>();
                            AFTime last = searchStartDate;
                            while (last <= searchEndDate)
                            {
                                times.Add(last);
                                last = searchStartDate.UtcTime.AddHours(1);
                            }
                            if (times.Last() < searchEndDate)
                            {
                                times.Add(searchEndDate);
                            }
                            return times;
                        }
                        

                         

                         

                        And the main delete method would be:

                         

                        private void DeleteEFsByHour(AFTime searchStartDate, AFTime searchEndDate)
                        {
                            List<AFTime> times = GetHourlyTimes(searchStartDate, searchEndDate);
                         
                            int deletedCount = 0;  //total deleted count
                            int chunkCount = 0;    //reset to 0 after each check-in
                            const int searchPageSize = 10000;
                            const int checkinPageSize = 1000;
                         
                            Parallel.For(0, times.Count - 1, i =>
                            {
                                AFTime startTime = times[i];
                                AFTime endTime = times[i + 1];
                                using (AFEventFrameSearch search = new AFEventFrameSearch(_dataBase, $"FindEvents_{i}", AFSearchMode.Inclusive, startTime, endTime))
                                {
                                    search.CacheTimeout = TimeSpan.FromMinutes(10);
                                    foreach (AFEventFrame item in search.FindEventFrames(0, false, searchPageSize))
                                    {
                                        if (!item.IsDeleted)
                                        {
                                            item.Delete();
                                            Interlocked.Increment(ref deletedCount);
                                            if (Interlocked.Increment(ref chunkCount) > checkinPageSize)
                                            {
                                                _dataBase.CheckIn();
                                                Interlocked.Exchange(ref chunkCount, 0);
                                            }
                                        }
                                    }
                                }
                            });
                         
                            if (chunkCount > 0)
                            {
                                _dataBase.CheckIn();
                            }
                        }