1 Reply Latest reply on Dec 16, 2008 6:07 PM by dvacher

    How to merge archive files with sub-second start/end times

    Claude Martineau

      Here is a short article I wrote about the difficulties of merging two sets of archive files when they have start and end times in sub-seconds.  It was originally intented as an internal document, but I guess it could benefit other PI users.

      If someone has a better way of doing it, please let me know!

      1          Problem Definition

      Archive files from system B are merged into Archive files from system A using the Offline Archive Utility.  Both systems have timestamps in sub seconds.

       

      When archive files are merged, the end time of the destination archive has to be specified.  However, if it is truncated to a round second, there will be an archive gap of a few milli-seconds between this file and the next one.  If it is set to the next round second, then this archive will overlap with the next one.

      2          Troubleshooting and Solution

      A typical command line used to merge archive files is as follows:

       

      d:\PI\bin\piarchss -id E:\OriginalArchives\IDConv.dat -if e:\OriginalArchives\temparchives\piarch.027  -of E:\MergeOutput2\ARCH_20050625_20050627   -filter "25-Jun-05 23:56:38" "27-Jun-05 12:39:49" -oet "27-Jun-05 12:39:49"

       

      A filter has to be used in order to define which time span of the source archive will be transferred into the destination archive file. 

       

      In addition, the output end time has to specified, mostly for a case where the source archive file ends after the destination archive file.  If an output end time is not specified, the end time of the destination archive will be set to the same end time as the source archive file.

       

      Many attempts were performed before to successfully merge the archive files.

      2.1       First attempt: no sub seconds

      In the first attempt, we did not realize that the archive files had start and end times in sub seconds.  This was due to the Archive Manager of PI-SMT which, by default, shows time in Windows format and therefore does not display sub seconds.  This is set in PI-SMT: Tools -> Settings

       

      The results were problematic in two ways:

       

      ·                 Some events were missing.  As an example, if an archive ends at 01:23:45 then an event at 01:23:34.12345 would not be part of it

       

      ·                 There was an archive gap between each archive files.  As an example, an archive ends at 01:23:45.00000 and the next one starts at 01:23:34.12345

      2.2       Second attempt: Use the Archive Manager to export sub seconds

      In the second attempt, the PI-SMT Archive Manager has been set to use PI Time with sub seconds.  The result was as bad as in the first attempt because the Archive Manager only displays a maximum of 3 digits for the sub seconds, with the last one being rounded.

       

      The results were once again problematic in two ways:

       

      ·                 Some events were missing.  As an example, if an archive ends at 01:23:45.123 then an event at 01:23:34.12301 would not be part of it.  If the next archive start at 01:23:45.12340, then this event has no place to go.

       

      ·                 Many archive files could not be registered because they were overlapping.  As an example, an archive ends at 01:23:34.12300, the next one starts at 01:23:12987.  They overlap and can’t be registered at the same time.  In other cases, there was an archive gap between 2 archive files.

      2.3       Third attempt: Use piartool –al to retrieve the exact start and end time

      In the third attempt, piartool –al has been used to retrieve the archive files start and end times (with 5 digits sub seconds).  This involved a few extra steps in order to extract only the required information from the piartool output.  The result was almost the same as the second attempt because the time output shown in piartool are rounded to the nearest hundred thousandth of a second

       

      The results were once again problematic in two ways: some events were missing and archive files were either overlapping by micro-seconds or had micro-second gaps.

       

      The explanation is that the smallest time granularity in PI is 1/65536 seconds.  As an example, when piartool displays 0.01180 s, the real value is 0. 0117950439453125, which has been rounded.  However, when the filter specifies 0.01180, the system does not round to the closest increment, it goes to the next possible one, which is 0.011810302734375.

       

      Therefore, when piartool shows the start time of archive A at 01:23:34.01180, it really is 01:23:34.0117950439453125.  If the command line used to merge the previous archive specifies to end the file at 01:23:34.01180, it will really end at 01:23:45.011810302734375.  These archive files will overlap.

       

       

      2.4       Fourth attempt: Use piartool –al to retrieve the exact start and end time, then correct to the nearest 1/65536 seconds

      In the fourth (and successful) attempt, piartool –al has been used to retrieve all the archive start and end times.  However we did not use these times directly.

       

      A lookup up table was used to translate the output of piartool in the proper 65536th increment, then this exact time was used as a filter and as the time for ending the archive file. 

        • Re: How to merge archive files with sub-second start/end times
          dvacher

          Thanks for this pertinent article on PI timestamp precision and rounding, in the context of offline archive reprocessing.

           

          I would like to mention that starting in PI Server version 3.4.364 (released 11/2004), the Archive Subsystem does not generate archive files with subsecond start/end times. During a shift, the Archive Subsystem will use the next round second when closing an archive and opening the next one. You may obviously find systems that have archive files initialized with older PI Server versions.
          Note: you can still create archive files with subsecond time boundaries using SMT or piartool.

           

          The bug in SMT (showing only 3 subsecond digits) was noted and will be addressed in a subsequent release. PLI number is 18725OSI8.