7 Replies Latest reply on Sep 21, 2018 2:16 PM by Rick Davin

    What is the correct time syntax/format for  get_recorded_values?

    ilima

      Hello,

       

      I am new to PI system in general and I am trying to get some data from my AF server (as a pandas data frame) through the PI Web API client libraries for Python. The data is a time-series, so I would like to select only a specific date (September 6th,2017) and time interval (1 hour). I am using the function get_recorded_values and I  have tried several data/time formats for the end_time, but, no matter what I try, I keep getting data with timestamps from 2018-05-08... Is there a specific syntax that I should use?

       

      Thanks in advance,

      Isabela

        • Re: What is the correct time syntax/format for  get_recorded_values?
          tramachandran

          Function get_recorded_values is available in GitHub repo:

          df1 = client.data.get_recorded_values("pi:\\\\PISRV1\\sinusoid", start_time="<START>", end_time="<END>")

           

          Keep in mind that PI Web API returns timestamps in UTC only and any conversions have to be done on the client side. So you need to take your time zone into account.

          • Re: What is the correct time syntax/format for  get_recorded_values?
            Rick Davin

            Have you tried Python's datetime.isoformat?  You definitely should keep the default separator of 'T'.  One possible conflict might if the returned string ends with a "Z".  A Python string without a time zone or "Z" is assumed to be UTC, whereas for PI Web API, which makes AF SDK calls under the hood and therefore relies upon .NET Framework, a string without a time zone or "Z" is assumed to be Local time.  Thus if your string lacks time zone info or a "Z", you may need to append a "Z" on  it before passing to the client.

              • Re: What is the correct time syntax/format for  get_recorded_values?
                ilima

                Thank you Rick and Thyagarajan for your answers.

                 

                I've tried passing the end time as a string and as a datetime object and they both produced the same wrong result... Please find below the code I used and the result of df.head():

                 

                Code:

                 

                ### Time as string-------------------------------------------------
                t1="2017-09-07T13:00:00.0000000Z"

                df_acc01 = client.data.get_recorded_values(afPath, start_time="-1h", end_time=t1,selected_fields=None,max_count=None,filter_expression=None,include_filtered_values=None,boundary_type=None,time_zone=None,desired_units=None)

                 

                print(df_acc01.head())

                 

                 

                ### Time as datetime object----------------------------------------------
                timezone = pytz.timezone("Europe/Berlin")

                t_local=datetime.datetime(2017,9,7,15,00,00,00)

                t_aware=timezone.localize(t_local)

                 

                df_acc02 = client.data.get_recorded_values(afPath, start_time="-1h", end_time=t_aware.isoformat(),selected_fields=None,max_count=None,filter_expression=None,include_filtered_values=None,boundary_type=None,time_zone=None,desired_units=None)

                 

                print(df_acc02.head())

                 

                Result (df.head()):

                 

                    Good Questionable                Substituted   Timestamp  \
                0  False     False    False  2018-05-08T10:42:19.4490051Z  
                1  False     False    False  2018-05-08T10:42:19.4330139Z  
                2  False     False    False  2018-05-08T10:42:19.4330139Z  
                3  False     False    False  2018-05-08T10:42:19.4180145Z  
                4  False     False    False  2018-05-08T10:42:19.4180145Z  

                 

                Any suggestions?

                  • Re: What is the correct time syntax/format for  get_recorded_values?
                    Rick Davin

                    Hello Isabela,

                     

                    Note how the times are in descending order.  Hard to see since they have the same date down to the second, but there is subseconds that clearly show it is descending.  That is because your start time of "-1h" really means Now minus 1 hour (or 1 hour ago), and your end time is a year ago.  Because the end time occurs before the start time, the recorded values retrieved are shown in descending order.

                     

                    Can your verify with another tool, perhaps PI-SMT, that you have more recorded values from the time range you've asked for?

                     

                    The other thing as I re-read your original post is that you want data values in 1 hour intervals.  This is not what get_recorded_values does.  get_Recorded_Values returns the compressed archive values as they are in the archives.  What you probably want instead is get_interpolated_values, where you may specify an interval.

                    4 of 4 people found this helpful
                      • Re: What is the correct time syntax/format for  get_recorded_values?
                        ilima

                        Thank you very much! If I now set the specific start time I need (instead of -1h), I get the correct time interval. Sorry about the silly mistake and thanks again!

                        • Re: What is the correct time syntax/format for  get_recorded_values?
                          ilima

                          Hi Rick,

                           

                          I have a question regarding your comment that I should use the get_interpolated_values function. I was originally using get_recorded_values, because I want to analyze data as it was input to the PI system, without any processing. By doing so, I realized that the max_count term is limiting the amount of samples I can get to 1000, when set to None, and it can only be increased up to a certain value. Is there a specific flag I could pass to the parameter max_count, so that the function will return all samples that belong to the time interval I'm interested, no matter how many they are? Or is the limitation of max_count the reason why you suggested the get_interpolated_values function?

                           

                          Thanks in advance,

                           

                          Isabela

                            • Re: What is the correct time syntax/format for  get_recorded_values?
                              Rick Davin

                              Hello Isabela,

                               

                              The reason I suggested interpolated values has nothing to do with max_count.  In your original post you wanted to look at data in 1-hour intervals for a time range.  This just screams for interpolated data.  And it also provides for data to be evenly aligned on the same time horizon, which is beneficial for your pandas.

                               

                              If you abandon the idea of 1-hour intervals, and instead the goal is to look at raw values in the archives, then yes the solution now screams for recorded values.  The default for max count is 1000 but you may increase it.  Note there is a practical limit to all data requests with the PI System.  This is determined by a PI Data Archive tuning parameter called ArcMaxCollect.  Its default setting is 1.5 million.  We do not advise changing ArcMaxCollect.  For one, it requires a server reboot, but for two, you really don't want everybody being able to fetch 10 million values at once.  If you have a huge amount of data, you may want to narrow your time range instead.

                              1 of 1 people found this helpful