15 Replies Latest reply on Apr 21, 2010 2:07 PM by jnissisolutions

    Recorded vs. Interpolated Values in Summaries

    jnissisolutions

      Is there a way to drop out intervals in a Summary Call that do not have any recorded values?  I would like to use a start/end time span of a month,  calculate Max of Averages within 1 hour intervals but only do that on the 1 hour intervals that have recorded values.  I do not want to interpolate the intervals or use any interpolated data in intervals that do not contain recorded values in the 1 hour time segments.

       

      So if 15 values are recorded during a month I only want to use the 1 hour segments where those intervals were recorded in the max of averages that the summary would do.

       

      The background consideration is that all this is in a process (C# program) that mixes sparse, occassional, hand entered data with continuous monitor based data.

       

      Dan

        • Re: Recorded vs. Interpolated Values in Summaries
          cescamilla

          Dan,

           

          I know that the cbEventWeighted parameter will only succeed if there is a value on each time range, I'll check if it skips a time range otherwise, I'll write some test code and post it in a bit.

            • Re: Recorded vs. Interpolated Values in Summaries
              hanyong

              I thought of something different from what Cristobal has suggested, using FilteredSummaries and "EventCount" in the filter expresssion.

               

              The calculation result is not dropped for the time range that does not have a result, instead you get a value of -2147219947 which indicates an error (you can get the error description from the ValueAttributes "ErrDescription"). Testing for PIValue.IsGood() should let you know if the value is an error or not

               

              Here's what I used in my test, calculating 2hour averages within a day:

              PISDK.PIPoint CDT158 = myserver.PIPoints["CDT158"];
              PISDK.IPIData2 data = (PISDK.IPIData2)CDT158.Data;
              PISDKCommon.NamedValues nvValues = new PISDKCommon.NamedValues();

              string filexpr = "EventCount('cdt158', '*', '*+2h') > 0";

              nvValues = data.FilteredSummaries("*-1d", "*", "2h", filexpr, PISDK.ArchiveSummariesTypeConstants.asAverage, PISDK.CalculationBasisConstants.cbTimeWeighted, PISDK.FilterSampleTypeConstants.fstInterval, "2h", null);

              object reference = "Average";
              PISDK.PIValues vals = (PISDK.PIValues)nvValues.get_Item(ref reference).Value;

              object reference1 = "ErrDescription";

              foreach (PISDK.PIValue val in vals)
              {
                      if (val.IsGood())
                              // if value is good, show value and timestamp
                              MessageBox.Show(val.TimeStamp.LocalDate + ":" + val.Value);
                      else
                              // else show error description
                              MessageBox.Show((string)val.ValueAttributes.get_Item(ref reference1).Value);
              }

              • Re: Recorded vs. Interpolated Values in Summaries
                andreas

                Dan,

                 

                PISDK Help on cbEventWeighted

                Evaluate values with equal weighting for each event. No interpolation is done. There must be at least one event within the time range to perform a successful calculation. Two events are required for standard deviation.

                 

                I did a brief test with

                PISDK.PISDK _pisdk = new PISDK.PISDK();
                PISDK.Server _piserver = this.servPickList1.SelectedServer;

                _piserver.Open("");
                PISDK.PIPoint _pipoint = _piserver.PIPoints[txtTagName.Text];

                listView1.Clear();
                listView1.View = View.Details;
                listView1.Columns.Add("Time");
                listView1.Columns.Add("IsGood");
                listView1.Columns.Add("Value");

                PISDKCommon.NamedValues _piNdVals;

                _piNdVals = _pipoint.Data.Summaries
                                ("T",
                                 "T+12h",
                                 PISDK.BoundaryTypeConstants.btInside,
                                 PISDK.ArchiveSummariesTypeConstants.asMaximum,
                                 12,
                                 PISDK.CalculationBasisConstants.cbEventWeightedIncludeBothEnds,
                                 null);
                 
                PISDK.PIValues _pivalues;
                object _o = "Maximum";
                _pivalues = ( PISDK.PIValues)_piNdVals.get_Item(ref _o).Value;

                foreach (PISDK.PIValue _pivalue in _pivalues)
                {
                    PISDKCommon.NamedValues _piNdValAtts;
                    _piNdValAtts = _pivalue.ValueAttributes;
                    ListViewItem _lvi = 
                        newListViewItem(_pivalue.TimeStamp.LocalDate.ToShortTimeString());
                    _lvi.SubItems.Add(_pivalue.IsGood().ToString());
                    if (_pivalue.IsGood() == false)
                    {
                        _lvi.SubItems.Add("ERROR");
                    }
                    else
                    {
                        _lvi.SubItems.Add(_pivalue.Value.ToString());
                    }
                    listView1.Items.Add(_lvi);
                }

                As you can see from the result (and my code ) the pivalues collection will contain also the intervals where there is no value - but it will not interpolate that, it will simply be a bad value.

                 

                Hope this helps,

                • Re: Recorded vs. Interpolated Values in Summaries
                  cescamilla

                  What summary function are you using?

                   

                  IPICalculation.ExpressionSummaries
                  This method creates a data stream from an arbitrary expression and calculate one or more summaries on the expression result.

                   

                  IPIData2.Summaries2
                  The Summaries2 method retrieves several summary types in a single call over the specified range, and for each interval within the range.

                  IPIData2.FilteredSummaries
                  This method, when supplied a filter expression that evaluates to true or false, evaluates it over the passed time range.

                   

                  PIData.Summaries
                  The Summaries method retrieves several summary types in a single call over the specified range, and for each interval within the range.

                   

                  The only one that could do what you want is PIData.Summaries, but then again, the parameters you could pass to that function are:
                  StartTime, EndTime, BoundaryType, SummaryType, nIntervals, CalculationBasis, AsyncStatus

                   

                   

                   

                  Do you input the data by hand as an hourly average?

                   

                  Are you doing Event or Time Weighted Averages?

                   

                  Do you want to use the maximum of both, the entered value and the one the pi server responds with?

                   

                  Do you want to perform both functions on the PI Server?

                   

                   

                   

                  You could request the averages from the pi server event based for each hour in the past month and then filter out (programatically) the values that you did not return a value, add this (or copy these) to another object to mix the manual entry data and then use the object as needed.

                   

                  I may not be understanding properly what you are trying to do, so please respond to this thread directly if this is not correct.

                    • Re: Recorded vs. Interpolated Values in Summaries
                      jnissisolutions

                      What I have in this case is the need to calculate expressions on dense data and not-so-dense data.  The dense data is working fine with the ExpressionSummaries function.  In that case I analyze the data in 15min intervals, take the average of the interval and find a maximum value over a time range.

                       

                      I would like to use the same function for the sparse data, but I'm not sure if I can.  In order for it to work, the intervals that do not have recorded data should drop out of the maximum result set.  What appears to be happening (regardless of whether I use the suggestions made by Andreas) is that the result uses interpolated values for each of the intervals.

                       

                      The research that I did for this is not complete though.  Next I need to determine if I am working with more recorded values that I think I am.  I am under the impression that it is sparse data for certain points, but I could be mistaken.

                       

                      I'll get back to this tomorrow. Thanks for all the good input.

                       

                      Dan

                        • Re: Recorded vs. Interpolated Values in Summaries
                          jnissisolutions

                          Ok.  I checked the recorded values for the tag that I am working with.  There are only a half a dozen readings since the beginning of the year.  Sparse data. 

                           

                          The reason I decided on ExpressionSummaries is: 1.) I'm already using it for the dense data formula.  2.) It is a formula that I am evaluating and not simply a tag value.

                           

                          I did some initial tests with a few different settings.  The one that initially looked favorable was...

                          peakProcItem.maxNamedHr = piCalcVal2.ExpressionSummaries(startTime, endTime,
                          null, peakProcItem.formulaHR, PISDK.ArchiveSummariesTypeConstants.asMaximum,
                          PISDK.CalculationBasisConstants.cbEventWeightedExcludeEarliestEvent,
                          PISDK.SampleTypeConstants.stRecordedValues, "1h", piHrStatus[hrCount]);

                          The parameter PISDK.CalculationBasisConstants.cbEventWeightedExcludeEarliestEvent, did what one would expect.  It excluded the beginning value.  And I thought I had found the solution.  Then I came upon a date range where the ending value was interpolated to be greater than the recorded value.  In that case, I have the same problem.

                           

                          Is there a way with this function to exclude the interpolation at both ends of the time range?

                           

                          Dan

                            • Re: Recorded vs. Interpolated Values in Summaries

                              Dan Jablonski

                              Is there a way with this function to exclude the interpolation at both ends of the time range?
                              Unfortunately, no. You might want to file this in as an Enhancement Request with our regular Technical Support team, i.e.: add 'cbEventWeightedExcludeBothEnds' to the PISDK.CalculationBasisConstants.

                                • Re: Recorded vs. Interpolated Values in Summaries
                                  jnissisolutions

                                  Steve,

                                   

                                  Ok.  Thanks, I will do that.

                                   

                                  For this circumstance I can switch functions and apply the one that Andreas mentions above for the sparse calculations.  In the future, it would be helpful in case there is a short burst of data that needs to be analyzed in incremental sections for the max of a average and either end of the date range excluding interpolation.  I'm surprised it doesn't exist already.  Seems like it would be a common problem on any archival date range expressionsummary calculation that has irratic data density.

                                   

                                  Dan

                                    • Re: Recorded vs. Interpolated Values in Summaries
                                      jnissisolutions

                                      I'm still having some problems with this.  I thought I had solved by using a calculate function, but interpolation appears to rule there also, even when I used the recorded values parameter...

                                      valueqtr = piCalcVa2.Calculate(startTime, endTime, formula, PISDK.SampleTypeConstancts.stRecordedValues, duration, null);

                                      I set the duration for the entire time period, so I only get one value in the result.  I thought I would get a result based on the RecordedValues,  if I supply max of a tag as the formula, I thought I would get the max of the Recorded values for that duration and time range, but that is not the case.  I get the interpolation for that time period.  Somehow that seems incorrect for your system.  I am supplying a parameter that is stRecordedValues.

                                       

                                      Thanks,

                                       

                                      Dan

                                       

                                       

                                        • Re: Recorded vs. Interpolated Values in Summaries

                                          Hi Dan,

                                           

                                          Note that the Calculate method will not return recorded values based on the sampling type parameter - it will use recorded values to evaluate the passed formula.

                                           

                                          Speaking of whic, what is the "formula" parameter here? (I assume something like "Max('yourTag')" ?)
                                          And are you certain there is at least 1 recorded value within the time range you provide?

                                          • Re: Recorded vs. Interpolated Values in Summaries
                                            hanyong

                                            Dan Jablonski

                                            valueqtr = piCalcVa2.Calculate(startTime, endTime, formula, PISDK.SampleTypeConstancts.stRecordedValues, duration, null);

                                            I set the duration for the entire time period, so I only get one value in the result.  I thought I would get a result based on the RecordedValues,  if I supply max of a tag as the formula, I thought I would get the max of the Recorded values for that duration and time range, but that is not the case.


                                            Just like to add something about Calculate method here, even though the discussion seem to have moved off in another direction.

                                             

                                            The duration in Calculate method is a sampling interval, and is only valid when stInterval is used as the Sampling type. So setting it with stRecordedValues will not have any effect. Judging from what you described, it seems to me like you want the max value of a tag within that time range (from start to end time). If thats the case, you can consider using PIData.Summary(starttime, endtime, SummaryType, CalculationBasis. AsynchStatus).

                                              • Re: Recorded vs. Interpolated Values in Summaries
                                                jnissisolutions

                                                Han,

                                                 

                                                Yes, that's what I have been seeing.  stRecordedValues has no effect on the calculate function.

                                                 

                                                In regard to what I'm going after, there is a little more to it than that.  It's not simply a tag, but an expression.  Also, of course, it's within a system that is constructing the time ranges and calculations determining if it is a sparse or dense set of values.  For dense data, the expression summary function works fine.  If it is sparse data, well, that is where this thread started.  The expression summary function won't work well for the reasons I mentioned above.

                                                 

                                                Thanks for your interest.

                                                 

                                                Dan

                                        • Re: Recorded vs. Interpolated Values in Summaries
                                          ken

                                          Dan,

                                           

                                          If you are using event weighting it shouldn't be interpolating any values. I don't quite understand your comment:

                                           

                                          Dan Jablonski

                                          Then I came upon a date range where the ending value was interpolated to be greater than the recorded value.

                                           

                                          Does that mean that the maximum for an interval was greater the expression's value within that interval? Can you supply the expression, the archive data, and the expression summary result for that interval?

                                           

                                          The purpose of the CalculationBasis settings: cbEventWeightedExcludeEarliestEvent & cbEventWeightedExcludeEarliestEvent is to prevent the double counting of events when doing an event weighted summary. For an event which falls on an interval boundary, it lets you select which interval (previous or following) it should belong in. Excluding events which fall on an interval boundary from both the previous and following interval means that event won't be included at all. This would cause problems for boundaries other than the start and end time of the call.

                                           

                                          Ken___

                                            • Re: Recorded vs. Interpolated Values in Summaries
                                              jnissisolutions

                                              Ken,

                                               

                                              Ok.  That helps explain the Event Wighted concept. 

                                               

                                              I am still unsure about how to evaluate a sparsely populate area of a larger range though.  As an example let's say I am evaluating 3 months worth of recorded values.  The first and third months end and start with high values respectively.  My starttime and endtime are the beginning and ending of the 2nd month.  The recorded values in that month are low.  I am using an expression that takes an average for each day, but I need to exclude days (at least the start and end days) that do not contain any recorded values as the day averages will be evaluated as a maximum for specifically that month. Currently what I am seeing is that the intervals (days in this example) at the beginning and ending of the month are evaluated and used even though they do not contain any recorded values.  This causes the maximum to be too high for the overall month as the end of the first month and the start of the third month contain higher values.

                                               

                                              My next attempt will be to programmatically reset my start and end time to coincide with the earliest and latest recorded values of the 2nd month. 

                                               

                                              Thanks for your interest.

                                               

                                              Dan