Caution with converted UOM and removing values with the UpdateValues method

Discussion created by Rick_Davin_3.0 on Nov 13, 2020
Latest reply on Nov 13, 2020 by TimCarmichael

This post explores some subtle details of using the AF SDK UpdateValues method when using Remove as the AFUpdateOption.  Note that usually a call to RecordedValues is first made, and then those recorded values are pushed back to UpdateValues AS IS.  Quick code demo:


var values = attribute.Data.RecordedValues(timeRange, AFBoundaryType.Inside, null, null, true);
attribute.Data.UpdateValues(values, AFUpdateOption.Remove, AFBufferOption.BufferIfPossible);


A work colleague recently contacted me with news she may have discovered a bug when she was trying to delete archived values using the UpdateValues method.  News of a possible bug is mixed.  On one hand, there is a certain flush of geek pride to have discovered a bug.  On the other hand, there is sobering notion that your code doesn't work and it may require a work-around that may or may not exist.


In this particular case, she could not delete negative values, but when she described the issue more, I realized quickly that (A) it had nothing to do with the values being negative, and (B) the greater hunch is that this was not a bug but a quirky artifact related to UOM conversions.  


Before going further, let's understand the expected behavior of UpdateValues (or even UpdateValue) when using AFUpdateOption.Remove.  You will pass in an AFValues collection, and for each AFValue in that collection, it will be removed from the data archive IF a match currently exists in the data archive.  This means the Timestamp and Value must be the same for the compared AFValues (i.e. the ones in your list and the ones in the archive), otherwise the archived value will not be deleted.  Thus, for this to be a bug, it would have to mean that the method was not behaving as expected.


My hunch to my colleague was that the values she was using were most likely not an exact match.  Though this post is about AF SDK methods, I can demonstrate this without code. 


Let's start with the AFAttribute, which I named Temperature.  It will use a UOM of °F but it's underlying PIPoint has an EngUnit of K or Kelvin.  Note that the AFAttribute is a Single and the PIPoint is a Float32 (or the same),



Most of us are familiar with the DefaultUOM property but there is also a read-only property called SourceUOM.  In many attributes, these are the same but AF does allow them to be different, and that is certainly the case here.


To see what's going on, let's add 2 more attributes.  The first will be the raw PIPoint, which I will just let be unitless although using K would be acceptable as well. 



The next demo attribute will be called Roundtrip.  It uses a Formula that converts the Temperature from °F back to K.



Let's pause to review.  We have 1 PIPoint that is a Float32, and 3 AFAttributes that are Single. The Temperature attribute will have an implicit UOM conversion done when data is read from the archives in K but passed to Temperature as °F.  Roundtrip will then explicitly convert that °F back to K.  We can compare that to the Raw PIPoint to see if they are the same.



Uh oh.  These look the same.   Did my colleague discover a bug? 


No.  Do keep in mind that they appear the same but appearances may be deceiving.  If you review my attribute configurations above, you will note I was using the default Display Digits of -5.  Watch what happens when I declare Display Digits to be -20:



And there it is.  We now visually see all the decimal places, and more importantly, we can see that the value Roundtrip is just ever so different from the Raw PIPoint.  Roundtrip and Raw PIPoint will fail an equality test, which is why my colleague was unable to delete the archived values.  Since she passed in values that did not match, they were not being deleted, and this is not a bug since it is the defined behavior.  




Consider what a developer is doing.  They will make a RecordedValues on the Temperature attribute.  They then turn immediately around and push those recorded values right back to the data archive to be deleted.  It should work since you haven't altered the values.  Just because the developer hasn't altered the values doesn't mean the values haven't been altered.  They have been, and the reason is because there is a whole lot of implicit things happening.


The first is with RecordedValues.  Archived values are fetched from the data archive.  Next they are assigned a UOM of K.  Then each K needs to be converted to °F.  For this to happen, the Float32 or Single values are widened to a Double.  Then the conversion expression is evaluated, resulting in a Double.  Finally, that Double is then truncated to a Single, since Temperature is a Single.  This truncation results in lost precision from a Double with 16 significant digits to a Single with only 7 significant digits.  This last thing is the most damaging in the process.


The second thing is calling UpdateValues with the truncated values that were converted to °F.  To go back into the data archive, they will need to be converted back to K.  Like above, the Single °F value will be widened to a Double, the conversion is evaluated resulting in a Double.  This Double is then truncated to a Float32 to be passed to the PIPoint.  But it fails due to the original conversion from K to °F,



The good news is that there are at least 3 possible workarounds I can think of.


ONE, define the Temperature attribute to be a Double.


The biggest failing in the process is when the original Kelvin value is converted to °F and then truncated to a Single.  If you don't truncate to a Single, the round trip conversion back to Kelvin will work.  But now Temperature looks ugly, or it looks ugly with DisplayDigits of -20.



TWO, specify the SourceUOM in the RecordedValues call.


A lot of developers overlook the SourceUOM property, but it can be used to save the day here.


Change from:


var values = attribute.Data.RecordedValues(timeRange
, AFBoundaryType.Inside
, null
, null
, true);


To (note line 3):


var values = attribute.Data.RecordedValues(timeRange
, AFBoundaryType.Inside
, attribute.SourceUOM
, null
, true);


THREE, use the ReplaceValues method instead


Deleting a handful of recorded values isn't a big deal.  You fetch data from RecordedValues and then push those values back with UpdateValues.  This is 2 trips to the PI data archive.  What if you have a million values to delete?  You have to pull all 1 million into the client memory, only to turn around and push all 1 million back across the network.


The ReplaceValues is a one-trip method.  To delete values within the request time range, pass in an empty AFValues collection. Not only is this more performant, but you have totally circumvented any UOM conversion issue.


I personally would go with ReplaceValues.


Until next time, happy coding.