1 of 1 people found this helpful
What happens if you try if (timestamps[i] == timestamps[i+1] && (flags[i] & AFValueStatus.Substituted))?
What are the values of flags[i] and flags[i+1] for your duplicates?
What version of AF SDK are you using?
The AFValue.GetValueArrays returns what was provided to the source AFValues collection, perhaps its source was a GetValues or RecordedValues call. But definitely if your data stream has duplicates in its source, and those duplicates end up in the PI Data Archive, then yes the GetValueArrays will contain duplicates as well.
In addition to inspecting the flags when a duplicate event is found, you can also verify against your original afvals object. When timestamp[i] equals timestamp[i + 1], you can double-check that afvals[i].Timestamp equals afvals[i + 1].Timestamp as well. If duplicates exist in afvals, then we can eliminate GetValueArrays from being a problem. You then move up the chain to see why afvals has duplicates. What data method are you calling to fill afvals?
You say it's random, which makes it hard to troubleshoot. Oftentimes someone may post a problem that is very repeatable on their end that we can't repeat on our end. It's more challenging when the original post can repeat on their end.
Good questions, I had inspected flags in VS, but not included them here. Adding little more to the logs:
Duplicate for tag ACCA_EMS.LINE.88B.MW at index 2153, t1 10/18/2017 1:10:25 PM, f1 Good, v1 14.3, t2 10/18/2017 1:10:25 PM, f2 Good, v2 14.3
t1 is timestamp at timestamps[I], f1 is flags[I] (status), v1 is values[I]. t2, etc., are identical but for the i + 1 values.
Source is indeed a RecordedValues call:
ppl.RecordedValues(range, AFBoundaryType.Inside, null, true, new PIPagingConfiguration(PIPageType.TagCount, 100));
AF SDK Version is 184.108.40.20601.
I didn't originally look the afvalue -- I always assume perhaps incorrectly that afvalue is a façade ontop of the arrays. I checked these and they too are identical / duplicate.
It is an interesting issue -- the values and flags are identical. I had thought that replaced values would be marked as substituted, but all events in array are good. I looked at SMT to inspect the archived values, no substituted, only a single event. Checked logs and no issue showing for this tag...
Intermittent events == fun, right? Thanks! Phil
They appear to be identical but if I remember rightly, PI supports subsecond times, are you sure these aren't 10/18/2017 1:10:25.250 PM and 10/18/2017 1:10:25.750 PM for example?
From the AF SDK documentation:
> Actual storage of an AFTime value may cause a loss of accuracy depending on the target storage. The PI AF Server will store timestamps to an accuracy of 3.33 milliseconds. A PI 3.x Server stores timestamps to an accuracy of 15 microseconds.When these values are then fetched by the AF SDK they may then be truncated so that they are the same client side, but not server side.
1 of 1 people found this helpful
Actually, the AFValues collection as retrieved from RecordedValues is the true source, and populating GetValueArrays is fed directly from it. I would suggest you can omit GetValueArrays and iterate over afvals directly.
Nicholas Alderton the subseconds are not at play here. Usually that occurs the opposite direction, that is when someone visually sees 2 values in SMT that have the same timestamp but the code says they aren't the same (because the subseconds are hidden from their view). But AF SDK does not truncate the times with PIPoint.RecordedValues. Actually the Timestamp of an AFValue can have greater precision than 15 microseconds! It's only when attempting to write back to the system of record that the time will be truncated - by that system of record. For the documentation, the AF Server storing timestamps would only refer to actual values stored on the AF Server, and that would be for static AFAttributes or event frames StartTime and EndTime.
See A Detailed Exploration of AFTime Precision for more.
Ah that makes sense, it was a bit of a stretch that one. Great link!
Thanks for the suggestions and information. I also looked at timestamp ms and can confirm there is no difference.
Rick, information on value arrays is interesting. I had assumed the opposite because the documentation says:
This method can perform higher than extracting individual AFValues or PIValues because less objects need to be created.
That of course could mean a lot of things Maybe more clarification in the documentation might help.
Performance aside, for my use it's still preferable to use the array structures as the timeseries work I'm doing can directly load data. Less code, and a little more performant. Regardless, after looking at the AFValues, and still finding the phantom duplicate, I'll have to work around that. Right now I can change duplicates in the array to DateTime.MinValue.AddSeconds(dupcntr++) to avoid uniqueness constraints, then filter out anything in year 1. It's more work, but works.
I don't know if I should open up a tech support case at this point? Others may experience this in similar conditions but may not notice or care...
Thanks for the thoughts, Phil
If the problem was with code, you can keep this thread alive. But the problem doesn't seem to be with your code. The larger issue seems to be with your environment. What is it about the interfaces or flow of data that is causing duplicates? And that may be more of a configuration or installation problem where TechSupport can help.
I too find the documentation remark confusing. First off, "can perform higher" could really be interpreted as "might possibly perform higher, or might not". And GetValueArrays must be called from an AFValues collection, which has already extracted all of the data client-side so I am a bit lost on how that is more efficient. I had always considered GetValueArrays a feature for legacy PI SDK code (see PI SDK GetValueArrays or Load), or as a mechanism to pass data to another layer that doesn't support AF SDK objects directly.
The AFValueStatus remarks says something the same: When obtaining large numbers of values, it is more efficient to access the values and status using the AFValues.GetValueArrays method which returns the status information as flags.
I have checked with our development team and received this reply:
You are correct. GetValueArrays is a convenience method for interoperating with other systems. Same with the constructor for AFValues that takes arrays.
I would suggest you are better off ditching the GetValueArrays call. Meanwhile I will start an internal discussion on clarifying the documentation.
Thank you Rick.
If someone from OSIsoft does look at this in future, I would very humbly suggest it's a defect:
- Just looked at the custom code which is inserting the large quantities of dups. Unfortunately I can not touch it or this wouldn't be an issue, but it is inserting duplicates with AFUpdateOption.NoReplace
- Documentation says this will "Add the value to the archive only if no value exists at the same time. If a value already exists for that time, the passed value is ignored."
- If this is the case, there should be no side effect on RecordedValues. If the interface was only doing updates that were strictly duplicates, RecordedValues should be idempotent. In this case, at random occasions, it is not.
Thanks for looking into the documentation as well -- sure a few other folks have read that and said "...lets have a go with that..."
Again, many thanks for your thoughts on this, Phil
1 of 1 people found this helpful
It is a known PI Data Archive behavior that NoReplace does not apply to snapshot event, it will behave as Insert.
There are several posts on this issue:
2 of 2 people found this helpful
Last year I wrote a little diagnostic/learning tool named "Timestamp Inspector" specifically for an extremely easy way to detect:
- Duplicate timestamps, or
- Subsecond timestamps
The executable and source code are on GitHub.