Is there any chance to retrieve original value when the value was substituted ? Any idea is appreciated.
Once the value is Substituted you cannot retrieve the old value. It is only possible to know which events were substituted by looking at the flag set on it.
Note: If you have the archive backed up before substituting the values you will be able to get to the old values. I would recommend this only if some critical values have been replaced and for some reason you want to get to the original state.
Thank you for the reply. That's pity, I was hoping on something called "Event Frames". Backups is not an option.
Event frame is not an option to see previous value... In PI Data Archive, there is a PI Audit function. So if you enable it, Audit records will contain previous value. From PI Audit Viewer, you can see the previous value. The problem is that you need to go to PI Data Archive machine to see these values. Currently there is no way to see it from PI Web API, AF SDK, ODBC etc...
So basically you need to put the previous events in somewhere manually if you want to see the previous values.
It is old technology but PI tag annotation with PI SDK may work if you have knowledge of PI SDK coding.
(But remember that PI SDK is old technology. PI Web API cannot read PI tag annotation.)
If you want to use the latest coding technology, the question is where should we put these previous values. The possible answers may be different tags or AF/ event frame attribute etc...
It is out of curiosity, why do you need to see previous values?
I need to track all the "substitutions" in PI instances across several plants, the old values are very desirable for ease of audits and security investigations.
With regard to implementation, it may be different tags, it may be a dedicated journal, it may be an annotation containing replaced value, it may be "insert-only" approach with data versioning and a specific API call to extract all the verions (I understand this approach would significantly affect architecture).
This is where you should look at enabling audit logging on your PI server. When you enable auditing for the snapshot and archives, value substitutions will be logged in the audit trail, and you can see information such as original value, new value and the user that made the substitution (see the Live Library)
Auditing is enabled in the Operation => Tuning Parameters section of PI-SMT (PI Server Auditing).
I know about the auditing, but I can't seriously consider that option due to the limited capacities. Thanks anyway.
Unfortunately audit logging is the only real option available for tracking data substitutions as you've described. You could possibly build your own AFSDK application based around data pipes to capture out of order data writes that might end up overwriting an existing value and then log that information to your own log or journal, but I seriously doubt that it would be a) worth the effort, and b) not as accurate as the built-in audit logging.
For the scenario you have described, why do you consider the built-in auditing to be of limited capacity? Sure it is on a per server basis - you won't get an overarching log for all your PI instances, but then the approaches you were looking for would have the same limitations.
Well, if there is no reliable approach with WEB API, I would rather go for the insert-only implementation on server-side, that's possible in my case (means the data points will be ingested to our cloud insert-only db).
My engineers said that the built-in auditing would be significantly consuming disk space - 250-500MB daily, it's just not an option unfortunately.
One more thing to sort out, is there any "alerts" functionality in PI ? What if I setup a specific trigger upon value change which would log the alert, are you familiar with that ?
I'm curious as to how that value of 250-500MB per day was determined for audit log growth. There is fairly granular control over what is captured in the audit logs - if you only want to track changes to archive data then that is all that needs to be enabled. If tracking just those changes in the audit log consumes up to 500MB per day then I would be asking why so much archive data is being altered in the first place - that represents an awful lot of audit records.
There is a Notifications feature that is built into PI AF, however this may not be a suitable option for tracking changes to existing data. You could probably configure a trigger expression to check if the timestamp associated with the triggering value is older than the current snapshot timestamp, and you could send the Notification content to a webservice that generates an audit 'log' for you, but you would have to configure such a Notification rule for each PI point that you want to specifically track for changes. If this turns out to be any/every tag in the system, then AF Notifications is not the solution you are looking for either.
Okay, they probably meant such a log growth with auditing everything, let me check it out. The granular control can be an option, what is the API call for gathering auditing events, if you aware of ?
You can't read the audit log using any of the developer technologies such as PI WebAPI, AFSDK etc. You will need to use the PI Audit Viewer application, and this can only be used on the PI Data Archive server (see the Live Library documentation).
I see, thank you for the info. One more idea, what is about some ad-hoc tool, which would periodically gather archived data (API call /recorded?startTime=&endTime=) and discover the substitutions ? Will it negatevely impact PI performance ? What do you think ?
Sure, you could write an ad-hoc tool to periodically retrieve data and check for values with the substituted flag set as Thyagarajan Ramachandran mentioned previously. All this would tell you though is that certain values have been overwritten, but nothing more. As to whether such a tool impacts on system performance depends on how much data you retrieve each time, and the frequency at which you run the tool to perform the data retrieval.
Anton M John Messinger
I 100% agree with John that fine-tuning your audit settings sounds like the best solution.
Just wanted to chime in here regarding John's earlier comment that the audit log must be read with the PI Audit Viewer application. It actually is possible to programmatically access PI Audit records using the AF SDK's AFAuditTrail Class, although in general the PI Audit Viewer application is easier for most purposes.
EDIT: Per Kenji's comment below, the AFAuditTrail class is for AF Audit, not PI Audit
I think AFAuditTrail class is used for AF Audit. In this case, PI Audit records need to be accessed which is different from AF Audit.
Okay, thank you all for your help.
Retrieving data ...