Jerome Lefebvre

An AF/Vision experiment - Visualizing data loss

Blog Post created by Jerome Lefebvre on Oct 27, 2017

Hello, this is a simple experiment that I did in AF. It is probably not something that has great value, but mostly doing so for the fun of exploring what can be done.


Monitoring for data loss can be a hard thing to do. A gap in data could simply due to a period in time where data was particularly strongly compressed or there are natural gaps due to collecting certain data only during particular shifts and not continuously.

Thus, I was looking for ways to visualize when data was coming in and I took inspiration in this paper that looks at techniques to determine if certain tweets are generated by a bot or by a human.

To visualize when data was coming in, they created XY plots where the X and Y axis were determined as follows:



Recreating this in PI Vision (details to follow)  for a tag like cdt158 we see the following:

So, I can quickly pick out events that are unusual. That is, if I was worried about data loss, I would be looking at what happen during those time.



Other tags which are more affected by compression setting like Sinusoid have a more striking pattern.




■ How these graphs were created.

First, they use the XY plots which are a full featured symbols in PI Vision 2017 R2. To create the data of the graph I use the following AF setup:

I use the name of the element to grab the tag that I am curious about and then two attributes that measure the time before and after every event.


The PreviousEvent attribute is computed this way:

NextEvent is similar, but NextEvent will return bad values sometime (for example, if we are looking at the current value as there is no next event)


I'm still tweaking these expressions to see if I can better reveal patterns. But, let me know if you have any feedback on it.