25 Replies Latest reply on Dec 7, 2017 4:57 PM by skwan

# Linear Regression in AF

Looking for a means to determine the rate of change for a value over time. Was looking at performing a linear regression for a set time period and comparing that with the slope for the time period directly before and after it.

Any ability to use an Analysis template to perform the simple linear regression calculation?

• ###### Re: Linear Regression in AF

There is not a built in function for linear regression.  However, functions are available for you to apply numerical methods to calculate it.

• ###### Re: Linear Regression in AF

As far as i understand linear regressions, you must consider all values over time. Within AF, you can't really pull all values over a given time period into a formula and do some calculations over that.

Best option i can think of is a custom DataReference.

• ###### Re: Linear Regression in AF

I had to calculate linear regression in the project I am currently working on, and also found that it wasn't easily done in the AF analytics. I ended up calculating it in some custom AF-SDK code in PI-ACE using the MathNet.Numerics library (numerics.mathdotnet.com). Doesn't immediately address the issue of templating this, but longer term this is something I would like to do too. A custom calculation Data Reference is one option as Roger suggested, but won't suit my particular project requirement.

• ###### Re: Linear Regression in AF

John: Can you expand on the business use case? What kind of data is it? And, what do you do after the linear fit?  Is the time-range for the data, say last 24 hours etc.?

• ###### Re: Linear Regression in AF

Gopal,

The use case is to calculate the rate of gas leakage from SF6 insulated circuit breakers and transformers. We're running a linear regression on the previous 12 months of gas density readings to determine the leakage rate using the slope, and using the y-intercept to then predict the date at which the gas density will reach a pre-defined critical level. There is added complexity in the fact that I am required to apply a number of rules to filter the input data prior to running the regression calculation. The 'daily' readings we use are actually an average of the values between 10pm and 4am each day for the 12 month calculation period.

• ###### Re: Linear Regression in AF

What do you do with the predicted date? This might be just a report, e.g. through PI DataLink, Excel services etcetera that delivers the information when needed. So i thing the optimal solution in your case will be mainly dependant on the business process it is used in.

There is litlle that can beat the flexibility and ease of use of PI DataLink and Excel. So if that is applicable in your case, that would be my recommendation.

• ###### Re: Linear Regression in AF

The predicted date is used by the business to determine and plan maintenance on the equipment, specifically topping up the insulating gas. The results are reported based on these dates, with the soonest date predictions at the top. There are other drivers as well, such as OH&S for staff entering the substations  and potential reporting to environmental regulatory agencies.

Excel and DataLink aren't really options here, as we are performing these calculations on a monthly scheduled basis for some 4000+ elements in the asset hierarchy. Automating the calculations in a server based environment one of the business requirements - they are actually wanting to move away from an existing Excel based system.

Using PI-ACE with the AFSDK seems to be a good option for us in this instance, as it gives us the ability to collect and filter the input values according to the business rules as well as perform the actual calculations. It also is dynamic enough to automatically handle changes in the asset hierarchy as equipment is commissioned or decommissioned.

The key point of this was that it isn't that straight forward to calculate linear regressions using the built in AF functionality, either through the DR's that come out of the box, or with Asset based Analytics. While it would ultimately be desirable to be able to do this through templating and avoid writing custom code, at the moment it doesn't seem possible as you said yourself in response to the original post.

• ###### Re: Linear Regression in AF

Thanks for explaining the circumstances. Clarifies a lot on the requirements that drive the solution in a certain direction.

I'd personally still go for a custom DR, as the effort required for building an ACE module will be similar as for a custom DR. And the DR will be much more future proof. If you start on ACE now, you will likely need a strategic solution for the long term anyway.

And AF is very fast. I used to do online cost caclulations with a daily granulatity over one year for a few hundred assets. That query took just 20 seconds on OLEDB Enterprise, providing a full report on all assets. So using a well-built DR, online calculations might be suffucient, but be very, VERY careful in the access paths to highly aggregated online calculated values as this can explode quickly if you're not careful.

PS: I'll put a linear regression DR on my list of DR's i'd like to have...

PPS: Integration with R might be a nice one. See this post: vcampus.osisoft.com/.../predictive-analytics-using-pi-and-r.aspx

1 of 1 people found this helpful
• ###### Re: Linear Regression in AF

I have written a custom DR for calculating slopes of regression lines (with user-configurable duration), to determine slow leaks in SF6 switching gear. Works like a charm!

I've also written - and made public here on vCampus through my blog - a RegressionLine add-in for ProcessBook. This one includes the y-intercept as far as I can remember.

The code is similar, and although I don't currently output the y-intercept in my custom DR it shouldn't be much work to make it do that as well. We don't currently market this as a product, but if you are interested we can probably work something out (either ready-to-use dll, or complete code). If you are interested contact Amitec through our website www.amitec.no

1 of 1 people found this helpful
• ###### Re: Linear Regression in AF

Thought about a custom DR, but unfortunately we need multiple outputs from this particular calculation, not just a single output such as the slope. One of the outputs (the leak rate) needs to be output to an attribute configured with a PI Point DR, as the business wants to be able to historize and trend the calculated values over a period of time. From memory, I don't believe it's possible to meet either of these requirements with a custom calculation DR, and I don't want to write Frankencode to try and achieve this. If I somehow missed the bit where that has changed, I'd love to know, as I agree that something like a custom DR would be better in the long term than PI-ACE.

• ###### Re: Linear Regression in AF

I recognize this issue. First options i see are using ACE, or any other calculation outside of PI. E.g. i used a calculation in SQLserver before, reading back the results using the AF table DR.

ACE gives the option to perform the data and compute intensive calculations once.

But looking closely at Abacus, it does allow you to map the output of an analysis to multiple output attributes. Have not covered much ground yet with abacus , but one option might be to have the DR return the results in an array, and then split the array out to multiple output attributes. Hmm, should give that a try...once i get my hands on an up-to-date AF server (surrounded by old stuff at the moment).

• ###### Re: Linear Regression in AF

Hmmm...attributes with array data types. I'm not sure that Asset based analytics can deal with arrays (either reading from, or writing to) in Attributes. Haven't personally tried that avenue yet, but so far I haven't seen anything that indicates they can be used in an Analysis. Unfortunately I haven't had too much spare time to experiment with this either, so I can't say anything definite here. I'd be interested to hear about your results with this. I might just post my own results too if I get some time to look at this option in the next week.

• ###### Re: Linear Regression in AF

Did you see Xi Wang's blog post about creating your own custom dr? The example he presents allows you to pick individual array parts and present them on their own as single attributes. Combine it with your own custom dr returning an array and you should be ready to go

https://vcampus.osisoft.com/bloggers_place/b/weblog1/archive/2014/09/01/create-you-own-data-reference.aspx

• ###### Re: Linear Regression in AF

Thanks, I hadn't seen that post before - it's an interesting concept.Unfortunately I won't have time to try this out on the current stage of this project, but I will definitely explore the possibilities it presents as a possible "upgrade" for the project...

• ###### Re: Linear Regression in AF

... wrong account...

• ###### Re: Linear Regression in AF

... correct account...

Asle, thanks for chiming in!

I did some looking around. Still in the realm of 'it is possible, but it ain't very pretty'.

The key issue is that i cannot convince abacus to write data to an array. it throws an error: "Attribute xxxx is incompatible: The data method 'UpdateValue' is not supported on attribute 'xxxx'"

I have a simple Int16 array as input, an equation analysis, and an Int16 PiPoint array as output.

So we could revert back to a poor man's array: a CSV list of values, and split that using the Instr and Left functions. If you do the call to the DR in one expression to pull that into a CSV string attribute (and map this to a string PI Point if needed), you can then split the parts of the CSV and write those to individual PI Points of the datatype of your liking.

So then you just need one DR that returns the array of results in a CSV format.

So we can now expand our wish-list for AF:

- Proper syntax support to address individual values in an array (e.g. "Array[1]")

- Abacus support for writing to Arrays. Mapping single value outputs to single array members, or if the size matches, map the entire array across.

• ###### Re: Linear Regression in AF

LOL Roger

Please feel urged to not share SSO account information

Steve is traveling. I'll make him aware of your post.

• ###### Re: Linear Regression in AF

I need to calculate a regression slope for a project that I am working on.  I see that your Processbook add-in is public.  Is your DR available as well?

• ###### Re: Linear Regression in AF

Unfortunately not, it was written for a specific customer.

• ###### Re: Linear Regression in AF

It's coming in the 2017 R2 release natively :-)

--

Steve Kwan

1 of 1 people found this helpful
• ###### Re: Linear Regression in AF

Asle Frantzen Thank you for your reply.  It was worth a shot.  We have resources in the company that might be able to help.

Stephen Kwan Any idea when the 2017 R2 release is being released?  I might be able to put something together that will work until then.  We might be able to talk the customer into upgrading when it comes out.

• ###### Re: Linear Regression in AF

Evan,

It's currently planned for Q4 2017.  Depending on the feedback we get from the Community Technology Preview, it will most likely be December time frame.

--

Steve Kwan

• ###### Re: Linear Regression in AF

Any news on the release ?

--

Tue Berg

• ###### Re: Linear Regression in AF

We're trying to get it out the door per our estimate.  We're hunting down a couple more bugs reported during the User Acceptance Test.  Estimated release date is on our PI System Roadmap webpage.

--

Steve Kwan

2 of 2 people found this helpful
• ###### Re: Linear Regression in AF

Sweet!