Skip navigation
All Places > PI Developers Club > Blog > Author: Ahmad Fattahi
1 2 3 4 Previous Next

PI Developers Club

57 Posts authored by: Ahmad Fattahi Employee

I am pleased and excited to let you know that we will soon roll out two significant enhancements and a series of minor improvements to PI Developers Club:

  • If you are not a member yet but are interested in trying out PI DevClub member features you will be able to start a trial subscription right form the website. The goal is to let any interested person a chance to try and see how much value PI DevClub membership offers without having to pay upfront. The trial period is designed to last 45 days; at that point the subscriber will have the choice to renew and become a regular member. To be eligible two conditions need to be met: i) the individual cannot have been a member for the past two years; ii) the individual cannot have regular vouchers available to her/him to use and become a regular subscriber.
  • If you are a member already you will be able to unconsume (release) your voucher as a self-service feature; today you need to email us to do so. We believe this move gives more control to our subscribers on how to use their subscriptions.
  • Across the board we will be improving the user experience when it comes to purchase, renewal, and managing subscriptions.


We are planning to roll out these features over the coming weeks. Stay tuned!

Data and community are two major assets that cannot be invented or disrupted as quickly as technology. At the OSIsoft Users Conference 2017 in San Francisco I had the opportunity to witness firsthand how the PI community is pushing the envelope for a smarter manufacturing world. In the process they have at their disposal decades of real data sitting across several thousands of PI Systems around the world. An ever-increasing number of organizations are making good on the promise of turning time-series process data into decision-ready and predictive knowledge. Below is a short summary of my observations.


Activities and opportunities


  • Hackathon: we offered real operational data from Barrick Gold, the biggest gold producer in the world. The dataset spanned 6 months of sensor measurements from their massive haul trucks. 60 data streams from 30+ trucks made this a rich dataset. The trucks are big parts of Barrick’s operational cost. Each of these giants cost $4MM while one tire costs $50K. Their miles per gallon (mpg) is 0.3. That gives you an idea how costly this operation is and how efficiency can be vital to any mine operator like Barrick. When a truck is full the value of gold waiting to be extracted is $60K. It means if a truck goes down without notice a significant amount of capital can sit in the middle of nowhere for days before fix arrives.

Our hackers took advantage of the opportunity and built very innovative ideas. As a judge I was struck by the level of quality and maturity of the submissions in 23 hours. Several teams focused on advanced analytics and data as the cornerstone of their submission while others focused on software development. Most notably, the winning team merged machine learning with social engineering to design a system where drivers would earn points by driving “well”. And “well” was learned through sifting through sensor data, the strains and temperatures across the truck body, as well as geospatial qualities of the road.


  • Partners and customers: several OSIsoft partners and customers flexed their muscles around data science with PI data. We enjoyed several hands-on labs and presentations on the topic ranging from real customer stories to educational pieces on how to pull off a successful machine learning project with PI data. Most notably “United Utilities” (UK) presented how they built a demand forecasting engine which is critical to serving water to their customers efficiently. “Total” showcased how they use data form PI System to build and deploy a model and predict the percentage of gasoil in the residues of their distillation tower. Many conversations I had with attendees all point to the acceleration of more advanced analytics and data science in the PI world. This is all exciting because it shows how much business opportunity is out there waiting to be tapped.




Like everything else in the world the nice benefits don’t come for free. There are still significant challenges along the way:


  • Quality of the data: throughout the event a constant theme was challenges around data quality. While it’s typical to immediately focus on the machine learning algorithm or architecture it is evident that industrial data can be messy, vague, or flat out nonsense. The nature of these data sources and their paths to server make them susceptible to sensor error, noise contamination, process errors, mistakes in units of measure, unlogged changes over time, and lack of context to name a few. A significant amount of effort has to be spent on vetting, reshaping, cleaning, and reformatting data before a machine learning algorithm can be applied. Building a diverse and broad team of data scientists and subject matter experts seem to be the right strategy to alleviate such pain points.
  • Cultural and governance issues: preserving and sharing data may not be as easy as the technology that enables it. The industrial community has a long history of protecting itself against all sorts of malicious attacks and innocent mistakes, hence isolating itself from the rest of the world. The new needs and opportunities call for easing up some of the traditional requirements while guaranteeing security. It takes a significant cultural shift on top of technological and security advancements. Besides, addressing the data quality issues mentioned above takes a change in the mindset across the organization from top to bottom. To top this off with yet another layer, in many cases data is comprised of elements sitting in different jurisdictions which makes data sharing and aggregation even more challenging.


The opportunities for data science and machine learning in the PI world abound as do the challenges. However, challenges are nothing that we can’t overcome with the power of our smart and energetic community. All the signs are pointing to a wave of industrial organizations investing serious capital and resources in this area. After all it may be the differentiator between the survivors and failures of the coming decade. I ask of anyone in this community to share their thoughts, experiences, challenges, and ideas in this field. We at OSIsoft are committed to push this forward with your help.

UC San Francisco 2017 will offer a new series of major activities and content for the builders: PI System developers, data scientists, and architects. With a hackathon and 1.5 days of packed activity there will be a lot to take home. To provide added focus all of these activities will happen at Parc55 hotel which is separate but just next to the main conference venue. Here is a high level overview of the activities:


  • Hackathon – Monday-Tuesday
    • Barrick Gold will be our data sponsor and the theme is IIoT. Our sponsor offers real and rich datasets coming from their massive haul trucks
  • Developer Track – Wednesday afternoon and all Thursday
    • 10 unique hands-on labs covering novice to expert personas on a multitude of PI System and data science concepts
    • 11 technical presentations by customer, partner, and OSIsoft presenters
    • Focus on data science with a lab and 3 presentations
    • Developer Lounge for PI Geeks to stop by product booths and geek it out
    • Vox Pop sessions for ad hoc roundtable discussions whose topics are decided by the voice of attendees
    • General session with a developer-centric roadmap talk
    • Award ceremony and Geek Night


I encourage all PI Geeks to take advantage of this opportunity.

It is that time of the year again! Before jumping into this year results I like to open with a short but very important point. We recognize that our community has matured quite a bit. As a result and to leverage our bigger and more diverse community we are considering adding new categories to our All-Star program next year. It is to recognize new comers as well as experts in certain technologies. If you have any ideas to share with us along these lines please share your thoughts.


We are very pleased to announce our 2016 edition of the PI Developers Club All-Stars! These individuals brought tremendous amount of value to the PI Geeks community by constant presence in discussion, creating content, and helping others. We would not be where we are without great contributions from these individuals.


PI Developers Club Community All-Stars 2016

  1. Asle Frantzen, Amitec AS
  2. Rhys Kirk, TGG
  3. Roger Palmen, CGI


They win:


  1. Recognition in the community
  2. One year free subscription or renewal to PI Developers Club
  3. One time free registration at UC/TechCon (UC EMEA 2016 or UC San Francisco 2017 - not including labs)
  4. Q Grant Fossil Smart Watch


PI Developers Club OSIsoft All-Stars 2016

  1. Dan Fishman, OSIsoft
  2. Stephen Kwan, OSIsoft
  3. Mike Zboray, OSIsoft


They win:


  1. Recognition in the community
  2. Q Grant Fossil Smart Watch


Please join me in congratulating these All-Stars for 2016! Are you a 2017 All-Star?

OSIsoft is now officially present on GitHub! This move facilitates version control, reuse, and collaboration for the whole PI community on their projects. We have made the choice of using GitHub to provide a significant complementary piece to PI Developers Club and PI Square. We highly encourage the whole PI community to not only take advantage of the projects, but also share and collaborate with others. This document walks you through the steps you need to take in order to take advantage of this space.


OSIsoft has opened up PI Developer Technologies to the public for development purposes. Now everybody with a PI System can create open source projects and share them on GitHub; that's what we have started doing in our space; we invite you to do the same and take our community to the next level.

It's that time of the year again! With pleasure we are announcing our PI Developers Club All-Stars for year 2015. These are the individuals with exceptional skills and passion to share knowledge, help others, and elevate the community as a whole. The factors we took into consideration are technical knowledge, quality of the posts, frequency of the posts, and peer feedback. Similar to previous years we have 3 Community All-Stars along with 3 OSIsoft All-Stars. Unlike OSIsoft All-Stars, the Community All-Stars are members who are not OSIsoft employees. Community All-Stars 2015 are:



Roger Palmen.pngRoger Palmen, CGI
Karl Hodgkiss.pngKarl Hodgkiss, DAI
Rick Davin.pngRick Davin, Nalco Champion


They win:

  1. One time free subscription to UC/TechCon
  2. One year free subscription to PI Developers Club
  3. Recognition in the community
  4. A Microsoft Band


OSIsoft All-Stars 2015 are:


PI Geek.pngMike Zboray, OSIsoft
David Hearn.pngDavid Hearn, OSIsoft
PI Geek.pngDan Fishman, OSIsoft


They win:

  1. Recognition in the community
  2. A Microsoft Band


Congratulations to our PI Developers Club All-Stars 2015. With all the rising stars in the community there will be a fierce competition to defend the title for 2016!

2014: We had a very exciting year 2014. We moved from our home of 6 years (vCampus) to our shiny new platform PI Developers Club in December. In the meantime we integrated with the bigger PI community, beefed up our development support crew, and opened up content to everybody in the community. We are confident these changes will help us grow the developer community even further and also integrate better with the broader PI System users community in PI Square. In 2014 we handled more than 900 new discussion threads. Looking forward to reaching for higher targets in 2015.



2015: Enabling development of applications for the PI System is a high priority for OSIsoft. We will continue the trend that was started last year and add more valuable pieces to this ecosystem.


  • PI Developers Club: we will continue nurturing and improving the features of our program, PI Developers Club. Customers on active SRP can enjoy specialized support on phone and email for their development-related questions. Share your other improvement ideas with me.
  • App Exchange: we will start looking into a more formal way to exchange applications in the community. There is a tremendous amount of energy and skill in PI Developers Club. We want to make it as easy as possible to build and share useful applications with the community. In 2015 we will talk to people from the technical and business sides and investigate best practices. Please share your thoughts on this topic with me! Tell me what would make it easier or more profitable for you to build applications for PI System. We love to know how we can help you do well building and distributing such applications!
  • Development PI System in the cloud: Cloud is not only a cost saver or matter of convenience; it fosters growth. We have already started investigating ways to offer PI Developers Club software package (aka. your development PI System) in the cloud. The idea is to offer a new way for developers to spawn a full development PI System in a matter of minutes. Expect to hear more on this issue in 2015 from us. Note that this will be in addition to the on-premise version of the development PI System.
  • UC TechCon 2015: in Users Conference 2015 we will start the technical conference in its new format (formerly known as vCampus Live!). We call it UC TechCon. We will have several rich technical hands-on sessions for developers and system admins to learn and connect with their peers. The good old Programming and Security hackathons are back; the theme of the Programming hackathon this year is Internet of Things (IoT). You can already register for the event.


I am very excited about the year ahead of us! 2015 will be a great year for the PI Community and, in particular, PI Geeks. Now that the new platform is in place it's time to innovate and build more apps together!

PI Developers Club (PI DevClub) is almost here! OSIsoft vCampus will soon start its transition into the new and improved PI Developers Club. The vCampus program has been very successful in enabling you to develop more and better applications for the PI System. That’s why we are doubling down on the program to provide you with an even better service. To that end we are improving paid and public benefits of our developer program. We are also adding phone and email development support for our SRP customers. Let’s walk through these changes:


Q: What do I need to do? Nothing! Regardless of you being a member or not we will take care of all of these changes. If you are a member you will see the newly added benefits automatically. If you are not a member we encourage you to take advantage of the public content and engage with the developer community. We encourage you to consider becoming a member to enjoy all of the exclusive member benefits.


Q: When does it happen? The first phase, including several feature improvements, will be delivered in December 2014. Some transactional aspects, such as invoices and licensing, will come in future phases.






Q: Is the price changing? No! Pricing remains the same as vCampus.


Q: What is included in the paid program? On top of the existing member-only features we are adding three new exclusive benefits for members:

  1. Enjoy Better Support: though the forums will be open to all, we will make sure that questions by paid members enjoy priority support by OSIsoft staff.
  2. Get Discounts: we offer member discounts for those attending yearly technical events or taking online workshops covering programming topics.
  3. Get Recognized: members should be recognized in the community for their investment. We have devised features in the community for PI DevClub members to be clearly identified.







Q: What benefits will become publicly available? Most of our existing vCampus content will open up to the public. Such content includes discussion forums, blogs, webinars, and most of the technical papers. Our goal is to lower the barrier for PI users to participate in development-themed discussions and to foster more overall engagement for members and non-members alike.





Q: What is the relationship between PI DevClub and OSIsoft Community, PI Square? We are very excited to roll out these major improvements together with the bigger OSIsoft community, PI Square. PI Square is the open community for all PI professionals to get together, exchange ideas, and get help and support on their questions. The social features of PI DevClub will be delivered in the context of PI Square. At the same time, we are excited to bring together our partners, developers, customers, and OSIsoft employees all in one community. Please do not hesitate to contact us if you have any questions or comments about these new changes.




We are excited to be one step closer to a more open development environment on PI System! At UC San Francisco we provided more details about future development community and PI System Access technologies. While we are working to iron out final details, these are some highlights that you may be interested to know:

  • We will be opening up licensing and documentation to many of our development technologies soon. This means you will no longer be required to be a paid member of our development program to have the right to develop on these "Open" technologies.
  • We will define different stages of the lifecycle and sensitivity of our extensibility technologies clearly. Everybody will be clearer about what to use and what not to use for their future development projects.
  • Along the same lines of "more hands on the PI System" we are planning to elevate both paid and non-paid levels of our developers community. While we will open up most of the "content" that is currently locked in vCampus (such as forums and blogs) we preserve or add some key benefits to the paid program. Among the exclusive paid benefits are software downloads, Virtual Learning Environment, exclusive previews and betas, premium support, recognition in the community, and member discounts to our technical events and development workshops.
  • To further bring all PI System professionals together we are combining the Users Conference and developer events (TechCon) together starting in 2015. This will help us grow the developers community and facilitate the travel for attendees who like to attend both events.

Stay tuned as we roll out these exciting changes over the coming months! For a more detailed narrative of what future holds for us see our presentation at the Users Conference in San Francisco last week.



OSIsoft vCampus Live! 2013 was the 5th and the biggest of our technical events by a huge margin. We hosted 300 guests from all over the world in 24 hands-on and 8 lecture sessions. We had the opportunity to make some important announcements about the future direction of the community. PI Geeks provided us with lots of valuable feedback, comments, and ideas. Below you can find the summary of the event.

  • Key Announcement: the vCampus program is expanding to reach more developers and provide better service all around the world. This goes hand-in-hand with the opening of our recommended data access technologies such as AF SDK, RESTful API, PI OLEDB Enterprise, among others. When it comes to the vCampus program we are planning to open up the discussion forums, document library, webinars, and blogs to the world to reach more smart developers. At the same time, the paid program will only get better. On top of the free components we are considering the following offerings for the paid program: a development PI System, access to the Virtual Learning Environment, exclusive access to early adopters program, discounts to our development events and training workshops, guaranteed response time in forums, and a limited number of escalations of key issues to our support team.
  • Key Announcement: Building on top of the vCampus community success and in order to serve our customers better, OSIsoft will offer an online community for the customers and end users of the PI System. Our Technical Support team will have a strong presence in that community to handle technical discussions and questions. It will also provide a unique opportunity for OSIsoft partners to see what customers need and get engaged in discussions to promote their skills and capabilities.
  • Key Announcement: Starting in 2015, we will combine vCampus Live! and Users Conference events in order to allow more PI System professionals take advantage of both events. While the venue and timing will be combined, the character of vCampus Live! as the main technical event for PI Geeks will be preserved. In the coming months we will be officially announcing these developments.


  • Keynotes: on Day 1 we saw our engineering roadmap presentation by Ray Hall and John Baier, VP and Director of Engineering, respectively. They covered the upcoming products and the widening reach of the PI System in the connected world of tomorrow. Among all exciting developments cloud-based products and services formed an outstanding part of the presentation. Ray and John were followed by OSIsoft VP of Customer Support and Services, Steve Nye. He made the announcement on upcoming OSIsoft Community and changes to the vCampus program as summarized above. On Day 2, we had a fantastic visionary talk by Jared Spool on "The unintuitive nature of creating intuitive designs". I followed him to gave a quick overview of the "State of the Community" and announced Community All-Stars (Lonnie Bowling, Rick Davin, Rhys Kirk, Roger Palmen) and OSIsoft All-Stars (Steve Kwan, Chris Manhard, Mike Zboray). Congratulations to all of our super All-Stars for year 2013!
  • Hackathons: similar to last year we held both programming and security hackathons. Teams worked tirelessly through the night to create their apps in the programming hackathon and came up with fabulous creations. The winners of the programming hackathon were Dong Energy, Plenty of Sun for the REST of the week, and The Sunny PI Guys holding 1st to 3rd places, respectively. Security Hackathon awards went to Yes! Team, Lucky 8, and WTF Security teams. Congratulations to all!


Note: We apologize in advance for the audio and video quality.


  • Hands on and lecture sessions: through 24 hands on and 8 lecture sessions we covered many of our existing and future development technologies including AF SDK, SharePoint, future data, RESTful API, Business Intelligence, client tools, and lots of best practices. Every year we shape the sessions in a way that conveys more value to you; please let us know if you have any suggestions or comments for us.
  • Networking lounge: this was a great opportunity for developers and partners as well as Enterprise customers to get together and mingle or just relax. There were numerous ad hoc and follow up meetings as well as great snacks in the lounge .
  • Vox pops: for those who are not familiar with them, these are the roundtable ad hoc sessions where the community decides the topics of discussion. This year we had sessions on Big Data analytics, future data, and Esri integration. It was great to see the high interest level of PI Geeks in such topics. These are great learning opportunities for OSIsoft Engineering and Product Management teams.
  • Future: we are getting better and bigger continuously! Our biggest vCampus Live! to date was the proof. Our goal is to bring the whole PI Community, customers, partners, and developers in one place. To help this community flourish we will see some very important developments in 2014. Establishing the OSIsoft Community, opening up the recommended development technologies, improving the vCampus program, combining vCampus Live! with the Users Conference, and a much improved partner program are among the improvements we have been working on. Stay tuned as we will be announcing these exciting pieces in near future!

Feedback: We are eager to hear your feedback and comments. If you you like to share anything with us about the upcoming changes or the event itself please don't hesitate to leave a comment here, contact the vCampus team, or contact me privately.

We are up to a very nice and energetic start to vCampus Live! 2012. The two Hackathons, Security and Programming, are already in full action with folks jamming in the rooms. While teams in the Security Hackathon are sweating over their ideas to use car data in PI System, the folks in the Security Hackathon are defending their PI Systems against OSIsoft attackers (aka. bad guys). We are going to conclude tonight with a reception. The great event will officially open Thursday morning California time with fantastic keynotes and lots of hands-on and expert presentations.


Security Hackathon folks defending their turf against virtual attackers while folks in the back taking pictures of the breathtaking views of San Francisco from 36th floor windows:



Following my previous blog post here, I am focusing on the single variable statistics for a PI tag using R. The goal here is to dig deeper into the data we have collected in the PI System and enable the Power of Data. As we will see, some very small code snippets can generate huge analytical and visual value out of data. All of this comes at no monetary price as long as you have your PI System in place; R is a free tool!


We base our operations on the same dataset as in the previous example, being the power and temperature data gathered from OSIsoft headquarters building in San Leandro, CA. We assume that the sampled data is already exported to a CSV file using PI DataLink or other methods such as PIConfig. The data is sampled one per day going back for a full year.


First we read the data,  from the CSV file into a variable called PowerTemp.df. The type of the data is called data frame in R. A data frame is pretty much like a table having rows and columns. The values in each column are usually of the same type representing a variable, such as temperature or time. Each row will be an entry or observation. Note that the whole structure is easily read from the CSV file. The names of the variables are automatically set from the headers in the CSV file:

#Read the data from the CSV file
PowerTemp.df <- read.csv(file='C:\\Users\\afattahi\\Documents\\R\\Examples\\SL - Power - Temp - 1year - Cleaned.csv', header=TRUE)

#Converting the Power and Temperature to numerical vectors
power.numeric <- as.double(as.vector(PowerTemp.df$Power))
temperature.numeric <- as.double(as.vector(PowerTemp.df$Temperature))

Next step will be to look into the individual distribution of each variable, temperature and power consumption. The histogram will give us this information. This should give us more insight into how the variables have been behaving over the desired period.

#Plotting the simple histogram of the power with 20 bins
hist(power.numeric, breaks=20, col='blue')



It clearly shows the behavior we witnessed before: there is two types of behavior or distribution; one related to weekends (base power) and the other one to the working days. Now let's fit a density function to the histogram above using function density() for a smoother description of our data:

#Calculating the density function: we get an error due to NA in data. We need to clean it out.

d <- density(power.numeric)

Error in density.default(power.numeric) : 'x' contains missing values

Oops! We get an error. The problem is that our dataset contains some NA values (Not Available). The density() function cannot handle that. This is a classic example of the need for data cleaning. There is a saying that 80% of a data scientist or engineer's time is spent on cleaning data and 20% on algorithms and generating insight! So, let's take the NA out. R can do this very efficiently. In the snippet below we clean out the data, calculate the density and plot the density function. The density object, d, contains all the statistical description of the graph:

#Cleaning the data
power.numeric.clean <- power.numeric[!]

#Create the density and plot it
d <- density(power.numeric.clean)
polygon(d, col="red", border="blue")



Now it is evident that the behavior to the right (right lobe - weekdays - 5 days a week) is the dominant behavior as opposed to the left one (left peak - base power - weekdays - 2 days a week) - Beautiful!


To put the icing on the cake, let's look at the distribution of the power in different seasons and compare them. We define seasons as: Jan-Mar as winter, Apr-Jun as spring, Jul-Sep as summer, and Oct-Dec as Fall. The first step is to extract the vector of months (1-12) from the timestamps in our dataset and clean it:

#Extract the vector of months and clean it
months.vector <- as.numeric(format(as.Date(PowerTemp.df$Time, format="%d-%b-%Y"), "%m"))
months.vector.clean <- months.vector[!]

Here is a very important step. We need to bin this vector according to seasons. In other words, take the vector of months and attach the corresponding season to each entry in the vector based on our definition of seasons. We use the function cut() to do so. It generates a factor which is another data structure in R. A factor is an ordered vector of objects; every realized value in the whole list is called a level. Factors are very good to represent observations of categorical values, in this case seasons.

#Create the factor of seasons
seasons <- cut(months.vector.clean, breaks=c(0,3,6,9,12), labels=c("Winter", "Spring", "Summer", "Fall"))

Now we are ready to compare the distributions of the values of power per season. To do so we use the function That's why we load the package sm first. The beauty of it is that once we know what we are doing everything is done with very few lines of code and becomes intuitive.

#Compare the distribution of power consumption by season
require(sm), seasons, xlab="Power consumption by season")
legend("topright", levels(seasons), fill=2+0:(length(levels(seasons))-1), legend=c("Winter", "Spring", "Summer", "Autumn"))



It shows that the dual behavior is again observed in each individual season; so this is an intrinsic behavior of the underlying process. The only curious point is that in spring the baseline power is dominant. It can be because of the moderate weather in California in spring time where there is very little energy used to cool or heat the building. To see the different behavior by season we can look at the box plots of the power consumption by season:

bwplot(power.numeric.clean~seasons, ylab="Power Consumption")



The intent of this post is to delve deeper into single variable statistical analysis of the data. To do so we need to import data from PI System into R, clean it up, and use appropriate analysis and graphics. R proves to be very efficient in enabling the Power of Data!

Ahmad Fattahi


Posted by Ahmad Fattahi Employee May 10, 2012

As geeks we all love puzzles! How about attacking a cool one using R and solving it in an elegant way? (Even if you don't know R you will probably enjoy it! I found the problem and solution here; I just tweaked it a little bit.) The syntax and code efficiency is really impressive. Here is the description:


We are given integers between 1 and n. We would like to order them in a line, when possible, so that every adjacent pair adds up to a square number. For example, for n=15 a solution would be:


9, 7, 2, 14, 11, 5, 4, 12, 13, 3, 6, 10, 15, 1, 8


There are multiple ways to model the solution; an elegant way is to model this as a graph. Each number corresponds to a vertex. If two integers add up to a square number there will be an edge between the corresponding vertices. We have a solution if there is a Hamiltonian path in the graph (a path that traverses all the vertices once and only once).


Now let's implement this in R. The comments in the code should explain each line.

#We consider numbers between 1 and n
n <- 15

#Creating the graph adjacency matrix and giving rows and columns names
d <- outer(1:n, 1:n, function(u,v) (round(sqrt(u+v), digits=6)==sqrt(u+v)))
rownames(d) <- colnames(d) <- 1:n

#Defining the graph object based on the adjacency matrix
g <- graph.adjacency(d, "undirected")

#Labeling each vertex with the actual integer it represents in the graph
V(g)$label <- V(g)$name

#Plotting the graph with a readable layout
plot(g, layout=layout.fruchterman.reingold)

 And here is the result for n=15, n=10 (no solution), and n=20 (no solution).







"The Power of Data" - sounds familiar? It was the tag line for OSIsoft Users Conference 2012 in San Francisco, and not for no reason. After the awesome PI Systems collect data from different parts of an enterprise we need to extract useful information out of raw data. This is especially true given how easily the new and mighty PI System can handle millions of tags (data streams) and events in a blink of an eye. To really get to "the Power of Data" we have no choice but to use more advanced analytics to massage the raw data and seek insight out of this huge volume that comes to us very fast.


To this end, I would like to share some of the current efforts we are taking here at vCampus with the help of some of my OSIsoft colleagues as well as third-party partners to enable more advanced analytics on PI Data. I would also love to hear your comments, feedback, and ideas along the way:


1) PI System and MATLAB: we have been working on such integration for about two years now. Not only we have a white paper in the Library describing the integration, we have also presented some machine learning applications during vCampus Live! 2011 and other places. MATLAB is a very powerful tool to do general and specialized analytics across several disciplines including machine learning, statistics in general, mathematical optimization, signal processing, and control systems among others. It has good penetration in research communities and academia as well as some industries. We have been working with Mathworks over the past year and will continue the joint venture.


2) PI System and R: R is the emerging and de facto language of Big Data Analytics. A lot of organizations are actively using R for development, in production, or are adopting the technology. The open source nature of the platform makes its use ubiquitous for many different applications. The language is specifically designed for handling data; therefore, it is extremely powerful in making tables, joins, selecting, and performing statistics on the data with a very efficient and smart syntax. Another huge advantage of R is its powerful graphics which make data much more readable and improves interpretability. We have been working with Revolution Analytics, who commercialize R in their product Revolution R Enterprise; it offers a much better development environment (like an IDE) and also offers supported packages for parallel computing. The following infograph shows the trend in terms of the number of books sold on different programming languages. The light green box on the bottom is about R with 127% annual increase and on par with some other major languages.


In my previous blog post I showed a way to use PI Data in R. I will follow up with other integration methods and more cool applications and graphics in the weeks to come.




3) PI System and Python: Python is a programming language that lets you work more quickly and integrate your systems more effectively. It is free to use (even for commercial purposes) and runs on multiple platforms. We have recently started looking into using PI Data with Python. A group of our enthusiastic engineers at OSIsoft have some good experience in doing so. We hope we can offer more concrete documents and results in the months to come.


Do  you use any of such analytical tools with your PI Data? Would you be interested to do so? Do you face any particular challenges? What would be the most valuable result out of a successful analytics package running on your PI Data?

Following my previous posts (here and here) on integrating PI System with R, this post takes the story one step further by analyzing the data to find correlations between them. By doing this we make two major points; first, R is extremely powerful and efficient in analyzing large amounts of data and manipulating it statistically. You see from the code snippets that we get all we do here with about 10 lines of code! Second, we see the power of good graphics to interpret data. In fact, the same data set can reveal many different things if it is shown in different ways. Appropriate graphics make data talk.


We work with some real-world data here. In our San Leandro office, we collect outside temperature as well as the instantaneous power consumption level of the building in a PI Server. I exported the data (sampled at around 4pm once a day) over the past 1 year to a CSV file (using PI DataLink or the piconfig utility). Therefore, we have a CSV file with 3 columns: Timestamp, Power, Temperature. This file is attached for your reference. Note that there may be some data cleaning necessary before feeding it into the graphs.




The next step is to read the data into a "dataframe" object in R. You can think of a dataframe as a super table which R can do lots of things with it.

#Read the data from the CSV file
PowerTemp.df <- read.csv(file='C:\\Users\\afattahi\\Documents\\R\\Examples\\SL - Power - Temp - 1year - Cleaned.csv', header=TRUE)

 The "\" character happens to be the escape character. First step is to convert the power and temperature to numeric values (they are read as "factors" in R):

#Converting the Power and Temperature to numerical vectors
power.numeric <- as.double(as.vector(PowerTemp.df$Power))
temperature.numeric <- as.double(as.vector(PowerTemp.df$Temperature))

 Now let's plot power vs. time:

#Plot the power and temperature vs time as is
plot(PowerTemp.df$Time, power.numeric, xlab="Date", ylab="Power") 



As we see the date objects are just imported as strings and there is no real date order and the x-axis looks really bad. To fix this, we convert the date into "Date" objects in R and redo the plot for temperature and power:

#Plot the power and temperature vs time as Date objects
plot(as.Date(PowerTemp.df$Time, format="%d-%b-%Y"), power.numeric, xlab="Date", ylab="Power")
plot(as.Date(PowerTemp.df$Time, format="%d-%b-%Y"), temperature.numeric, xlab="Date", ylab="Temperature")



As is intuitively expected as well, there seems to be some sort of correlation between the temperature and power consumption at the building. The higher the temperature the higher the power consumption due to AC at the building. To see this better lets plot power vs. temperature:

#Plot the correlation between temperature and power using plot and smoothScatter
plot(temperature.numeric, power.numeric, xlab="Outside temperature", ylab="Power consumption")



Now it is way more obvious that there is some strong correlation between the temperature and power demand. The only problem with this graph is that it can fail showing the density of points in crowded areas as dark points tend to overlap each other; two overlapping dark points look the same as 10 overlapping points. To fix this issue let's use smoothScatter:

smoothScatter(temperature.numeric, power.numeric, xlab="Outside temperature", ylab="Power consumption")



Now it looks beautiful! Not only the correlatoin is obvious, but also it shows some bifurcation in behavior; there are two obviously separate branches. In other words, it shows that there should be other parameter(s) that the relationship is conditioned on. The intuition of the underlying problem tells us that the weekends should be much lighter on the power because the AC set points are adjusted to save power. To test this let's add the "day of the week" to our dataframe and plot power vs. day of the week:



#Get the week day out and plot power grouped by week day
PowerTemp.df <- transform(PowerTemp.df, Weekday=weekdays(as.Date(Time,format="%d-%b-%y"), abbreviate=TRUE))
qplot(PowerTemp.df$Weekday, power.numeric,  position=position_jitter(w=0, h=1), xlab="Day of the week", ylab="Power")



Now it clearly shows that Saturdays and Sundays behave differently than the workdays. These two days correspond to the lower branch in the correlation graphs above.


As we saw above, R provides mighty tools to analyze large amounts of data and produce actionable graphics out of them. Together with PI System they make a power house in turning data into action. There is a reason we spend so much resources to collect many many pieces of data.





Filter Blog

By date: By tag: