Skip navigation
All Places > PI Developers Club > Blog > 2012 > February
2012

I attended the O'Reilly Strata 2012 today (Tuesday). It was the day for workshops; the conference will officially start on Wednesday morning and will last two days.

 

It has been great so far and much bigger than I expected. The motto is "Making data Work" which comes so close to heart for all of us here I attended two workshops today. The first one was on "R" (the open source version) and "Revolution R" (the commercial version) presented by Revolution Analytics. It was a great review of the power of R in general as well as what the commercial version, Revolution R, can offer in statistical analysis and machine learning.

 

The second workshop was titled "The two most important algorithms in predictive modeling today".  The speakers, Jeremy Howard and Mike Bowles, talked about two major machine learning approaches today: random forests and glmnet. It was a very exciting and interactive session with some hands-on as well as theoretical discussions (Mike Bowles is a former MIT professor).

 

Overall, so far it has been a very well-attended (sold out!) event. On the algorithmic side there is a lot of buzz around R. On the implementation side, you hear a lot of Hadoop and MapReduce. There are big company names around both as sponsors and presenters such as Microsoft, IBM, Netflix, EMC, LinkedIn, vmware, Oracle, Amazon, to name a few. Will post more updates soon!

 

2772.IMAG1213.jpg

OSIsoft provides several data access methods to PI System data. Most of us would be familiar with most of them in our day-to-day PI System development efforts:

  • PI Web Services
  • PI SDK
  • AFSDK
  • PI OLEDB
  • PI OLEDB Enterprise
  • AFSDK (RDA)

There is a good reason for having these multiple data access methods. Different circumstances prefer a different data access product. If you are developing your own high performance PI Interface, you might want to resort to PISDK for performance reasons. If you are developing a Silverlight app,  you probably want to resort to PI Web Services, because of the ease of access. If you are working on more of a BI product, and want to really cross examine your data, then PI OLEDB or PI OLEDB Enterprise could be your method of choice.

 

It's difficult to really state beforehand when you should use what product. It's really dependent on the environment, goal and architecture.

 

One thing that could be measured is the performance of the different data access products in your architecture and situation. The last couple of weeks I've been working on a concept where measuring the performance of the different data access methods should be a walk in the park. In this post, I want to elaborate a bit on this concept, and ask you for your opinion. 

 

Introducing PI System DataAccess Profiler

 

The goal of this concept is to create a 'one stop shop' for all performance measurements involving PI System Data Access technologies. But why stop there, why not also measure the performance of different programming paradigms like sequential, async, multi threading and parallel programming. 

 

Some example use cases I have in mind:

  • You want to get insight into the performance of a data access method in your (network) environment
  • You want to compare the performance of PI SDK vs PI Web Services for data retrieval in order to make an informed architecture decision.
  • You want to know the performance using PI OLEDB to all the PI Servers in your Enterprise.
  • You are creating a new .NET product, and you are wondering if multithreading or parallel programming would allow the biggest data throughput to your PI Server using PI SDK
  • ...

Overview

 

The main idea of the application is to generate detailed performance reports, according to your configuration. You create a 'Execution Plan', which holds all the information.

 

An execution plan holds one or more 'Connections' and one or more 'Execution Groups'. A Connection is a dedicated connection using a certain Data Access product. For instance, you can configure one or multiple PISDK or PI Web Services connections. The application should support PI Web Servces, PI SDK, AF SDK, PI OLEDB and AF SDK RDA. An Execution Group is dedicated to a certain Connection, and consists of Operations. The application supports the following Operations (for each connection type):

  • Get Product/Server Version
  • Get Snapshot
  • Get Archive Values
  • Get Summaries

These are the basic operations that all Data Access Products support, and therefore we can cross measure performance. 

 

The application looks like this when you open it. It presents you with an empty execution plan on the 'Execution Plan Explorer' on the left pane. The middle pane is where the report will show up, the right pane is the 'object explorer', where you can make configuration changes. The lower pane is a log pane, where any information and error messages appear. The toolbar on top let's you

  • Create a new Execution Plan
  • Open an existing Execution Plan
  • Save the current Execution Plan
  • Save the current Execution Plan As (specify filename)
  • Run the current execution plan and generate a report
  • Save the generated report to HTML
  • Get information about PI System Data Access Profiler

2352.screen1.png

 

 

 

Creating a Performance Report

 

If we want to create an execution plan, we will first start off by adding one or multiple connections

 

0184.screen2.png

 

After that, we can configure the connection (using the object explorer). You can rename a connection to make it more descriptive.

 

6560.screen3.png

 

After that, we can add our first execution group, using one of the previously defined connections.

 

4807.screen4.png

 

Once we have our execution group, we can configure it. We can rename it (to make it more descriptive, for instance 'PISDK to PISRV101 Parallel'). We can also specify 'iterations'. This number indicates how many times the entire group is iterated for the performance test. The default is 1, but if you really want to see how it performs getting data a few hundred times, you can configure it here. The ExecutionType configured the way these iterations are handled. You can choose from the following:

  • Serial (will be renamed 'Sequential). Uses a typical for loop to iterate
  • Parallel. Uses Parallel Extension to the .NET Framework to execute the iterations in Parallel
  • Async. Uses async mechanism (if available for this particular Data Access Product). Works well with for instance PI Web Services or the PISDK
  • MultiThreading. Spawns new threads for every iteration. 

1614.screen6.png

 

After we have configured our Execution Group, we can add operations to our Execution Group. You can add as many different operations as you like. The same goes for adding more execution groups.

 

4530.screen5.png

 

Now we can configure the operation. Here is an example of the 'Get Snapshot' configuration

 

4657.screen7.png

 

And here is an example of configuring and adding Requests to a 'Get Summary Data' operation. Again, you can add as many as you like.

 

1346.screen8.png

 

Here is an example of running a Execution Plan that uses PI Web Services to get snapshots. In this example we compare the results of calling 'GetPISnapshotData' 20 times sequential, and 20 times in parallel. You can see the report (in HTML) in the middle pane. Before executing the Execution Plan, there are latency checks (ping) and a traceroute to the target server to get some insight in the network performance. You will also get a summary about the performance of the different execution groups (in this case, using the same operation but one sequential and one parallel executed). You can clearly see the difference in performance when comparing Parallel vs. Sequential.

 

1106.screen10.png

 

Here are some more details from the generated report (Sequential group)

 

3618.screen11.png

 

and for the Parallel group

 

4353.screen12.png

 

You can imagine using this to generate a performance report for PI SDK vs PI Web Services, or vs AF SDK, or any other combination. You can also use it to see the performance difference using the same technique to different PI servers in your organization. This could give some great insight into choosing the right data access method for your environment or project.

 

Status of the application

 

The application that is showed here is in a very early stage of development, and should be considered a concept application. Development has come up to a point where it is feasible to show, and to a point where I need more input. I could personally think of something like this as a Community Project.

 

At this point, I cannot promise anything beyond what I just showed. I'm not sure how this will develop, and if this will ever reach maturity. I would certainly like to come up to a point where I can share the code.

 

The application is a .NET 4.0 WPF application, written in C#. 

 

Questions that need input

 

I'm personally convinced this could be a great tool to get insight into the performance of the different PI DataAccess products. I would like to get some input on the following:

  • What's your overall thought after reading this?
  • Will something like this be useful in your PI application development efforts?
  • What features should definitely be present in a concept like this?
  • Any further comments/input.

 

 

If you have made it to the bottom, thanks for reading! I hope you leave comment to provide some input!

 

 

PI Data Access 2010 R3 Releases

Posted by spilon Feb 21, 2012

Dear vCampus Community,

 

We are pleased to announce the PI Data Access 2010 R3 wave of releases, the last piece of the greater PI Server 2010 R3 release! Indeed, we just released the 2010 R3 version of PI OLEDB Enterprise, PI JDBC and PI Web Services, which complete the family portrait with the release of the classic PI OLEDB Provider 2010 R3 back in early January. These releases focus on performance improvements and provide several other enhancements and bug fixes:

  • PI OLEDB Enterprise
    • Support for AF attribute searches and AF attribute value searches anywhere in the asset structure, without the need to traverse and reference AF elements from the hierarchy
    • Support for the LEFT OUTER JOIN statements, which is very useful for more sophisticated queries that combine asset metadata and time series data.
    • Significant performance enhancements for both time series data access and asset metadata access
    • Reduced memory consumption
    • Ability to configure error handling options (Embed Errors, Ignore Errors) globally, via the Connection String
    • Ability to change the Catalog in use dynamically ("USE <catalog name>" statement)
  • Classic PI OLEDB Provider
    • This release includes an important performance enhancement, specifically around archive queries that involve many tags but few values per tag
    • Bug fixes, including Punchlist Item (PLI) #22196OSI8
    • This version comes bundled with PI SDK 2010 R2 which enables Client-Side Buffering (buffered data writes and writing to High Availability PI Server)
  • PI JDBC
    • Benefits from all enhancements made to PI OLEDB Enterprise and the classic PI OLEDB Provider, as listed above
    • Support for the Java Runtime Environment (JRE) version 7
  • PI Web Services
    • Performance enhancements
    • Support for "data by exception" for PI tags (commonly referred to as "signup for updates")
    • Improved installation experience
    • Support for additional data insertion modes
    • Support for hosting PI Web Services in Windows without Microsoft Internet Information Services (IIS)
    • This version comes bundled with PI SDK 2010 R2 which enables Client-Side Buffering (buffered data writes and writing to High Availability PI Server)

Of course you can consult individual product Release Notes for a more detailed listing of changes/fixes.

 

While I certainly hope the user manuals provide all the information you need on the new functionality, we will be hosting a webinar to discuss the PI Data Access 2010 R3 wave of releases, show performance numbers and go through the new functionality. More details to come.

 

I look forward to reading about how the improved performance and the new features of these products are helping you bring projects to fruition! Please do not hesitate to share snippets of your projects (e.g. screenshots) with me, as I love to hear how you make use of our products, what problems you solve, as well as what you would like to see next. Of course you can contact me directly at spilon@osisoft.com if you don't want to provide this kind of feedback publicly, on the forums

 

Now what? You can look forward to the PI Data Access 2012 releases, whose main purpose will be to deliver PI Event Frames capabilities! Please reach out to me if you are interested in being an early adopter for these!

This has become a pretty big story, scroll down for the TL;DR; (Too Long; Didn't Read;)

 

I'm always really excited to go to conferences. One of the conferences I always try to attend is the local Microsoft Developer Conference in The Hague. It was previously known as Microsoft DevDays, but it has now been rebranded to Microsoft TechDays.

 

It is a 3 day event. The first day focussed on 'IT Professionals' (mainly administrators), and the last 2 days focus on 'Developers'. Development is in my blood, but I still decided to go the full 3 days, and also attend the 'IT Professionals' day.

 

When I first arrived, I found it funny that the IT Professionals (read: admins) attending the first day are a very different bunch. I was wondering where all the people with their laptops were, scattered around the conference site, writing code, drinking coffee and chatting. That's what developers do, but it seems that admins are a different bunch. Also in terms of enthusiasm: admins tend to cheer and clap a lot less during exciting announcements. But this could be my persional bias as a developer...

 

The keynote was presented by two really funny dudes, and there was a lot of talk about Windows 8, Windows Server 8 and System Center 2012. After that there were 5 different parallel tracks. Topics ranging from general system administration, to 'what's new in Windows Server 8' to advanced topics of windows Debugging and security. I decided to focus on security. I attended a total of 4 security topics, and these were all pretty good.

 

The first two sessions were presented by  Hasain Alshakarti . The topic of some big security breached came up, specially the one at Symantec, DigiNotar and RSA. The message was clear: it's almost impossible to know who you can trust anymore. The security breach at DigiNotar had a big impact on the way we think: hackers were able to issue their own signed security certificates, which caused a lot of trouble. Especially, here in the Netherlands a lot of govermental agencies were using certificates by DigiNotar. After the security breach it was impossible to know for sure who had issued these, and it was almost impossible to recognize if you were on the real goverment site, or a spoofed one (with all risks attached). 

 

The last two security topics were presented by  Paula Januszkiewicz . This was very refreshing. She was very knowledgable, and she brought the topics with a lot of fun and charm. She did tons of demo's, which really impressed the audience. I wasn't aware that in this day and age, the Windows operation system allowed for so many malicious actions. She demonstrated retrieving passwords, being very annoying with the Windows debugger, to totally crashing entire enterprise IT systems.

 

The last topic was about the risk of disgruntled IT administrators. The point of the presentation was to make people aware of the power that (Enterprise) administrators have, and the risk that comes with that. If an administrator turns evil, for instance because he is unsatisfied with the company, or he is at the brink of being fired: he can do a lot of damage and (almost) bring a company down. There were a few nice examples where disgruntled administrators can do tons of damage using malicious (Active Directory) configurations or some evil scripts.

 

All and all I had a very good first day, even though there no development topics. If you ever had the chance to attend topics by  Paula Januszkiewicz , make sure you do!

 

The second day was the first of two development conference dayS. When entering the venue, you could immediatly feel that developer vibe and atmosphere. Everywere people are talking geek, and the energy was very motivating.

 

The keynote was very interesting, and had a lot of small demonstrations. Offcourse, the mandatory Windows 8 demo, but also a lot about Metro development and Windows Azure. I was looking forward to hearing Scott Guthrie speak. I'm always very inspired by him and the way he speaks. Offcourse, he had an awesome demo with code. The small demo made use of the Azure Service Bus, and allowed people in the audience to write messages with their phones to the big screen. This was reason for a lot of hilarity, as people send messages ranging from 'Hello Scott, you are awesome', to 'format c:\' and '; -- drop table users;'. 

 

The first breakout sessions I attended was called 'Rocking your enterprise with Microsoft Kinect'. As you may know, I developed some Kinect with PI AF demo's a few months ago, so I was very anctious to see what was next. The overall message of the presentation was very nice. I was glad that other people are also enthusiastic about the Kinect sensor and what it could do for business applications. I was a bit dissapointed by the examples that were presented, as I already studied those before. One really funny thing was the fact that one of his demo's looked suspiciously like my Kinect + AF + Bing Maps demo that I uploaded to YouTube a few months ago. It also was a data visualization demo, with a carrousel of country flags...

 

The rest of the day was filled with very nice topics surrounding MVC4, WebAPI, Metro, C# 5.0, etc. 

 

In the evening there was something called 'Geek night', which basically meant that there were a lot of fun sessions. I went to a Kinect track. The first session was very interesting, as the presenter  Kay Hofmeester  (who works on the Xbox team) raised a lot of interesting questions about Natural User Interfaces in a non-gaming environment. It was very nice to hear someone struggling with the same questions around this subject. I will definitly contact him to get more insight.

 

The second presentation in the evening was presented by Rob Miles . What an awesome presentation! Everything broke down, demo's stopped working and applications crashed. The guy presented the entire presentation with the charme and jokes of a standup comedian. It had been a long time since I had to laugh this hard during any presentation. The guy was just funny as hell. If you ever have the chance to attend one of his presentations, please make sure you do!

 

The last day was filled with a lot of real in depth technical tracks. I made sure to attend the introduction to MVC4 WebAPI framework. This is a very interesting concept, and I will definitly research that further and blog about it.

 

Then the sessions came where I really looked forward to: C# 5.0 async 'behind the scenes', and a talk about 'Project Roslyn'. 

 

C# 5.0 offers true asynchronous programming trough a set of new keywords and methods. Bart de Smet took us trough some of the internals of async, and how the compiler handled the async calls and keyword. This was a very good session, where the internal state machine that the compiler creates was programmed out in detail.

 

The last session for me was about Project Roslyn. As you may know, I'm a big fan of the project. Roslyn opens up the compiler, and offers API's to get directly involved in the compilation process. I did a few blogposts about it here and here.

 

 

 

TL;DR;

 

Microsoft is focussing all client development on Metro and ASP.NET (MVC). There is not too much mention of Silverlight, but Silverlight developers will have acquired the right tools to hit the ground running with Metro development. Windows 8 and Windows Server 8 will bring a lot of great innovation. I'm personally very anctious to get my hands on the Consumer Preview of Windows 8 that will be released end of this month. 

 

My focus on getting a Natural User Interface with Kinect to an office/enterprise environment is shared with a lot of people. This was very refreshing to notice. There are a lot of uncertainties and undiscovered realms here. A very good analogy by a presenter was when video recording became available, it was not that populair because the concepts of zooming/editing and moving were not yet introduced. Introducing Kinect as a controlling device for your PC also needs some new ways of thinking.

 

Roslyn is going to be awesome! There are so much possibilities here. I will be sure to dive deeper into it, and blog about it here.

 

Long story short: I don't possibly have enough time to dive deep into all this new stuff in the short term, which annoys me

 

 

skwan

PI AF SDK - Rich Data Access

Posted by skwan Employee Feb 10, 2012

Hi folks,

 

For those of you who attended the last vCampus Live! in November, you may have overheard or even participated in conversations with us on the work that we've been doing with rich data access (RDA) and the PI  AF SDK.  For the benefit of those who do not know what this is, in a nutshell, we have been busy enhancing the AF SDK so that it would allow you to directly access PI Server data directly in AF Attributes using the AF SDK.  Some of you may have been developing .NET-based multi-threaded applications using a combination of the AF SDK and the PI SDK in order to access PI AF and PI Server data respectively.  In this case, you may have run into STA/MTA threading issues, as well as performance issues with COM Interop.  The AF SDK with RDA functions will allow you to use a single .NET 4 based SDK to access data from the PI System, i.e. PI AF objects and PI Server data, thereby avoiding those issues.  In addition, the AF SDK with RDA functions may provide performance improvements depending on the architecture and implementation of your application.

 

The PI AF Client 2012 CTP setup kit is now available in the vCampus Download Center.  This kit consists of two versions of the AF SDK; a .NET 3.5 version that retains backwards compatibility with your existing applications and a .NET 4 version that implements RDA, and includes PI System Explorer and PI AF Builder.  Along with the PI AF Client setup kit, there is also an updated PI AF Developer Tools setup kit and a README that describes the improvements and changes.  We will continue to ship both the .NET 3.5 and .NET 4 versions of the AF SDK so if you do not have the need for the new RDA functions, you do not have to do anything and your applications will continue to work.  If you wish to use the new RDA functions, you will need to implement the methods provided to you in the two new namespaces that have been added to the AF SDK: OSIsoft.AF.Data and OSIsoft.AF.PI.

 

Of course the PI SDK continues to be fully supported.  It’s important to note that if your application is based on the PI SDK and you are not using the AF SDK or if you have no need for the RDA functions in the AF SDK, then you do not have to do anything.  Rest assured that your applications will continue to work and be supported going forward.

 

This CTP is just that, a preview, so it is not intended nor recommended to be used in a production environment.  There are some features and functions that are not yet finalized at this time and much refinement is to be done in many areas.  This SDK is evolving and will continue to do so over the next few releases.  The initial release will happen later this year along with PI DataLink.  Keep your eyes on the PI System Roadmap for updates.  It is worthwhile to remind everyone that this is one element of our PI Data Access product suite and as such its usage is governed similarly, i.e. vCampus license for Development use and PSA license for Runtime use.

 

I urge you to spend some time evaluating this CTP.  In addition to any bug reports, we are extremely interested in any performance comparisons of the RDA functions in your applications that you are willing to share.  You may of course post to the vCampus forums or you can send them to us via email.  Any data you send directly to us will be kept strictly confidential.  Please use the BetaAF@osisoft.com mailbox to report back any comments, questions or problems that you may have with this CTP.  The BetaAF mailbox is monitored by myself, the PI AF product specialists and the PI AF development team.  Please do not contact our regular Tech Support as they are not prepared to answer your questions regarding this CTP.  Lastly, keep your eyes out for an upcoming vCampus webinar on this CTP later this month.

 

And oh, one more thing, check out the enhancement to AF Tables

 

--

 

Steve Kwan, PI AF Product Manager

As promised here we will explore more of the PI SDK in C++. This time we are going to use an EventPipe (AKA sign up for events).

 

So we start with our preparation exactly as in this post (Note - the PI SDK has changed since that post, PITimeServer.DLL is now in the PISDK folder), but before we start with _tmain, we need something to handle our event pipe.

 

Here is a little function that takes all events from an EventPipe, counts them and prints them on the command line. We are going to use an EventPipe from a PointList - so the events we are receiving are PointValues, not PIValues.

 
double PrintEvents(EventPipePtr spEvPipe)
{
     // the number of events we are going to read
     long nEvent = 0;

     try 
     {
          while (spEvPipe->Count > 0) 
          {
               // loop through the events
               nEvent++;
               // pointer to the Event object
               _PIEventObjectPtr spEvObj = spEvPipe->Take();
               // the variant to the event data
               _variant_t vT = spEvObj->EventData;
               IDispatchPtr pDispatch = vT.pdispVal;
               // we are signed up for snapshot events, so we get PointValue objects
               PointValuePtr spPV = pDispatch;
               // print the tag, time, value we received in the event
               MyPIValue mPV(spPV->PIValue);
               std::cout << "Eventpipe: ";
               std::cout << spPV->PIPoint->Name << " ";
               std::cout << mPV.bstrTimeStamp << " ";
               std::cout << mPV.bstrValue << std::endl;
          } 
     }
     catch(_com_error Err)
     {
          std::cout << "Error in GetEvent: " 
                    << Err.Description()
                    << " : "
                       << Err.Error()
                      << std::endl
                      << "Unable to load PISDK."
                      << std::endl;
     }
     // return the number of events read
     return nEvent;
}

 Now we can continue with the main function. Similar to last time we initialize COM, check for the command line parameters and finally create the PISDK. After this has been done, we print out the PI SDK version:

 

 

 
int _tmain(int argc, _TCHAR* argv[])
{
     // Initialize COM
     ::CoInitializeEx(NULL,COINIT_APARTMENTTHREADED);
     // Check the command line switches
     if (argc < 3) {
          std::cout << "Command Line:" << std::endl
                      << (_bstr_t)argv[0] << " SERVERNAME TAGNAME(s)";
          return (1);
     }
     try                                    
     {
          // Create an instance of the PI SDK
          spPISDK.CreateInstance(__uuidof(PISDK));
          // Print out the PI SDK version
          spSDKVersion = spPISDK->PISDKVersion;
          std::cout << std::endl << "PI-SDK Version " 
                      << spSDKVersion->Version << " Build "
                      << spSDKVersion->BuildID << std::endl;
          // get the PI Server
          spServer = spPISDK->GetServers()->GetItem((_bstr_t)argv[1]);

 

 

So far everything is similar. But now we need to create a PointList and populate it by the tags we have supplied at the command line:

 
          // we want to use a pointlist
          _PointListPtr spPointList = NULL;
          spPointList.CreateInstance(__uuidof(PointList));
          
          // You can use more than just one tagname
          for (int i = 2; i< argc; i++) {
               // add tags one by one to a pointlist and print out the tags
               std::cout << "Adding " << (_bstr_t)argv [ i ]  << std::endl;
               spPIPoint = spServer->PIPoints->GetItem((_bstr_t)argv [ i ] );
               spPointList->Add(spPIPoint);
          }

          // Print out the number of tags
          std::cout << std::endl;
          std::cout << "Signed up for snapshot evenvts for ";
          std::cout << spPointList->Count;
          std::cout << " tags.";
          std::cout << std::endl;

 

 

Now let's have a look at the EventPipe:

 
          // the EventPipe will handle the events, 
          // the secondary interface is only required if we want to change the defaults
          // like the poll interval
          EventPipePtr     spEvPipe = NULL;             /* the event pipe */
          IEventPipe2Ptr     iep2 = NULL;                  /* Secondary interface */

          // get the eventpipe
          spEvPipe = spPointList->Data->EventPipe;
          // get the secondary interface 
          iep2 = spEvPipe;
          // set the poll intervall to 5000 ms
          iep2->PutPollInterval(5000);

 

 

to finish this simple example I am going to wait for the first ten events in my EventPipe and exit my little application:

 
          // for the sake of simplicity we just wait 
          // for some events in the eventpipe.
          double dblTotCnt = 0;
          while (dblTotCnt < 10)
          {
               Sleep (5000);
               dblTotCnt += PrintEvents (spEvPipe);
          }

     }
     catch( _com_error Err )
     {
          std::cout << "Error: " << Err.Description() << " : " << Err.Error() << std::endl;
          return (1);
     }

     return 0;
}

 

 

Done for this time. Stay tuned for more!

Filter Blog

By date: By tag: