26 Replies Latest reply on May 28, 2010 7:08 PM by lpagemorin

    StreamInsight: Input and Output Adapter for PI

    MichaelvdV@Atos

      I'm having a look at StreamInsight and PI (looks pretty exciting). I'm using the StreamInsight CTP2 with SQL Server Compact.

       

      I haven't found any (examples) of Input / Output adapters for the PI system. At this moment I'm trying to see what it will take to develop one to get an impression of StreamInsight with PI. Did any of you found the time to have a look at StreamInsight with PI, and maybe developed a (prototype) adapter ?

       

       

        • Re: StreamInsight: Input and Output Adapter for PI
          hanyong

          Hi Michael,

           

          There is some news about StreamInsight with PI mentioned in the OSIsoft vCampus Live!, you might not have to develop one yourself just to get a feel of how it would work. I'll check with the people involved in this and get back to you on this.

           

           

          • Re: StreamInsight: Input and Output Adapter for PI
            richard

            OSIsoft has been working with Microsoft for over a year on StreamInsight. We previewed PI System Adapters at TechEd 2009 last Spring and I had hoped we would have shown off our CTP at OSIsoft vCampus Live but I was unable to attend.

             

            Currently our plans are to provide a CTP of our PI System Input and Output Adapters which focus on Time Series PI Server Data on OSIsoft vCampus for the community to start working with. This should be available within the next 30 days.

             

            There is also an AF Input and Output Adapter in the works which enables Element Relative streams of data to be specified. This code is still in test and will be made available when stable enough for a CTP.

             

            All in all, we expect to simship the PI System Adapters for StreamInsight with the release of Microsoft SQL StreamInsight in mid 2010.

              • Re: StreamInsight: Input and Output Adapter for PI
                MichaelvdV@Atos

                Thanks Richard and Han,

                 

                I'm really glad to hear that there will be a CTP available pretty soon. I missed the fact that you previewed the adapters on TechEd.

                 

                Altough it doesn't hurt to have a look at adapter development myself, it is very good to know that there will be a CTP available within the next month.

                 

                Will there also be PI usage examples of Explicit/Implicit server development and the Observable pattern available with the CTP ?

                • Re: StreamInsight: Input and Output Adapter for PI
                  krachtr

                  Richard Beeson

                  Currently our plans are to provide a CTP of our PI System Input and Output Adapters which focus on Time Series PI Server Data on OSIsoft vCampus for the community to start working with. This should be available within the next 30 days.

                   

                  Richard, have you got any updates on the schedule/status?

                   

                   

                   

                  Rusty

                    • Re: StreamInsight: Input and Output Adapter for PI
                      richard

                      Thirty days has come and gone and unforeseen health issues halted progress shortly after I last posted. We are working on adjusting our resources to get this moving again. In the mean time we don't have a new date. It is in everyone's interest to get this out so we are doing everything we can. I will post back here when there is a more definitve answer.

                        • Re: StreamInsight: Input and Output Adapter for PI
                          MichaelvdV@Atos

                          I had a beer with John Bayer in Antwerp last wednesday, and he told me there where some bumps on the way developing the input and output adapters for StreamInsight.

                           

                          Is OSIsoft willing to share the difficulties with the community? I'm just really curious about what it takes to develop the high troughput adapters

                            • Re: StreamInsight: Input and Output Adapter for PI
                              richard

                              I am not entirely sure what John Baier was alluding to, but I can share a bit on our experience.

                               

                              First, since we began working with Microsoft early in their development cycle, we went through a number of interations in our code as their design and adapter surface area evolved. Of course tracking a moving target offers some bumps, but I can say that where they ended up is solid for a v1.

                               

                              Writing a high performance, scalable adapter requires understanding both the scheduling behavior of the StreamInsight engine and the intimate behavior of the "data access" layer (which is our case is "talking to PI".)

                               

                              To support multiple instances of adapters across multiple instances of queries, we found that we could not let the engine manage the threads that the adapters ran on. Instead we ended up taking over the adapter "worker threads" and managing those ourselves (in the adapter factories.) We then use the "message" passing mechanism of the engine to manage lifetimes of our adapters. While the samples provided in CTP2 are very instructive, they will not scale horizontally. If you try to run them through a scale test, you'll find what we found, which is that the managed thread pool gets saturated quickly. We will also be moving to asynchronous calls to PI to futher boost performance and simplify the thread management challenge. 

                               

                              As far as the "data access" layer to PI, since adapters are required to provide managed code interfaces to bind to the engine, we encountered performance issues due to marshalling across managed - unmanaged boundaries as well as performance issues when pushing the envelope in the PI SDK. As a result we decided to implement our adapters in C++/CLI which allowed us to minimize (basically better control) the managed - unmanaged boundary. We also decided to avoid the COM overhead and instead of using the PI SDK implement the adapters as an internal "subsystem" or as an internal "utility" not too different from an application like piconfig.exe or piartool.exe.

                               

                              So as to not get into too much trouble I need to point out that the performance of the PI SDK has continued to improve and will continue to improve. The COM model however, introduces some significant contraints to performance in our particular scenario which given our performance goals where unacceptable. Remember COM (OLE!) was introduced for VB and automation developers - not generally a high performance goal.

                               

                              We are also working on another improvement to the PI System which will significantly improve performance and robustness of analytics. Later this year we hope to introduce a new version of our PINS node which includes an updated buffering subsystem and a PINS node update manager. This will allow true edge processing by allowing analytics to intercept data (that is, sign up for exceptions) directly on the machine where the data is being collected instead of picking up the data streams from the server. This will allow StreamInsight analytics to be deployed on PI interface nodes which not only reduces latency but allows analytics to be distributed across all interface nodes and allows analytics to keep running even if the data flow to the server is disrupted removing the "surge" phenomena on our server based analytics engines when data flow is restored. For analytics that require joining data from different collection nodes it will still be necessary to pick up the data from the common servers but a significant number of scenarios will benefit from this.

                               

                              I hope that provides some insight...yes pun intended.