11 Replies Latest reply on Aug 29, 2011 8:31 PM by opelosi

    Poor performance when retrieving annotated values

      Hi guys

       

       

       

      I´m developing an application that sends a big file annotation (16+ MB) and retrieves it on ProcessBook VBA using PI SDK. The application takes only a few seconds to send the value but it takes a lot of time (1 - 2 min) to retrieve it from ProcessBook or any PI client (PI SMT, PI SDK application), even when the point´s annotations flag is set to 0.

       

       

       

      The ProcessBook VBA script will get theses values and show them in a ListBox. The user choose one option and click in a button that will save the annotation into a file on user´s computer.

       

       

       

      The server´s Snapshot_EventQueuePageSize tuning parametrer is set to 32768 (32 MB). Any clue to solve this?

       

       

       

      Thanks

        • Re: Poor performance when retrieving annotated values
          dtakara

          Hi Orlando,

           

          Could you share a little more about your use case? What is this big file that you are adding as an annotation to PIValues?

           

          It is known that extensive use of PI Annotations can impact PI Server Performance, as documented in Technical Support Knowledge Base Article #405.

            • Re: Poor performance when retrieving annotated values
              Ahmad Fattahi

              To add a more general resource to what Daniel pointed out you can watch this webinar on how to optimize your PI SDK applications.

                • Re: Poor performance when retrieving annotated values
                  jlakumb

                  Perhaps you are running into this known issue - PLI #24572OSI8

                    • Re: Poor performance when retrieving annotated values

                      Hi again

                       

                       

                       

                      Thanks everyone for the answers.

                       

                       

                       

                      One thing that I observed is if the point's snapshot have an event with a smaller annotation, the retrieval of the archived events is quite fast, even the ones with a large annotation.

                       

                       

                       

                      We are developing a system that monitores the signal generated by a power generator. When the system detects an anomaly, it sends a register file (COMTRADE file in a zip) to PI that contains the data of the event. Because of the data's volume (15 ks/s * 96 channels for 30 seconds), the PI's time resolution (16 micro and the system uses 1 micro) and the event rate (1 per week), we decided to not use the PI points for this purpose to avoid overflow on server and data precision loss.

                       

                       

                       

                      By the way, our PI server version is 3.4.375.99 with PI SDK 1.3.8.388.

                       

                       

                       

                      Thanks

                       

                       

                       

                       

                        • Re: Poor performance when retrieving annotated values
                          Ahmad Fattahi

                          So you are using the annotations for real-time data to avoid overflow? If that's the case you may want to reconsider that. Annotations are meant to contain peripheral meta data on the side of real values. For real time data acquisition, values, and time stamps your best bet is the actual values and time stamps.

                          • Re: Poor performance when retrieving annotated values
                            dtakara

                            Orlando L. Pelosi Jr.

                            One thing that I observed is if the point's snapshot have an event with a smaller annotation, the retrieval of the archived events is quite fast, even the ones with a large annotation.

                             

                             

                             

                             

                            This is interesting. Could you tell me more about it?

                            • What’s small and what’s big (in the snapshot annotation)?
                            • What archive call is being made? Does it always involve the snapshot event?
                            • How long does it take in all cases?
                            • Have you tried outside of PB/VBA (in a standalone SDK application)?
                              • Re: Poor performance when retrieving annotated values
                                dtakara

                                An alternative approach you may want to consider is to store the data with microsecond resultion in a table in a relational database and see these data as if it was stored in PI tags using the PI OLEDB COM Connector.

                                 

                                One thing to note though is that, when accessing this through the PI OLEDB COM Connector tags, the timestamps lose resolution when the data is viewed in DataLink or ProcessBook, for instance, although the full timestamp keeps stored in the relational database with no loss of precision. This is because the communication layers (PINET protocol, PI SDK) do not have enough bytes in the timestamp to maintain the full precision.

                                 

                                To illustrate this, here is a test scenario:

                                 

                                some sample data in a table in a Microsoft SQL Server database, where the type of the time column is datetime2

                                 

                                0131.SQLServer_2D00_TimeSeriesData.png

                                 

                                UTC timestamps and values retrieved from the corresponding PI OLEDB COM Connector tag with a VBA script using the PI-SDK

                                 

                                2248.ExcelVBADialogBox.png 

                                 
                                Dim myServer As Server
                                Dim Cxn As New PISDKDlg.Connections
                                Dim myPoint As PIPoint
                                Dim myData As PIData
                                Dim myValues As PIValues
                                Dim val As PIValue
                                Dim msg As String
                                
                                Set myServer = Servers("DANIEL-WIN7")
                                If Not myServer.Connected Then
                                    Set myServer = Cxn.Login(myServer, , , True, True)
                                End If
                                Set myPoint = myServer.PIPoints("OLEPointX")
                                Set myData = myPoint.Data
                                Set myValues = myData.RecordedValues("22-aug-11", "*", btInside)
                                msg = "UTC timestamps and values" & vbCrLf
                                For Each val In myValues
                                    msg = msg & "ts: " & Format(val.TimeStamp.UTCSeconds, "0.000000") & " - value: " & val.Value & vbCrLf
                                Next val
                                MsgBox msg
                                

                                 

                                 

                                timestamps and values retrieved from the corresponding PI OLEDB COM Connector tag with piconfig

                                 
                                @table piarc
                                @mode list
                                @timf 9
                                @istru tag, starttime, endtime, mode
                                @ostru value, status, time
                                @ostru ...
                                OLEPointX,22-aug-11, *,comp
                                
                                1.,GOOD,22-Aug-11 17:20:00
                                2.,GOOD,22-Aug-11 17:20:00
                                3.,GOOD,22-Aug-11 17:20:00
                                4.,GOOD,22-Aug-11 17:20:00.40000916
                                5.,GOOD,22-Aug-11 17:20:00.50000000
                                6.,GOOD,22-Aug-11 17:20:00.60000610
                                10.,GOOD,23-Aug-11 08:00:00
                                11.,GOOD,23-Aug-11 09:00:00.50000000
                                

                                Also, the PI OLEDB COM Connector is not provided as part of your vCampus membership. If you are interested in trying it, you may ask your account manager about the possibility of getting a demo license.

                                 

                                In the spite of this loss of resolution in the timestamps when retrieving the data using the approach, it could still be interesting if getting all the values in the right sequence in the period of interest is enough and if you don't actually need the timestamp of each individual value.

                                • Re: Poor performance when retrieving annotated values

                                  Hi again

                                   

                                   

                                   

                                  Thanks everyone for the answers.

                                   

                                   

                                   

                                  The size of the files used are ~700kb (smallest) and ~16mb (largest). We are sending the files with the following code:

                                   

                                   

                                   
                                  PIAnnotations annotationsShort = new PIAnnotations();
                                  annotationsShort.Add("Short", "Short register", filepath + ",S.zip", true);
                                  
                                  PIValue valueShort = new PIValue();
                                  valueShort.Value = timestamp;
                                  valueShort.TimeStamp = timestamp;
                                  valueShort.ValueAttributes.Add("Annotations", annotationsShort);
                                  
                                  this.shortRegisterPoint.Data.UpdateValue(valueShort, 0);
                                  

                                   Sending the files takes only a few seconds (in worst case), but the retrieval takes 2+ minutes. We also developed an application that retrieves theses files using the following code:

                                   

                                   

                                   
                                  PITime ts = new PITime();
                                  ts.LocalDate = date;
                                  PIValue value = this.eventPoint.Data.ArcValue(ts, RetrievalTypeConstants.rtAtOrBefore);
                                  NamedValues nvs = value.ValueAttributes;
                                  NamedValue nv = nvs["annotations"];
                                  PIAnnotations anns = (PIAnnotations)nv.Value;
                                  foreach (PIAnnotation an in anns)
                                  {
                                         if (an.IsFile)
                                                an.Export(path);
                                  }
                                  

                                   The files are correctly exported but it takes a long time to the ArcValue function returns.

                                   

                                   

                                   

                                  Using an external database would solve this problem but there are some security restrictions of our costumer network that prevents this approach. Also, our customer wants this data in COMTRADE format.