23 Replies Latest reply on Feb 21, 2013 11:36 AM by BenSmith

    Help in using updatevalues method

    BenSmith

      Hi,

       

      Some one can help me in using updatevalues method?  I'm trying to save bunch of data in PI Server, but the values are getting lost even if i use insertduplicates.

       

      My data size is around 13000, so i created a PIValues object and added all the values with different timestamps.  The calling the updatevalues method, but only very few values are saved in pi server.

       

      I thought may be updatevalues was not working properly if large collections passed, so i made it small chunks like 10 values for each call, still i cannot see the complete list in the pi server. 

       

      For testing i added only 50 points, but in server i can see around 30 values..and everytime this is getting changed. I'm setting different timestamp for each value and passing insertduplicate const.

       

      If i try using update value in a loop, but still this also doesnt work as expected and its very slow.

       

      Sample code here,

       
      PISDK.PIValues newPiV = new PISDK.PIValues();
                              newPiV.ReadOnly = false;
                              for (int i = 0; i < numrows; i++)
                              {
                                  float v = Convert.ToSingle(frm.selectedMatrix.GetElement(i, col));
                                  pitime.LocalDate = ldt
      ;
                                  newPiV.Add(pitime, v, null);
                                  if ((i+1) % 10 == 0)
                                  {
                                      newPiV.ReadOnly = true;
                                      pi.Data.UpdateValues(newPiV, PISDK.DataMergeConstants.dmInsertDuplicates);
                                      newPiV = new PISDK.PIValues();
                                      newPiV.ReadOnly = false;
                                  }
                              }
                              newPiV.ReadOnly = true;
                              pi.Data.UpdateValues(newPiV, PISDK.DataMergeConstants.dmInsertDuplicates);  
      

       

       

      BTW all the data timestamps are in ascending order.

       

      Thanks in advance!

        • Re: Help in using updatevalues method
          BenSmith
          now i've set compressing = 0, still same issues :(
          
            • Re: Help in using updatevalues method
              Ahmad Fattahi

              Here are a couple suggestions. To narrow down the issue, I would make the values and timestamps manually entered into your code (as opposed to being retrieved from a table somewhere) so we clearly know the values are numeric and good. Also, the timestamps should be in the past, and do not go older than the oldest archive in your PI Server.

               

              Another point of caution would be buffering. Do you have buffering (PIBuffss) set up and configured on your client machine? If yes, you may want to disable it and see if that makes any difference.

            • Re: Help in using updatevalues method
              hanyong

              Hi Rajani,

               

              Just looking at the code that you posted, it seems like the problem may not be because of how it is written. Since you say only some values are lost, so I am guessing that your application is authenticated and given privilege to write data to the server. Just trying to understand a bit more of what you are observing:

               

              When you say values are lost and only few values are saved, does the saved values have recent timestamps whereas the lost values have older timestamp earlier than some time?

               

              When you test writing values, are you repeatedly writing at a specific set of timestamps? If so, when you check the values of the PI tag in tools like DataLink or PI SMT, do you see multiple values at the same timestamp?

                • Re: Help in using updatevalues method
                  BenSmith

                  @ahmed, My timestamps are different and actually these are the data from last 3 years collected in a plant! And the values are numeric/float32 as my application handles only that data. Also originally i've not setup buffering in my client, but later after this issue popping up, i setup buffering and again disabled as there was no improvments.

                   

                  Han Yong

                  Since you say only some values are lost, so I am guessing that your application is authenticated and given privilege to write data to the server.

                  Yes our authentication is based on domain/AD.

                   

                  Han Yong

                  When you say values are lost and only few values are saved, does the saved values have recent timestamps whereas the lost values have older timestamp earlier than some time?
                  I dont feel the values are lost based on the timestamps.  Its just random and each try i can see different results.  Everytime i start writing, i remove all the existing values and the timestamp in the PI server set it to year 1970.  Then i start writing the whole 13000 values at a time, but i see only around 200 or 250 or some time 150 values in the archive values. I verified this with my own data import application and also the SMT/archive editor.

                   

                  Han Yong

                  When you test writing values, are you repeatedly writing at a specific set of timestamps? If so, when you check the values of the PI tag in tools like DataLink or PI SMT, do you see multiple values at the same timestamp?
                  Since these are the data from past 3 years, has corresponding timestamps from 2011, i'm writing everything at a time.  

                   

                  After more debugging, i made my code to write 10 values at a time, i find some strange issues :( and all these are not consistent. For eg, after 5 iteration, one time i can see 30 values, and remove the values and restart the test... after 5th time, i can see 21 values...  While continuing testing like this i also found, the existing stored values completely overwritten. For eg, after 20 iteration i can see around 130 values(actually it should be 200 values)., in the 21st iteration i can see only 7 values :(.  So what happened to the previous 130 values?

                   

                  Actually all these values differ by marginally like 123, 123.4, 125.2, 121.4.. so i thought deviation value in the point attribute might be causing the trouble.. i set to 0 for compdev and corresponding % attribute also to 0.  but no change in the results.  I've one more column of data which has const values of 99.898 for the all 13000 and for this i can see only one value in the archive after using updatevalues. So i'm thinking there is some problem in the server configuration, as its not honouring insertduplicates const?

                   

                  So finally i'm currently using UpdateValue instead of updatevalues, after 5000 iteration, i can see 4800 values and atleast in this case % lost values is less. In this test , parallely i also setup my client to read based in the event pipes. Surprisingly i'm getting all the data without any lose in my PI import client which is based on events. So this means, server receives the data properly, and i can also get it my own client which is setup to run based on the event pipes. The problem lies, only when i try to retreive archive data or while seeing through archive editor in PI mangement tool.

                    • Re: Help in using updatevalues method
                      hanyong

                      Just to make sure here, have you created the archive files for the time range that you are writing values to on the PI Server? or what is the time range that your set of archive files cover at the moment?

                       

                      You can check this using PI SMT, Archives plugin under Operation category

                        • Re: Help in using updatevalues method
                          BenSmith

                          Thanks Han Yong, I think that could be the problem... i'm attaching the archive list here... my data is from 2011 but the archive date set from 2012 or something? Can you please tell me, how i can configure or modify this date?

                            • Re: Help in using updatevalues method
                              hanyong

                              Yeah, basically PI Server uses a set of archive files to store the time-series data in the system. Each archive file has a start and end time indicating the time range that the file is storing data for. You can see that in your archive list as well. There should be 1 archive file with a start time, but end time as "Current Time". We refer this as primary archive and new data that is collected goes into this archive file.

                               

                              To backfill data, you have to first create archive file for the required time period. Using the Archives plugin in PI-SMT, you can create new archive file and specify the size and the start and end time. Just look at the toolbar on the top of the Archives plugin and you should find the button create a new archive file. Do take note that you cannot create or register archive with overlapping time period.

                               

                              For your development server, this probably doesn't affect you as much. But most of the time, we would set a reasonable time-period when we create archive files for backfilling. For example instead of a single archive for an entire year, we might break it down into months or weeks.

                                • Re: Help in using updatevalues method
                                  BenSmith

                                  Thanks Yong! i will try these settings and will get back to you.  Meanwhile another info.. i also tried add current time stamp.. by setting DateTime.Now, still the updatevalues was not working as such. My understanding is, even if we have same timestamp, if we select insertduplicates, it should allow all the duplicated records.  So this should have worked for me with the default starttime and current time setting in the archived files.

                                   

                                  Also another question, is there way i can call updatevalues for the set of points/listdata?

                                    • Re: Help in using updatevalues method
                                      hanyong

                                      No problem.

                                       

                                      Yes, you are right that Insertduplicates option should allow you to write multiple values at the same timestamp. I guess we would need more information to investigate into it.

                                       

                                      Using PI SDK, you will not be able to do that. You can write values to a group of tags using AF SDK 2012, PIServer.UpdateValues call, instead. There is quite a number of advantage using AF SDK 2012, especially if you are using PI SDK in .NET now, I would recommend you look at that. But bear in mind that writing data using AF SDK 2012 doesn't go through buffering.

                                       

                                      You can find the reference to AF SDK hereThere is a section that shows you what are the matching PI SDK methods if you want to move from PI SDK to AF SDK 2012. And you can also refer to my blog post on using PIServer.UpdateValues

                                        • Re: Help in using updatevalues method
                                          BenSmith

                                          Hi Han Yong, Thanks for your input on the archive file.  After creating new archive file for the required time frames, the issue got solved.  

                                           

                                          Actually we are currently experimenting with saving and retrieving the data in PI using PI SDK. So before i start updating the values, i try to delete the existing values using RemoveValues by passing start time as "-4y" and current time as "*".  Even though this function deletes all the data for the particular tag, its throwing an exception of "0 success" or something related. But if i see the archive editor all the values are getting removed.

                                            • Re: Help in using updatevalues method
                                              hanyong

                                              Glad that the writing issue is resolved 

                                               

                                              As for the error with RemoveValues, I think we need more information for this... Can you copy the exception message and the code that you are using?

                                              • Re: Help in using updatevalues method

                                                Hello Rajani,

                                                 

                                                Rajani Davuluri

                                                its throwing an exception of "0 success" or something related

                                                 

                                                What's the exact exception you are getting?

                                                 

                                                I've tried the following without seeing any exception being thrown:

                                                 
                                                    Sub Main()
                                                        Dim mySDK As New PISDK.PISDK
                                                        Dim myServer As PISDK.Server = mySDK.Servers.DefaultServer
                                                        myServer.Open()
                                                        Dim myPoint As PIPoint
                                                        myPoint = myServer.PIPoints.Item("SINUSOID")
                                                        myPoint.Data.RemoveValues("*-2h", "*", DataRemovalConstants.drRemoveAll)
                                                        myServer.Close()
                                                    End Sub
                                                

                                                 

                                                  • Re: Help in using updatevalues method
                                                    BenSmith

                                                    Error message is "Failed to remove one or more requested events. [0] Success" and the Error code is -2147219609.  My code is like this..

                                                     
                                                    PISDK.PointList _ps = _server.GetPoints("descriptor = 'DEMO_TAGS'");
                                                    foreach (PISDK.PIPoint pi in _ps)
                                                                        {
                                                                            try
                                                                            {
                                                                                pi.Data.RemoveValues("-3y", "*", PISDK.DataRemovalConstants.drRemoveAll);
                                                                            }
                                                                            catch (Exception ex)
                                                                            {
                                                                            }
                                                    }
                                                    

                                                     

                                                      • Re: Help in using updatevalues method

                                                        Hello Rajani,

                                                         

                                                        You can use PISDKUtility.exe -> Error Lookup to get a translation for error codes.

                                                         

                                                        -2147219609 translates into: Failed to remove one or more requested events.

                                                         

                                                        I recommend listing / retrieving events before deletion and to perform the deletion in smaller chunks in order to find out what's causing the exception. Alternatively, please review the PI Server Log for related messages.

                                                          • Re: Help in using updatevalues method
                                                            BenSmith

                                                            Hi Gregor, I will look into the server log.  But i didnt understand your first part.. "I recommend listing / retrieving events before deletion".  Also i just want to reiterate, that my removal action does succeed.  When i get the exception, if i see server through SMT, there is no values(No Data with an time stamp of 1970).  So currently i'm just catching this exception and continue to remove the values for other tags.

                                                              • Re: Help in using updatevalues method

                                                                Hello Rijani,

                                                                 

                                                                The exception may be caused at the beginning, in between or at the end of the period you pass. I can imagine that an archive gap - a period of time within the 3 years that is not covered by any archive -  might be causing the exception. If this is the case, you may get an exception with the attempt to retrieve events for the 3 years period.

                                                                 

                                                                When retrieving events before and after deletion, you should be able to see those event(s) causing the exception. You've stated, that all events become deleted but sometimes you recognize something when looking more detailed.

                                                                 

                                                                I have tried the 3 years period without having an archive online at the beginning of the period and also after unregistering an archive in between. Because I still don't get any exception returned, I conclude something is wrong with your data / setup. The PI Server log should contain some hint.

                                                                  • Re: Help in using updatevalues method
                                                                    BenSmith

                                                                    Gregor Beck

                                                                    The exception may be caused at the beginning, in between or at the end of the period you pass. I can imagine that an archive gap - a period of time within the 3 years that is not covered by any archive -  might be causing the exception. If this is the case, you may get an exception with the attempt to retrieve events for the 3 years period.

                                                                     

                                                                    When retrieving events before and after deletion, you should be able to see those event(s) causing the exception.

                                                                    This could be the problem...Here is the log,

                                                                     

                                                                    Postevent failed: [-11073] Target event for removal not found in record
                                                                    Point ID: 93, Connection ID: 33, User: CAMO\manjunath, User ID: 12, mode: remove
                                                                    Event time: 11-Sep-12 17:04:38, value: DS_0_302

                                                                     

                                                                    More info on this.. One archive file end time is 11-Sep-12 17:04:37 and another archive file start time is 11-Sep-12 17:04:37. So this failure event occured when there is removal attempt between this end time and the start time of the archive files?

                                                                     

                                                                    If there is no record or archive for the given period, is it right throw an exception?  

                                                                     

                                                                     

                                                                     9/11/2012 5:04:37 PM

                                                                     

                                                                      • Re: Help in using updatevalues method

                                                                        Hello Rajani,

                                                                         

                                                                        Let's start with the explanation what DS_0_302 means. DS indicates a Digital State. In case the point type is not Digital, it's usually an error code. 0 is for the SYSTEM Digital State Set and 302 is the sate number. It translates "Arc-offline" and I understand that there is no archive online at the time 11-Sep-2012 17:04:38. The event Timestamp is 1 second after an archive shift.

                                                                         

                                                                        Rajani Davuluri

                                                                        One archive file end time is 11-Sep-12 17:04:37 and another archive file start time is 11-Sep-12 17:04:37

                                                                         

                                                                        This is confusing because I don't see where the gap should be. The start time of the later archive is equal to the previous one or did you possibly report wrong times?

                                                                         

                                                                        However assuming a gap, the DS_0_302 status is generic - it doesn't really exist. Because I have tried similar and didn't receive an error there are 2 possible reasons I can imagine:

                                                                         

                                                                        1. The user you are using has limited privileges. I was using piadmin.

                                                                         

                                                                        2. The issue may show for a specific version or version combination. I have tried it with PI 2010 SP1 (3.4.385.77) and PI SDK 1.4.0.416.

                                                                         

                                                                        Can you try using piadmin user (e.g. by changing / creating a correspondent trust) just for a test? If this succeeds we can evaluate what permissions are required.

                                                                         

                                                                        On the other hand, if you do have an archive gap, the best way to address it would be closing it by either creating an archive that fits into the gap or by reprocessing the archive before or the one after the gap and set the period covered by the output archive. 

                                                                         

                                                                          

                                                                          • Re: Help in using updatevalues method
                                                                            BenSmith

                                                                            Hi Gregor, Thanks for your reply.

                                                                             

                                                                            There is no time gap in the archive file list and i posted the correct time stamps. the gap i talked about was just my assumption of PI system not treating it properly, sorry for bringing confusion. Also my point type is float32.

                                                                             

                                                                            1. The user you are using has limited privileges. I was using piadmin.

                                                                             

                                                                            Do you mean, the user a/c under which i'm trying to remove? Currently we enabled to use to AD/domain user accounts.  Also if there is limited priveleges setup, then is it possible to remove the data? BTW sorry for posting the previous event log from pi server. I dont think that log is during the exception. Now i tested again, during the exception i dont get any logs in the PI message log.  Also i dont get exception when i specify the endtime as "-1y"  for eg,   pi.Data.RemoveValues("-3y", "-1y", PISDK.DataRemovalConstants.drRemoveAll);

                                                                             

                                                                             

                                                                             

                                                                            2. The issue may show for a specific version or version combination. I have tried it with PI 2010 SP1 (3.4.385.77) and PI SDK 1.4.0.416.

                                                                             

                                                                            We use exactly same setup.

                                                                              • Re: Help in using updatevalues method

                                                                                Hello Rajani,

                                                                                 

                                                                                You will see Digital States from SYSTEM Digital State Set for all point types - as well for float32.

                                                                                 

                                                                                Does the tag with point ID 93 belong to the set of tags you are deleting from?

                                                                                 

                                                                                Does the exception occur for the past one year?

                                                                                 
                                                                                pi.Data.RemoveValues("-1y", "*", PISDK.DataRemovalConstants.drRemoveAll);
                                                                                

                                                                                Rajani Davuluri

                                                                                Also if there is limited priveleges setup, then is it possible to remove the data?

                                                                                 

                                                                                As I said before DS_0_302 might be generic and hence privileges necessary may differ from the privileges necessary to delete existing events. 

                                                                                  • Re: Help in using updatevalues method
                                                                                    BenSmith

                                                                                    Hi Gregor,

                                                                                     

                                                                                    Yes, Point ID 93 is the from the tag list which i'm trying to delete.  As i said, that log related to DS_0_302 may not be related to the removal action.

                                                                                     

                                                                                    The exception occurs exactly, for this call,

                                                                                     

                                                                                    _point.Data.RemoveValues("-161d", "-160d", PISDK.DataRemovalConstants.drRemoveAll);

                                                                                     

                                                                                    But if i change the code to below, then no exception.

                                                                                     

                                                                                    _point.Data.RemoveValues("-161d", "-161d", PISDK.DataRemovalConstants.drRemoveAll);

                                                                                     

                                                                                    _point.Data.RemoveValues("-160d", "-160d", PISDK.DataRemovalConstants.drRemoveAll);

                                                                                     

                                                                                    So -160 days refers 11th Sep 2012 and for that day i've two different archive files as i specified earlier.

                                                                                      • Re: Help in using updatevalues method

                                                                                        Hello Rajani,

                                                                                         

                                                                                        You say that even the exception is thrown, the events you intended to delete are gone. What happens, if you execute the delete a second time? Does the exception occur again?

                                                                                         

                                                                                        Can you do me a favor and retrieve events between "-161d" and "-162d" using DataLink (compressed data). Please ensure you are using the same credentials as with your application.

                                                                                         

                                                                                        Does that work? If so, can you upload the Excel sheet, please?

                                                                                         

                                                                                        Also, please dump an archive listing into a text file and upload it too:

                                                                                         

                                                                                        piartool -al > archivelisting.txt

                                                                                         

                                                                                        I would also be curious about the messages produced by PI Archive Subsystem at startup. If my expectation is correct that this isn't a productive PI System, please do the following:

                                                                                         

                                                                                        - Open a command prompt on the PI Server, navigate to \PI\Adm and execute pigetmsg -f > piserverlog.txt

                                                                                         

                                                                                        - Stop PI Archive Subsystem service (e.g. net stop piarchss) and start it again (e.g. net start piarchss).

                                                                                         

                                                                                        - Wait for about 2 Minutes and then interrupt the pigetmsg.exe process (e.g. [Ctrl] + [ c]).

                                                                                         

                                                                                        - Collect piserverlog.txt from \PI\Adm and upload it.