30 Replies Latest reply on Jun 1, 2017 12:36 PM by gregor

    Parallel CheckIns to AF Database allways fail.

    VarbanVarbanov

      Hi, we are using our custom WCF Service for writting assets in AF Database for quite some time.
      There has allways been a following problem : when two calls try to add assets they fail with the following exception:

       

      A first chance exception of type 'System.InvalidOperationException' occurred in OSIsoft.AFSDK.dll

      Additional information: Element 'XXX' with UniqueID '84403e58-fff4-4721-8947-18e4f999cb12' is not checked out by the requesting user.

       

      This also happens when somebody has checkout elements in Pi System Explorer.
      The service is just adding or editing (mainly) multiple elements and at the and makes this call:

       

        private void CheckIn()

              {

                  if (_afDb.HasItemsCheckedOutToMe)

                  {

                      _afDb.CheckIn(AFCheckedOutMode.ObjectsCheckedOutToMe);

                  }

              }

       

      NOTE! The described behavior does not change with any of the values of AFCheckedOutMode (except "All" which is not allowed). The service does not maintain session and does not use multilple threads inside (i.e only the calling thread). Does anybody knows solution to this problem?

        • Re: Parallel CheckIns to AF Database allways fail.
          David Hearn

          If your service is running under an account that is used by other applications, then it is possible that the other application checked in the changes while your service was making modifications. It is best to have your service run under a specific account that is not shared by other applications.

           

          You can check the CheckOutInfo on an object before attempting to make changes to determine if the object is already checked out by another user and take appropriate action. Maybe delay the change or log an error and skip the change.

          1 of 1 people found this helpful
            • Re: Parallel CheckIns to AF Database allways fail.
              VarbanVarbanov

              I supposed the behavior should be as described by you. However the behavior I'm describing is easy reproducible if no one else except the service is attempting to make changes. Just several parallel calls to that service are leading to this exception (that's why we are using sequential calls). The service is running under Network Service account. I assume that this exception happens because when separate changes are made to the AF Database by the service at the same time and when one attempts to check in it attempts to check in all the changes and this is not allowed. However  when the option “CheckoutToMe” is used – logically speaking this should not be a problem. On the other hand when ObjectsCheckedOutThisSession or ObjectsCheckedOutThisThread is used logically speaking the service should try to checkin only the changes it just made. But all these check in options does not work and produce same error when parallel calls to the service are made.

              Regarding your second proposal – it is difficult to apply it, because usually the service is making changes to many elements at once and make check in to all of them for improving performance. That’s why between the time of making change (and presumably checking the checkout info of the AF Element) and the time of the actual checkin anyone could make changes again. I guess this is solvable by locking, inspecting all element changes before check in using some of the AFDatabase methods and comparing to the changes attempted to be made by the service, but this is not a small change and also probably will hit the performance while most of the times it is not needed.

              So I just prefer to understand why the check in logic described above does not work as expected. I just want to achieve the following: the service to check in always its changes without an exception. I want to ideally achieve this regardless of the other changes made by the service itself in another call or if someone checked out something is Asset explorer. I think this should be done using check in options but i cannot understand why this is not happening.

              Forget to tell you: I call  _afDb.Refresh(); before starting to make changes.

                • Re: Parallel CheckIns to AF Database allways fail.
                  David Hearn

                  You say that you are not using multiple threads within your service, but are getting the error with parallel checkins. Is the application that is calling your service triggering multiple checkins at the same time?

                   

                  With what identity are you connecting to the AF Server (e.g. are you impersonating the user calling your service)? Are different users calling your service? How are you obtaining the PISystems object for each user (note the PISystems object maintains state information for each user)?

                   

                  With calls coming in on multiple threads, you should be using the ObjectsCheckedOutThisThread for the mode when calling CheckIn. But if you are using an async pattern or are using Tasks (which can re-use the same thread), this mode will not work either and you will need to keep track of the objects you have modified and do a checkin using PISystem.CheckIn and pass the list of modified objects.

                   

                  If another user already has the object checked out, then you will get an exception because you are not allowed to modify an object that is checked out by another user.

                    • Re: Parallel CheckIns to AF Database allways fail.
                      VarbanVarbanov

                      The service is WCF Service. It uses default instancing and concurrency mode for WCF, i.e Singe Concurency mode and Per Session instancing. But because we are not maintaining session, it is equal to Per Call instancing. Yes - you are right in your suggestion -  the client can initialize multiple calls to the service - in this particular case using Task.Parallel library, but this does not directly affect the service - it is most likely even on separate machine. Each incoming call is dispatched to the service by the runtime on the threads from the Thread Pool. Although these threads are eventually reused it is not possible that one thread is used for two calls at the same time.

                      The AF Server is called by the service and it is connected by using windows credentials of the machine where the AF Server is running (usually the same machine where the service is running). This code is used:
                       

                        _piSystems = new PISystems();

                       

                        _piSystem = _piSystems[afServerName];    

                        
                      _piSystem.Connect(GetCredentials());


                      private NetworkCredential GetCredentials()

                        {

                                  return new NetworkCredential(ConfigurationManager.AppSettings[SettingsUsername], ConfigurationManager.AppSettings[SettingsPassword]);

                        }

                      However because this code is very expensive to run on each request (it takes up to several seconds), this information (including PiSystems object) is cached and it is alive as long as the web site hosting the service is alive (until the service is restarted).

                      So, knowing this information, should the
                      ObjectsCheckedOutThisThread mode work provided only the service is making any changes? (According to me all modes should work in this case, Am I wrong?)

                      And what will happen if another user is checked out something? Are you saying that this always will result in exception, no matter which check in mode is chosen?

                        • Re: Parallel CheckIns to AF Database allways fail.
                          David Hearn

                          If another user has the same element checked out that you are attempting to modify, then you will always get an exception because the other user has the element locked for modifications. In this case, you will get the error when attempting to modify the object (which must do a checkout) and the checkin mode does not matter.

                           

                          The SDK's cache of loaded and checked out objects is based upon the user's thread identity at the time the 'new PISystems()' is called. The call to 'PISystem.Connect' where you specify the credential is just specifying the credentials to use when connecting to the server and does not affect the SDK's cache of loaded and checked out objects. Also, the Connect with credentials specified will close all connections to the server for the current user even in other threads (this does not appear to by your issue).

                           

                          In a web environment, I think the only mode that has the possibility of working is ObjectsCheckedOutThisThread because I think IIS can actually run multiple instances of your service on different threads and will periodically shut down your service and start it on another thread. If this mode does not work, then it would appear that somehow you are getting multiple calls using the same thread which is what would happen if Tasks are being used. In that case, I think the best solution is for you to put the elements that you have modified in a list and then use PISystem.CheckIn by passing in this list. Then you will be sure that you are only checking in the objects that you have modified during the current call.

                            • Re: Parallel CheckIns to AF Database allways fail.
                              VarbanVarbanov

                              Hi David, thanks a lot for the information!

                              I think we are getting closer to the answer

                               

                              You are right for the multiple instances of the service - this is what i meant by "it is equal to Per Call instancing". And all these instances can modify some data at the same time. However this applies for the instances of the actual class, implementing service contract and performing service logic. But if we think of a service as the web site which is hosting this logic - it is alive until it is stopped or restarted and there are no different instances. That's why the initialization of the Pi Systems object can happen once  in one thread (and it does exactlty this because of the current implementation) and the service logic will be performed on different threads. Just to confirm this i run some simple tracing in debug:

                               

                              Pi systems initialized at thread id: 13

                              Service called at thread id: 13

                              Service called at thread id: 9

                              Service called at thread id: 7

                               

                              On the first call Pi systems is initialized on thread with id 13, and at the same thread the call is performed.
                              Next calls are on different threads - they may and may not be the same - it depends on the currently available threads. It is possible that you get different calls using the same thread, but I believe - not at once (i.e when the thread is reused, it would already have checked in its previous changes).


                              Please tell me what does user's thread identity mean, but the threads in each service call are different, the user is allways one.

                               

                              So in this case - Pi Systems is initialized on one thread and calls are made on another threads - what would be the behavior?
                              As I mentioned - I do not find any difference in the diffrent modes - allways get an exception at some point with parallel calls. And one more thing - each of these calls does not modify same elements - maybe children of the same element, but not same elements themselves. Should in this case the exception happen?

                                • Re: Parallel CheckIns to AF Database allways fail.
                                  David Hearn

                                  I meant the windows user identity associated with the thread that is making the call to the SDK. So it appears that the user identity calling the SDK for all the threads is the same. So in this case, the behavior would be that the same cache of loaded and checked out objects would be used for each thread. That is why you would want to use the ObjectsCheckedOutThisThread mode when doing your checkins.

                                   

                                  Just to clarify, each time the service is called you are modifying one or more elements and calling AFDatabase.CheckIn before the call returns.

                                   

                                  When using the ObjectsCheckedOutThisThread mode and you get an exception, can you determine how it is related to the object you are checking in? For example is it a child element of an element being modified in another thread? What type of change is being made (modifying an existing element or adding new children)? Are you using Parent-Child reference types (you will have problems with Composition reference types)?

                                    • Re: Parallel CheckIns to AF Database allways fail.
                                      VarbanVarbanov

                                      So, as I understand based on our conversation:

                                       

                                      1. The cache which is taking care of checked out objects is initialized when the PiSystems object is created and stays the same as long as the user is the same and the PiSystems object is the same, no matter how many different threads are accessing the af data base through this PiSystems object at the same time. Is this correct?
                                      2. In this case when ObjectsCheckedOutThisThread is used this will attempt to check in only the changes made in the current thread (service call). Is this correct?
                                      3. If ObjectsCheckedOutToMe is used this will attempt to check in all changes made by the calling user in the context of the created PiSystems Object including the changes made in different threads than the calling one. Still this shouldn’t be a problem for the calling thread itself since its changes will be also checked in. Is this correct?
                                      4. The exception will happen regardless of the check in options if another user with different PiSystems object has checked out same objects (or not same objects - still you need to answer) because it is not allowed two users to modify the same data at once. Is this correct?

                                      If all correct i think that it is no strange that i receive this exception with any of the modes, because it is not related to the used check in mode, but to the fact that af sdk thinks that another user has checked out same (or not the same) af objects. The question is - why is this happening? Because no modifications by different users were made. Is this correct?

                                       

                                      However I'll test further with the ObjectsCheckedOutThisThread mode and I'll make sure the user identity, associated with the thread is always the same (although I cannot imagine how it wouldn't be) and tell you the result. Currently I have another puzzling exception and maybe I'll start another thread... In the meantime - if you can answer all the questions above and if you can make any suggestion why is this exception happening it would be great. Here is the information you've asked:

                                       

                                      David Hearn wrote:

                                      "Just to clarify, each time the service is called you are modifying one or more elements and calling AFDatabase.CheckIn before the call returns."

                                       

                                      Yes, exactly, this is correct.

                                       

                                      David Hearn wrote:

                                      "When using the ObjectsCheckedOutThisThread mode and you get an exception, can you determine how it is related to the object you are checking in? For example is it a child element of an element being modified in another thread? What type of change is being made (modifying an existing element or adding new children)? Are you using Parent-Child reference types (you will have problems with Composition reference types)?"

                                       

                                      In the past, until now I have used ObjectsCheckedOutToMe mode and i didn't noticed any particular pattern when this was happening. In the current situation, when I received the exception again - it was just a case when the different calls were adding child elements to the elements that belong to the same parent (for example there is element A, who has children A1, A2, A3 and the parallel calls are adding elements at the same time to the A1, A2, and A3). Then I changed the mode to ObjectsCheckedOutThisThread and ObjectsCheckedOutThisSession and received the same exception. I am using Parent-Child relationship with newly created elements (not references). If the referenced elements are used, again the Parent-Child is used, not Composite.

                                        • Re: Parallel CheckIns to AF Database allways fail.
                                          David Hearn

                                          Answers to items above:

                                          1. This is right with this addition: The same user can call 'new PISystems' multiple times and will still be using the same cache of checked out objects. The cache is based upon the windows user identity at the time the 'new PISystetms' was called.
                                          2. Correct. But if the same object is modified from multiple threads, the modified object is associated with the last thread that made the change.
                                          3. Correct. This would be a problem if one thread did the checkin while the other thread was making changes to the same object because it would be checked in before the changes were complete.
                                          4. Correct.
                                          5. This is the big question! Normally the error you originally reported is caused by a different client running as the same user and checking in the objects that you are modifying or another user calling UndoCheckOut with the force option set.
                                          1 of 1 people found this helpful
                                            • Re: Parallel CheckIns to AF Database allways fail.
                                              VarbanVarbanov

                                              Hi , 1-4 are quite clear.. if there wasn't 5...

                                               

                                              I think there are some contradictions - so please explain:

                                               

                                              1. "Normally the error you originally reported is caused by a different client running as the same user and checking in the objects that you are modifying..."

                                               

                                              According to 2 - if Thread 1 uses ObjectsCheckedOutThisThread mode and same user is modifying same object from different client (i.e different process, i.e diffetent thread) the check in of Thread 1 will attempt to check in only the changes which Thread 1 have made, however, according your clarification of 2, the modified object will be associated with the last thread that made the change. If this is Thread 1 - (as I understand) it check in all changes of the object, if this is the other thread - no check in will be made from me.
                                              According to 3 - if ObjectsCheckedOutToMe is used - no matter which thread check ins - it will check in all changes, because they are associated to the same user.

                                              2. "...or another user calling UndoCheckOut with the force option set."


                                              According to 4 - The exception will happen if another user has checked out same object (you confirmed Correct ).  If another user make UndoCheckOut with forse true - this will undo the changes to our user changes and when he attept to check in he will no have things to check in, i.e., in our case the call: afDb.HasItemsCheckedOutToMe will return false. Maybe the exception will happen if at the same time the one user is making UndoCheckOut(true) and the other CheckIn(), but generally it seems that UndoCheckOut(true) of diffrernt user will most likely prevent exception to happen, although no changes will be made (checked in). If the another user does not call UndoCheckOut, but keeps the changes Checked Out, exception will happen for sure (according to 4).

                                               

                                              Where am I wrong?

                                               

                                              Also - Is it correct that only changes upon same objects would interfere each other and may cause exception when check in, i.e in the scenario I described :

                                              "In the current situation, when I received the exception again - it was just a case when the different calls were adding child elements to the elements that belong to the same parent (for example there is element A, who has children A1, A2, A3 and the parallel calls are adding elements at the same time to the A1, A2, and A3). " - no exception in any case would be expected?

                                                • Re: Parallel CheckIns to AF Database allways fail.
                                                  David Hearn

                                                  You have to keep in mind the state of the cached objects that your application has in memory. You can only find out changes another application has made by calling Refresh.

                                                   

                                                  Answers to your questions above:

                                                  1. It is possible to have to different clients executing under the same user account making modifications to the same object. In this case, your app makes a change to Obj1 which causes it to be checked out on the thread. Then a different app executing as the same user does a database checkin with mode ObjectsCheckedOutToMe. Then when your app attempts to do a checkin with ObjectsCheckedOutThisThread you will get the error because it is no longer checked out and your app did not know another app had checked it in while you were in the process of making changes.
                                                  2. Similar to item 1, your app makes change to Obj1 causing a checkout on the thread. Then the other app calls UndoCheckOut with force true which removes the checkout that your app thinks you have. When your app attempts to call CheckIn, you will get the error because it is no longer checked out on the server.

                                                   

                                                  In order to add child elements, the parent element is checked out. So as long as the child elements A1, A2, and A3 were already checked in and not modified by any of the threads and only child elements were being added, then i would not expect an exception.

                                                  1 of 1 people found this helpful
                                                    • Re: Parallel CheckIns to AF Database allways fail.
                                                      VarbanVarbanov

                                                      Hi, David, thank you for the further clarifications.

                                                      Regarding the A1, A2 and A3 - they are checked in, but as you say they will be checked out when child elements are added to them. However the different calls are adding elements to different elements (i.e one call is adding to A1, another to A2, etc).

                                                       

                                                      In the meantime I made tests with "CheckoutThisThread" option and calls in parallel and again received the exception. As far as I remember I've never received exactly this exception if the calls are sequential in this particular test. However, as I said in the beginning - we have received this exception on multiple occasions in the past and the most reasonable guess based on the exception message is that elements have been modified at once by several requests. But this is not necessarily the reason. I personally think that there is a common problem in OSI Soft Sdk's  - sometimes the exceptions have misleading text. Since now we haven't found any logical reason for this exception in our case. So is it possible someone to check in

                                                      the SDK code for other possible reasons for it and share them? 

                                • Re: Parallel CheckIns to AF Database allways fail.
                                  VarbanVarbanov

                                  Hi David, sorry for my late reply but I did't had time for this issue. What I meant in my last post was that the developers of AF Sdk may look for other reasons for the exception. However I'm posting the relevant code here. Some things are missing, for example code for setting attribute values, categories, etc, but I think that this is not related to the issue, I hope you can replace this with dummy implementation. The ChekIn() code has been modified recently based on our discussions and it is not the original code we discussed (but it seems there is no difference). Again - note, that normally in our system nothing else is changind, editing, checkiong out asset elements besides the code below:

                                   

                                  [DataContract(IsReference=true)]

                                      public class AssetElement

                                      {

                                          [DataMember]

                                          public Guid ID { get; set; }

                                   

                                          [DataMember]

                                          public Guid ParentID { get; set; }

                                   

                                          [DataMember]

                                          public string Name { get; set; }

                                   

                                          [DataMember]

                                          public string Description { get; set; }

                                   

                                          [DataMember]

                                          public List<AssetAttribute> Attributes { get; set; }

                                   

                                          [DataMember]

                                          public List<AssetElement> SubElements { get; set; }

                                   

                                          [DataMember]

                                          public Guid ElementTemplateId { get; set; }

                                   

                                          [DataMember]

                                          public List<string> Categories { get; set; }

                                      }

                                   

                                   

                                    [DataContract(IsReference = true)]

                                      public class AssetAttribute

                                      {

                                          [DataMember]

                                          public Guid ID { get; set; }

                                   

                                          [DataMember]

                                          public string Name { get; set; }

                                   

                                          [DataMember]

                                          public AttributeData Data { get; set; }

                                   

                                          [DataMember]

                                          public string Description { get; set; }

                                   

                                          [DataMember]

                                          public List<AssetAttribute> SubAtributes { get; set; }

                                      }

                                   

                                   

                                   

                                  public List<Guid> AddOrUpdateAFElements(List<AssetElement> elements, string parentElementPathOrId, bool updateSubElements)

                                          {

                                              _connection.GetAFDatabase(_vesselID).Refresh();

                                              var result = new List<Guid>();

                                              foreach (var element in elements)

                                              {

                                                  result.Add(AddOrUpdateSingleElement(element, parentElementPathOrId, updateSubElements));

                                              }

                                              CheckIn();

                                              return result;

                                          }

                                   

                                   

                                          private Guid AddOrUpdateSingleElement(AssetElement element, string parentElementPathOrId, bool updateSubElements)

                                          {

                                              AFElement foundElement;

                                              AFElement parentElement = null;

                                              Guid result;

                                   

                                              if (!String.IsNullOrEmpty(parentElementPathOrId))

                                              {

                                                  parentElement = _af.GetElement(parentElementPathOrId);

                                                  if ((parentElement == null))

                                                  {

                                                      throw new ArgumentException("Parent element does not exist. Element cannot be added");

                                                  }

                                                  foundElement = parentElement.Elements[element.Name];

                                              }

                                              else

                                              {

                                                  foundElement = _connection.GetAFDatabase(_vesselID).Elements[element.Name];

                                              }

                                              element.Description = element.Description ?? String.Empty;

                                              if (foundElement == null)

                                              {

                                                  result = AddAFElementFromAssetElement(parentElement, element);

                                              }

                                              else

                                              {

                                                  result = UpdateAFElementFromAssetElement(foundElement, element, updateSubElements);

                                              }

                                              return result;

                                          }

                                   

                                  private Guid AddAFElementFromAssetElement(AFElement parentElement, AssetElement asElement)

                                          {

                                              // Note! if particular element is not already added in the Af DataBase, it is not possible to add any subelements to this element

                                              var afElement = CreateOneLevelAFElementFromAssetElement(asElement);

                                              if (parentElement == null)

                                              {

                                                  AddElementToAfElementsCollection(_connection.GetAFDatabase(_vesselID).Elements, afElement);

                                              }

                                              else

                                              {

                                                  AddElementToAfElementsCollection(parentElement.Elements, afElement);

                                              }

                                              RecursiveAddChildElementsToAFElement(afElement, asElement.SubElements);

                                   

                                              return afElement.ID;

                                          }

                                   

                                  private void RecursiveAddChildElementsToAFElement(AFElement element, IEnumerable<AssetElement> childElements)

                                          {

                                              if (childElements == null) return;

                                              foreach (var asElement in childElements)

                                              {

                                                  if (element.Elements[asElement.Name] == null)

                                                  {

                                                      var afElement = CreateOneLevelAFElementFromAssetElement(asElement);

                                                      AddElementToAfElementsCollection(element.Elements, afElement);

                                                      RecursiveAddChildElementsToAFElement(afElement, asElement.SubElements);

                                                  }

                                              }

                                          }

                                   

                                      private AFElement CreateOneLevelAFElementFromAssetElement(AssetElement asElement)

                                          {

                                              var elementName = ValidateNeme(asElement.Name);

                                              var afElement = new AFElement(elementName);

                                              SetAFElementProperties(afElement, asElement);

                                              return afElement;

                                          }

                                   

                                   

                                   

                                          private void SetAFElementProperties(AFElement afElement, AssetElement asElement)

                                          {

                                              var template = _connection.GetAFDatabase(_vesselID).ElementTemplates[asElement.ElementTemplateId];

                                              if ((afElement.Template != template))

                                              {

                                                  afElement.Template = template;

                                              }

                                   

                                              if (afElement.Description != asElement.Description)

                                              {

                                                  afElement.Description = asElement.Description;

                                              }

                                   

                                              SetAFElementCategories(afElement, asElement.Categories);

                                   

                                              foreach (var asAttribute in asElement.Attributes)

                                              {

                                                  AddAFAttributeFromAssetAttributesToElement(afElement, asAttribute);

                                              }

                                          }

                                   

                                  private void AddElementToAfElementsCollection(AFElements elements, AFElement element)

                                          {

                                              var name = element.Name;

                                              bool success = false;

                                              int count = 0;

                                              while (!success)

                                              {

                                                  try

                                                  {

                                                      elements.Add(element);

                                                      success = true;

                                                  }

                                                  catch (InvalidOperationException)

                                                  {

                                                      element.Name = String.Concat(name, count);

                                                      count++;

                                                      if (count == MaxReplaceNameAttempts)

                                                      {

                                                          throw;

                                                      }

                                                  }

                                              }

                                          }

                                   

                                     private Guid UpdateAFElementFromAssetElement(AFElement elementToUpdate, AssetElement asElement, bool updateSubElements = true)

                                          {

                                              asElement.Name = ValidateNeme(asElement.Name);

                                              if (elementToUpdate.Name != asElement.Name)

                                              {

                                                  elementToUpdate.Name = asElement.Name;

                                              }

                                              SetAFElementProperties(elementToUpdate, asElement);

                                              if (updateSubElements && asElement.SubElements != null)

                                              {

                                                  AFElement.LoadElements(elementToUpdate.Elements);

                                                  foreach (var childAsElement in asElement.SubElements)

                                                  {

                                                      childAsElement.Description = childAsElement.Description ?? String.Empty;

                                                      var childAfElement = elementToUpdate.Elements[childAsElement.Name];

                                                      if (childAfElement != null)

                                                      {

                                                          UpdateAFElementFromAssetElement(childAfElement, childAsElement);

                                                      }

                                                      else

                                                      {

                                                          AddAFElementFromAssetElement(elementToUpdate, childAsElement);

                                                      }

                                                  }

                                              }

                                              return elementToUpdate.ID;

                                          }

                                   

                                    private void UpdateAttributesOfAnElemenent(AFElement element, AttributeValue attributeValue)

                                          {

                                              if (element != null)

                                              {

                                                  var attribute = element.Attributes[attributeValue.AttributeName];

                                                  if (attribute != null)

                                                  {

                                                      bool isEnumSet = false;

                                                      if (IsAttributeValueForUpdate(attribute, attributeValue.Value, out isEnumSet))

                                                      {

                                                          SetAttributeValue(attribute, attributeValue.Value, isEnumSet);

                                                      }

                                                  }

                                                  else

                                                  {

                                                      throw new ArgumentException(String.Format(@"Attribute with name {0} does not exists in the requested AF Element ''{1}''", attributeValue.AttributeName, element.Name));

                                                  }

                                              }

                                          }

                                   

                                    private void CheckIn()

                                          {

                                              var afDb = _connection.GetAFDatabase(_vesselID);

                                              if (afDb.HasItemsCheckedOutToMe)

                                              {

                                                  try

                                                  {

                                                      // try to check in with different modes. Sometimes illogical exceptions happen

                                                      try

                                                      {

                                                          afDb.CheckIn(AFCheckedOutMode.ObjectsCheckedOutThisThread);

                                                      }

                                                      catch (Exception)

                                                      {

                                                          afDb.CheckIn(AFCheckedOutMode.ObjectsCheckedOutToMe);

                                                      }                

                                                  }

                                                  catch (Exception)

                                                  {

                                                      // if check in is not successful undo changes to not interfere with other attempt to change

                                                      afDb.UndoCheckOut(true);

                                                      throw;

                                                  }            

                                              }

                                          }

                                  • Re: Parallel CheckIns to AF Database allways fail.
                                    VarbanVarbanov

                                    Hi again. In addition to the previoud post, I'm adding the code for geeting the afDbObject. This is the implementation of code above you can see several times above:  _connection.GetAFDatabase(_vesselID);. Basically the AFDB is allways got from the PISystem object by name, the rest is custom logic. PiSystem object (along with PiSystems) is initialized once and shared through calls:

                                     

                                    public AFDatabase GetAFDatabase(int vesselID)

                                            {

                                                AddVesselInformationToCacheIfNotExisted(vesselID);

                                                AFDatabase result;

                                                if (!_veselConnectionInfosCache.ContainsKey(vesselID))

                                                    throw new InvalidOperationException(String.Format("PI AF not connected. Vessel id {0} does not exist in service cache", vesselID));

                                                else

                                                    try

                                                    {

                                                        result = _piSystem.Databases[_veselConnectionInfosCache[vesselID].VesselAFDbName];

                                                        if (result == null)

                                                        {

                                                            throw new Exception("Af database does not exist in the pi system");

                                                        }

                                                    }

                                                    catch (Exception ex)

                                                    {

                                                        throw new InvalidOperationException(String.Format("Cannot connect to Asset database. Probably configured AFDB for vessel with ID {0}  does not exist.", vesselID), ex);

                                                    }

                                                return result;

                                            }

                                      • Re: Parallel CheckIns to AF Database allways fail.
                                        bshang

                                        Varban,

                                         

                                        Is it possible to consolidate the code into a standalone console application that we can configure/run to reproduce the issue? Although we can take your code and set up our own custom WCF service, there will  invariably be things that are different that would hinder reliable reproduction. For instance, we will need example calls that the WCF clients are making, an export of the AF database, exact web.config configuration, IIS hosting parameters, etc. If it's possible to condense this into a console app, it may be faster to isolate the problem and find out what parts are necessary/unnecessary for reproducing the issue.

                                          • Re: Parallel CheckIns to AF Database allways fail.
                                            VarbanVarbanov

                                            Hi, Barry, thank you for your reply. If you want exactly the same conditions including the AF Db I have to post all service code in console application or not which I cannot do... And besides there is a lot of logic regarding different types of attributes, templates, categories, etc which seem unrelated to the issue. Regarding the service configuration - there is nothing special in it on the one side and on the other side there are custom logic behaviors which have not anything to do with this issue. As I mentioned earlier - default instancing and concurency mode are used. Here is the binding used:

                                             

                                             

                                            <bindings>

                                                  <customBinding>

                                                    <binding name="BinaryHttpBinding" receiveTimeout="00:10:00" sendTimeout="00:01:00" openTimeout="00:01:00" closeTimeout="00:01:00">

                                                      <binaryMessageEncoding>

                                                        <readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647"

                                                                      maxBytesPerRead="4096" maxNameTableCharCount="16384" />

                                                      </binaryMessageEncoding>

                                                      <httpTransport  maxReceivedMessageSize="2147483647" maxBufferSize="2147483647" />

                                                    </binding>

                                                  </customBinding>

                                                </bindings>

                                             

                                            Also nothing special in IIS configuration - you can use Default App Pool.
                                            Regarding the calling code - again - nothing special - just call in Parallel.ForEach the AddOrUpdateAssets method passing several asset

                                             

                                            And here is the exact calling code which I reproduced the issue for the last time (as you can see there is a lot things you do not need, just make sure you have tree levels of assets and at least 300 000 assets in total):

                                             

                                             

                                            private void ImportElementTree(AssetElement element)

                                                    {

                                                        try

                                                        {

                                                            if (_backGroundWorker.CancellationPending) return;

                                                            var parentId = element.ID != Guid.Empty ? element.ID.ToString() : null;

                                                            List<List<AssetElement>> childrenList = new List<List<AssetElement>>();

                                                            int count = 0;

                                                            var breakAtCount = 2000;

                                                            DeepCountChildrenInAssetElement(element, ref count, true, breakAtCount);

                                                             // if total children are less than 1000 remove them, passs object without children and lately call method for all children

                                                            var updateSubElements = count <= breakAtCount;

                                                            if (!updateSubElements)

                                                            {

                                                                foreach (var child in element.SubElements)

                                                                {

                                                                    // copy children to another variable

                                                                    var children = child.SubElements;

                                                                    childrenList.Add(children);

                                                                    // clear sub elemenets avoiding passing too much data to service at once

                                                                    child.SubElements = null;

                                                                }

                                                            }

                                                            WriteAssetElementChildren(element, parentId, updateSubElements);

                                                            // set children again and call the method for each child

                                                            // cannot use parallel requests, because they try to checkin each other

                                                            if (!updateSubElements)

                                                            {

                                                                Parallel.For(0, element.SubElements.Count, =>

                                                                {

                                                                    var subElement = element.SubElements[i];

                                                                    subElement.SubElements = childrenList[i];

                                                                    ImportElementTree(subElement);

                                                                });

                                                            }

                                                        }

                                                        catch (Exception ex)

                                                        {

                                                            AddErrorToCollection(ex, element);

                                                        }

                                                    }

                                             

                                             

                                                    private void WriteAssetElementChildren(AssetElement element, string parentId, bool updateSubElements)

                                                    {

                                                        // call service for all children

                                                        if (element.SubElements != null && element.SubElements.Any())

                                                        {

                                                            _assetsEditorClient.AddOrUpdateAFElements(element.SubElements, parentId, updateSubElements);

                                                            ReportProgress(element.SubElements.Count);

                                                        }

                                                        // set element references

                                                        if (element.ChildRefIds.Any())

                                                        {

                                                            _assetsEditorClient.AddElementsAsReferences(element.ChildRefIds, element.ID);

                                                        }

                                                    }

                                             

                                            private void DeepCountChildrenInAssetElement(AssetElement element, ref int count, bool breakCounting = false, int breakCountingAt = 2000)

                                                    {

                                                        if (breakCounting && count > breakCountingAt)

                                                            return;

                                                        if (element.SubElements == null) return;

                                                        count = count + element.SubElements.Count;

                                                        foreach (var child in element.SubElements)

                                                        {

                                                            DeepCountChildrenInAssetElement(child, ref count, breakCounting, breakCountingAt);

                                                        }

                                                    }

                                             

                                             

                                            So, can you try to reproduce the issue with available information?

                                              • Re: Parallel CheckIns to AF Database allways fail.
                                                bshang

                                                Thanks for the information, Varban. I have a few questions.

                                                 

                                                1) What is a typical value for elementSubElements.Count in the line: Parallel.For(0, element.SubElements.Count, ...

                                                It seems like it would be ~2000 but it's just my guess as an external observer of the code.

                                                 

                                                2) Does restricting the parallel call via ParallelOptions.MaxDegreesOfParallelism (=4 for example) avoid the exception?

                                                 

                                                3) David mentioned above :

                                                Answers to your questions above:

                                                1. It is possible to have to different clients executing under the same user account making modifications to the same object. In this case, your app makes a change to Obj1...

                                                 

                                                From reading the thread, I understand you have a grandparent element GP, 3 parent elements P1, P2, P3, and you are adding child elements (C_a, C_b, C_c, etc). to P1, P2, and P3. Can you confirm this? From David's comment above, it seems the exception may occur if two threads are adding child elements to the same parent (thread 1: P1->C_a; thread 2: P1->C_b), and one checks in before the other. Then the parent element is no longer checked out by the other thread, causing the error during checkin . Would this scenario be encountered in your case? In the UniqueID in the error, what level in the hierarchy does that element appear and is it consistent?

                                          • Re: Parallel CheckIns to AF Database allways fail.
                                            VarbanVarbanov

                                            Hi, Barry, thank you for your help!

                                            1) It is a bit random, but normally the structure is : We have 3 levels (or 4 if we count the parent element) - At first level there is about 100 assets. Each of these 100 has 0-50 children. Each of these children has 0-200 children. So each particular element has never 2000 children. However this 2000 you saw in code is about the count of all children in the tree structure of one element (including grandchildren, etc). Also the hierarchy may contain other random structures, but the most assets are included in the described structure.

                                             

                                            2) When I received the exception for the first time, I received it with MaxDegreeOfParallelism = 2, so it seems it does not help.

                                             

                                            3) "From reading the thread, I understand you have a grandparent element GP, 3 parent elements P1, P2, P3, and you are adding child elements (C_a, C_b, C_c, etc). to P1, P2, and P3"
                                            That is not entirely correct - this was just a sample. You can see the whole picture in the code above. We have a tree structure we want to import and this is done following the logic:
                                            - start from root (which is already written in the db, at first step it is the db elemenent itself) - check count of all element in the tree structure of this element.
                                            - if this count is less than 2000 pass to the service all children of the element including their grandchildren, etc (at first level it is way above 2000)
                                            - if this count is more than 2000 pass to the service only the direct children to avoid possible timeout
                                            - if you have passed only the direct children, repeat step 1 for each of these already written in the step above direct children as roots

                                            In the context of this logic it seems that is not possible that David scenario with the same parent is met (as we have agreed in our further posts) because the parallel calls are in this case always for children for different elements. I.e if you at one pass have written elements P1 , P2 and P3, the parallel calls are “Write children of P1”, “Write children of P2” and “Write children of P3”.

                                            Also, please note , than I don't think the exact structure, elements count, etc. are decisive for getting the exception. I mentioned in the previous posts, but let me clearly state, that I've seen this exception in different occasions including such when it seemed that there were no parallel calls and it was impossible that somebody has checked out elements – at least not anything in the Pi System Explorer (production environment). Although it seems that Parallel calls is certain scenario when this exception occur, I'm actually more concern to know the exact reason for it and to be avoided.  All that has been told up to now was not apparent reason for this exception in our cases.

                                              • Re: Parallel CheckIns to AF Database allways fail.
                                                Rhys Kirk

                                                In the code you posted there is a method "UpdateAttributesOfAnElemenent" but I don't see it ever called in the code posted. Is it not used?

                                                The reason I ask is that if one of your Attributes is a Config Item and you update the value then that would cause a Check Out, and if you have code to check in that Attribute Value change then it would Check In the containing Element which you might be updating on your other code path hence the error.

                                                • Re: Parallel CheckIns to AF Database allways fail.
                                                  bshang

                                                  I think one way to troubleshoot this is to run afgettrace.exe from %pihome64%\AF\

                                                  Capture AF SDK event trace output 

                                                   

                                                  From command line, you can run something like

                                                  afgettrace /Level:Information /PI- /LogFile:C:\trace.txt

                                                   

                                                  or run afgettrace /? to get the full list of options. It may help to use /MaskPID to ignore other AF clients on the machine.

                                                   

                                                  This will show the process id, thread id, and checkin/out info for every object. Then, when the error appears, we will need to scan the logs and determine the checkin/out pattern for that object. Can you run this test and attach the output of the trace to this thread?

                                                • Re: Parallel CheckIns to AF Database allways fail.
                                                  VarbanVarbanov

                                                  Hi!

                                                   

                                                  Rhys, I didn't post this code because I think it is irrelevant and I cannot post all code. I see what you mean, but there is no separate check ins on any AFObjects, including attributes. One checkin is performed at the end of the operation on the database object for perfoermance reasons.

                                                   

                                                  Barry, there is no afgettrace in PIPC/AF folder as stated in your link and the command cannot be started. This is both for 32 bit and 64 bit version. Installed AFSdk is version 2.5.2 

                                                  • Re: Parallel CheckIns to AF Database allways fail.
                                                    matthew.rivett

                                                    I am also receiving this error.

                                                     

                                                    ^Having Previously Logged Exception InvalidO

                                                    perationException^ with ^Message Element '89500101' with UniqueID 'f71b20dd-31c2

                                                    -11e7-80f7-0050568529e0' is not checked out by the requesting user.^ AND ^Output

                                                    s []^

                                                     

                                                    What I have found is that if my templates have analysis this issue occurs.  If I delete the analysis it works.  Were you using analysis?

                                                    Is anyone from OSIsoft watching and can let me know if this is a known issue?

                                                     

                                                    Thanks,
                                                    Matt