7 Replies Latest reply on Jul 13, 2010 11:06 AM by MSEConsultants

    PI-ACE sometimes returning "Calc Failed"

    MSEConsultants

      Hi,

       

      I'm new to PI so please excuse me if my question is elementry.

       

      I have set up a test server and I'm using DataLink to populate my "raw data" tags on a timer to simulate reading field data. I have also written a PI-ACE module which is set to read some of the raw data tags, calculate about 15 result values and these get written back to PI output tags. It is set to run every 30 seconds in PI-ACE Manager.

       

      Everything works really well for hours at a time, however on average once a day, the PI-ACE module stops working. All the tags I'm calculating say "Calc Failed". In PI-ACE Manager the current status is still "On". I can go back and recalculate and that does sporadic calculations (ie: not every 30 seconds even though I set the interval to 30 secs) and when I trend the recalculated values it doesn't join up most of the dots to make a trend. I can stop and start the PI-ACE module and it still says "Calc Failed".

       

      If I try to debug the module and step through, it works fine. I thought it could be data related so I've tried stepping through in debug mode a number of times thereby trying to catch it with different data. It always works fine when I debug.

       

      I have tried stopping PI and starting it again, but that doesn't make any difference. The only way I have found to get it working properly again is to stop PI (PISrvStop.bat) and reboot. After reboot everything is back to normal again for a good few hours.

       

      Here is a trend of one of my calculated values to try help explain the problem:

       

      http://www.mseonline.co.uk/temp/PIAceProblem.PNG

       

      Any tips? (thanks )

       

      Flic

        • Re: PI-ACE sometimes returning "Calc Failed"
          hanyong

          Hi Flic,

           

          "Calc Failed" is usually the result of an invalid calculation, I think an often case would be when the tags displays a system state (like "Shutdown", "I/O Timeout") and there is a mathematical calculation involved. I would agree with you that it could be data related.

           

          Did you setup value clamping and bad value substitution for your tags? These can help you to deal with values that are out of valid range, or replace bad values with a valid values.

           

          In your trend, you showed one of the output tags, what's the values of the input tags over the same time range?

           

          Felicity Gibbs

          I have tried stopping PI and starting it again, but that doesn't make any difference. The only way I have found to get it working properly again is to stop PI (PISrvStop.bat) and reboot. After reboot everything is back to normal again for a good few hours.

          This is pretty interesting... when you say stopping PI and starting it, I guess you mean starting and stopping with the scripts here. Since you did try to restart both PI and ACE without rebooting the machine, could it be the data simulator that is running on PI DataLink having a problem here?

          • Re: PI-ACE sometimes returning "Calc Failed"
            mhalhead

            Hi,

             

            Calc Failed normally implies an error in the ACE module/class library not an error in PI/ACE. For example your calculation has a divide by zero or your calculation is using data that does not exist; e.g. the interface is shutdown and you using the points in a a calculation.

             

            What does the pi message log say? are you outputting messages from you ACE class library to the pi message log? I would by writing some meaningful message to the message log and checking the input data when the calculation failed.

              • Re: PI-ACE sometimes returning "Calc Failed"
                MSEConsultants

                Hi,

                 

                Thanks for your replies - very useful suggestions indeed. It takes a long time to replicate the problem so debugging is slow so sorry for delayed reply. I didn't have clamping or bad value substitution configured so I've done that now. I also wasn't outputting to the message log but I'm trying to implement that now. I have put error traps now throughout the code, and just to test I put this at the end of a successful calculation:

                 

                PIACEBIFunctions.LogPIACEMessage( _
                   OSIsoft.PI.ACE.MessageLevel.mlCalculationExecuted, _
                   "Calculation OK @ " & Now.ToString("dd MMM yyyy - hh:mm:ss"), _
                   Context)

                 

                It works fine (ie: code runs without breaking) in debug mode and I implemented it but it doesn't appear in the message log. I'm looking in PI System Management Tools under Operation ... Message Logs ... Server & SDK Logs. I did check everything under "Set Message Log Level" in PI Ace Manager. The logs I am seeing are things like "ACE Context successfully locks mutex" & "ACE Context exits calculations", but these are not messages that I've generated. Am I looking in the wrong place for my messages perhaps?

                 

                Thanks for your help.

                 

                Flic

                  • Re: PI-ACE sometimes returning "Calc Failed"
                    hanyong

                    LogPIACEMessage() does write the messages to the message log. One thing you want to take note, if ACE is on a different machine as the PI Server then the messages are logged to the local message log instead of the PI Server's message log. In this case, I would run pigetmsg.exe (it should be in either \pipc\adm or \pipc\bin folder) to print out log messages from ACE. If both ACE and PI are on the same machine, then you should be fine.

                     

                    Other things to consider are like restarting the ACE executable after making changes and compiling the new dll. ACE automatically creates a copy of the dll (naming it <dllname>_aceshadowcopy.dll) and use that for running the calculation. Something to take note is when you restart the calculation, you should ensure that calculation is completely stopped before replacing the dll (that is, the corresponding PIACEClassLibraryHost no longer appears in the task manager). If the shadow is still in memory when the calculation starts, the old shadow copy will continue to be used.

                     

                    Or you can try this:

                    1. Stop the executable in ACE Manager
                    2. Verify that the executable has been stopped. The PI message log for the host server should show a termination message for each class module, followed by a termination message for the executable:
                      "0 PIACEClassLibraryHost.exe ,15-Jul-04 09:42:56,(Server)\(executable name) terminated"
                    3. Delete the aceshadowcopy of executable DLL in C:\Program Files\PIPC\ACE\ClassLibraries\"executable name"\bin
                    4. Start the executable using the PI-ACE manager (as well as the context(s))

                     

                    • Re: PI-ACE sometimes returning "Calc Failed"

                      By default only Error messages are sent to the message log from PI ACE; you have to manually enable additional log levels/categories from the PI ACE Manager. In your case you selected "mlCalculationExecuted", so you want to make sure the appropriate checkbox is checked in the "Set Message Log Level" dialog - you can edit logging settings by right-clicking on the desired context in PI ACE Manager and selecting Message Log Level...

                      • Re: PI-ACE sometimes returning "Calc Failed"
                        michaelh

                        LogPIACEMessage's third parameter is a bit tricky : the Context variable works for mlErrors level, but Name is the correct full path name required for mlUserMessage and other levels.

                         

                        Not sure if it's a confirmed ( or fixed ? ) bug or just something to show the value of an ACE Training class

                         

                         

                         

                        MyBase.Name is the full context name including server\executable\module\contextpath , MyBase.Context is just the contextpath.