A second thought is to create additional modules within the big exe. Would each module then run as a separate thread on a diferent CPU ?
I don't expect creating additional ACE modules under the same ACE executable to provide any benefit. Basically, scheduled calculations corresponding to all contexts within an executable (which may be coming from different ACE modules) are simply put on the .NET ThreadPool. Following this, its up to the system to schedule these threads - so as to take full advantage of the underlying hardware (multi-core or multi-processor).
Splitting the contexts in multiple executables may help in the sense that each host process would be using lesser memory and would have its own ThreadPool - which increases concurrency to certain extent. However, increased concurrency is only going to be helpful, if you were not reaching very high CPU utilization with the single executable.
Finally, before you explore any of these alternatives, it is important to make sure that this indeed is a loading issue. For most part, you would see "Calc Failed" event written to the output because of some unhandled exception in your calculations. As Han pointed, you should also monitor ACE performance counters to get more insight into this.
I guess I don't have all the answers that you are looking for. But it sure sounds like this is not going to be easy to troubleshoot.
It may or may not be true that server overloading can cause "Calc Failed", but I think there are certain things that you can do to prevent "Calc Failed" being written. Some suggestions can be found in this other post, Something to add on to it is if an exception is thrown in the ACE calculation, we can get "Calc Failed" as well. So perhaps you can have a try and catch in the ACE calculation like:
Public Overrides Sub ACECalculations()
[your ACE code]
Catch ex as exception
' indicate not to write value to PI, avoid "Calc Failed"
OutputTag.SendDataToPI = FALSE
' write exception message to message log
PIACEBIFunctions.LogPIACEMessage(OSIsoft.PI.ACE.MessageLevel.mlErrors, ex.message, MyBase.Name)
To verify if the ACE server is overloaded or not, we can have some performance monitoring. ACE exposes perfmon counters that we can look at like "Number of aborted calculations", "Number of skipped calculations" and "Time to complete calculations". You can probably monitor some of the basic system counters like "% Processor Time". These should let us have a better idea of what is happening to ACE server.
As for your questions about spreading the context to different exe and if modules in an exe are running on seperate threads, I'm afraid I do not have an answer to that.