I am not sure if my idea could solve this problem, but it could be try. When you use third-part add-in in the excel, I do not think the calculation result in the excel form will be the standard data type, like string, double, int or some others. Is that possible to try to convert your results into the standard data type before PI ACE calls them?
Based on the error that you get, I believe that loading the Excel add-in properly is indeed the issue.
One reason that I can think of is that the add-in cannot be found. Have you tried placing into the folder where the ACE 1.x executable calling it is located?
The second reason that I can think of is inappropriate access permissions and to me this appears to be more likely. When debugging through PI ACE Wizard, this happens in the context of the user, you are logged on with; when running the ACE Module through PI ACE Scheduler 1.x the user context is the one set up with the PI ACE Scheduler service. If you are on a development platform, you can try setting up PI ACE Scheduler service to run under the credentials of your log-on user account to verify my theory.
Is the Excel add-in you are using publicly available? What functionality do you use from the Excel add-in? Maybe similar can be done with PI ACE directly.