We have seen that the primary OPC interface will be connected but we don't see data coming across since the OPC server itself seems to have a problem. Since the connection is still active we won't failover to the secondary. We have to leave Notifications to look for IORATES to drop so we know we have to go and manually fail over to the secondary but this isn't good enough as we now have a data gap for the duration it takes for us to log in and failover.
What if the secondary interface actually buffered a configurable amount of time of data. Maybe we choose 6 hours, for example, and it just operates in a FIFO way so that you always can recover up to 6 hours of data. In practice you would be notified the IORATES was low and then the admin would log in and determine where the interface flatlined (for example, 30min ago). They could then set the recovery time back to 30min and stop the primary interface. The secondary interface now recovers the last 30min from the buffer on the secondary, the collection continues on the secondary interface and there is no data gap. You now can diagnose the issue on the primary interface (aka reboot the OPC server) and fail back to the primary.
It seems that all of the pieces are there: redundancy, buffering. We just need the secondary interface to actually start collecting a little bit of the data, even while the primary interface is running.