Individual Collective Members Health (data matches with other nodes)

Idea created by mbailey on Oct 3, 2018

    We would like a way to know if a node in a collective is not receiving the same data as the other nodes. It's my understanding that all nodes in the collective receive very similar data. The PI collective node members should be able to talk with each other to know if one node is not getting similar data.
    At times, a node receiving data from PItoPI or other methods, may not be getting data. The data could be buffering (a single path), hard drive space may have ran out on the PI server,  or a network issue may stop the individual PI server from receiving data.
    Many times, we don't catch this if the other nodes of the collective did receive the data. Whether it backfills in or not, we would like to know the health of the collective and have this as an indicator a node is not healthy.
    To add, Programs like the WebAPI or even ProcessBook don't know if data is not correct on a collective member. If there was a way to capture if a node is not healthy, it should filter into the other programs accessing the data, and steer "clients" away from using that node, as if the node were offline.
    I have a hard time with a HA architecture when a node in collective may not have accurate up to date data, and a client connected to it doesn't know, therefore providing bad data to a client or other services polling the data.