AnsweredAssumed Answered

export big data from pi server

Question asked by ElsaHuang on Apr 6, 2018
Latest reply on Apr 10, 2018 by Lonnie Bowling

Hi gurus,


Hope you could spent few minutes to read my questions and do me the favor, please~~~!


our company have encountered a tough problem that, we are using pi server to track/store the Latitudelongitudespeedheight and direction angle real-time data sent by GPS devices, which installed on the automobile.


we do have nearly 28.6TB real-time data (40 million points/tags,  PI version: 2008) separately stored in 3 groups of PI servers, and need to export from pi server as .txt format and do some data clean work, then import into HBase, otherwise the 3rd one is almost full and we have to add the 4th group of PI server to support explosive real-time data, its not a smart working way but we do not have any good solution currently. There were some problems while designing the database at the very beginning, which result into the redundancy of data and it only need one group pi server indeed.


Our architecture estimated it requires 4-5 months to do the export data work by writing a script to read the data 100kb/sec per sub-process, and only 60% success rate. Generally it requires 1-2 years to do the export work of historical data, however it doesnt make sense to spend such a long time.


I believe there must be some more better way to achieve this goal, could any one do us a favor for this? Thanks a lot!