I'm currently merging 1 year of data into one file and it will take me 120 Days... (quick math)
Since i have another ssd driven test machine at my disposal, OSI suggested i did a restore of the environment on that machine and do the merge on it.
A good approach but during my battle some questions came to mind.
Today we hit disk i/o issues. Now we moved the current archive file to a separate drive and have the other archives on 2 other disks.
Is there a possibility to have the data archive , upon shifting the archives, move them to another drive ?
This would result in having:
writes from our sensors and reads for our near real time applications on one disk (current archive)
Historical queries (analyses / integrator / sdk) on another, maybe even one disk per quarter/year.
Another question probably already answered somewhere...
Is there any performance gain using 1 big archive file that contains a quarter or even a year of data VS multiple smaller files ?
Corruption is not an issue since we backup daily ... and to be honest, we have PI running for multiple years and never had any corrupted files (yet ?)
Now i do realize we can trow metal at it and have ssd's all over the place but in case of moving to a cloud solution, this will impact the price.
Having ssd's for older > 2 year data somewhat feels like waste of $
Thanks for any feedback and experiences.
Kristof De Mey
Lead DBA at Deme group