AnsweredAssumed Answered

Archive Disk i/o best practice and possible improvements ?

Question asked by KristofDM on Oct 20, 2016
Latest reply on Dec 15, 2016 by KristofDM

I'm currently merging 1 year of data into one file and it will take me 120 Days... (quick math)

Since i have another ssd driven test machine at my disposal, OSI suggested i did a restore of the environment on that machine and do the merge on it.

A good approach but during my battle some questions came to mind.

 

Today we hit  disk i/o issues. Now we moved the current archive file to a separate drive and have the other archives on 2 other disks.

Is there a possibility to have the data archive , upon shifting the archives, move them to another drive ?

 

This would result in having:

writes from our sensors and reads for our near real time applications on one disk (current archive)

Historical queries (analyses / integrator / sdk) on another, maybe even one disk per quarter/year.

 

Another question probably already answered somewhere... 

Is there any performance gain using 1 big archive file that contains a quarter or even a year of data VS multiple smaller files ?

Corruption is not an issue since we backup daily ... and to be honest, we have PI running for multiple years and never had any corrupted files (yet ?)

 

Now i do realize we can trow metal at it and have ssd's all over the place but in case of moving to a cloud solution, this will impact the price.

Having ssd's for older > 2 year data somewhat feels like waste of $

 

Thanks for any feedback and experiences.

 

Kristof De Mey

Lead DBA at Deme group

Outcomes