StreamInsight for Automatic Tuning of Compression Attributes

Discussion created by merighm on Jan 14, 2010
Latest reply on Jan 19, 2010 by spilon

Since StreamInsight can process raw/snapshot data (I mean data that has not been compressed yet), StreamInsight (SI) could be used to help tune (or perhaps auto-tune) PI tags' compression (CompDev) parameters.  The importance of compression is clear to everybody and it was the main topic of the very first talk at the 2008 PI Dev Conf.  This is well explained here:


The  compression  methods  in  process  information  management  systems  (PIMS)  are  needed  to  store  the  masses  of  data  measured  in  large-scale  production  processes.  In  practice,  the  quality  of  the  compressed  data  is  often  unsatisfactory  due  to  inadequate  compression settings. This can lead to significant delays and high cost when production  problems  occur,  because  the  compression  settings  must  be  corrected,  and  more  data  must be collected before the problem can be analysed and solved.

The  reason  for  inadequate  compression  settings  is  the  high  cost  associated  with  the  PIMS configuration, as it is mostly done manually. The piecewise linear compression methods  in  use  today,  like  the  swinging  door  algorithm or  the modified    Boxcar-Backslope    algorithm must    be parameterized  for  each  process  variable  separately.  As  a  consequence,  the  parameters  are often left at their default values, typically 1% of the maximum value. This is often larger than the actual noise variance in the data, resulting in loss of information.


The paper from which I copied these explanations describes a method for adjusting PI compression parameters.  With some knowledge of Wavelet analysis, it should not be too hard to implement in SI.