Hi for an analytics project I have the requirement to bring data from the PI archives of the last X years for 10K tags (in total 80K) into an external data lake (or swamp)
Since this concerns a lot of data to process (a little under 1TB) i am looking for a way to process the archives in firehose mode, instead of the traditional tools ( PI OLEDB (enterprise), PI-SDK )
AFAIK all current solutions except AF-SDK, have limitations and don't cut it in terms of performance, storage, usability . Ideally I am looking for an .NET API that taps *directly* into the archives.
Is anybody in the community aware of a *standard* solution that would fit above case ?
Thanks in advance..
btw: So far I have assessed the following solutions.
|PI OLEDB||multi-hreaded, easy to use||Custom development, no bulk fetches, so archive walk for each tag|
|PI Integrator||multi-hreaded, easy to use||Streams only SQL Server so inefficient storage|
|PI config||simple, crude||works on tag-for-tag basis, single threaded / single core . 70-ties interface, no bulk fetches|
|PI-Datalink||simple||not suitable for massive data loads , no bulk fetches|
|PI-SDK||simple||single threaded, no bulk fetches|
|AF-SDK||versatile, multi-threaded||supports bulk fetches, custom coded solution|