AnsweredAssumed Answered

Apache Spark's Structured Steaming with PI Web API Channels

Question asked by BoSun on May 23, 2018
Latest reply on May 24, 2018 by tramachandran

Has anyone tried to connect PI Web API Channels with Apache Spark's (i.e. Databricks) Structured Steaming (Structured Streaming Programming Guide - Spark 2.3.0 Documentation ) successfully?

We're thinking to use Databricks for all PI data ETL process which is currently done by a Python program, but I cannot figure it out how to feed PI data (via Web API Channels) to Databricks streaming.

I found OSIsoft published a Python client for PI Web api (GitHub - osimloeff/PI-Web-API-Client-Python: PI Web API client library for Python generated using the Swagger specificat… ). Not sure if I could leverage this library with Databricks' streaming.

 

Much appreciate if anyone could shed some light on this topic.

Outcomes