zzoldan

Sending public JSON data to the PI Connector for UFL silently and continuously

Blog Post created by zzoldan Employee on Aug 19, 2016

Sending public JSON data to UFL Connector Silently

This post will discuss sending public JSON data to the PI Connector UFL, and the steps which can be taken to run this process continuously in the background. Please have a look at Jerome Lefebvre’s post on sending pubic JSON data to the PI Connector for UFL.

 

This post is based on a summer internship project, undertaken at the OSIsoft Montreal office. For more details, check out "Monitoring Smart City Assets with the PI System"

 

 

Running the GET/ PUT script continuously

When reading data from a live source, there is often a need to read data from the source at a regular interval. This means the Python script must be called at a specified interval, such as every 10 minutes. I found this was best done by creating a batch file which calls the Python script repeatedly, and then adding a 10-minute sleep command within the Python script. The result is a script which is executed once every 10 minutes continuously.

The Python script is called within a batch file in order to pass necessary arguments (Connector’s REST endpoint location and data page URL).

 

The batch file is as follows:

 

:begin
py putjsondata_BIXI_service.py "https://YUL-SRV-INT01:5460/connectordata/REST/" "https://secure.bixi.com/data/stations.json"
GOTO begin

 

Once the batch file is launched, a command prompt window will open showing the status of each time the Python script was run. The Python script is set to print a message to the terminal window confirming if the script was run successfully, or if any error occurred.  It may show “data sent successfully over https” or “sending data failed due to error 4XX”.

 

Running the Python script as a windows service

When we wanted to migrate this script from our development to production environment, one of the requirements was that the script had to be running in the background and not under a single user account. This was first accomplished by running the Python script as a windows service. NSSM (Non-sucking service manager) was used to take the launcher batch file and implement it as a windows service.  NSSM is a service helper, created by Iain Patterson – more information on it can be found here.

 

Calling NSSM from the command line will bring up a GUI allowing us to configure the service. We can customize our service to run under a specific user account, add startup dependencies and more.


 

Writing the output to the Windows Event Logs

Since our Python script is now running as a windows service in the background, we can’t rely on the command prompt window to see whether it is functioning properly.  We created a workaround for this by writing to the windows event logs. This is possible thanks to the Win32API Python library, which allows direct access to the windows event logs.  A basic outline of the script is available here.

 

We can include statements to open communication with the  windows event logs before executing GET/PUT requests:

ph = win32api.GetCurrentProcess()
th = win32security.OpenProcessToken(ph, win32con.TOKEN_READ)
my_sid = win32security.GetTokenInformation(th, win32security.TokenUser)[0]
applicationName = "UFL_Service"
eventID = 1
category = 5    # Shell

 

Then, in the section of our script where we handle printing the status to the terminal, we can write to the windows event logs. This method must be embedded in the Python GET/PUT script which can be found in Jerome Lefebvre's blog post. The print message is hard-coded in each Python script. In our case, each Python script dealt with a different data source, so the print message had to be changed for each. The one below deals with data from Philadelphia's public bike sharing systems, Indego.

 

if response.status_code != 200:
    print("Sending data to the UFL connect failed due to error {0} {1}".format(response.status_code, response.reason))
    desc= ["Sending PHL Bike data to the UFL connect failed due to error {0} {1}".format(response.status_code, response.reason)]
    data = "Application\0Data".encode("ascii")
    myType = win32evtlog.EVENTLOG_WARNING_TYPE
    win32evtlogutil.ReportEvent(applicationName, eventID, eventCategory=category,eventType=myType, strings=desc, data=data, sid=my_sid)

else:
    print('The data was sent successfully over https.')
    print('Check the PI Connectors event logs for any further information.')
    desc = ['The PHL Bike data was sent successfully over https.']
    data = "Application\0Data".encode("ascii")
    myType = win32evtlog.EVENTLOG_INFORMATION_TYPE
    win32evtlogutil.ReportEvent(applicationName, eventID, eventCategory=category,eventType=myType, strings=desc, data=data, sid=my_sid)

 

Instead of printing status to a command prompt window, the script will now print to the windows event logs. The logging level has been set to WARNING ( myType = win32evtlog.EVENTLOG_WARNING_TYPE) when the script encounters an error, and INFORMATION (myType = win32evtlog.EVENTLOG_INFORMATION_TYPE) when the script functions properly.

 

 

 

Alternative: running the Python script using Windows Task Scheduler:

Instead of running the Python script as a Windows service, we can call it at a regular interval using Windows Task Scheduler. In our case, certain arguments must be passed to the Python script every time it is run, so this is included in a batch file which calls the Python script with the  necessary arguments.

 

There are several advantages to using Windows Task scheduler vs a Windows Script. We can set the application to run at a specified interval or a specific event – on computer shutdown, sleep, etc. In addition to our Event
logging present in the Python script, Windows Task Scheduler will give us a confirmation of whether it was run successfully or what error it encountered. Windows Task Scheduler can also be configured to retry the script a specified number of times if an error was encountered.

 

Below is an example of our use case, we call a Batch file which reads data from six different websites and then sends them all to the PI Connector for UFL via REST endpoint.

 

 

py putjsondata_BIXI_service.py "https://YUL-SRV-INT01:5460/connectordata/REST/"  "https://secure.bixi.com/data/stations.json" 
py putJSONdata_Bos_Bikes_service.py "https://YUL-SRV-INT01:5460/connectordata/Bos/" "http://feeds.thehubway.com/stations/stations.json" 
py putjsondata_NYC_service.py "https://YUL-SRV-INT01:5460/connectordata/NYC_Live/" "https://gbfs.citibikenyc.com/gbfs/en/station_status.json" 
py putJSONdata_phl_service.py "https://YUL-SRV-INT01:5460/connectordata/PHL_Live_Bike/" "https://gbfs.bcycle.com/bcycle_indego/station_status.json" 
py putjsondata_sf_bikes_service.py "https://YUL-SRV-INT01:5460/connectordata/SF_Live_Bike/" "http://www.bayareabikeshare.com/stations/json" 
py putJSONdata_To_Bikes_service.py "https://YUL-SRV-INT01:5460/connectordata/Toronto_Live_Bike/" "http://feeds.bikesharetoronto.com/stations/stations.json"

Outcomes