Jerome Lefebvre

Collect OpenWeather data using UFL

Blog Post created by Jerome Lefebvre on Jul 19, 2019

Collecting weather is a frequent request, and a little search will reveal on PI Square or GitHub will reveal many examples. What is new in this example is that it makes uses of the PI Connector for UFL's newer functionality to parse JSON data and make requests automatically against a REST api. Below, when I refer to UFL, I mean the PI Connector for UFL version 1.3.

 

 

OpenWeather

OpenWeather requires an API key to make use of any of its functionality. The free tier https://openweathermap.org/price allows access to the current weather. Which is what we will be using in this example. If you are interested in using the ini file below, your first step will be to register an account with OpenWeather at https://home.openweathermap.org/users/sign_up.

 

PI Vision

The goal will be to collect data from multiple cities with UFL doing as much as the work as possible.

(In the graph below, the images of the clouds are shown using a PI Vision custom symbol)

 

API

The first thing we need to understand is the API. As I read the doc https://openweathermap.org/current, I typically use Postman https://www.getpostman.com/ to explore the various functionalities of the API.

 

Here is one example response, we can see several data that we would like to historize such as temperature, humidity, etc. Also, interesting metadata about a particular city such as longitude & latitude and timezone. More importantly, we see something called an id, according to the docs, this is an unique identifier for the city; this makes it a good choice for inclusion our tag names.

The request was made using a city name, but one can use coordinates to request information and once you pick the exact location you want, you can then make use of that id to do further requests.

 

UFL's INI

When dealing with JSON data in general, the ideal would be to use dot notation. That is, the temperature would be access by something like "$.main.temp". Very short and succinct. 

 

Let's see how the equivalent is done in UFL.

The $ represents the complete message, in UFL this is done by the variable __MESSAGE.

The . represents access, in UFL this is done by the JsonGetValue method.

 

Thus, the equivalent to $.main.temp is the following:

main = JsonGetValue(__MESSAGE, "main")
temperature = JsonGetValue(main, "temp")

 

 

Tag and element creation

For any data source, something has to be in charged of data creation. Even as UFL has the ability to create both tags and AF structure, as one wants very fine control over both, it is usually a combination of both UFL doing the initial creation and refinements being done afterwards. In this example, what gets created by UFL will be perfectly usable, the additional AF work will be gravy on top of it.

 

Data type

Now that UFL will create tags, we need to make sure that the tags created are of the correct type. By default all data types are assumed to be strings and in the INI file you need to make sure the string gets cast to the correct data type. In UFL, casting is done using variable assignment. In the above example, the main and temperature variable would have the following types:

 

FIELD(1).NAME="main"
FIELD(1).TYPE="string"

FIELD(2).NAME="temperature"
FIELD(2).TYPE="number"

 

As casting as to be done a lot and strings are the default type, I typically simply uses the following:

FIELD(1).NAME="main"

FIELD(2).NAME="ValueNumber"
FIELD(2).TYPE="number"

 

main = JsonGetValue(__MESSAGE,"main")
ValueNumber = JsonGetValue(main,"temp")
StoreEvent(TagPrefix + "main.temp", "Temperature", TimeStamp, ValueNumber)
DynamicAttributes = Add(TagPrefix + "main.temp")

 

Timestamp and TagPrefix

To use the StoreEvent method above, I need to a timestamp and a full tag name.

In the API, timestamps are provided in unix UTC time (also called epoch time). This is a supported format in UFL, thus we can define out timestamp as:

FIELD(3).NAME="TimeStamp"
FIELD(3).TYPE="DateTime"
FIELD(3).FORMAT="SECONDS_GMT"

 

TimeStamp = JsonGetValue(__MESSAGE, "dt")

 

For the tag information, I want to use the unique id provided in the response message. I also plan to use other APIs of OpenWeather later in the future, such as forecast data, thus I add the name of the API in the tag name as well.

TagPrefix = "OpenWeather." + JsonGetValue(__MESSAGE, "id") + ".Weather."

With this knowledge, 90% of the INI can be written.

 

Meta data

To collect information such as longitude, we can use StaticAttributes, in this case as well, casting is required. If not done, the attributes that will be created will have string type.

 

coord = JsonGetValue(__MESSAGE, "coord")
ValueNumber = JsonGetValue(coord, "lon")

StaticAttributes = Add("Longitude", ValueNumber)

 

Optional values

In the API we can see that the amount of rain will only optionally be sent (that is, only after it has rained). The way that we can handle this is when there is no rain information being collected, we can instead send a value of "No Data" instead. This allows PI Vision and other clients to correctly show when there was and wasn't rain, without interpolation giving the wrong impression to the user.

 

The small technical trick here, is the first response does not contain rain information, we still want a double tag to be created, thus if the rain is null, I will send a dummy value of 248.1 to inform UFL that the type should be a Number.

 

rain = JsonGetValue(__MESSAGE,"rain")
If (rain is NULL) then
      StoreEvent(TagPrefix + "rain.1h", "Rain 1h" , TimeStamp, 248.1, 248)
else
      ValueNumber = JsonGetValue(rain,"1h")
      StoreEvent(TagPrefix + "rain.1h", "Rain 1h" , TimeStamp, ValueNumber)

endif

DynamicAttributes = Add(TagPrefix + "rain.1h")

 

The rest of INI file

The rest of the INI file is simply a lot of copy and paste of the above.

The final information needed is the creation of the country and city elements.

 

StoreElement(CountryName, "Country")

StoreElement(CountryName + CHAR(92) + CityName, "City", DynamicAttributes, StaticAttributes)

 

The data source configuration

UFL can collect data from a REST server, to do so we configure the data source type to be REST

 

To collect data from multiple cities, I will use the UFL_Placeholder to collect have UFL run the same request, but against different city names:

https://api.openweathermap.org/data/2.5/weather?q=UFL_Placeholder&APPID={{APPID}}&units=metric|Tokyo,JP|Montreal,CA|Barcelona,ES|Vancouver,CA|New Orleans,US

 

AF

This is the resulting AF structure that will be built.

 

Future

In the future, I may also go deeper into some additional modifications that can be done on the AF structure built.

Outcomes