rborges

PI Web API and Node-RED

Blog Post created by rborges Employee on May 21, 2019

1. Introduction

1.1 What is Node-RED?


Before going to the juicy bits of this blog post, let me start by explaining what Node-RED is for those who are not familiar with this tool. Quoting their own website, Node-RED is a flow-based programming tool for the internet of things. What this means, in simple English, is that you can drag and drop functional boxes for wiring together devices and online services.

 

This is what a Node-RED flow looks like:

A group of nodes is called a flow. Each node is responsible for manipulating an object called message. In this flow, Node-RED is listening for UDP packets and, based on the destination port, it can either pull high GPIO 15 or ground GPIO 13 while sending an HTTP request to a given API (and keep trying while not succeed).

 

Another selling point is the community support. Being an open source project (Apache 2.0 if you are curious), community engagement has skyrocketed and you can find thousands of different nodes: from OPC UA and Modbus nodes to Instagram and other social media platforms. From FFT to sentiment analysis. So most likely you won't even need to write a single line of code.

 

1.2 Why should I use it?

 

Rafa, usually graphical programming is just a dumbed-down version of actual programming. Why should I use it if I can write my own application? Because it's easy and fast. As a software engineer, I agree that a tailor-made firmware in C is more reliable and efficient. But can you create one in seconds? Can you easily include new features or make changes to it? This is an old debate where, on one hand, we have practicality and, on the other hand, robustness.

 

Let's just remember that we are talking about IoT devices, frequently located outside the boundaries of the main network and sending data through a non-wired channel. Unpredictability and unreliability are a given and your logic can go easily go sideways with an unforeseen condition. So would you prefer to update your code through a web interface or going on the field with a computer and a USB UART dongle to reflash your firmware?

 

1.3 Caveats


Everything sounds great, but it's not all roses. Although it has a very small footprint, you can't deploy Node-RED on small microchips (ESP8266, if you are wondering) or limited hardware like Arduino. Right now, the only "IoT hardware" capable of running it is Raspberry PIs and Beagle Bone Black. This poses a challenge if you are considering deploying the platform on the field, but there are alternatives if you need to use minimalist hardware.

 

Another caveat is the engine itself. Its capabilities are as big as the capabilities of the device hosting it. If you deploy it on a small Raspberry PI, don't expect the performance of deployment on a full-blown server. It may sound like an obvious observation, but because IoT devices can scale up easily on a mesh network, sometimes we forget that the host doesn't scale up that easily. So, your mileage may vary when it comes to performance.

 

1.4 Architectural Considerations

 

In today's example, we will have multiple sensors sending data directly through an access point (a Raspberry PI 3 B+ in our case). A rough representation is this:

 

For the geeks out there who are curious about the setup I'm using, here's more info: I'm using BME280 for temperature, humidity and pressure and a DS1307 for real-time clock. All of them are using the I2C bus to send data through an ESP-01 breakout. The ESP8266 is running a custom firmware that creates a mesh network. The Raspberry PI host is connected to the main corporate network through its own wi-fi module, but it's also connected to the mesh network through its own ESP-01 breakout. Each sensor is sending its data to specific MQTT topics on a Mosquitto broker running on the same Raspberry PI. I have three sets of sensors: one is sitting on my desk, the second one is at the office's kitchen, and the last one is in a meeting room, about 20m (65 ft) from my desk.

 

 

Although the hardware side is not related to PI, leave a comment below if you would like to hear a little bit more about what I'm doing. By the way, this is a preparation for an ongoing project, where a more professional looking version this will eventually be deployed around our office here in London.

 

Keep in mind that this is not a suitable production-grade architecture as there is no data buffer, no redundancy, no scalability, and no fail-safe mechanism. If you need any of these items (and you need if data is critical to your operation!), you should consider giving a look at EDS (Edge Data Store), our answer to data management on the edge (watch this presentation about EDS, you won't regret it).

 

2. Sample Flows

 

2.1 Reading Values from the Sensors

 

This is not a Node-RED tutorial, but let me just show how I'm capturing the data. After all, this is how everything starts. As I mentioned before I'm running an MQTT broker and each sensor data has it's own topic. Because I like to sort my data by data domains, my topics are the following:

 

DeviceMeasurementTopic
Desk SensorTemperatureoffice/temperature/desk
Desk SensorHumidityoffice/humidity/desk
Desk SensorPressureoffice/pressure/desk
Kitchen SensorTemperatureoffice/temperature/kitchen
Kitchen SensorHumidityoffice/humidity/kitchen
Kitchen SensorPressureoffice/pressure/kitchen
Meeting Room SensorTemperatureoffice/temperature/meetingroom
Meeting Room SensorHumidityoffice/humidity/meetingroom
Meeting Room SensorPressureoffice/pressure/meetingroom

 

This allows me to easily get to all temperatures by subscribing to office/temperature/# or get all office data by subscribing to office/#. I can also get only my desk data by subscribing to office/+/desk. Here's an example of how to get the data using an MQTT node:

Dead simple, right? The green node is a debug node that outputs all the content of the payload. Here's the output for a single topic. Keep in mind that we receive several messages like this (one for each topic we are subscribed to).

 

 

Because I'm trying to keep things as streamlined as possible, the sensors are already streaming an output that is pretty much what we need to send to PI, including the AFPath where we are sending data to. On my custom ESP8266 firmware I have this hardcoded:

 

const String baseAFPath = "\\\\RBORGES-AF\\IoTDemo\\Rafael Desk Environmental Data|";
const String tempAFPath = baseAFPath + "temperature";
const String humiAFPath = baseAFPath + "humidity";
const String pressAFPath = baseAFPath + "pressure";
const String dpointAFPath = baseAFPath + "dewpoint";

 

In a more real-life scenario, instead of a hardcoded string, you should do something like this:

 

const String chipID = ESP.getChipId();
const String tempTopic = "office/temperature_"+ chipID + "/desk";
const String humiTopic = "office/humidity_"+ chipID + "/desk";
const String presTopic = "office/pressure_"+ chipID + "/desk";
const String vccTopic = "office/vcc_"+ chipID + "/desk";
const String ipTopic = "office/ip_"+ chipID + "/desk";

 

Then, on your Node-RED server, you would correlate the chip ID with the attributes you are sending data to. That has the extra benefit of easy maintenance, in the case of you moving the sensor to a different location. There's no need for you to rewrite the firmware, just update the reference table.

 

By the way, talking about real-life scenarios, if you are using the PI System infrastructure, you should consider OMF as your format standard, in the near future, you will be able to send OMF data right away to PI Web API. This will free you from the transformations we have to do in the next section.

 

2.2 Sending Values to PI


Now that we have the data, we need to send it to PI. As the blog title suggests, we will use PI Web API as our data entry point. In order to do this, we have to make some transformations on our data. First, we have to add the WebID for the attribute we want to write data to and then we have to execute an HTTP POST with a JSON body containing the data itself.

 

The first thing is to get the WebID. Using Web ID 2.0, it's actually pretty easy to encode the AF path into a valid Web ID. I strongly suggest you Christopher Sawyer's excellent post on how to encode, decode, and some basic concepts behind it.

 

Going back to Node-RED, in order to encode the path into a valid Web ID, we have to execute custom code. This is easily done with the function node, where you can run any arbitrary javascript code. It exposes the whole message as a plain JavaScript object and allows you to manipulate it in a programmatic way.

 

path = msg.path
if (path.substring(0, 2) == "\\\\") {
path = path.substring(2);
}
var encoded_path = new Buffer(path.toUpperCase()).toString('base64');
var count = 0;
encoded_path = encoded_path.replace('+', '-').replace('/', '_');
for (var i = (encoded_path.length - 1); i > 0; i--) {
if (encoded_path[i] == "=") {
count++;
}
else {
break;
}
}
if (count > 0) {
encoded_path = encoded_path.slice(0, -count);
}
msg.webid = "P1AbE" + encoded_path
return msg;

Do you get the idea? I'm just getting the path variable from our message, encoding in base64 and concatenating with "P1Abe". The P1Abe WebID header means it encodes the path and refers to an AFAttribute that is a child of an AFElement. Once again, if you have not checked Chris' blog post, stop reading this article now and go read it! A final note for those JavaScript nerds wondering why I'm not using btoa(). Node-RED, as the name suggests, run on Node.js and it doesn't expose btoa() / atob().

 

At this point, this is our message object:

 

Our payload is ready and we have the Web ID encoded, so we are pretty much ready to send the data to PI. To do this, we now need the HTTP Request node.

 

It works by getting the payload content and sending it as a JSON body to a given URL. Here's the configuration for our example:

We define it as a POST to the Stream Controller Update Value Method, we set the URL, I enable SSL to properly handle the SSL certificate and finally, I select basic authentication. Let me just call your attention to the URL I'm using https://rborges-af/piwebapi/streams/{{webid}}/value. See the {{webid}}? It's a template system called Mustache Notation. It allows us to get a value from the message object and use it to feed the template.

 

Here's the full flow:

A recap: we subscribe to an MQTT topic, we convert the string into a valid JSON, we do some housekeeping by moving the path information out of the payload, we use the path to encode our WebID and we finally post the data. I added the debug node as an output of our request, so we can get the HTTP Response object and see if it's everything working properly. Here's the output:

 

 

The 202 status code on our response means that the data was sent. We can now see it on AF:

 

 

2.3 A Periodic Calculation Engine

 

Another cool thing that we can do with Node-RED, is to deploy flows that are triggered periodically. The Inject node allows us to not only inject an arbitrary JSON but also do it in a regular fashion. Here's an example. We first inject a JSON with two important tags that will be used for a given analysis:

 

 

Then we configure it to do it periodically. You can define a frequency (e.g., every 10 minutes), an interval between times (e.g., every 10 minutes between 8 AM and 3 PM), or at specific times (every noon on weekdays). For our example, every minute.

 

Now we can use it as the start trigger for our flow.

 

In this example, we are injecting the JSON every minute, we split it to get data from the PI Web API for each PI tag, we join the messages into a single message, we pass it through a generic splitter where we prepare it for FFT and the result we send it to our maintenance database. I'm using our beloved CDT158 and SINUSOID, but it could easily be vibrational data so we can log vibration information from a maintenance perspective.

 

Keep in mind that the inject node is just on way to do it. There are plenty of other ways for you to trigger a flow. Another possibility is to listen to multiple tags and start it only when a condition is met. I helped a customer a couple of years ago to wire up some electronics and trigger a flow when a door was opened and closed. This was used to log on PI when the lab door was open.

 

3. Conclusion

 

3.1 The PI System and IoT data

 

Today we saw how easy it is for you to wire up sensor data with PI, using nothing else than PI Web API requests. It's simple, cost-effective and fun to execute. I actually use this same architecture for my home automation system with PI (let me know in the comments if you would like to know why I use PI at my home).

 

But let me stress once again that this is just a proof-of-concept. An enterprise-grade project would never ever send data directly like that as it's a security breach and the lack of buffer makes it very unreliable. Once again give a look on EDS if you need to send edge data to your PI System.

 

3.2 Reference Material

Outcomes