PI Web API, AF Byte Arrays and Sensor Data

Blog Post created by rborges Employee on Jun 11, 2019



Last week, just after publishing the blog post about how to send data from Node-RED to PI Web API, I got a call from a friend asking me how he could use a very similar setup to send raw binary data from the sensor to PI System and process it there. The answer is surprisingly simple and, after a quick look at several PI Square questions, I realised that this is a common question. So let's see how can we accomplish this in a very simple way, shall we?




The setup we will be using today is very similar to the one on my last post, so I strongly recommend you to give it a look before proceeding here. I will just go deeper into details about the sensor I'm using because we will need this information later on.


For this blog post, I will use my old and faithful BME280, a small temperature, humidity and pressure sensor that has a builtin calibration mechanism. Today we will only retrieve the temperature and calibration information, so we can keep the code simple and readable, but the procedure is pretty much the same for every sensor out there.


Because we will be dealing with binary data, it's important we start by giving a look at the sensor's datasheet so we understand how it's organized and where to get the information we need.



So here it is. Raw sensor data is available from 0xF7 to 0xFE (with temperatures on the first three bytes) and the calibration is a long sequence starting at 0x88 (if you read the datasheet you will see we only need the initial 6 bytes as the rest is for the other readings). The compensation formula is given in Appendix A and we will need to implement it on AF in order to process this data.


Sending the Data to PI


Once again, it's the same configuration from my last post, so I won't waste your time explaining all the details, but here's the Node-RED flow we will be using for this example:

This is pretty simple to follow. We start with an injection node that triggers the flow every five seconds. From there we go to an I2C node that reads data straight out of the I2C bus. Then we make some small adjustments to make it PI Web API friendly and we finally send it through a simple HTTP POST. The big difference here from the last time we did the same thing, is that now the I2C node creates a byte array as our payload:



So how do we POST an array to PI Web API? It's actually pretty simple. Considering that the PI Tag and AF Attribute are configured properly (we will see how to do that on the next secion), the JSON body should simply contain an array as the Value key. Then you POST to the Stream controller's UpdateValue method and you are good to go. Here's an example using Postman:



PI Tag and AF Attribute Configuration


There are two things we must consider in our configuration: the PI Tag type that will be used and the AF Attribute. Let's start with the AF Attribute configuration, where things are straightforward as the engine already exposes native array types. For this demo, we will use a byte array. Here's the config string of my attribute template, already set to tag creation: \\%Server%\%Element%.%Attribute%;ReadOnly=False;pointtype=Blob.




Because the sensor we are using sends data as a byte array, this will be the data type we will use. Keep in mind that it's not uncommon to see sensors sending data as Int arrays.


Now on the Data Archive side of our project, let's address a very common question: how do we store array data in the PI System? How should I configure a PI Tag to store array data? Here's a quote from LiveLibrary:


BLOB is the PI point type typically chosen for an arbitrary unstructured array of bytes. The value of a single event for a BLOB point is limited to binary data of up to 976 bytes in length. Data for larger binaries must be split into multiple events for the BLOB point or the data must be stored as an annotation.


So here's our answer: we must configure our PI Tag as a blob (by the way, blob means binary large object).


Processing the Data


At this point we already have data flowing in, an 8-byte array for our raw data a 6-byte array for the compensation factors:




Now we have to extract meaningful information out of it by implementing some transformations that will be able to convert the binary data into our final temperature value. In order to do this, we first have to check the sensor's datasheet to see how we convert the calibration factors into actual numbers that will go in the conversion formula. We will start with our calibration parameters and here's the info from the sensor documentation:




As I said before, we only need the first six bytes from the calibration information. So we now need to convert the bytes into actual numbers. Also, from this table, we now know T1 is an unsigned short and the other two are signed, so the transformation is simple. Here's how it's done in Python:


dig_T1 = cal[1] << 8 | cal[0]
dig_T2 = cal[3] << 8 | cal[2]
dig_T3 = cal[5] << 8 | cal[4]

if dig_T2 > 32767:
dig_T2 = dig_T2 - 65536

if dig_T3 > 32767:
dig_T3 = dig_T3 - 65536


In order to do that in AF, you have to use a little math because it doesn't offer the bitwise operators available in Python or C. The bitwise left shift (<< n) is equivalent of multiplying your number by 2^n while the binary OR ( | ) is a simple sum. Finally, we have to check if T2 and T3 are above 32767 because this is how signed ints work. This is how our final implementation is in AF (important: arrays on AF use one-based indexing! So to access the first to elements, we will use [1] and [2]):



Now we have to go back to the sensor's datasheet to see how can we convert the raw data into an actual number. Here's the information we need: 



On my Node-RED flow, I'm requesting data from 0XF7 to 0XFE so I don't need to make several requests to the I2C bus. This is important because, on our array, the MSB will be on position [4], LSB on [5] and XLSB on [6]. The Python script that does the bitwise operation to convert it into a decimal number is quite simple:


temp_raw = (block[3] << 16 | block[4] << 8 | block[5]) >> 4


In a similar fashion as before, we convert it to an AF Analysis script by using simple math:




We are almost there! We have all our reading as numbers and we can finally apply the conversion formula available on the sensor's documentation. Here's the C code they've provided:


double BME280_compensate_T_double(double adc_T)
double var1, var2, T;
const double K1 = 1024;
const double K5 = K1 * 5; // 5120
const double K8 = K1 * 8; // 8192
const double K16 = K1 * 16; // 16384
const double K128 = K1 * 128; // 131072

var1 = ((adc_T / K16) - (dig_T1 / K1)) * dig_T2;
var2 = ((adc_T / K128) - (dig_T1 / K8)) * ((adc_T / K128) - (dig_T1 / K8)) * dig_T3;
T = (var1 + var2) / K5;
return T;


This is easy peasy lemon squeezy for AF and I'm sure you will have no problem implementing this logic. Here's my complete analysis, where final is the output temperature in celsius:




Do I need this?


I reckon this is not for everyone. Most of the time we only need the final sensor reading. But some modern sensors and instruments are able to send more meaningful and important data, like maintenance flags, reading status and other parameters that may be useful for some teams, like instrumentation and maintenance.