I decided to pick up my efforts in integrating the PI System with the Kinect sensor again. I'm taking a bit of a new approach to this. My first step is to create a Kinect to PI interface. This 'interface' should be able to store all skeletal data (joint positions) in the PI System.
For this I'm using the new AF SDK 2.5 CTP. All is going reasonably well. I'm able to smoothly write all skeletal data. By the way, it's pretty fun to consider your body and joints as assets in AF . The skeletal data flows in at 30 fps, with about 20 joints with 3 coordinates. That's 1800 values per second. That's no issue for the PI System or the AFSDK CTP.
Next challenge I set for myself is to be able to store the depth and RGB camera information. This is a bit more tricky. First I will focus on the depth information.
Now, I know that the PI Archive was never intended to store this kind of information. But, I think it's always good to look at new ways to use the PI System. PI AF is part of the PI System, and PI AF offers quite some extensibility and programmatic options, so there has to be a nice way to do this.
I'm not looking for a production-grade solution, more of a lab/research solution, that can run and operate in a lab/demo environment.
The depth information comes in different resolutions. You can decide what depth frame resolution to use in your application. For this purpose, I'm using the resolution of 320x240 depth pixels. Other options are 80x60 and 640x480. All resolutions are 30 frames per second.
This comes down to a byte array of 76800 bytes per depth frame. In an optimal situation this will be 30 x 76800 bytes per second. That's 2304000 B p/s, or 2.223MB p/s. That's 133 MB per minute.
I have considered different options:
Storing as blob data in PI tags
For me this was an obvious first choice to try. If I take a blob tag, and write the frames as blob values. Now -- as you may know -- a blob event has a limit of 1024b per event. This means I have to spread the data over multiple events (about 150 per frame). I can store these as duplicate values, or as values with a fraction of time in between (say, 1 ms). That's 4500 very large values per second. If you consider that a blob value this large counts for about 100 int32 values in the archive, and that I'm already writing 1800 values for the skeletal data... This could be an issue on my little laptop.
This has proven to work, but it is dead slow considering you have write that large amount of data. The PI Archive was never intended to store these kind of values in such a rate, but it does work! It's not pretty, and my framerate drops to about 10 fps for the depth information. The advantage of this is that you can use standard AF PI Point DataReference to retrieve the values. Still, not pretty, and not too performant.
Storing as a varbinary in SQL Server
First I looked into storing the data in an SQL Server table, linked to an AFTable. This did not prove to work (I was not aware that linked AFTables are readonly). So, I thought I could write them directly to a SQL Server table (name, timestamp, blob). I'm using varbinary(max) for this, and the new FILESTREAM datatype in SQL Server would also be an option.
The upside to this would be that the data would be available across the network (using OLEDB or the SQL Native Client), and it would be easy to write a small AF DataReference to retrieve and store the frames.
I got some decent performance out of this, with about 15- 20 fps on a laptop. The issue here becomes the memory consumption of my application and of SQL Server, and I quickly encountered OutOfMemory exceptions after less than a minute of 'recording'. I did some optimizing, but I fear that the managed nature of .NET does not allow me to have much tighter control over memory consumption. I guess that using some unmanaged parts would be a solution, but I didn't want to go there yet.
Storing in a file, with stream positions in PI tags
This was my backup option: not the nicest one, but almost sure it would work. The idea is simple: just append the binary data to a file, and write the positions to PI tags. This way, it will be easy for a AFDataReference to retrieve the positions in the file from PI, and read the frames using a filestream. This is (obviously) the most performant option yet. It just flies. One disadvantage I see here, is that I cannot let my datafile grow beyond 2gb, or the position index (int32 in PI) will overflow. So, for larger files I have to have some paging mechanism. That will probably mean a second PI tag with the filename/fileindex.
Downside to this: it is not available across the network (local files). You could share the files, or have some web service in between. Still, the files grow really fast (to hundreds of megabytes in a few minutes).
The main reason I'm writing this is to share my experiences, and maybe discuss different options. My next option would be to use some sort of compression (gzipstream) to compress the imageframes before I store them. I have yet to see the performance (really want to be able to keep that 30 fps). Do you see any better or other options?
Again, this is not a day to day use of the PI System. But, I think it's good to look for more unconventional uses of a system from time to time. For me it is also a great opportunity to test out the new AF SDK CTP, and get more insight into performance options.
I will make sure I will blog more about the why I'm connecting the Kinect sensor to the PI System, and how I do it in a little while