26 Replies Latest reply on Jul 2, 2010 4:43 PM by MichaelvdV@Atos

    Historising Video Streams

      (Random topic)

       

      Has anyone, OSI or partner, ever looked in to a mechanism for storing video streams along side process data in PI?  What I had the vision of is having the video stream stored in a PI tag archive directly and not using annotations, so the actual time the frames of the video are captured, the frames of data are sent and stored at that time in a PI tag.  Then you have a video stream viewer control to watch the stream directly from the PI tag data - all usual video controls (play, rewind, pause), via usual PISDK methods.  Like when you create a slider on a trend to view the data at that time, the video control could would display the frame(s) at that exact time.  Playing the video stream you would see the slider move along the trend that has associated process data displayed as traces.

       

      Imagine you are investigating flaring, you see spikes in a flow meter to the flare.  If at the same time you had the video stream stored, you could watch the actual footage of the flare whilst reviewing the flow meter data together.  (Just an example, possibilities are huge - e.g. production line quality monitoring).

       

      Now data storage is an obvious issue when you talk about video streams but with some clever compression & buffering and the fact storage is relatively cheap (seperate PI archives for video streams via PI Point partitioning) it could work.

      Thoughts?

        • Re: Historising Video Streams
          Asle Frantzen

          I see others have it already: http://www.longwatch.com/in-the-news/pr100109.php

           

          If there was an independent video historian out there, one could probably make something useful with AF - using external data lookup into this other video historian.

            • Re: Historising Video Streams

              OK, I will have a chat with Rich from AR, nice to see they are looking in to this.

               

              Still, I see this could be integrated in to the core PI historian "quite easily" (easy when looking in from the outside ) so wonder if OSI have already evaluated such an enhancement to PI.  I even could see Event Frames playing a role in this when an event frame is generated...to perform a thorough analysis, the relevant video streams could be pulled in, in addition to the data from DR's.

                • Re: Historising Video Streams

                  This is a really interesting idea! Do you have any particular use cases in mind?

                    • Re: Historising Video Streams
                      Asle Frantzen

                      I have one:

                       

                      In my last job (software development for industrial robots) we installed webcameras inside the robotics cells, mainly so that we could give better remote support. In our software suite we also implemented logging of all the robot movements (coordinates and joint degrees) so that we could re-play the scenarios in our 3d graphic simulator/visualizer - in the event of something going terribly wrong (yes - it happened).. hehe

                       

                      One of the plans we had (never did come that far though) was to continously store the last 60 seconds of video stream, and in the event of a system stop we would always be able to play back the last 60 seconds.

                       

                       

                       

                      It would have been a huge advantage though, to be able to go back into all situations and see what happened, with the help of a video historian.

                        • Re: Historising Video Streams
                          MichaelvdV@Atos

                          Intriged by this topic I tried to create a simple client application that takes an .AVI file, and write each frame to an event on a blob tag.

                           

                          It seems the frame (around 40kb) is too large for the blob to contain. I haven't worked with blobs, so I don't know if this can be expanded, but this could be a limitation at this point. So, to continue I take a small portion of the frame to save to the event.

                           

                          Here's the relevant code (.NET 4.0) (not working, see below):

                           public void Upload(string path, string tagName, DateTime startTime)
                          {

                              PISDK.PIPoint point = server.PIPoints[tagName];

                              PISDK.PIValues values = new PISDK.PIValues();
                              values.ReadOnly = false;

                              for (int i =0; i < 50; i++)
                              {
                                  Bitmap bitmap = AviBase.GetBitmap(path, i);
                                  Bitmap second = new Bitmap(bitmap.Width, bitmap.Height);
                                  MemoryStream s = new MemoryStream();
                                  var g = Graphics.FromImage(second);
                                  g.DrawImage(bitmap, new Rectangle(0, 0, 100, 100));
                                 
                                  // g.DrawImage(bitmap, new Point(0, 0));
                                  g.Dispose();
                                  bitmap.Dispose();
                                  second.Save(s, ImageFormat.Png);

                                  values.Add(startTime.AddSeconds(i), s.GetBuffer(), null);    
                              }

                              point.Data.UpdateValues(values, PISDK.DataMergeConstants.dmReplaceDuplicates);
                          }

                          public Bitmap Download(string tagName, int frame, DateTime dateTime)
                          {
                              PISDK.PIPoint point = server.PIPoints[tagName];

                              PISDK.PIValues values = point.Data.RecordedValues(dateTime, dateTime.AddHours(1));
                              var blob = values[frame].Value;
                              MemoryStream s = new MemoryStream(blob);
                              Bitmap bitmap = new Bitmap(s, true);
                              s.Close();
                              return bitmap;
                          }

                          The PIValues collection in the first method gets nicely filled with 50 events. The UpdateValues gives me no exception, but I don't see any values back (in the SMT archive editor. When the download function is called, there are no events found (so it seems I'm doing something wrong)

                            • Re: Historising Video Streams
                              Asle Frantzen

                              Michael @ Atos Origin

                              Intriged by this topic I tried to create a simple client application that takes an .AVI file, and write each frame to an event on a blob tag.

                              It seems the frame (around 40kb) is too large for the blob to contain. I haven't worked with blobs, so I don't know if this can be expanded, but this could be a limitation at this point.

                               

                              Cool! I checked out Steve's blog post What do you want to BLOB (not Blog) about?? and if you see the first comment, even 1500 bytes is too large for a BLOB.

                                • Re: Historising Video Streams
                                  MichaelvdV@Atos

                                  Asle Frantzen @ Amitec

                                  Cool! I checked out Steve's blog post What do you want to BLOB (not Blog) about?? and if you see the first comment, even 1500 bytes is too large for a BLOB.

                                   

                                  So, that is creating a bit of a problem.... I have no other idea as to how to store a frame (about 50 - 80 kilobytes) into an event. Any suggestions? It wouldn't really make sens to store a reference to a file (part) would it?

                                   

                                  I wonder if it makes sense in the PI Server thought.

                                   

                                  Well, at first maybe not IMO. But if you look at the (projected) I/O to PI, it could make sense to store this kind of data... If you look at the individual frames, it is all timeseries based.

                                    • Re: Historising Video Streams

                                      Michael, this is exactly my point.  The PI server in it's current state is not 'optimised' for this type of storage - otherwise others would have already done this in PI .  There should be a seperate point type/mechanism for storing this type of data.

                                       

                                      Video footage of a process or piece of equipment/plant is time series data, which is exactly what PI does.  Those PI server developers can't be very busy so it will give them something to do   I wouldn't say the data would be continously streamed, but batched across in time slots from an interface.

                                       

                                      There are various applications for this, the one I got the lightbulb moment () for this topic was when talking to someone about flaring.  Operators have live footage of flares and then they have their data, they never really have them both together.  If they are reviewing data for the flare they can't view the video footage with the historical data - operators want everything in a single window with 1 click of the mouse!

                                       

                                       

                                       

                                       

                                        • Re: Historising Video Streams
                                          MichaelvdV@Atos

                                          I was unaware that it wasn't possible, so I just gave it a go

                                           

                                          I guess if you have a large enough blob container, you can store the raw frame data. The client and interfaces can do the encoding/decoding.

                                           

                                          Rhys @ RJK Solutions

                                          Those PI server developers can't be very busy so it will give them something to do

                                           

                                          Maybe it's just a mather of changing:

                                          #DEFINE MAX_BLOB_BYTES 1024

                                          into

                                          #DEFINE MAX_BLOB_BYTES 1024 * 1024 * 1024

                                           

                                            • Re: Historising Video Streams

                                              The maximum size of a BLOB value is 976 bytes and this is because of the PI Archives structure: the records within an archive (both Primary and Overflow records) are 1Kb large. That's 1024 bytes by the way, not 1000   The first 26 are reserved for "header data" and that leaves 998 bytes for actual data. Both String and BLOB values can be up to 976 bytes (you need to leave some room for the timestamp and quality bits).

                                               

                                              So it's not just #DEFINE MAX_BLOB_BYTES... it's a matter of changing the internal structure of the archive.

                                               

                                              With that said, there is a relatively easy way to make use of BLOBs to store data that's larger than 976 bytes: simply break it in <976 chunks and store them close to each other (say at 1 millisecond intervals) in a linked list fashion - i.e. you have information indicating whether there's a "next chunk" at the end of a data chunk. I discussed that in this other post and Jeremy Tinkler used the same technique (see his comment on my blog post).

                                               

                                              You need to serialize that data to store it in binary format anyways, so it's not much extra work to add that type of information in there - it's either "there's a next data chunk" (and its location/timestamp) or "that's the end of the file".

                                               

                                              Just thinking out loud here... wouldn't this be an awesome Community Project?? Some sort of "video to PI" client application, along with a player (could be a .NET ProcessBook add-in)...

                                                • Re: Historising Video Streams
                                                  MichaelvdV@Atos

                                                  Steve Pilon

                                                  The maximum size of a BLOB value is 976 bytes and this is because of the PI Archives structure: the records within an archive (both Primary and Overflow records) are 1Kb large

                                                   

                                                  Right, hadn't thought of that!

                                                   

                                                  Steve Pilon

                                                  So it's not just #DEFINE MAX_BLOB_BYTES... it's a matter of changing the internal structure of the archive.

                                                   

                                                  That was 100% meant as a mather of joke in reply to Rhys

                                                   

                                                  Steve Pilon

                                                  With that said, there is a relatively easy way to make use of BLOBs (...)

                                                   

                                                  Brilliant idea! That could work!

                                                   

                                                  Steve Pilon

                                                  Just thinking out loud here... wouldn't this be an awesome Community Project?? Some sort of "video to PI" client application, along with a player (could be a .NET ProcessBook add-in)...

                                                   

                                                  It totally would! Who's in?

                                                    • Re: Historising Video Streams
                                                      Asle Frantzen

                                                      It actually does seem possible then (or 'plausible', to use Mythbuster terminology), to achieve a framerate of 25 frames/second - same as the PAL format.

                                                       

                                                      If you have appr. 1 kb available for the blob, you have 1000 milliseconds in one second - and use 1 millisecond as storage rate, a frame takes appr. 40 kb to store. 1000 / 40 = 25

                                                        • Re: Historising Video Streams
                                                          MichaelvdV@Atos

                                                          Asle Frantzen @ Amitec

                                                          It actually does seem possible then (or 'plausible', to use Mythbuster terminology), to achieve a framerate of 25 frames/second - same as the PAL format.

                                                           

                                                          If you have appr. 1 kb available for the blob, you have 1000 milliseconds in one second - and use 1 millisecond as storage rate, a frame takes appr. 40 kb to store. 1000 / 40 = 25

                                                           

                                                          Does this mean you are in for a community project ?

                                                           

                                                           

                                                          • Re: Historising Video Streams

                                                            Asle Frantzen @ Amitec

                                                            If you have appr. 1 kb available for the blob, you have 1000 milliseconds in one second - and use 1 millisecond as storage rate, a frame takes appr. 40 kb to store. 1000 / 40 = 25
                                                            The 1 millisecond was just an example, you can actually achieve better...

                                                             

                                                            As per this other post of mine, PI supports timestamps with a minimum resolution of 15.26µs - this means you could theoretically store up to 65535 values per second.

                                                              • Re: Historising Video Streams
                                                                MichaelvdV@Atos

                                                                Steve Pilon

                                                                As per this other post of mine, PI supports timestamps with a minimum resolution of 15.26µs - this means you could theoretically store up to 65535 values per second.

                                                                 

                                                                It keeps looking better!

                                                                 

                                                                @Asle: Experiencing the same issue, still working at 00:10 AM on a Friday night, browsing vCampus and reddit to take my mind off! I don't have a lot of spare time at the moment, but that doesn't mean we can't discuss it and maybe begin to start something up!

                                                                  • Re: Historising Video Streams
                                                                    Asle Frantzen

                                                                    Sure, I guess we could start planning something.

                                                                     

                                                                    But I don't want to step on some partner's toes by developing an open source solution competing with their own product - if they should have a product ready for sale.

                                                                     

                                                                    I agree this perhaps should be supported by OSIsoft in the pi server directly (or via a plugin), but many cool features start off as partner developed add-ins, and get integrated as a product feature with the manufacturer after some time. Microsoft as an example - often buys small companies to acquire their technology, and I know a few OSIsoft products as well - which started off outside of OSIsoft offices :)

                                                                     

                                                                     

                                                                     

                                                                    Rhys, did you talk to AR? Do you know if they have a solution for this yet?

                                                                      • Re: Historising Video Streams

                                                                        I have spoken with Rich (Rich Winslow, member in here too), he will try and get some time to comment. 

                                                                          • Re: Historising Video Streams
                                                                            MichaelvdV@Atos

                                                                            I'm not familiair with Rich, is he an authority on this subject?

                                                                              • Re: Historising Video Streams

                                                                                Michael, he is talking about Richard Winslow from Automated Results (which he abbreviated AR) - they pointed out that AR might have a solution along these lines, and they do not want to just start developing something and possibly stepping on some other partner's toes.

                                                                                  • Re: Historising Video Streams
                                                                                    MichaelvdV@Atos

                                                                                    Steve Pilon

                                                                                    Michael, he is talking about Richard Winslow from Automated Results (which he abbreviated AR) - they pointed out that AR might have a solution along these lines, and they do not want to just start developing something and possibly stepping on some other partner's toes.

                                                                                     

                                                                                     

                                                                                    Right, didn't get that. Thanks!

                                                                                      • Re: Historising Video Streams
                                                                                        formerpigeek

                                                                                        This is probably over my head but if you could "bookmark" a video and store the bookmark pointers in PI.  Then you could stream the video in processbook and update the rest of the display according to the bookmarks.

                                                                                         

                                                                                        You might also want to come up with a way of storing short clips around specific events.

                                                                                         

                                                                                        Just storing the whole thing in 1ms 1k records would eat up a lot of archive space fast.  You might be able to reduce that through some time based compression in PI.  But there is also value in storing the entire 60 second clip around an incident.

                                                                                          • Re: Historising Video Streams
                                                                                            Asle Frantzen

                                                                                            I like that idea, David!

                                                                                             

                                                                                            If the bookmark specifies the datetime of the video start time, we could collect the length of the video and then open it at the correct time of an event.

                                                                                             

                                                                                            Maybe the camera could embed a precise datetime to the video stream as well, making it easier for the users while watching.

                                                                                              • Re: Historising Video Streams
                                                                                                MichaelvdV@Atos

                                                                                                Asle Frantzen @ Amitec

                                                                                                I like that idea, David!

                                                                                                 

                                                                                                If the bookmark specifies the datetime of the video start time, we could collect the length of the video and then open it at the correct time of an event.

                                                                                                 

                                                                                                Maybe the camera could embed a precise datetime to the video stream as well, making it easier for the users while watching.

                                                                                                 

                                                                                                 

                                                                                                I also thought about this, but I'm not really sure. If you create a pointer to a file + location, you will still have to access the file. So, the file still has to be accessable via either http, windows share, ftp, <other forms of file sharing>. This will make things more complicated.

                                                                                                 

                                                                                                Another thing: since PI can store multiple values at the same timestamp. Can't we do the 'distribution of the data' on the same timestamp using multiple values?

                                                                                                 

                                                                                                 

                                                              • Re: Historising Video Streams
                                                                cescamilla

                                                                I though of something similar for my car, in case I'm in a car accident, to have the last 60 seconds of playback would be great, having it auto stop when a collision is detected would be even better.

                                                                 

                                                                However, the storage space for this is non trivial an keeping just 60 seconds should not require more than one meg of data storage in a decen quality with a good compresion algorithm and maybe with sound.

                                                                 

                                                                It does make sense when you have something to watch, your robot use case was great!

                                                                 

                                                                Perhaps recording the last 60 seconds of a site before the signals dies would be good, I can think of somethings about the same lines.

                                                                 

                                                                I wonder if it makes sense in the PI Server thought.