8 Replies Latest reply on Mar 2, 2010 7:23 AM by oshafie

    PI Point Database


      I am in the process of building out a new set of PI servers and will need to bring the historical archive data over form the old ones.  We have been using PI for a long time and it has been upgraded (including from 32bit to 64bit) and moved from server to server.  I would really like to start with a fresh build but need to keep the point database in tack. 


      I asked tech support if it is possible using some advanced secret technique to create PI tags and specify a PointID and RecNo but have been told it is impossible.


      The only other idea I have it to only bring over the point database and not a full PI system.  Has anyone done this before?  I talked to some OSIsoft employees at vCampus Live and they said this should be possible but if we have any point level security it could cause issues because the users would not exist.  As far as I know our points are using default security so this shouldn't be an issue.


      Has anyone done something like this before?  Any ideas?



        • Re: PI Point Database



          I wonder why you are concerned with the moving process. Anyhow, beside the point database you will have the ModuleDB, the digital state sets, the security settings like trust, users, groups, point classes, and many other things that I am probably missing.


          Probably you want to post more information: are you going to split the server into multiple ones? join them to multiple ones?



            • Re: PI Point Database

              Maybe I am just paranoid; we have been using PI since 2001 and recycling these files each time we upgrade.  A lot has changed over the years and I'm sure there is junk in them that doesn't need to be there. 


              We are basically going to have 2 full PI systems running in parallel with two different data sources.

                • Re: PI Point Database

                  that's just 8 years - we have people updating, moving, migrating the PI system since more than 25 years


                  Your files should be clean - probably the archives carry some junk if you have done a lot of tag deletion, but that would need time consuming work (archive reprocessing). Other sources for data that could be obsolete would be all deletion you have done (users, digital states, digital state sets).


                  Let's see what the community has to say but I seriously would not consider some "traces" of history in the PI System a reason for going through that hassle and I am absolutely sure that they do not cause any issues on the performance or reliability of the PI System. If you are realy a bit paranoid - you could check the archive integrity, but this comes back to archive reprocessing and than you could move all the data with a new & clean point database to a new and clean PI Server (you can force the pointID when you create a tag, and the reprocessing of the archive would than match the pointid with the new RecNo).





                    • Re: PI Point Database

                      1453 archives totaling over 1TB of data.  No I do not want to even think about reprocessing archives...  Maybe I shouldn't be worrying. :)

                        • Re: PI Point Database

                          Having a temporary, dedicated PI Server just to do that archive reprocessing would definitely help here (that's our recommendation anyways) ;) Honestly, it wouldn't be a bad idea to start reprocessing/reorganizing those archives... it might end up simplify the administration and backup burden by having a smaller number or smaller archives - especially if you did a lot of tag deletion/creation over the past 8 years.


                          As far as the other files you seemed to be worried about when reusing an older backup... well as Andreas pointed out, there shouldn't be that many of these obsolete/orphaned files and they definitely won't be hurting anything. If there's anything in particular you are worried about, I would definitely  suggest you route this through tech support and get the appropriate mentors/developers involved.

                        • Re: PI Point Database

                          Andreas is correct.  There is really no reason to think that the configuration database files are in some respect "suspicious" just because they have been through a large number of PI system upgrades.  The actual structure of these files doesn't change with every revision to PI, so they probably haven't really been "upgraded" as many times as it would appear.


                          Regardless, the way you'd have to go about what you suggest is to build an entirely new PI server.  You would then have to manually create everything that you wanted to have on this new system.  Much of it can be copied from the old system, for instance use can use Excel and the SMT add-in to extract the point definitions from the old server and create them on the new.  Likewise for the MDB definitions.  Other things like the security settings, ACE context definitions, tuning parameters, etc. would all have to be done manually using PI-SMT tools.  There's no way to verify that you've got everything - it's trial and error.


                          Then comes the real work.  All of your archive files would have to be reprocessed to get them aligned to the new PointDB.  8 years worth of data at your data rates is going to require a monster server (or several servers), a lot of leg work, and significant time to grind through all of the archives.


                          Once done with all of this you’d have a completely "clean" PI setup, which will be almost indistinguishable from the one you have now...

                            • Re: PI Point Database

                              Alright I'm convinced.  I knew all these other methods would have been a lot of work.  My real hope was that you could create a tag and specify a PointID/RecNo.

                                • Re: PI Point Database

                                  When you delete a tag, it is removed from the pipoints.dat (points file - managed by PI Base Subsystem), piarcmem.dat (snapshot table - managed by PI Snapshot Subsystem), and the primary archive (managed by the PI Archive Subsystem).  For performance reasons, data in older archives for a deleted tag is not deleted unless you reprocess those archives.  We delete all the data in the primary archive so that we can efficiently use all the space if someone creates / deletes lots of points.


                                  It may be an interesting exercise for some folks to do an archive walk (piartool -aw) for a given recno to see the various point ids in effect at different points in time (aka in different archives).


                                  The space vacated by a deleted point will get re-used as new points / data are created.


                                  PointID and Recno are unsigned 32-bit integers, which is 4+ billion numbers, so even if you were creating / deleting a bunch of points per day, you have quite a lot of numbers to consume.


                                  So like others, I really do not understand the concern here.  PointID gaps mean nothing; recno gaps get re-used; and file space gets re-used.


                                  In PI HA, when we replicate a point creation from the primary to the secondary, we actually are specifying the point id and recno for the secondary based on the values from the primary.


                                  Thus, it's not that it cannot be done technically, it's just that we have not really seen a very good common use case, especially given that these two identifiers are internal and subject to all kinds of assumptions which may change in the future.  For example, new points must always be created with a poind id in monotonically increasing order.  And you definitely do not want to use the same recno for a point that's already in use -- just search all of our fixed bugs over the past few years for when the PI Server accidentally assigned the same recno to multiple points.


                                  The only situation that would make sense would be the recovery situation (with no valid pipoints.dat backup), so you do not have to reprocess each of your archives with id conversion files after re-creating all the points.  But even if we eased the pain of that troubleshooting scenario, which is not the scenario being proposed, it would still have to be tightly controlled.


                                  And it would definitely be a mistake to just copy the pipoints.dat or some part of the configuration.  This will almost assuredly result in a fatal inconsistency, as the pipoints.dat references many other files: the point classes, users, groups, digital sets, etc.  We see this all the time in support where folks just take one file from backup and replace it without realizing all the potential problems, especially if the backup is very old.  And in 3.4.380, the pipoints.dat has even more references -- to the ACL table, for example.  Unless there are exceptional circumstances, you should always restore the PI Server configuration as a consistent unit.  And if you cannot for some reason, you definitely want to be working with Tech Support to limit the potential consequences.