6 Replies Latest reply on Jan 16, 2019 8:41 AM by Roger Palmen

    Deployment and version control for AF Development


      I wanted to ask the community on how you guys are maintaining development and deployment across different AF environments. In my organization, we have Development à Test à Production AF environments. Multiple developers work on development AF server to make changes. Once the solution is ready, we migrate the entire AF structure through Excel in QUT. Once it passes the QUT testing, we deploy it to Production via Excel as well.


      Often time we struggle on identifying the changes that have been occurred since last deployment as there are multiple developers working on it and find it difficult on keeping the list of changes. Hence, we end up migrating the entire structure to include all the changes. We find this process crumble some and error prone.


      Is there an effective way to implement version control on AF server? How do we compare two databases across different AF server to identity a delta and only import the changes?


      One solution I was thinking of to develop custom AFSDK application that fetches the entire AF structure on daily bases (similar to format we see on PIAFBuilder) and storing them in SQL server. This serves as a version control and could be used to identify changes that have occurred during development period.

        • Re: Deployment and version control for AF Development
          Eugene Lee

          Hi Rubin,


          You can vote for such an enhancement request here.

          Looking for a feature  which makes merging multiple databases seamless with a version control system like Git – Customer…


          You can also enable the Audit Trail feature to track AF Server changes. This requires you to be sysadmin on the SQL Server on which PIFD resides.

          Audit Trail

          • Re: Deployment and version control for AF Development
            Roger Palmen

            That is a tricky problem, never really solved it to my full liking.

            Two approaches i could suggest:

            • Change Process: have every developer develop on a Dev database, and when unit testing done, create a changelog and implement in the shared test environment. That way all CRUD operations on the database should be known
            • Export based: export the updated library objects and elements. Either using XML export or PI Builder. PI Builder allows to add any deletes which the XML export does not.
            • Re: Deployment and version control for AF Development

              The issue here resides with the AF Tree and AF Library, operated via AF Browser, as well as PI SQL. Version control and configuration management of AF catalogues are practically non-existent, in my view. The current version tracking (Audit Tracking) is not really intended for full-scale concurrent version control and configuration management in the software systems engineering sense. Furthermore, there is potentially a significant performance hit when enabling audit trail on large AF trees, particularly under intensive PI SQL query load. It appears that AF has not been designed with professional software configuration management in mind at the application point.


              Under the bonnet, AFSDK is a rather beautiful design, and that level of development is easily accommodated in formal software concurrent version control systems, such as SVN and Git, and as such is not the point of this discussion. If one purely developed AF deployments via C# programming, targeting the AFSDK, the situation was partially resolved - but only for professional programmers with C# and AFSDK backgrounds, and only for structuring templates and a tree. How to manage AF configuration settings, such as AF tables, UOMs, DRs, remain unresolved. Some mention PI Builder, others resort to XML exports, etc. Neither of these solutions are truly seamless, integrated concurrent version control as professional software development expects.


              In my application area - Mining and Minerals Processing - we deploy fairly elaborate AF trees with standardised outlines and AF libraries. However, we must accommodate continuous improvement, as well as particular specialisations per deployment. Also, our team does deployment and development in concurrent fashion, whereby more than one team member may be operate on the same AF catalogue in PIFD (also called an AF Database). Since each deployment is for a different enterprise, we partition our solution such that each PIFD contains a single enterprise AF Database. However, a number of short-comings emerge from the lack of formal version control features:


              a) Layering specialisations across AF libraries is virtually impossible. One ends up duplicating unchanged content.

              b) Merging libraries via version control comparison and merge is unsupported.

              c) General version control features, such as tagging, branching, merging, reverting (roll-back), and change annotations are practically unsupported (Audit Trail does not really qualify as a player here, although it does make an appearance).


              A similar version control fate befalls PI SQL (whether the old PI OLEDB Enterprise, or the new RTQP edition).

              PI Archive is entirely deserted in this sense. It does not even attempt to enter the stage of formal version control in the established software sense.

              SQL tools for version control exist, but then one must wade through native PIFD models (similar for the XML efforts).


              We resort to a very basic XML export every week of our AF libraries and trees, and PI SQL catalogues, which files are archived in SharePoint   with versioning active. On PI Archive, we schedule daily backups and will be forced to restore at archive level, should a major roll-back be required. I suppose some incantations of shell commands may allow finer grain restoration of some PI Archive databases, but dragons lurk there.


              Quo Vadis, OSIsoft... We turn our lonely eyes to you.

                • Re: Deployment and version control for AF Development
                  Roger Palmen

                  In my experience, versioning and configuration control requirements can be very different across the board. From light-weight requirements in any professional environment, to very strict regulatory requirements in Validated environments for Pharma.I think there will not be a one-size fits all set of functions that will resolve all our combined needs, and as with any application, there are other items to include into versioning and configuration management than just the PI related stuff.


                  So while there is little support for this in PI, i've never really found that to be a major issue, even less in those enviroments (like Pharma) where procedures are very strict. Basically, being able to export / import configurations is the minimum required to make anything work, and all other items can be covered by deployment runbooks.


                  Hope my thoughts help in this

                  1 of 1 people found this helpful
                    • Re: Deployment and version control for AF Development

                      It is all very well in circumstances where an AF configuration is produced and maintained in-house for only that one enterprise. But the cracks appear, as described in my previous response to the opening post of this thread, when one develops AF configurations for various enterprises, using a standardised approach with specialisations per enterprise. In other words, when one approaches AF from the perspective of a value added software service provider in manufacturing information systems. In such circumstances, the lack of the full gamut of software & system configuration management and version control for AF and its related technologies become a bit of a head-ache.