13 Replies Latest reply on Dec 17, 2013 5:41 PM by matthew.rivett

    AF Server Monitoring Model

    matthew.rivett

      First off I'm not sure where to put this.  It's not really programming but I think the audience on vCampus is what I'm looking for.  

       

      We have a lot of PI servers at PJM and I'm responsible for monitoring them.  In the past I used some custom ACE code that works a lot like a simple version of TopView.  Put in a PI tag and can do some simple things like higher than, equal to, etc.  I want to model this all out in AF instead and use notifications.  Has anyone done something like this?  Would anyone like to help me do this?  It's a lot of tedious boring work.  Here's a screen shot of what I'm working on.

       

      1323.server-monitoring.jpg

       

       

          • Re: AF Server Monitoring Model
            matthew.rivett

            So maybe tedious and boring weren't the right words to use.  It'll be fun and awesome!

             

            No one is interested in monitoring their servers?  I see this as a community thing where if you're an expert in one area you could improve the model.  Maybe you're a network admin and know routers inside and out.  You could build that up and add it to the model.  

             

            I'm making it as flexible as possible.  There's a table with interface configuration information and an enumeration set.  Once they are setup for your environment you should just have to add elements and if the tags already exist they'd show up.  If not you could use AF to create the tags.

             

            8468.Interface-Configuration.PNG

             

            4011.Interface-Template.PNG

             

            1464.Interface-UniInt-Template.PNG

             

            Right now I'm mostly using Perfmon and specific interfaces we use at PJM.  You could easily add SNMP, Ping, and any other interface.  

              • Re: AF Server Monitoring Model
                Roger Palmen

                In my experience, the Monitoring for PI Environments is highly dependant on the service organisation responsible, and the tools and practices they use. I see everything from some ProcessBook displays giving an health overview, to fully IT-management integrated solutions that automatically create and dispatch support tickets when a specific monitoring point breaches it's limits.

                 

                I know we (CGI) also do Monitoring in a similar manner using PI Notifications, but that is customized on the specifics of the environments to monitor.

                 

                Why so little interest in this topic? In my view @ vCampus you'll find more developers than support or operational people. And developers mainly build new functions, not monitor environments.

                  • Re: AF Server Monitoring Model
                    matthew.rivett

                    Yes that is true.  I guess here we do have some enterprise monitoring but we also monitor PI on our own.  It is so different than most of the other applications that our enterprise monitoring group doesn't really know what to do.  Sure they can monitor services and performance counters (typically they'll monitor CPU, disk usage and memory) but they don't really understand how it all works.  

                     

                    Maybe we are unique.  Our PI group installs, maintains, and monitors PI.  We also develop PI applications.  

                     

                    Thanks for your feedback.  I wasn't thinking about it from purely a developers perspective.  I do think it is something most "Geeks" would find interesting though.  

                      • Re: AF Server Monitoring Model
                        matthew.rivett

                        The biggest obstacle is trying to add all the information into AF.  There are thousands of possible pieces of information.  I'm also not an expert it all types of hardware and software.  I can really only do so much so that is why I was thinking having a community involvement would be good.  

                         

                        Even if I can find people who are interested I'm not sure how to make it a community project.  It's a SQL database, not source code.  Trading around copies of exported XML wouldn't work very long.

                          • Re: AF Server Monitoring Model

                            Hello Matt,

                             

                            I am personally a big fan of community projects. The idea of working together on a joined solution is great mainly because of the different aspects I expect coming from people working together on such a project. We can easily create such a community project within the forum software but looking at the existing community projects, I have some doubts. A community project isn't done by just writing it's name on an empty envelope. You need comrades working with you. This said, I propose making some advertisement for the project you suggest. Not all vCampus Community members email subscribe to all forums. Hence only a smaller number of members may have received notification emails about updates to this thread. You may want to advertise this thread by creating new threads within other forums e.g. AF SDK Development and PI System Integration.

                             

                            Thinking about how to approach your task, is something we can discuss i.e. with replies to this thread. I am not sure if exchanging a database structure would be useful because of personal preferences different users may have. My idea would be an approach involving some coding. I would be starting to think about the tasks more abstract. What would be a reasonable asset structure? One could think about countries, sites, facilities, buildings and rooms but what happens in case equipment becomes moved or replaced. One could also think about using IP subnets as criteria. The more I think about what might be reasonable, the more I recognize this might depend on individual preferences within companies. Could the tag naming convention be used for automation? Would it make sense to create a service which would become installed on all machines to be monitored. This service would create the corresponding structure in AF but what about the tag assignments? Sorry for just bothering you with my thoughts but I believe I found a solution that you may find useful. When PI Interfaces become configured using the ICU, the structure is stored within the PI Servers Module Database (MDB). The MDB lists interface nodes, interfaces installed on these nodes and has also information of PointSource and Interface ID assigned to each particular instance. This information allows to identify the PI tags serviced by the particular instance. If equipment is monitored remotely, there will be the name or IP address of the machine included into the tags configuration. What tag attribute has these information depends on the type of interface (PI Ping, PI SNMP, PI Performance Monitor) which is again information available in the MDB.

                             

                            To summarize, I would look at the information available in MDB and with MCN Health Interface tags and use those information to create the corresponding structure in AF. I further suggest using AF Element Templates whenever possible. I also like to repeat Lingli's hint that Asset Based Analytics will make your life a lot easier, compared to do calculations within PI ACE, as soon as it is released. You may want to make yourself familiar with PI AF 2014 beta. The setup kit is available at vCampus Download Center under category "Pre-Release"

                              • Re: AF Server Monitoring Model
                                matthew.rivett

                                Gregor,

                                 

                                I agree you really need to make sure people are interested and want to help.  I also need to make sure I have the time necessary as well.

                                 

                                The hierarchy would have to be flexible.  At PJM we have very few geographic sites so I didn't even consider that however I can see how it would be useful for other people.

                                 

                                I didn't really think about using the configuration information (%OSI) from the module database.  That's an interesting idea and can provide some of the necessary configuration attributes for AF.  I haven't attempted to create an AF structure using code before so I have been manually creating the templates in system explorer.  If it was done using code that would make the community effort easier since there are many code repositories around for this reason.

                                 

                                I'm not sure about MCN.  We are licensed for it but I don't use it anymore.  I haven't seen any new templates created in a long time and don't see why I'd want to fill up our MDB with information that I don't use.  It was a great idea but seemed to get neglected.  This project would essentially be a replacement for MCN.

                                 

                                Matt

                                  • Re: AF Server Monitoring Model

                                    Hello Matt,

                                     

                                    Matt Rivett

                                    I'm not sure about MCN

                                     

                                    I wasn't referring to MCN but to the interfaces (PI Interface for Ping, PI Interface for Performance Monitor, PI Interface for SNMP and PI Interface for TCP Response) bundled within the MCN Health package because I was assuming you are using them for monitoring. If this isn't the case or at least not the complete list of what you are using, please let me know what kind of measures you are looking at. This would allow me to come up with some additional suggestions. As far as PI Interfaces are concerned and I can assume that you are looking at Interface Health tags, IO Rate tags and possibly UNIINT Interface failover tags, you will find the configuration within the MDB in case interfaces have been configured using PI ICU.  

                                      • Re: AF Server Monitoring Model
                                        matthew.rivett

                                        Yes these are the interfaces I am using.  I also use ISU, health points, IO Rate, uniint points and in some cases interface specific points.  For example some of the PDC status tags used for C37118.  I also just began using DNP3 so will implement the IIN points for RTUs.  

                                         

                                        My model is mostly a skeleton right now.  I add things when I have time and as we install new products.  I'll add a blank template as I come up with ideas in the hope that some day I'll be able to work on it.

                                         

                                        I'll dig through the %OSI interface modules to see what is in there.  I've never really looked that closely.  I normally only work with the ACE modules when moving from one server to another etc.

                                          • Re: AF Server Monitoring Model
                                            hogan

                                            Hello Matt,

                                             

                                            If you haven’t seen it yet, check out the presentation Steve Kwan and Frank Batke gave at the Paris user conference:

                                             

                                            The Asset-Centric PI System: The Value of Asset-Based PI Now & in the Future

                                             

                                            These new “smart” interfaces they mention auto create an AF structure for you (with attributes and tag mappings).
                                            Looks like it will take some of the pain out of the process in the future and may have some impact on what you're working on.

                                             

                                            Where I work we use the PI-SNMP interface extensively and are coming to terms with how best to represent this data using AF.
                                            Next year we’re looking at using the PI-WMI interface to monitor and potentially restart Windows services on some of our application servers.
                                            I would love to see something which auto creates an AF structure which represents the PI system and all the health metrics.

                                             

                                             

                                              • Re: AF Server Monitoring Model
                                                mhamel

                                                @Matt and mgh: The recent vCampus Live! 2013 showed a new "smart" interface (PI IPMI) for data centers. This hands-on lab was demonstrating the capacity of this new interface to create the AF structure needed to represent well the assets and populate the archives.

                                                 

                                                The abstract of this lab was: "Come and learn firsthand about the new generation of PI Interfaces. You will setup and configure a PI Interface to automatically build assets in PI AF and tags in the PI Data Archive. Experience how fast you can get up and running from installing the PI Interface all the way to using PI Coresight to look at the data in terms of the assets the data source provided."

                                                  • Re: AF Server Monitoring Model
                                                    matthew.rivett

                                                    Thanks mgh & Mathieu.  

                                                     

                                                    I watched the video and I'm interested to know how/if these interfaces are using templates.  What was show in the example was only  a small number of the performance counters that are available.  Maybe this is because it was a demo and not released yet.  Either way not everyone is going to want every single performance counter so I'm not sure how you'd handle that with templates.

                                                     

                                                    I like the idea of the PI interfaces creating the AF structure for you.  My only concern is that each interface is populating a different element in AF.  Lets say I'm using Perfmon, SNMP, Ping, and other interfaces that are all monitoring the same asset.  I'd have 3 different elements in AF instead of 1 with all the attributes.  Maybe I could manually link them together but it wouldn't be as automated as I'd like.

                                                     

                                                    Thanks,

                                                     

                                                    Matt

                                  • Re: AF Server Monitoring Model
                                    lZheng

                                    Hi Matt,

                                     

                                    I have seen customers using AF database monitoring servers, and it seems you have already built a pretty robust database, combining the usage of templates, inheritance tree, automatic tag creation, etc.  I'm wondering what are the current problems/obstacles you are facing ?

                                     

                                    Next version of AF (2.6) will support attribute analysis, some ACE calculations can be easily replaced by creating analyses in AF.