39 Replies Latest reply on Sep 11, 2012 11:00 PM by RJKSolutions

    PI Sever performance

    ygalipeau

       Good afternoon

       

      i don't know if this is the right place to ask, but is there any official OSI documentation that is talking about the capability of the historian (i.e. number of tags it can treat per seconds, etc)? I know it depends on a multitude of factors, but as a general rule of thumb, that is what i am looking for as documentation.

       

      thanks

       

      yannick

        • Re: PI Sever performance
          Ahmad Fattahi

          As you pointed out yourself, it will depend on many factors which makes putting a number on these specs very difficult, if not misleading. On top of your hardware specifications (memory, CPU, network, etc.) there is a lot of variance in different types of data. For example, how often you scan your tags and how the values pass through the exception/compression significantly affect the actual load on the PI Server.

           

          Do you have a specific example in mind we can talk about?

            • Re: PI Sever performance
              ygalipeau

              i understand that this question can be ask in many ways.

               

              Let's say i have 40 000 tags that are sampled each second. could the server handle this volume of data without any problem ?

               

              yannick

                • Re: PI Sever performance
                  mhalhead

                  Yannick,

                   

                  I can give you a users perspective. We have numerous servers handling more than 40k tags at a 1s scan. The servers handle this with easy plus they have numerous performance equations and totalisers running on the PI server, and we aren't talking massive hardware. Bear in mind that scanning at 1 second does not mean that you archive all the data.

                    • Re: PI Sever performance
                      ygalipeau

                      Thanks all for your response.

                       

                      yannick

                        • Re: PI Sever performance
                          morten.staalesen

                          Hi,

                           

                          (I hope it's ok that I dig up an old thread with relevant info instead of posting a new one)

                           

                          We are currently trying to calculate the performance of how much data PI can handle (snapshot/s and archived events/s).

                           

                          The data is sent to either one or several interfaces (custom made for our company) which then sends it to PI. I can't really go into details on how our system is built up, but a quick example would be that we have ~10 000 points which receives data every 20ms.

                           

                          At the moment we have observed the following:
                          With 1 interface, PI can handle roughly 100.000 snapshots/s and 60.000 archived events/s
                          With 2 interfaces: rougly 120.000 snapshots/s and 75.000 archived events/s
                          With 3 interfaces (or more) it has stabilized at around 144.000 snapshots/s and 80-90.000 archived events/s.

                           

                          The system that was used to calculate this data has the following specs:

                          • Intel Core i7 2820QM CPU @ 2.30 GHz
                          • 16 GB RAM
                          • Windows 7 (64-Bit)
                          • PIServer 2010.

                          So, our observations is that we have approximately half of what is stated in the official statement. Is there something we can do to increase the performance?

                           

                          I will be happy to explain in further details if things in this post was unclear.

                            • Re: PI Sever performance

                              Hi Morten...welcome to vCampus.  

                               

                              I am going to steal Jay's thunder and say that you should get yourself on to the PI Server 2012 beta and re-run your tests

                               

                              Anyway, what are the latencies and bandwidth from your interfaces to the PI Server?  Are you using PIBufss?  Your archive rate seems good for 2010 R3 with the 3 interfaces.  

                                • Re: PI Sever performance
                                  hanyong

                                  Welcome to vCampus Morten! And a good first post for a start.

                                   

                                  On top of what Rhys had asked. Did you monitor the other system resource usage (like CPU, Disk I/O and memory) during the test? 

                                   

                                   

                                   

                                   

                                   

                                   

                                    • Re: PI Sever performance
                                      morten.staalesen

                                      Hi again,

                                       

                                      First: thanks for the quick response.

                                       

                                      At the moment I'm not able to use PI Server 2012 beta as I am currently trying to establish what the bottleneck will be for PI Server 2010 (which we are currently using in production). But as soon as I have the time to do so, I would love to check out the beta.

                                       

                                      At the moment, everything is happening on my machine, so this could be one of the reasons that I see the values that I do. So I assume that latency should be very low? :)

                                       

                                      As of right now, we have not enabled PIBufss, but we are looking into it. We have among other things noticed that we loose some data during archive shift, I assume PIBufss would fix this? (not to derail the thread...)

                                       

                                      I am monitoring the following at the moment (from performance monitor):

                                      • "\Memory\Available bytes"
                                      • "\processor(_total)\% processor time"
                                      • "\PI Snapshot Subsystem\Snapshots/sec"
                                      • "\PI Archive Subsystem\Archived Events/sec"
                                      • "\Process(IF)\Working Set - Private" (Our Interface)
                                      • "\Process(IF)\% Processor Time"  (Our Interface) 
                                      • "\Process(w3wp)\Working Set - Private"
                                      • "\Process(w3wp)\% processor Time"

                                      I have been thinking about monitoring piarchss and pisnapss too, but at the moment that is not implemented in my test cases. What we see is that as soon as we reach ~150.000 variables with 3 or more interfaces, the interface start allocating more and more memory untill the interface crashes. My theory is that this is because PI is unable to handle more data at the time, and the interface is somehow buffering all the data it is receiving untill PI can handle the data. I am not expecting a clear answer from you on this, since this is a customer specific interface, but do you think this is theory make sense? My main reason for this theory is that as soon as I stop sending more data, the interfaces memory allocation drops accordingly.

                                       

                                      The interface operates at around 27-29k memory untill the bottleneck is reached. Then they (one by one, depending on the blocks it is subscribing on is increased or not) start allocating memory until they crash. CPU usage is also rather low. However I'm not 100% sure if it is the correct counter I am monitoring for CPU usage, as I've seen values as high as 140 on some of the processes (when the system is really struggling). 

                                       

                                      To sum things up, the values I've seen seems to be approximately where we can expect to be at the moment with the hardware used for the testing, the version of PI server etc?

                                       

                                      Once again thanks for getting back to me so quick.

                                       

                                       

                                       

                                       

                                       

                                       

                                        • Re: PI Sever performance

                                          My personal preference is to always simulate the infrastructure that would occur in production by having interface(s) on a separate node from the PI Server, to avoid competition of resources.  You could put all three interface on one node, the PI Server on another.  Stick the PI Buffer Subsystem with the interfaces to better monitor and control the update rate between the interfaces and the PI Server, plus you may experience some improved performance because of PIBufss internal protocol - but that is speculation as I don't know what Interface you are using.  PIBufss tends to handle disconnects etc better and will buffer during archive shifts, which on 2010 can take ~1 minute per GB.

                                           

                                          Out of interest, how often are your archives shifting?

                                            • Re: PI Sever performance
                                              morten.staalesen

                                              The interface we are using is based on the uniint interface (as far as i know atleast). 

                                               

                                              As of right now, our software for handling the data together with the interfaces and PI Server are all located on the same server (node?). So that's the reason I'm also running the tests like this. I've been looking into the possibility of making the system more distributed like you explain above, but the question is if this is necessary or not.

                                               

                                              At the moment our archives have a lifetime at around 40 days. (512 MB size) However this is with a smaller set of data that is being logged than with our new update. When running these tests, i've seen archive lifetimes at around 40 minutes. This is however to stress the system as much as possible, this is not anywhere close to the amount of data that will be generated.

                                               

                                              We have also some yet to define correct Exception Deviation and Compression Deviation for the different values. The goal of these tests is to determine if our system can handle 10.000 points being updated every 20ms (with the current specs). If that is not possible (as seems to be the case as of right now), we need to distribute the system. Your input on this matter confirms what I had in mind, thanks alot!

                                               

                                               

                                                • Re: PI Sever performance
                                                  andreas

                                                  Morten - there are basically two aspects in your test:

                                                   

                                                  - The amount of data the PI Server can write to disk, and

                                                   

                                                  - the amount of data the PI Server can receive from a single interface or interface node.

                                                   

                                                  I would expect that if you distribute the data collection you will see an increase. But note that the number of events provided by a data collection node is limited and you might have to go after >3 nodes to see increasing performance.

                                                    • Re: PI Sever performance
                                                      morten.staalesen

                                                      From the information you are giving me now, I think I can make a quick conclusion about the following:

                                                       

                                                      * The numbers I've seen seem feasible for the specs/PI Server version we are running.

                                                       

                                                      * Distribution needs to be investigated (should the amount of data increase above 90% of the "bottleneck that I've seen today)

                                                       

                                                      * PIBufss should be enabled on the interfaces.

                                                       

                                                      Thanks for all this valid input!

                                                       

                                                      /Morten

                                                        • Re: PI Sever performance
                                                          jlakumb

                                                          I agree with everything recommended so far (thanks to Rhys for cheerleading the PI Server 2012).  It sounds like this application has high performance requirements (10K tags @ 50 Hz = 500K events/sec).  This would require multiple PI Server 2010 nodes today, or one PI Server 2012 node in near future.  Morten, I will follow up with you offline to see if there is interest in participating in PI Server 2012 TAP and testing the beta to determine if it meets your needs...

                                                           

                                                          Jay Lakumb, PI Server PM

                                                            • Re: PI Sever performance
                                                              andreas

                                                              Morton, meanwhile I did some research - yes, your interface is uniint based but it is a very special one. It is probably better to have this discussions offline with the interface developer.

                                                                • Re: PI Sever performance
                                                                  morten.staalesen

                                                                  Hi again,

                                                                   

                                                                  I was looking for more "general" performance options. I should have noted that I am in contact with the interface developer also, sorry for not doing so.

                                                                   

                                                                  /Morten

                                                                    • Re: PI Sever performance
                                                                      jlakumb

                                                                      Regarding the following "We have among other things noticed that we lose some data during archive shift".  I forgot to mention that shift performance is 2-6x faster with PI Server 2012.

                                                                        • Re: PI Sever performance
                                                                          morten.staalesen

                                                                          Evening all. I hope it is OK that I dig up this thread once again, since it contains some background information regarding the setup I'm working on.

                                                                           

                                                                          Hopefully some of you will remember some of it, or are OK with reading the few posts I've posted in this thread. (Again, sorry if this is not the normal way of doing things on these forums!)

                                                                           

                                                                          To our (current) problem/challenge that has rendered us rather baffled:

                                                                           

                                                                          Until now the testing of performance has been on a laptop, thinking that a server with better hardware would give us an even better performance from the PI Server (measured in snaps/sec and archived/sec). After starting testing on two different servers, it looks like that might not be the case. First a little information regarding the setup we are running:

                                                                           

                                                                          PI Server: All 3 units are running the same version of PI Server 2010, with the same version of PI SDK.
                                                                          Our application: The exact same setup / version of our application is installed on all three servers. (Meaning that there should be no difference in that).
                                                                          Operating System(s): Laptop is running on 64-Bit Windows 7. The two servers are running on Windows Server 2008 R2.

                                                                           

                                                                          Hardware setup:

                                                                           

                                                                          LAPTOP:

                                                                           

                                                                           

                                                                          • Intel Core i7 2820QM CPU @ 2.30 GHz
                                                                          • 16 GB RAM
                                                                          • Windows 7 (64-Bit)
                                                                          • PIServer 2010.
                                                                          • Harddrive: 500GB Serial ATA-300. 7200rpm. 

                                                                           

                                                                           

                                                                          SERVER #1:

                                                                          • Intel Xeon CPU E5-2620 @ 2.00 GHz (2x6cored processors. In total 24 "Processors/CPUs?" in Hyperthreading)
                                                                          • 32 GB RAM
                                                                          • Windows Server 2008 R2
                                                                          • PIServer 2010
                                                                          • Harddrive: SSD in Raid5 or 6. not 100% certain about the make and model. Will update with this information tomorrow.

                                                                           

                                                                           

                                                                          SERVER #2:

                                                                           

                                                                           

                                                                          • Intel Xeon CPU E5640 @ 2.67 GHz (2 Quad cored processors. In total 16 "Processors/CPUs?" with Hyperthreading)
                                                                          • 36 GB RAM
                                                                          • Windows Server 2008 R2
                                                                          • PIServer 2010
                                                                          • Harddrive: Not 100% certain about the make and model or type. Will update with this information tomorrow.

                                                                           

                                                                           

                                                                           

                                                                           

                                                                          To elaborate a little more on the problem. What we have seen is that on the laptop, the machine with the "worst" specs, we get 2-3 times higher results in snapshots/s, and also higher archived/s. The scenario that we are testing is with approximately 115000 events/s entering our custom made interface. On the laptop this is not a problem. Even 130 000 is not a problem. But for the two servers, this is not the case. The interface is overloaded on both computers. They are only able to handle approximately 40-50 000 events/second. We have roughly 2500 pipoints that are receiving data at approximately 50 packets per second. The test is conducted on the LOOPBACK interface.

                                                                           

                                                                          We have struggled for some time now trying to determine what the bottleneck could be. We've looked into the following:

                                                                          • Made sure that all three computers are running on the same versions of all the PI-related programs.
                                                                          • The setup in PIServer is identical on all computers
                                                                          • Disabling Hyperthreading on the servers did not do anything. (We noted that 
                                                                          • There are no "hidden" processes allocating resources on either of the two servers. On the laptop I am simulatenously running Visual Studio, Word, Excel, Outlook, Chrome etc while running the test, without this seemingly making an effect on the systems performance.
                                                                          • The software we are testing is totally identical for all three machines.

                                                                          The list goes on, but these are some of the major parts. We even tried setting up so that the interface transferred data from an interface on the laptop to one of the servers. This unfortunately did not work as intended, the snapshot / archived values where even worse than earlier. The reason is most probably congestion on the internal network without me being 100% certain about this.

                                                                           

                                                                          As it might be clear by now, we are kind of in the dark on this matter. So I was hoping that you all could be able to perhaps get us in the correct way of what we should do or perhaps have some ideas about what locations to look into. It really would mean alot to us!

                                                                           

                                                                           

                                                                            • Re: PI Sever performance
                                                                              jlakumb

                                                                              Hmm, this could be a little tricky. Question, is buffering enabled on any of the machines? If so, could you turn off buffering on all nodes to help eliminate another variable? Next, perhaps you can enable PI SDK tracing for a short time and review the timing of the methods used to write data (UpdateValues?) in the trace logs. That would help to understand the performance from application perspective. These are just my initial thoughts. I am sure others will jump in too...

                                                                                • Re: PI Sever performance
                                                                                  morten.staalesen

                                                                                  Hi Jay.

                                                                                   

                                                                                  As of right now, none of the servers are running with buffering enabled. So this should not be the case.

                                                                                   

                                                                                  As I've understood it, we might need to update PI SDK to enable tracing. Perhaps you could confirm this? We are currently running PI SDK 1.4.0 416.

                                                                                   

                                                                                  Thanks for getting back to me so quick!

                                                                                    • Re: PI Sever performance
                                                                                      jlakumb

                                                                                      Actually, PI SDK tracing has been available going back many releases (and many years), so it is available in PI SDK 1.4 and in previous versions. These vCampus resources discuss tracing:

                                                                                       

                                                                                      vcampus.osisoft.com/.../6265.aspx

                                                                                       

                                                                                      vcampus.osisoft.com/.../pi-sdk-performance-improvement-in-pi-3-4-380-network-manager.aspx

                                                                                        • Re: PI Sever performance

                                                                                          Are the two servers running on VMWare?  e.g. Is the block they are running on heavily loaded.

                                                                                           

                                                                                          PI Server is the same version, are all tuning parameters the same?

                                                                                           

                                                                                          Is the bottleneck consistently 40-50k events/sec on each test run?

                                                                                           

                                                                                          Is the PI API version the same?  Thought I read somewhere your interface was uniint based.

                                                                                            • Re: PI Sever performance
                                                                                              morten.staalesen

                                                                                              Hi Rhys,

                                                                                               

                                                                                              These servers are not running on VMWare, they are running on standalone servers.

                                                                                               

                                                                                              The tuning parameters are the same for all the servers.

                                                                                               

                                                                                              The bottleneck is consistently 40-50k events/sec every time.

                                                                                               

                                                                                              The API Version is the same. It is correct that our interface is based on uniint.

                                                                                               

                                                                                              At this point it might not even be the PI server that is the problem. Have any of you guys encountered different performance on Win7 (64-bit) and Windows Server 2008 R2?

                                                                                               

                                                                                              I'm currently looking at the threads you posted Jay, will post back when I have more information.

                                                                                               

                                                                                              As always, thanks for getting back so quick!

                                                                                                • Re: PI Sever performance

                                                                                                  Hello Morton,

                                                                                                   

                                                                                                  I understood that you've already found the data source being slow when running on Windows Server 2008 R2. PI-SDK tracing will not help here if the interface is UNIINT based because PI-API will be used to update data in PI.

                                                                                                   

                                                                                                  If you are uncertain if your data source or the PI Server is the bottleneck, please check with task manager if you can identify one of the processes involved being extremely busy - possibly on a single core only - what shouldn't happen with PI Subsystems.

                                                                                                   

                                                                                                  Please also check with Resource Monitor especially for the hard disk performance. Is the disk busy at almost 100 %? What about the queue length?

                                                                                                   

                                                                                                  With regards to PI System performance, the first thing to check is the PI Server log. Next check the snapshot (piartool -ss), event queue (piartool -qs) and archive (piartool -as)subsystem statistics.

                                                                                                   

                                                                                                  Also very useful are the subsystem thread statistics in PI-SMT -> Operation -> PI Services. Select a subsystem and look at the "Thread Details for Selection" section - especially to column "CurTime". Threads showing with anything >> 0 are busy threads. Check what they are doing (column RPCName).

                                                                                                   

                                                                                                  Have you checked the tag configuration on all installations? Are tag attributes equal - especially the point type?

                                                                                                   

                                                                                                  Is your data source sending events in timely order (check for "Out of Order Events" with archive subsystem statistics).

                                                                                                   

                                                                                                  I believe the first thing you should verify is at what end the bottleneck sits - the data source or the PI Server. Can you run a second instance of your data source in parallel with each instance set up to service 50% of the tags? Is your data source running in a single thread or is it multithreaded? Is it operating synchronous or asynchronous?

                                                                                                   

                                                                                                  Gregor

                                                                                                    • Re: PI Sever performance
                                                                                                      mhamel

                                                                                                      @Morten: Two things brought my attention in with your problem description: utilization of RAID vs. non-RAID disks and server versus workstation.

                                                                                                       

                                                                                                      First, you might know that some RAID techniques add some delays on read and/or write operations. RAID 5 particularly is adding some write delays to calculate the parity blocks. The performance cost for writing will be higher with random access to disk and small writes.

                                                                                                       

                                                                                                      Do you know the IOPS metrics for the hard drives in each configuration? You could identify a possible bottleneck at this level that would affect the PI Server writing operation performance.

                                                                                                       

                                                                                                      As second, Windows Server OSes (such as Windows Server 2008 R2) are optimized to prioritize system/background processes compared to workstation OSes (such as Windows 7) that focus on foreground ones. Is the custom interface developed is run as a Windows service or foreground application?

                                                                                                       

                                                                                                      If it is not the case, can you try your test again comparing with all system/background processes?

                                                                                                       

                                                                                                      Let me know if that helped.

                                                                                                      • Re: PI Sever performance
                                                                                                        vegard

                                                                                                        Afternoon guys.

                                                                                                         

                                                                                                        First, let me introduce myself. I am a colleague of Morten, and we are both currently looking into this issue.

                                                                                                         

                                                                                                        I'll start by answering the questions by Gregor Beck. Thanks for your input by the way!

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        If you are uncertain if your data source or the PI Server is the bottleneck, please check with task manager if you can identify one of the processes involved being extremely busy - possibly on a single core only - what shouldn't happen with PI Subsystems.

                                                                                                         

                                                                                                        On our three PI servers, we tried replacing our custom interface (i.e. data source) with the PI Random Interface. Each interface is configured with 2500 tags with a scan rate of 10ms for each tag. The PI-server running on the laptop (with an i7 Sandy Bridge CPU) is able to push 220k snapshot events/sec to the locally running PI server. Our two (rather expensive) rack servers (with 2x Xeon Sandy Bridge/Nehalem CPUs) peaks at ~100k snapshot events/sec and ~85k snapshot events/sec, respectively. Currently, we have not been able to pinpoint exactly what is causing this behavior - but it appears to have no relevance to our custom interface. Instead, it looks like it is PI Server or hardware related. 

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        Is the disk busy at almost 100 %? What about the queue length?

                                                                                                         

                                                                                                        The disk on the laptop is a normal HDD. Possibly at ~7000RPM. On one of the servers, the partition where the PI server is running (also in which the PI archives and Event Queue is stored), is located on a RAID1+0 array consisting of 6x10k RPM SAS HDDs. The second server has two Seagate Pulsar.2 SSD drives running in RAID1. Queue length on both servers is ~0.02, as reported by Performance Monitor - so I/O does not seem to be the issue here.

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        Also very useful are the subsystem thread statistics in PI-SMT -> Operation -> PI Services. Select a subsystem and look at the "Thread Details for Selection" section - especially to column "CurTime". Threads showing with anything >> 0 are busy threads. Check what they are doing (column RPCName).

                                                                                                         

                                                                                                        The average CurTime value of all threads on Snapshot, Archive and Base subsystem is 0. Occasionally, a thread on the Archive subsystem increases to 15 for a short period of time.

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        Have you checked the tag configuration on all installations? Are tag attributes equal - especially the point type?

                                                                                                         

                                                                                                        Yes. The tags are automatically generated by a script, so all attributes are equal.

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        Is your data source sending events in timely order (check for "Out of Order Events" with archive subsystem statistics).

                                                                                                         

                                                                                                        "Out of Order Events" is 0 on all PI servers.

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        I believe the first thing you should verify is at what end the bottleneck sits - the data source or the PI Server.

                                                                                                         

                                                                                                        As we replaced our custom interface with the PI random interface, it appears that this is a PI Server or hardware related issue.

                                                                                                         

                                                                                                        Gregor Beck

                                                                                                        Can you run a second instance of your data source in parallel with each instance set up to service 50% of the tags? Is your data source running in a single thread or is it multithreaded? Is it operating synchronous or asynchronous?

                                                                                                         

                                                                                                        I am leaving for the day in 2 minutes, so I'll get back to you on this question tomorrow. Thanks again for your input and suggestions!

                                                                                                         

                                                                                                        Tomorrow, we will try installing Windows 7 on one of our servers  to check whether the OS version has any impact on the performance.

                                                                                                          • Re: PI Sever performance

                                                                                                            Hello Vegard,

                                                                                                             

                                                                                                            Vegard Haugland

                                                                                                            On our three PI servers, we tried replacing our custom interface (i.e. data source) with the PI Random Interface. Each interface is configured with 2500 tags with a scan rate of 10ms for each tag. The PI-server running on the laptop (with an i7 Sandy Bridge CPU) is able to push 220k snapshot events/sec to the locally running PI server. Our two (rather expensive) rack servers (with 2x Xeon Sandy Bridge/Nehalem CPUs) peaks at ~100k snapshot events/sec and ~85k snapshot events/sec, respectively. Currently, we have not been able to pinpoint exactly what is causing this behavior - but it appears to have no relevance to our custom interface. Instead, it looks like it is PI Server or hardware related.

                                                                                                             

                                                                                                            Random doesn't support sub second scans. The minimum frequency is 1s. With 100k events and 2,500 tags assigned to each Random instance I calculate 40 Random interface instances. For sure you would need >> 100k tags as well to achieve sending 220k events/s. I am a little bit confused. Can you confirm the numbers?

                                                                                                             

                                                                                                            Vegard Haugland

                                                                                                            As we replaced our custom interface with the PI random interface, it appears that this is a PI Server or hardware related issue.

                                                                                                             

                                                                                                            From your reply the PI System appears quite relaxed as well as your server hardware.

                                                                                                             

                                                                                                            Vegard Haugland

                                                                                                            The average CurTime value of all threads on Snapshot, Archive and Base subsystem is 0. Occasionally, a thread on the Archive subsystem increases to 15 for a short period of time.

                                                                                                             

                                                                                                            Please check the subsystem thread statistics of all basic PI Subsystems.

                                                                                                             

                                                                                                            - PI Network Manager (not showing for me but please verify)

                                                                                                             

                                                                                                            - PI Message Subsystem

                                                                                                             

                                                                                                            - PI License Manager

                                                                                                             

                                                                                                            - PI Update Manager

                                                                                                             

                                                                                                            - PI Base Subsystem

                                                                                                             

                                                                                                            - PI Snapshot Subsystem

                                                                                                             

                                                                                                            - PI Archive Subsystem

                                                                                                             

                                                                                                            Vegard Haugland

                                                                                                            Tomorrow, we will try installing Windows 7 on one of our servers  to check whether the OS version has any impact on the performance.

                                                                                                             

                                                                                                            I had the same idea but was hesitating to make the suggestion before having a better understanding what limit might be approached.

                                                                                                             

                                                                                                            Last but not least, how does the PI Server log look like? What do you find when searching for "[-" ?

                                                                                                             

                                                                                                            Gregor

                                                                                                             

                                                                                                             

                                                                                                             

                                                                                                             

                                                                                                              • Re: PI Sever performance

                                                                                                                The Random interface I use supports sub second scan classes...  For example, I generate data for 1 tag @ 100ms defined by a scan class and I get 10 values per second archived.

                                                                                                                 

                                                                                                                I wonder if it is related to something in the TCP windowing arena rather than anything in the PI Server software.  Like TCP auto-tuning differences, IPv6 loopback, ...  

                                                                                                                  • Re: PI Sever performance
                                                                                                                    morten.staalesen

                                                                                                                    Hi Gregor,

                                                                                                                     

                                                                                                                    I can confirm that on my laptop we are able to run approximately 220 000 events/sec for the 2500 tags when using the random interface.

                                                                                                                     

                                                                                                                    I've been told that the random interface is also based on uniint, which means that both our custom made interface and the random face are using the PI API for pushing data from the interface into PI.

                                                                                                                     

                                                                                                                    As of right now the theory we have is that it is the difference in either operating system or CPU architecture that is the reason for the difference in performance that we are experiencing.

                                                                                                                     

                                                                                                                    Rhys: That might actually be a really good point. I'll check what differences there could be. We have also asked ourselves if it could be Bus-related (in how the OSes lets PI API send data) or that the custom interface can create so many threads (24 processors avail in windows due to hyperthreading) which means that the overhead for all the threads is affecting the performance.

                                                                                                                     

                                                                                                                    As the clock is getting close to half past ten here in Norway I'll return with more information for you tomorrow. Again, thanks for all the input. We'll try to adress it tomorrow!

                                                                                                                     

                                                                                                                    It should be stated that we have also created a ticket with tech support on this manner, as we understand that the problems we are experiencing might be out of scope for these forums.

                                                                                                                    • Re: PI Sever performance
                                                                                                                      andreas

                                                                                                                      Rhys @ Wipro

                                                                                                                      The Random interface I use supports sub second scan classes...  For example, I generate data for 1 tag @ 100ms defined by a scan class and I get 10 values per second archived.

                                                                                                                       

                                                                                                                      Rhys, you get 10 values per second - but not with subsecond timestamp. If you want sub second timestamps, you need to use a sinus wave form.

                                                                                                                       

                                                                                                                      In other words, if you create a high speed version CDT158 - you get 10 values per second, but all at the same full second timestamp - if you create a 1s sinsuoid version, you get sub second timestamps.

                                                                                                                        • Re: PI Sever performance
                                                                                                                          vegard

                                                                                                                          Good afternoon gentlemen!

                                                                                                                           

                                                                                                                          As I mentioned in my last post, we replaced the OS on one of our servers today to see if this had any impact on the performance. To be honest, we did not expect any significant changes  - but the actual result really left us quite baffled.

                                                                                                                           

                                                                                                                          Now, the server performs equally well as the laptop with an i7 processor running Windows 7 using our custom interface. However, we now suspect that this is most likely a Windows related problem, and not specific to PI or the CPU architecture. Perhaps the settings on the loopback device is different in Windows 7 as opposite to Windows Server 2008 R2. Maybe it is caused by something in the TCP windowing arena as Rhys suggests? Currently, I am unaware of how one can adjust the settings on the Windows loopback device, but this is perhaps something worth looking into.

                                                                                                                           

                                                                                                                          Using the PI Random interface,  the server is now able to push through ~130k snapshot events per second (as opposed to ~80-90k snapshot events per second with Windows Server 2008 R2) with a scan frequency of 10ms - so it looks like this is not specific to our custom interface - but a OS version problem.

                                                                                                                           

                                                                                                                          edit: added some details to the last paragraph.

                                                                                                                           

                                                                                                                           

                                                                                                                           

                                                                                                                           

                                                                                                                            • Re: PI Sever performance
                                                                                                                              jlakumb

                                                                                                                              Vegard, just curious if you are running Windows 2008 R2 SP1 or no SP1? We just had another customer report a significant performance difference (factor of 2x) when applying 2008 R2 SP1 with PI Server 2010 R3. We are looking into possibly reasons why but I just wanted to confirm with you and see if that may be the case here too.  If no SP1, would it be possible to upgrade and try another test?

                                                                                                                                • Re: PI Sever performance

                                                                                                                                  The following resources may be useful finding and addressing performance issues with Windows 2008 R2:

                                                                                                                                   

                                                                                                                                  - Performance Testing Guide for Windows

                                                                                                                                   

                                                                                                                                  - Performance Tuning Guidelines for Windows Server 2008 R2

                                                                                                                                   

                                                                                                                                  Gregor

                                                                                                                                    • Re: PI Sever performance

                                                                                                                                      Sorry, I just recognized # 1 applies to Windows 7 and Windows Vista and hence do not apply here.

                                                                                                                                       

                                                                                                                                      Gregor

                                                                                                                                        • Re: PI Sever performance
                                                                                                                                          dvacher

                                                                                                                                          Morten, Vegard,

                                                                                                                                           

                                                                                                                                          The delta in performance between Windows 7 and Windows Server 2008 R2 might be due to something totally different.  Windows Servers support Non-Uniform Memory Architecture (NUMA) and your servers most likely have 2 NUMA nodes, one for each processor socket.  You can view NUMA computers as multiple machines in one box, with a bus connecting more than one bank of RAM (NUMA node).  Performance is great when everything happens on the same node, but if collaborating processes/threads are scheduled across nodes, significant delays can occur.  The OS doesn’t really know what to do and will typically schedule things in a round-robin fashion.  So in your case, Windows may be assigning your PI Interface and the PI Server subsystems to different NUMA nodes, which could explain the lower overall throughput.

                                                                                                                                           

                                                                                                                                          To test this theory, you can either go into the BIOS configuration of the server machines and disable NUMA (it’s usually under a ‘Performance’ section), or also from the BIOS or from BOOT.INI, you can try to disable the 2nd processor.  If performance is comparable to your laptop, then NUMA is the culprit.

                                                                                                                                           

                                                                                                                                          That being said, you might want to keep in mind that 100K+ events/sec into the Snapshot Subsystem is well into the “red” zone of PI Server 2010 on any hardware.  I guess I'm curious how much goes to disk (Archived Events/sec) when you claim 200K ev/sec.  You probably understand I would recommend giving PI Server 2012 a try to see how it performs in your environment.  It should be much nicer.

                                                                                                                                           

                                                                                                                                          You can download our PI Server 2012 Release Candidate (RC0) from our techsupport website.

                                                                                                                                            • Re: PI Sever performance
                                                                                                                                              morten.staalesen

                                                                                                                                              Evening all,

                                                                                                                                               

                                                                                                                                              The server we have been testing on was running Windows Server 2008 R2. After trying to determine the problem for a whole week, we have determined that we will not use any more time on this performance issue at this time.

                                                                                                                                               

                                                                                                                                              Regarding disabling NUMA: As this server has 2xhecta core processors, the result of disabling NUMA was that the server slowed down immensly. So unfortunately this did not solve the problem either.

                                                                                                                                               

                                                                                                                                              On the 200k events/sec, we where using the random interface. The 200k events/sec was on the snapshot, there was nothing being archived. (Which I assume is why the events/s was so high).

                                                                                                                                               

                                                                                                                                              Even though 100k events/sec is in the red zone, it still does not explain why we are seeing the large difference in performance on a laptop and a server that is specced for the job.

                                                                                                                                               

                                                                                                                                              I guess for now this will be our last entry in this thread, I thank you all for all your good input during the last week!

                                                                                                                                                • Re: PI Sever performance

                                                                                                                                                  Is someone from OSIsoft (Tech Support/CoE?) going to continue the investigation of the performance difference?  

                                                                                                                                                    • Re: PI Sever performance

                                                                                                                                                      Hello Rhys,

                                                                                                                                                       

                                                                                                                                                      The observation that performance is a lot poorer with Windows Server 2008 R2 x64 compared against Windows 7 x64 has surprised us. Just for completeness, please allow me to mention that Windows 7 doesn't belong to the operating systems we recommend.

                                                                                                                                                       

                                                                                                                                                      The PI3 team is still very much intrigued by this disparity but it's is not on their list of high priorities.

                                                                                                                                                       

                                                                                                                                                      Performance was in the focus of developers when working on PI 2012. As we are excitedly looking forward to the official release, please allow me providing some numbers measured in our labs.

                                                                                                                                                      PI 2012 PI 2010 R3 (3.4.385.77)
                                                                                                                                                      Max point count 20M+ 2-3M
                                                                                                                                                      Startup time < 30 sec/Mpts > 10 min/Mpts
                                                                                                                                                      Data In (Snapshot) > 1M events/sec < 200k events/sec
                                                                                                                                                      Data In (Archive) > 500k events/sec < 100k events/sec
                                                                                                                                                      Data Out (Archive) > 10M events/sec < 1M events/sec

                                                                                                                                                       

                                                                                    • Re: PI Sever performance
                                                                                      jlakumb

                                                                                      Hi Yannick,

                                                                                       

                                                                                      Here is the official statement we communicate to our customers when we receive RFIs/RFPs -

                                                                                       

                                                                                      The PI Enterprise Server is highly scalable, given proper capacity planning for hardware and software platforms. In general, the PI Server has no predefined limits with regard to performance and scalability.

                                                                                       

                                                                                      The PI Server supports the latest Microsoft Windows 64-bit operating systems allowing it to leverage large amounts of physical memory for millions of data streams and large in-memory time-series data caches. The PI Server is a multi-process, multi-threaded application, which allows it to scale on multi-processor and multi-core platforms. Adding processing resources expands the ability to service multiple concurrent workloads, like many simultaneous queries from users, in addition to collecting data from many PI Interfaces at the same time.

                                                                                       

                                                                                      The PI Server can also scale across multiple servers, referred as a PI Collective, to provide high availability, load distribution, and data redundancy for disaster recovery. The same PI Server software is used from small systems (thousands of points) all the way up to very large systems (multi-million points), making it easy to scale up over time. Additional scalability is achieved by scaling out multiple PI Server nodes or PI Collectives, which are accessible to end users as one system by OSIsoft’s Enterprise Asset Framework (PI AF), or through application servers such as PI WebParts using Microsoft SharePoint.

                                                                                       

                                                                                      A single 64-bit PI Server can easily support up to 3,000,000 points, and it may support more points when optimized for certain applications. A single PI Server has been successfully tested to sustain collecting up to 300,000 time-series events in real-time, and archiving to disk up to 100,000 events per second. A single PI Server can support thousands of concurrent client connections. A single PI Server has also been successfully tested to deliver hundreds of thousands of update events per second, across all client connections, through an efficient publish and subscribe mechanism.

                                                                                       

                                                                                      That said, as Ahmad points out "your mileage may vary".  If you can provide details on what you are trying to accomplish, we may be able to give guidance on how to achieve those goals.

                                                                                       

                                                                                      By the way, the PI System Roadmap shows the next release will focus on higher performance and scalability.  Stay tuned for more info in the coming months...

                                                                                       

                                                                                      Jay