31 Replies Latest reply on Jul 5, 2018 5:37 PM by vwolf

Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?

Ahlywog Jun 7, 2018 10:03 PM

The pi buffer is filled from an interface. The buffer is filling up and nothing is getting to the server. I've tried my normal steps, stop the pi buffer subservice, remove the pibufq_<server name>.dat file, restart the pi buffer subservice. This has always fixed the problem. It didn't fix it this time.

When checking pibufss -cfg everything appears to be connecting but the server status shows SendError. I can't find anything on the net in regards to troubleshooting this beyond checking for a corrupt buffer file. What else should I be looking at?

Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?

vwolf Jun 8, 2018 2:01 PM (in response to Ahlywog)
Hello Steven,

There are several ways to troubleshoot buffering issues. Using the pibufss command itself, you could use the following parameters:
pibufss -health
pibufss -testcon <SERVER_PATH>

You should also search the pi logs for relevant messages. We also have a video on buffering troubleshooting. The issue may be related to networking or security.

Make sure the buffering node is able to connect to the PI Data Archive machine using the 5450 port using the following powershell command (change the hostname):

(New-Object Net.Sockets.TcpClient).Connect(“HOSTNAME”, 5450)

The SendError revels an error while sending data, so it may not be a queue issue. It is actually not recommended to remove the queue files without confirmation of a queue problem. These should not be considered "normal steps".

All our buffering documentation is summed up on KB01437 - Buffering.

Regards,

Victor
2 of 2 people found this helpful
Like Show 0 Likes(0)

Actions
- Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
  
  Ahlywog Jun 8, 2018 7:21 PM (in response to vwolf)
  
  Thanks for the reply.
  
  I'm running pibufss version 3.4.380.79 so no -health or -testcon. Would installing a newer version disrupt current settings or break compatibility with the LoggerNet Interface or other PI products?
  
  Machine is running Windows XP so no native powershell. It also has no internet connectivity so getting powershell on it is going to be tricky...
  
  I have the log file but I'm not sure what I'm looking for...
  
  The system gathers data every 15 minutes, so 00:00, 00:15, 00:30, 00:45, etc. The last entry in the archive is at 5:15 yesterday morning so something happened between 5:15 and 5:30. Limiting the log file to just that time span still gives a nearly 70meg text file. Is there something I should be looking for specifically to narrow down my search?
  
  Like Show 0 Likes(0)
  
  Actions
  - Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
    
    Steve Boyko Jun 8, 2018 8:12 PM (in response to Ahlywog)
    
    You should be able to run the AboutPI-SDK / PI-SDK Utility and check your ability to connect to PI without using Powershell.
    
    In the logs you are looking for pinetmgr or pibufss, if that helps you narrow things down.
    
    Like Show 0 Likes(0)
    
    Actions
    - Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
      
      Ahlywog Jun 11, 2018 5:08 PM (in response to Steve Boyko)
      
      I'm seeing a lot of:
      
      "Snapshot post failed (5000 events); [-10002] Unable to Activate Object" from pibufss.
      and
      "Deleting connection: PISDKUtility.exe(196); Asynch read failed. [109] The pipe has been ended.; ID: 2205 :0" from pinetmgr
      
      The PI-SDK Utility connects just fine though.
      
      Like Show 0 Likes(0)
      
      Actions
      - Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 11, 2018 5:12 PM (in response to Ahlywog)
        
        Your best bet is to contact OSIsoft tech support so they can do a remote session with you and see the system.
        
        It does sound like the buffer is being corrupted.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 5:21 PM (in response to Steve Boyko)
        
        This is a government system behind a host of firewalls. It would take weeks to get the proper approvals and holes punched to make that happen. What else can I check?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 7:37 PM (in response to Steve Boyko)
        
        Error [-10733] writing data to PI. [-10733] PINET: RPC Resolver is Off-Line.. pisn_flushputsnapqx failed. Previous status = 0, New status = -10733.
        
        Looking at About PISDK in the PI SDK Utility on our Archive server I get that message.
        
        "For example, if a client makes a request for historical data, then that request goes through pinetmgr first and is then sent to the PI Archive subsystem. In order for this process to work, pinetmgr needs to know which requests should be sent to the PI Archive subsystem. When the PI Archive subsystem starts up, it tells pinetmgr which requests it can handle. These requests are known as a remote procedure calls (RPCs). Pinetmgr keeps track of all the RPCs for all the subsystems in a table."
        
        Restarted the PI Network Manager and that error goes away. Now I have:
        
        PINS service may have stopped (error code=-2147220368)
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 11, 2018 8:51 PM (in response to Ahlywog)
        
        Does the "PINS service may have stopped" error persist? (-2147220368 looks like 0x80040470 which is referenced here but doesn't seem applicable)
        If it does persist, then the interface service probably has to be restarted.
        What version of PI-SDK are you using?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 8:59 PM (in response to Steve Boyko)
        
        It did not. Current SDK status is "Ok" and no warnings, errors or critical messages in the Message Log in the last hour.
        
        PI SDK on the server is 1.4.4 Build 486.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 9:00 PM (in response to Steve Boyko)
        
        Still getting "Snapshot post failed (5000 events); [-10002] Unable to Activate Object" from pibufss on the client, though.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 11, 2018 9:25 PM (in response to Ahlywog)
        
        I am not clear where the error messages are being logged - on the PI interface machine ("client") or the PI server.
        
        The "Snapshot post failed" message is on the client - by client you mean the computer hosting the PI interface? This could be known issue 144141 https://techsupport.osisoft.com/Troubleshooting/Known-Issues/144141 where they recommend stopping the PI Buffer subsystem, and moving all queue files, including the STATS file within the queue folder, not just the "pibufq" file(s).
        
        The PI Buffer Subsystem on the PI server is irrelevant to your client issues. It only affects interfaces and client programs running on the PI server itself.
        
        Are the default PI interfaces on the PI server (random (pt source R) or ramp-soak (pt source 9)) writing to PI OK?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 9:45 PM (in response to Steve Boyko)
        
        These two errors are coming from the client (the computer running the interface)
        "Snapshot post failed (5000 events); [-10002] Unable to Activate Object"
        "Deleting connection: PISDKUtility.exe(196); Asynch read failed. [109] The pipe has been ended.; ID: 2205 :0"
        
        All other messages I was talking about were on the server which appears to be running fine now.
        
        And the default PI Interfaces on the PI server are writing ok.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 11, 2018 10:04 PM (in response to Ahlywog)
        
        The problem is probably with pibufss on the client then. Please follow the steps in KB00389 to remedy the situation. You should confirm the version of PI Buffer Subsystem on the client before choosing which method to follow. I think you said it was 3.4.380.79.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 10:10 PM (in response to Steve Boyko)
        
        "Running piartool -bfs on the PI Data Archive shows that PI Buffer Subsystem is registered. This indicates that PI Buffer Subsystem successfully connected and registered to the PI Data Archive."
        
        Running this command on the client (computer running the interface) gives "Query Status: [-10727] PINET: RPC is Non-Existent"
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 11, 2018 10:13 PM (in response to Ahlywog)
        
        Check your Windows services on the client to ensure that all "automatic" services for PI are actually running. It sounds like pibufss isn't running.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 11, 2018 10:16 PM (in response to Steve Boyko)
        
        Checked, it was running.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 12, 2018 3:30 PM (in response to Steve Boyko)
        
        So should we just try a good ol' fashion reinstall at this point? Or is there something else we can do?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        vwolf Jun 12, 2018 5:26 PM (in response to Ahlywog)
        
        Hello Steven,
        
        This does look like one of the following issues:
        
        24149OSI8 - When two PIbufss instances register with PIsnapss with the same GUID, data from one of them won't get to the server. This can often happen when a PINS node in cloned. This has been fixed
        
        144141 - PI Buffer Subsystem might get stuck reading from the queue after an ungraceful shutdown, due to a problem with Queue Recovery
        
        I would recommend trying the workarounds described on the issues above, starting with 24149OSI8. For your buffer version, the folder structure is under %PIHOME%\dat, so it is a little bit different from the description of the workaround. Back up everything before deleting/removing.
        
        Also, you should check what is causing the huge log files. If the log files are so big that you are not able to search through them, then they are not useful at all. Probably there are debug parameters turned on on interfaces.
        
        If the workarounds fix the issue, I would recommend upgrading everything (including OS) to the latest version possible, to avoid old bugs.
        
        Regards,
        
        Victor
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 12, 2018 6:06 PM (in response to vwolf)
        
        24149OSI8 My current version of pibufss doesn't have the -guids command. Nor does it seem to have pibufss.dat or pibufss_backup.dat files associated with it.
        
        144141 I've tried several times now to no avail.
        
        Seems like the best option now is to just install the newest version of pibufss and hope for the best. Will installing the newest version overwrite the current settings?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 12, 2018 6:16 PM (in response to vwolf)
        
        New problem: The client computer is running Windows XP because of the requirements of other applications running on that machine.. It looks like the newest versions of pibufss require Windows 7 or better. Is there a work around? Would using the OSIsoft Prerequisites Kit help with this?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 12, 2018 7:12 PM (in response to Ahlywog)
        
        Is switching to API buffering an option for you, given that you are using Windows XP and not able to use the later versions of PI Buffer Subsystem? That might be a way to keep buffering on that system.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 12, 2018 7:32 PM (in response to Steve Boyko)
        
        Would that work with the interface?
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 13, 2018 12:44 AM (in response to Ahlywog)
        
        Probably, but I don't know what interface you are using, so I can't say for sure.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        vwolf Jun 13, 2018 5:41 PM (in response to Ahlywog)
        
        Hello Steven,
        
        For the buffer version that you are using, try doing the following procedure to resolve 24149OSI8, after backing up the buffer files: Stop pibufss and the interfaces on the buffered node, remove pibufmem.dat and restart interfaces and pibufss.
        
        Regarding the OS, Windows XP was first released in 2001. The latest release (SP3) was more than 10 years ago. Since then, Microsoft ended support for it, and several security threats were discovered. The OS is targeted by hackers and attacks include ransomware. As you see, the recommendation for upgrading the OS is not only so that it is compatible with our newest products, instead, it is rooted in many important topics.
        
        I do not recommend reverting to API buffering. It is a legacy product and the real issue here is using old technologies. Reverting to API buffering is yet another step in the wrong direction.
        
        I would recommend first verifying the possibility of upgrading the application that requires Windows XP into a more recent version or at least trying to separate the interface and buffer so that they are on another machine with a more recent OS version, allowing for a buffer upgrade.
        
        Regards,
        
        Victor
        1 of 1 people found this helpful
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 13, 2018 7:13 PM (in response to vwolf)
        
        Believe me, I would if I could but this isn't my computer to upgrade. I'm merely tasked with keeping the PI applications running on it and making sure the data they collect gets into the archive. All the hardware is going to be upgraded before the end of the year, but until then...
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Steve Boyko Jun 13, 2018 8:08 PM (in response to Ahlywog)
        
        What happens if you turn buffering OFF temporarily? (in PI-ICU) Does the data flow to the PI server then?
        
        Maybe your short term "fix" until this system is upgraded is to leave buffering off. I know Victor Wolf recommended against API buffering, but given that you cannot upgrade the computer, perhaps using a legacy product is better than no buffering at all. That is your choice.
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 13, 2018 8:19 PM (in response to Steve Boyko)
        
        I'm about to go try the last 2 recommendations and I'll let you know.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 13, 2018 9:47 PM (in response to vwolf)
        
        It works! Here's what I did:
        
        stopped the pibufss and interface services.
        moved pibufq_<SERVER_NAME>.dat, pibufr_<SERVER_NAME>.dat and pibufmem.dat to another folder.
        started the previously mentioned services again.
        aaah!
        Profit.
        
        I've moved the pibufq_* and pibufr_* files multiple times since this problem started and it never fixed the problem. So it's pibufmem.dat that got corrupted this time? Is moving that a list ditch kind of thing like the moving the previous two files or should I add this to my tool bag of PI fixing?
        
        Like Show 2 Likes(2)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        vwolf Jun 14, 2018 2:24 PM (in response to Ahlywog)
        
        Hello Steven,
        
        I'm happy to hear that it was resolved!
        
        I wouldn't call this a corruption of pibufmem.dat. I believe it is more like a registration error with the pisnapss. It looks like there was more than one buffer registration with the same info. Without a thorough look into log files and architecture, it would be hard to pinpoint the exact reason this happened.
        
        That being said, I would not recommend applying this fix again without confirming that it is the precise same situation, or without Tech Support guidance. I think you did the correct thing reaching out to the community!
        
        Regards,
        
        Victor
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        Ahlywog Jun 14, 2018 4:09 PM (in response to vwolf)
        
        Why would you not recommend applying this fix again, for the record?
        
        Due to the nature of the current system (not necessarily PI) corrupted buffers is just par for the course, so removing them and restarting the services has pretty much been the ONLY fix I've applied thus far. And, as far as I can tell, it hasn't caused any further problems.
        
        Like Show 0 Likes(0)
        
        Actions
        
        Re: Pibufss -cfg server status shows SendError. Buffer is filling up. How do I troubleshoot this?
        
        vwolf Jul 5, 2018 5:37 PM (in response to Ahlywog)
        
        Hello Steven,
        
        Depending on the version of the PI Buffer Subsystem, queue problems may be fixed with different approaches. I always check the KBs so that I can be sure I'm following the correct approach. The latest version of the PI Buffer Subsystem is much more robust and should not get corrupted easily, but still, if you do experience an issue while using the most recent version, the KB or Work Item might tell you to contact Tech Support so that we may document what happened and possibly fix it in the next release.
        
        Regards,
        
        Victor
        
        Like Show 0 Likes(0)
        
        Actions

Go to original post