System stopped functioning after hard reset. Restored from backup but did not fix issue.
PI system hanging when using pisrvstart cmd. Hangs on "piartool -block pisnapss -verbose" Server 2015
Running on MS 2008 R2.
It looks like from the message logs that you may have corrupted event queues. I would recommend:
1. Running the pisrvstop.bat script to shut everything down for the moment while the next step is accomplished
2. Go to %piserver%\queue, the default location for event queue files. Move the oldest one (has the lowest number of pimqXXXX.dat) out of the directory and onto the desktop or somewhere else you'll be able to retrieve it from.
Note: if your event queue files are not in this location, you'll need to turn on the network manager, basesubsystem, and snapshot subsystem. Open SMT > Operations > Tuning Parameters > Snapshot to check the location of your queue files
3. Run the PIsrvstart.bat and check the message logs to ensure that the server is starting properly and you don't run into more MMQ errors.
4. Use the following KB article to help reprocess the data from that event queue into the Data Archive
It could be many things. Try running \pi\adm\pigetmsg -f in a separate command window to monitor real time PI messages, then start the PI Snapshot Subsystem service and see what messages come up.
I tried that and the results are in the attached file.
Looks like corrupted or missing files. Not sure which ones it needs.
Thanks, That worked.
Robert I thought everything was ok. Everything stated up and looked like it was working. I was out for a couple of weeks and when I got back was informed of no data being collected.
Looking at the queue files the data is being collected in the queue but nothing being written to the archives. I checked and Archive is set to 3. Not seeing any errors. I can read prevoius data from Archives. Also ran archive check utility and no corrupted archive.
I'm stumped at this point.
When I try to do the recovery as if it is a corrupted queue file the first step is turn off snapshot and wait for queue files to go to zero. That never happens.
With snapshot off it events number in queue doesn't change. With Snapshot on it continues to climb.
It sounds like the event queue between the snapshot and archive is the one that's corrupted.
See KB00735: https://techsupport.osisoft.com/Troubleshooting/KB/KB00735
When I try this as I stated, the event queue never goes to 0 files when I turn off the Snapshot sub system.
Ah, I think I understand where the confusion is coming from. I believe the instructions you're look at are for returning moved queues to the event queue file if you mistakenly moved them out of the directory.
Are you working with PI Data Archive 2012 or newer? If so the PI Data Archive should be capable of recovering from queue corruption automatically on startup. I would recommend:
"Deep Recovery Required. Beginning deep recovery with file # 0"
"Queue file load/recovery complete; commencing normal operation"
It's possible the recovery will take long enough that the pi archive subsystem will timeout waiting for the recovery to complete. If so you will see this message:
"Fatal error in PI subsystem piarchss: fatal error detected while attaching event queue,status: 258 The wait operation timed out"
At which point you'll need to let Queue recovery complete and restart the PI archive subsystem at the end of recovery.
It's 2015. It never goes into deep recovery and never shut down anything and never show any errors. Just keeps filling up the queue files. 37 of them now.
You probably should call OSIsoft tech support to resolve this issue.
Retrieving data ...