It would be nice if osisoft would get some benchmark to evaluate and report the latest pi afsdk against different pi data archive and after versions, at least the latest release.
These reports should answer most of the questions of pi afsdk developers in terms of performance, stability, diagnostics and release improvements.
A discussion around what kind of dataflows are relevant in the benchmark would help, don't you think?
I like this presentation.
https://www.osisoft.com/Presentations/Best-Practices-for-Building-AF-SDK-Applications/
Good thing is that at least OSIsoft explains how to get performance by AFSDK.
Also some of the post shows performance.
AFSDK Bulk summary calls slower that other bulk calls
$listResults = $pointlist.RecordedValues($tr,[OSisoft.AF.Data.AFBoundaryType]::Inside,"",$true,$pagingconfig,100000)
Total Seconds : 4.1796506
Values From Call : 3135593
It is only one example and it depends on the environment, network performance etc... So it is good to test in your environment too.
bench mark is helpful for user but I believe it depends on the test condition. Also which functions do you want to see the bench mark?
Sometimes user want to get 10000 tags current values, sometimes recorded values, there are a lot of methods in AF SDK.
How about some reports or information how to reach 500k events/second for
500k pinpoints using fan out pibufss on a pi ha setup.
Then how to reach higher like 1M/s for more pi points.
Regards,
Stephan
Are you looking for these kind of data rates for data being sent to the Data Archive, or being retrieved from it? In both cases, these kind of data rates would require a combination of both reasonably high end hardware, and well optimised code for bulk data handling (AF SDK bulk data methods, appropriate parallel processing etc). There will be other factors involved as well such as network environment, PI Data Archive tuning, etc.
Basically using pibufss means writing events to PI tags performance.
I believe the AFSDK developer team should have some of the internal performance test results.
David Moler , can you comment for it? Do you guys have the test results that is possible to share it within the PI Dev Club community?
Yes I do agree, around standardized benchmarks in order to reproduce and follow with changes accross releases (both pi server and sdk).
Yes, i need to evaluate in and out 500k+/s at least for 1 update/s/tag, but not only tag values update.
I did evaluate bulk vs serial vs parallel but still have some contentions between afsdk pibuss and pi servers.
Does anyone have already tested 500k+/s update with 1update/s/tag with fan-out on a pi server in ha.
Anyway, testing each release of afsdk and pi server is a lot of work, also as we don't have access to afsdk and pi development team and code, how we can troubleshoot to find answers to our questions, and to deliver solutions that match our objectives.
Don't you think it should done by osisoft afsdk team and just issue a performance report based on a set of benchmark?
Hi Stephan,
We run many different performance tests internally. Because our infrastructure permits a lot of flexibility in configuration and architecture and is run in production on a wide variety of hardware, it is impossible to give a specific measurement that is going to be widely applicable. In past instances, these measurements have been taken without the context in which they were generated and extrapolated to other use cases where they don't apply. So I hope you can understand our hesitance.
I can see your point as well; you are trying to figure out what are the most performant ways to execute certain tasks or what are the rough capabilities of the system to determine if a particular approach is appropriate. I'll see about making something available in the proper format - perhaps blog posts covering optimization for different scenarios. In the mean time, I can give you some general guidance for your situation. Do I understand correctly that you want to know the throughput through updates on a PI Data Archive? Can you describe your use case and architecture in more detail?
Hi David,
Thank for your answer, I do understand the problem to publish data and get as a result many out of context questions.
Yes, you do understand the context.
Basically I do need to manage a steady flow of update which should come in order and very very rarely out of order. You can conzicon a flow of segment of ordered update and once in a while a few segment out of order of ordered updates.
Hi Stephan,
500k events per second via updates is possible but is pushing the limits. A few things that could get in the way of reaching this target:
One important thing to note is that if you aren't getting the throughput you expect, the backup will almost certainly manifest as a back-up in the data ingest rate (e.g. write RPCs to PI Data Archive queuing up, buffers building up).
If you have more questions - please don't hesitate to ask.
Thank for the prompt reply.
I did already what you suggested, consuming data seems to be fine at 500k/s with few pipe instances. But updating data using a fan out pibufss is not easy, even without a single out of order event. I guess advanced pi server and hardware/network tuning might help but I am not sure that pibufss + pi data archive could handle the load.
Do you know if someone has been tested successfully a single client updating w/ pibufss + fan out to a PI collective at very high rate 500k+/s?
Ah - one other thing I forgot to mention is that pibufss can be a bottleneck, too. You probably need at least 2 independent nodes running pibufss to reach this level of throughput.
I tested also with two nodes but it doesn't help.
By the way is it possible to run two independent pibufss on a single node, I mean without virtualization.
Is it also possible to tune pibufss to increase it's resources thread, memory ...
PIBuffSS is usually CPU bound and it is not currently possible to scale-out threads accepting data. There is also no way to run multiple instances of pibuffss on a node without virtualization. When you tried scaling out write nodes, how did you do it? You'll want to partition the tags so that each pibuffss node is processing a similar amount of data (for load balancing) but it is critical that each node has its own set of tags (a tag should always be written on one node and not another). When you say "doesn't help" what do you observe? Is your writer app backing up making buffered writes? Which version of AFClient/PIBuffSS are you running?
Of course I set on each node a different set of tags, both updating at steady speed. But each node pibufss starts to buffer a lot of events without being able to keep up. Each node is running with enough CPU RAM and SSD only and pibufss and the client app were using too much of resources. The network bandwidth and latency didn't seem to be the problem also.
Afsdk latest with the pibufss coming along the installer. I didn't install pisdk.
If the values are actually getting into the buffer, then the next thing to check would be the PI Data Archive. Is there much data being written to this PI Data Archive that is not going via buffer? Are the buffer instances able to keep up when there are no signups on the points they are writing?
During the test, only the updating node was actually using the pi data archive. I do think that a good tuning of the pi data archive is in order to remove it as a major bottleneck. but how to tune the pi data archive for this kind of workflow? Suggestions? CPU 6/12 ht cores 32G ram and SSD only. Does the synchronization parameters between primary and secondary can be a bottleneck?
Generalized metrics are often difficult to compare with a specific case as there are a multitude of factors (memory, network latency, data density, I/O speed, concurrent applications etc.) that can affect the performance of an application.
However with AF 2.9+ you should be able to generate metrics for your environment and use this as benchmark to verify performance improvements on code changes/upgrades.
Using metrics in your AF SDK code
For a discussion on Serial/Parallel/Bulk calls for data retrieval using AF SDK refer to KB01216 - AF SDK Performance: Serial vs. Parallel vs. Bulk