7 Replies Latest reply on Sep 29, 2016 3:00 PM by Rick Davin Branched from an earlier discussion.

    Need help updating over 11,000 values per hour with AF SDK

    ebelaski

      I have a use case where I also am in need of a method to update a lot of data values for PI points. I am currently doing this recursively one Point at a time and it takes about 10 minutes. I would love to speed this up and make far fewer calls to the PI Server.

       

      My use case is:

      Pull hourly 48 hour weather forecast data from source and store values in PI future tags. The weather data consist of various attributes like temperature, humidity etc... 16 total. I have created an AF model to manage the Locations where the weather data is desired. Currently I have about 15 locations configured. Run an application each hour and update the PI values with the latest forecast values.

       

      An application in C# scrapes the AF model to get the PI tags for each Element (location). The forecast data is mapped to the correct PI tag and the forecast timestamp is added to the AFValue. Each location has 16 attributes with 48 hours of data = 768 updates per Location.

      With 15 location (and growing) I am making 11,520 updates per hour.

      I thought I could create a list like this :

       

      IEnumerable <PIPoint> pitag_oEnum = new List<PIPoint>()

       

      and then send the list all at one time but It look like I go through the same recursion to build the PIPoint list.

       

      Any suggestions?

      Thanks,

      Eric B.

       

      Message was edited by: Rick Davin.  Removed first sentence from another thread so that this thread stands on its own.

        • Re: Need help updating over 11,000 values per hour with AF SDK
          Rick Davin

          Eric Belaski  Could you start a new thread for your question?  The original Eric wanted to update PIPoint configurations, whereas you want to update the values or data.  Entirely different problem with different calls, so please make it a different thread.

           

          Update: after 12 hours, I moved branched this question to its own thread.  Original thread:

          AF SDK Update many tags attributes

          1 of 1 people found this helpful
          • Re: Need help updating over 11,000 values per hour with AF SDK
            Rick Davin

            Hi Eric,

             

            You definitely could benefit from bulk calls, and your use case seems well suited for it.  You could issue one bulk call of 11,520 values every hour.  Or you could issue 15 bulk calls (one per Location element) for 768 values (16 PIPoints times 48 hourly values).  Parallel calls could possibly be used if you issue 15 separate calls, but you would have to ask how that would affect the order of your logging, if any.

             

            To me, the most efficient solution would be one bulk call of 11,520 values once an hour.  It doesn't need parallel calls.  If you add more Location elements, the one bulk call still be efficient.  Unless you add lots and lots of new Locations and increase the values per Location.  I would like to think before it comes to that point, that you would have already modified your app to perhaps run every 15 minutes to break up the work load.

            • Re: Need help updating over 11,000 values per hour with AF SDK
              ebelaski

              As a follow-up post here are the results of revising the application.

              I revised to program to send AFValues to PI once for each location. As mentioned it could also be done very easily where values for all points are collected and then sent one time (see code below)

              My original attempt at this took approximately 12 minutes to run. After this change the program completes in about 10 seconds, a dramatic performance improvement.

              Thanks Rick!

               

              This is an abbreviated version of the code base.

               

              foreach (Forecast location in forecastList)
              {
               AFValues afValues = new AFValues();
               if (location.isGoodForecast)
               {
              
                 AFElement curLocation = forecastData.afElement;
              
                 foreach (HourForecast hrReading in forecastData.hourlyForecast)
                 {
                  try
                  {
              
                    Datetime curTimestamp = ah.UnixTimeStampToDateTime(hrReading.time);
                    AFTime curTime = new AFTime(curTimestamp);
                    afValues.Add(new AFValue(curLocation.Attributes["Temperature_2DFut"], hrReading.temperature, new AFTime(curTime)));
                    ... (repeat for all tags)
                  }
                  catch 
                  {
                   continue;
                  }
              
               }
              }
              // Update all PIPoints in the AFValues
              piserver.UpdateValues(allValues, AFUpdateOption.Replace);