AnsweredAssumed Answered

System.OutOfMemoryException Error encountered with Bulk data methods (AF SDK)

Question asked by amitpatil.vit on Aug 1, 2019
Latest reply on Aug 6, 2019 by rdavin

Hello All,

I tried bulk data extraction for multiple tags using a pointlist and associated bulk data retrieval methods of AF SDK. It works for few thousand tags but somehow program is encountering System.OutOfMemoryException after fetching data for few thousand tags.


I think I need to clear resources (paging configuration) dynamically. Can someone guide me on how can I tackle it effectively?

A part of code is enclosed. I have added multiple tags to a PointList and then dividing a large duration into smaller duration for which data extraction per tag will be performed (with loops).


Private Sub PITag_Compressed_Data()
        'PI data retrival for each tag for all the months
        Dim i, j, k, statuscode, pagenumbers As Integer
        Dim tagvalout As String

        Dim timeRange As AFTimeRange
        Dim pointid, pointtype, timestampstring, datastring, Data_Filename As String

        Dim Start_Time_Derived_Temp, End_Time_Derived_Temp As Date() 'Array to hold start and end time values for no. of months
        Dim Time_Derived_Temp1, Time_Derived_Temp2 As Date   'Temp date holders
        ReDim Preserve Start_Time_Derived_Temp(Months_Derived - 1)
        ReDim Preserve End_Time_Derived_Temp(Months_Derived - 1)

        'Logic to generate array of start & end times in order to split the duration
        For i = 0 To Months_Derived - 1 '1st loop for number of months
            'Loop to generate start time, end time array to spilt verall duration
            'Start time and to end time array
            If i = 0 Then
                Start_Time_Derived_Temp(i) = Start_Time_Derived   'valid condition for 1st run only
                'Initialize first start time from input provided in apps.config
                Start_Time_Derived_Temp(i) = End_Time_Derived_Temp(i - 1) 'Previous end time becomes next start time
            End If
            Time_Derived_Temp1 = DateAdd(DateInterval.Month, 1, Start_Time_Derived_Temp(i))
            Time_Derived_Temp2 = Time_Derived_Temp1.AddDays(-(Time_Derived_Temp1.Day - 1))  'Deduct number of days-1 to arrive at 1st day of month
            If End_Time_Derived < Time_Derived_Temp2 Then
                End_Time_Derived_Temp(i) = End_Time_Derived
                End_Time_Derived_Temp(i) = Time_Derived_Temp2
            End If
        Next i  'Loop complete for start & end time arrays

        If TagListArray_Derived.Count > 0 Then
            pagenumbers = 10  'Set to some number
            'pagenumbers = TagListArray_Derived.Count
            pagenumbers = 1
        End If

        'Loop1 - Initiate the tag processing for number of months
        For k = 0 To Months_Derived - 1

            timeRange = AFTimeRange.Parse(Start_Time_Derived_Temp(k), End_Time_Derived_Temp(k), myPIServer.ServerTime, Nothing) 'Specify reference time (imp for relative time format)

            ' Holds the results keyed on the associated point
            Dim resultsMap As New Dictionary(Of PIPoint, AFValues)
            Dim config As New PIPagingConfiguration(PIPageType.TagCount, pagenumbers)
            Dim listResults As IEnumerable(Of AFValues) =
                    PointList1.RecordedValues(timeRange, AFBoundaryType.Inside, Nothing, False, config)

            Dim temp_af_val As AFValue = Nothing
            Dim m As Double ' For counting records

            'Loop2 for point attribute values extraction and data file creation
            For Each pointResults As AFValues In listResults

                ' Map the results back to the point
                resultsMap(pointResults.PIPoint) = pointResults
                'Create Data files per tag per month & Write output strings to data files  e.g. Filename: Tagname_201612231125.csv
                pointid = pointResults.PIPoint.GetAttribute(PICommonPointAttributes.PointID).ToString()
                pointtype = pointResults.PIPoint.GetAttribute(PICommonPointAttributes.PointType).ToString()

                Data_Filename = pointid & "_" & Start_Time_Derived_Temp(k).ToString("yyyy_MMM")
                Data_Filepath = Output_Folderpath & "\" & Data_Filename & DataFileExtension
                'Check If data file exists Or Not, Create if file Not available.
                If Not System.IO.File.Exists(Data_Filepath) Then
                End If
                Dim objWriter As New StreamWriter(Data_Filepath, True)    'System.IO.StreamWriter

                If pointResults.Count = 0 Then
                    Call Write_Log(Log_Filepath, "No data available for tag " &
                                pointResults.PIPoint.Name & " , Duration- " & timeRange.ToString &
                                " , Number of events extracted :" & m & ", Filename: " & Data_Filename & DataFileExtension) 'm will be 0
                    GoTo Next_PointResult 'skip value extraction and jump to next PointResult duration loop
                End If

                'Loop3 for PI data extraction and printing
                For Each temp_af_val In pointResults
                    'loop for writing results to flat files
                    'Check timestamp
                    timestampstring = Today.ToString 'value assigned just for initializing
                    If TimeFormat = "Local" Then
                        timestampstring = temp_af_val.Timestamp.LocalTime.ToString(TimeStampFormat)
                    ElseIf TimeFormat = "UTC" Then
                        timestampstring = temp_af_val.Timestamp.UtcTime.ToString(TimeStampFormat)
                    End If
                    'Check if good (0) or bad (1) 'Optional (can be commented if status is required as good or bad)
                    statuscode = 0 'value assigned just for initializing
                    If temp_af_val.Status.ToString = "Good" Then
                        statuscode = 0
                        tagvalout = temp_af_val.Value
                        statuscode = temp_af_val.Value * (-1)          'Changed for bad data results
                        tagvalout = ""
                    End If
                    'Write data value in data file

                    datastring = temp_af_val.PIPoint.Name & DataSeparator & timestampstring &
                                    DataSeparator & tagvalout & DataSeparator & 1 & DataSeparator & statuscode &
                                    DataSeparator & temp_af_val.Questionable & DataSeparator & temp_af_val.Substituted &
                                    DataSeparator & temp_af_val.Annotated & DataSeparator
                    objWriter.AutoFlush = True  'Important line when using streamwriter, if not used then last few chars are not written properly

                    m = m + 1
                Next    'Loop3 complete (nested)

                Call Write_Log(Log_Filepath, "PI data extracted for tag " &
                                temp_af_val.PIPoint.Name & " , Duration- " & timeRange.ToString &
                                " , Number of events extracted- " & m & ", Filename: " & Data_Filename & DataFileExtension)
                m = 0     'Reset event count
            Next        'Loop2 complete (nested)

            Threading.Thread.Sleep(10)    'Delay in ms before processing data extraction for next duration
        Next k      'Loop1 complete (extract data for next duration)

        Call Write_Log(Log_Filepath, "PI data extraction has been completed for " &
                            TagListArray_Derived.Count & " PI tags" & " from " & StartTime.ToString & " to " & EndTime.ToString &
                            " , Output Timezone Format " & TimeFormat & " .")
        Call Write_Log(Log_Filepath, "PI_Data_Extractor_OnDemand.exe process exited.")

        End 'Flush memory
    Catch canceledEx As OperationCanceledException

        ' Errors that occur during bulk calls get trapped here,The actual error is stored on the PIPagingConfiguration object
        Call Write_Log(Log_Filepath, canceledEx.Message)
    Catch otherEx As Exception
        ' Errors that occur in an iterative fall-back method get trapped here
        Call Write_Log(Log_Filepath, otherEx.Message)

    End Try

End Sub


[Moderator edits by Rick Davin: included attached code as instream text, and marked thread as a Question.]