How to use NextBus Vehicle Data as a Data Source for the PI Integrator for Esri ArcGIS

Blog Post created by DLopez on Mar 3, 2015

To get started quickly with the PI Integrator for Esri ArcGIS (for more information, see OSIsoft and Esri: Bringing Space & Time Together - YouTube), you can use freely available data from NextBus, which provides real-time (updating every minute to every few minutes) for mobile assets, which can make for a very fascinating real-world data source.


The following PowerShell script (note: you can copy this code into a text file on your own machine and save it with a .ps1extension; you may have to run PowerShell as an Administrator and change your executionpolicy before the script can run; see How to Allow the Execution of PowerShell Scripts on Windows 7):


#write-host "Downloading NextBus Bug Data"
# Create the folder that will hold all of the downloaded files (if it does not exist yet)
$MyFolderName = "DownloadedNextBusData"
$MyFileExtension = ".csv"
$MyDestinationFolder = "C:\" + $MyFolderName + "\"
If(!(Test-Path -Path $MyDestinationFolder))
    New-Item -ItemType directory -Path $MyDestinationFolder

# Create a Web Client object that will make the requests
$MyWebClient = new-object system.net.WebClient

# Define the URL that you will be downloading from; see http://www.nextbus.com/predictor/simpleRegionSelector.shtml
# After selecting an agency and route in that web site, look at the page URL in your browser address bar, for example:
#              http://www.nextbus.com/predictor/simpleStopSelector.shtml?a=sf-muni&r=49
# Your agency is the parameter after "a=", and your route number is the parameter after "r="
$MyBusAgency = "actransit"
$MyBusRoutes = @("89")
#$MyBusRoutes = ""

for ($i=0; $i -lt $MyBusRoutes.Count; $i++)
    $MyBusRoute = $MyBusRoutes[$i]
    $MyURLString = "http://webservices.nextbus.com/service/publicXMLFeed?command=vehicleLocations&a=" + $MyBusAgency + "&r=" + $MyBusRoute + "&t=0"

    # Create a new file
    $MyStreamWriter = [System.IO.StreamWriter] ($MyDestinationFolder + $MyBusAgency + "_" + $MyBusRoute + $MyFileExtension)
    # Download the data 
    $MyContent = $MyWebClient.DownloadString($MyURLString)

    # Sanitize the content (remove bad characters, etc)
    $MyContent = $MyContent -replace """ ", ","
    $MyContent = $MyContent -replace """", ""
    $MyContent = $MyContent -replace "<\?", ""
    $MyContent = $MyContent -replace "\?>", ""
    $MyContent = $MyContent -replace "</", ""
    $MyContent = $MyContent -replace "/>", ""
    $MyContent = $MyContent -replace "<", ""
    $MyContent = $MyContent -replace ">", ""
    $MyContent = $MyContent -replace "=", ","

    # Split into lines
    $MyContentArray = $MyContent.Split("`n")
    $MyContentarray_Sorted = $MyContentArray | sort

    $LineNumber = 0
    for ($i=0; $i -lt $MyContentarray_Sorted.Count; $i++)
        if ($MyContentarray_Sorted[$i] -like "vehicle id*")
            # Write the data to the file
            $MyStreamWriter.WriteLine("" + $LineNumber + "," + $MyContentarray_Sorted[$i])
            $LineNumber = $LineNumber+1

    # Dispose now that the downloads are complete




... can be set up to run as a Windows scheduled task (see Use Scheduled Tasks to Run PowerShell Commands on Windows - Hey, Scripting Guy! Blog - Site Home - TechNet Blogs and http://blogs.technet.com/b/heyscriptingguy/archive/2011/01/12/use-scheduled-tasks-to-run-powershell-commands-on-windows.aspxSchedule a task - Windows Help), scheduled to execute every two minutes, for example (after downloading it--see attachments--simply try double-clicking the script to make sure that it works).  It will yield a text file output that looks like the following:


0,vehicle id,5030,routeTag,89,dirTag,89_43_1,lat,37.7271041,lon,-122.1477813,secsSinceReport,60,predictable,true,heading,330,speedKmHr,11
1,vehicle id,5034,routeTag,89,dirTag,89_44_0,lat,37.6863212,lon,-122.1510848,secsSinceReport,60,predictable,true,heading,178,speedKmHr,0
2,vehicle id,5040,routeTag,89,dirTag,89_43_1,lat,37.681179,lon,-122.1569442,secsSinceReport,60,predictable,true,heading,135,speedKmHr,16
3,vehicle id,5050,routeTag,89,dirTag,89_44_0,lat,37.7182464,lon,-122.1796035,secsSinceReport,60,predictable,true,heading,330,speedKmHr,10
4,vehicle id,5137,routeTag,89,dirTag,89_43_1,lat,37.7058982,lon,-122.1299057,secsSinceReport,60,predictable,true,heading,0,speedKmHr,0


This output is perfectly suited to the PI Interface for Universal File and Stream Loading (PI UFL) (see OSIsoft: What is PI UFL? - YouTube, or the PI UFL user manual).  This PI Interface can take in this output and write the data to PI tags.


Once the data is listed within PI tags, it can be organized using PI AF Element Templates (an example is attached) (since each vehicle always sends the same information each time: unique vehicle ID, route name, direction name, latitude, longitude, the time in seconds since the last report, whether the route is predictable or not (you won't use this), the heading in degrees (see Course (navigation) - Wikipedia, the free encyclopedia), and the vehicle speed, in kph), so that the PI Integrator for Esri ArcGIS can be used to create a live map layer from that AF Element Template.




Additionally, all sorts of interesting analyses can be done using this data; for example:


  • Event frames can be set up to track periods when a vehicle's speed is over a certain threshold, such as a local speed limit
  • PI AF Analyses can be used to calculate (using great-circle trigonometry: http://www.movable-type.co.uk/scripts/latlong.html) the distance between a vehicle and a specific location
  • A preventative maintenance routine can be set up to monitor the time since the last report--specifically, so see whether that time slowly increases over time, which could be indicative of an impending failure of the radio transmitter
  • Driving behavior analyses, using the standard deviation of the vehicle speed; since the vehicles almost always are driven on a prescribed route, the standard deviation of their speed should be fairly constant; a sudden change in standard deviation would thus indicate radically different driving behavior


An interesting note here, concerning how you would set up a PI UFL interface:


You might be tempted to create a PI UFL interface to automatically create PI tags using the vehicle ID, which is unique for each vehicle.  HOWEVER, not every vehicle is always on the road; there are general many more vehicles that exist in the bus depot, as opposed to vehicles that are actually on the road.  Thus, if you create a PI UFL interface for all of the vehicle ID's that you see in your first data download, you should know that those tags may not always update very regularly.  There may always be 9 active vehicles, and your data download might thus always return 9 or so lines of data with each run, but those 9 vehicles may not always be the ones you saw when you first ran the script.  This might be fine with you, or you might really want to always see at least a few moving assets, no matter what.


The solution, then, is to change your PI UFL interface to instead not store data by unique vehicle ID, but instead to store data based on the line number.  The PowerShell script appends a line number to each line in the data output, and you can use that line number to create, for example, 9 sets of tags, for the first 9 vehicles on the route (which correspond to the first 9 lines of data in the downloaded file); then, you can set up the PI UFL interface to send data to each of those groups of tags in turn, which will guarantee that even though the first bus on the route might be a different vehicle every day, regardless of vehicle ID, data for that first bus will be stored to the same set of PI tags, so you'll always have 9 moving vehicles on your layer.


The below PI UFL interface .ini file is an example of how this can be done; you can create a PI UFL interface and use this .ini file to parse the data files in the manner I described.  Your PI AF database will then look something like this:


AF Screenshot.png


Rather than listing all of the different vehicle ID's, for a given route (for example, route 89) upon which there are always 5 buses (at different points along the route), there will be 5 elements, one for each of those five buses (which again, have different vehicle ID's each day, because different vehicles from the depot are assigned to different routes).


The UFL .ini file:



' You may have to change this path if you change the download location

' You may have to change the log paths depending on your installation  
MSGINERROR=C:\Program Files (x86)\PIPC\Interfaces\PI_UFL\Logs\vehicles_err.log
OUTPUT=C:\Program Files (x86)\PIPC\Interfaces\PI_UFL\Logs\vehicles_out.log


' I recommend building tags manually, or, better yet, building tags via the PI AF element template

' Filter to only select lines with the vehicle id
MSG_1.FILTER=C1=="*vehicle id*"


' You can see that the PI tag names aren't based on the vehicle ID, but instead on the 
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_VehicleID"), ,Now(),VehicleID, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Route"), ,Now(),RouteTag, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Direction"), ,Now(),Direction, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Latitude"), ,Now(),Latitude, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Longitude"), ,Now(),Longitude, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Seconds Since Last Report"), ,Now(),SecSinceReport, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Predictable"), ,Now(),Predictable, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Heading"), ,Now(),Heading, , )
StoreInPI(CONCAT(CONCAT(CONCAT(RouteTag,"_"),NEWVehicleNumber),"_Speed"), ,Now(),SpeedKmH, , )


If you're wondering what PI tags you need to create, you need to create 9 tags per vehicle--in this case, these tags are for a bus on route 89, and specifically, for the first bus on that route (line number 0, not vehicle id 0).


agencyname_linenumber_Seconds Since Last Reportfloat32


I've included a sample PI tag list file that you can use to create tags in bulk with PI tag configurator; hopefully they can help you get started creating tags quickly.


All in all, this is a great way to research and explore geo-spatial data with the PI System, and in a manner that is completely free of charge, and that doesn't require any extra equipment.  I hope it proves useful for your research!