Case Research: Unstructured Location Descriptions of Automotive Accidents
Knowledge Assortment and Preparation
To check out and quantify the geocoding capabilities of LLMs, a listing of 100 unstructured location descriptions of auto accidents in Minnesota had been randomly chosen from a dataset that was scraped from the online. The bottom fact coordinates for all 100 accidents had been manually created by using varied mapping purposes like Google Maps and the Minnesota Division of Transportation’s Site visitors Mapping Utility (TMA).
Some pattern location descriptions are featured under.
US Hwy 71 at MN Hwy 60 , WINDOM, Cottonwood County
EB Freeway 10 close to Joplin St NW, ELK RIVER, Sherburne County
EB I 90 / HWY 22, FOSTER TWP, Faribault County
Freeway 75 milepost 403, SAINT VINCENT TWP, Kittson County
65 Freeway / King Highway, BRUNSWICK TWP, Kanabec County
As seen within the examples above, there are broad number of potentialities for the way the outline will be structured, in addition to what defines the placement. One instance of that is the fourth description, which incorporates a mile marker quantity, which is much much less more likely to be matched in any form of geocoding course of, since that data sometimes isn’t included in any form of reference knowledge. Discovering the bottom fact coordinates for descriptions like this one relied closely on using the Minnesota Division of Transportation’s Linear Referencing System (LRS) which gives a standardized strategy of how roads are measured by out the State, with which mile markers play an important function in. This knowledge will be accessed by the TMA utility talked about beforehand.
Methodology & Geocoding Methods
After making ready the info, 5 separate notebooks had been set as much as take a look at out completely different geocoding processes. Their configurations are as follows.
1. Google Geocoding API, used on the uncooked location description
2. Esri Geocoding API, used on the uncooked location description
3. Google Geocoding API, used on an OpenAI GPT 3.5 standardized location description
4. Esri Geocoding API, used on an OpenAI GPT 3.5 standardized location description
5. OpenAI GPT 3.5, used as a geocoder itself
To summarize, the Google and Esri geocoding APIs had been used on each the uncooked descriptions in addition to descriptions that had been standardized utilizing a brief immediate that was handed into the OpenAI GPT 3.5 mannequin. The Python code for this standardization course of will be seen under.
def standardize_location(df, description_series):
df["ai_location_description"] = df[description_series].apply(_gpt_chat)return df
def _gpt_chat(input_text):
immediate = """Standardize the next location description into textual content
that might be fed right into a Geocoding API. When responding, solely
return the output textual content."""
response = openai.ChatCompletion.create(
mannequin="gpt-3.5-turbo",
messages=[
{"role": "system", "content": prompt},
{"role": "user", "content": input_text},
],
temperature=0.7,
n=1,
max_tokens=150,
cease=None,
)
return response.decisions[0].message.content material.strip().cut up("n")[-1]
The 4 take a look at circumstances utilizing geocoding APIs used the code under to make API requests to their respective geocoders and return the ensuing coordinates for all 100 descriptions.
# Esri Geocoder
def geocode_esri(df, description_series):
df["xy"] = df[description_series].apply(
_single_esri_geocode
)df["x"] = df["xy"].apply(
lambda row: row.cut up(",")[0].strip()
)
df["y"] = df["xy"].apply(
lambda row: row.cut up(",")[1].strip()
)
df["x"] = pd.to_numeric(df["x"], errors="coerce")
df["y"] = pd.to_numeric(df["y"], errors="coerce")
df = df[df["x"].notna()]
df = df[df["y"].notna()]
return df
def _single_esri_geocode(input_text):
base_url = "https://geocode-api.arcgis.com/arcgis/relaxation/providers/World/GeocodeServer/findAddressCandidates"
params = {
"f": "json",
"singleLine": input_text,
"maxLocations": "1",
"token": os.environ["GEOCODE_TOKEN"],
}
response = requests.get(base_url, params=params)
knowledge = response.json()
attempt:
x = knowledge["candidates"][0]["location"]["x"]
y = knowledge["candidates"][0]["location"]["y"]
besides:
x = None
y = None
return f"{x}, {y}"
# Google Geocoder
def geocode_google(df, description_series):
df["xy"] = df[description_series].apply(
_single_google_geocode
)df["x"] = df["xy"].apply(
lambda row: row.cut up(",")[0].strip()
)
df["y"] = df["xy"].apply(
lambda row: row.cut up(",")[1].strip()
)
df["x"] = pd.to_numeric(df["x"], errors="coerce")
df["y"] = pd.to_numeric(df["y"], errors="coerce")
df = df[df["x"].notna()]
df = df[df["y"].notna()]
return df
def _single_google_geocode(input_text):
base_url = "https://maps.googleapis.com/maps/api/geocode/json"
params = {
"deal with": input_text,
"key": os.environ["GOOGLE_MAPS_KEY"],
"bounds": "43.00,-97.50 49.5,-89.00",
}
response = requests.get(base_url, params=params)
knowledge = response.json()
attempt:
x = knowledge["results"][0]["geometry"]["location"]["lng"]
y = knowledge["results"][0]["geometry"]["location"]["lat"]
besides:
x = None
y = None
return f"{x}, {y}"
Moreover, one ultimate course of examined was to make use of GPT 3.5 because the geocoder itself, with out the assistance of any geocoding API. The code for this course of regarded almost equivalent to the standardization code used above, however featured a special immediate, proven under.
Geocode the next deal with. Return a latitude (Y) and longitude (X) as precisely as doable. When responding, solely return the output textual content within the following format: X, Y
Efficiency Metrics and Insights
After the varied processes had been developed, every course of was run and several other efficiency metrics had been calculated, each by way of execution time and geocoding accuracy. These metrics are listed under.
| Geocoding Course of | Imply | StdDev | MAE | RMSE |
| ------------------- | ------ | ------ | ------ | ------ |
| Google with GPT 3.5 | 0.1012 | 1.8537 | 0.3698 | 1.8565 |
| Google with Uncooked | 0.1047 | 1.1383 | 0.2643 | 1.1431 |
| Esri with GPT 3.5 | 0.0116 | 0.5748 | 0.0736 | 0.5749 |
| Esri with Uncooked | 0.0001 | 0.0396 | 0.0174 | 0.0396 |
| GPT 3.5 Geocoding | 2.1261 | 80.022 | 45.416 | 80.050 |
| Geocoding Course of | 75% ET | 90% ET | 95% ET | Run Time |
| ------------------- | ------ | ------ | ------ | -------- |
| Google with GPT 3.5 | 0.0683 | 0.3593 | 3.3496 | 1m 59.9s |
| Google with Uncooked | 0.0849 | 0.4171 | 3.3496 | 0m 23.2s |
| Esri with GPT 3.5 | 0.0364 | 0.0641 | 0.1171 | 2m 22.7s |
| Esri with Uncooked | 0.0362 | 0.0586 | 0.1171 | 0m 51.0s |
| GPT 3.5 Geocoding | 195.54 | 197.86 | 199.13 | 1m 11.9s |
The metrics are defined in additional element right here. Imply represents the imply error (by way of Manhattan distance, or the whole of X and Y distinction from the bottom fact, in decimal levels). StdDev represents the usual deviation of error (by way of Manhattan distance, in decimal levels). MAE represents the imply absolute error (by way of Manhattan distance, in decimal levels). RMSE represents the foundation imply sq. error (by way of Manhattan distance, in decimal levels). 75%, 90%, 95% ET represents the error threshold for that given % (by way of Euclidean distance, in decimal levels), which means that for a given share, that share of information falls inside the ensuing worth’s distance from the bottom fact. Lastly, run time merely represents the whole time taken to run the geocoding course of on 100 information.
Clearly, GPT 3.5 performs far worse by itself. Though, if a pair outliers are taken out of the image (which had been labelled by the mannequin as being situated in different continents), for essentially the most half the outcomes of that course of don’t look too misplaced, visually at the very least.
It’s also attention-grabbing to see that the LLM-standardization course of truly decreased accuracy, which I personally discovered a bit stunning, since my complete intention of introducing that element was to hopefully barely enhance the general accuracy of the geocoding course of. It’s price noting that the prompts themselves may have been part of the issue right here, and it’s price additional exploring the function of “immediate engineering” in geospatial contexts.
The final major takeaway from this evaluation is the execution time variations, with which any course of that features using GPT 3.5 performs considerably slower. Esri’s geocoding API can also be slower than Google’s on this setting too. Rigorous testing was not carried out, nevertheless, so these outcomes must be taken with that into consideration.
