23. Online Search Trends (Illustrative)#
Knowing what people are searching online or the change in their searching patterns can be used to understand people’s behavior or needs after a crisis. Google Trends is an API that allows users to obtain data about what people searched online in real time and in the past for a region of interest. This dataset has been widely used for monitoring COVID-19 outbreak, for understanding goods shortages, etc.
This notebook shows how to work with Google Trends data through an example that leverages this data to understand the impact of Morocco’s 2023 earthquake to tourism.
23.1. Learning Objectives#
23.1.1. Overall goals#
The main goal of this class is for students to understand Google Trends data and how it can be leveraged for economic monitoring.
23.1.2. Specific goals#
Understand Google Trends data:
Introduce the dataset and its different forms.
Present the different filters that can be applied to the obtained disaggregated datasets.
Analyze Google Trends data:
Learn how to access the data.
Define the filters for a particular problem.
Visualize the data across the years.
Predict search term behavior.
Apply Google Trends data:
Study the earthquake impact on the tourism industry .
Detect population’s needs after the earthquake.
23.2. About the data#
Google Trends provides a random sample of aggregated, anonymized, and categorized Google and YouTube searches. It lets you analyze interest in a particular query or topic from around the globe or at city-level geography.
The values obtained from Google Trends are not search volumes. The data is normalized to make comparisons between terms easier. Search results are normalized to the time and location of a query by the following process:Each data point is divided by the total searches of the geography and time range it represents to compare relative popularity. Otherwise, places with the most search volume would always be ranked highest.
The resulting numbers are then scaled on a range of 0 to 100 based on a topic’s proportion to all searches on all topics. (Applies only to public tool).
Different regions that show the same search interest for a term don’t always have the same total search volumes.
The data can be accessed for free through Trends Explore Tool or through their API if you have an agreement with Google.
The search in Google Trends can be done by adjusting:
Country: Where the searches are being done.
Date range: It can be from today till 2004.
Terms or Topics: Terms, like for any other Google search are the terms to look for. Topics, on the other hand, are generally considered to be more reliable for Google Trends data. They pull in the exact phrase as well as misspellings and acronyms, and cover all languages. This is more useful, particularly when looking at world data. More information about this data can be found in the Basics of Google Trends.
Categories: For example, All, Art and Entertainment, Real Estate, etc.
Where to search: Web Search/Images Search/News Search/Google Shopping/YouTube Search.
As an example, we downloaded Google Trends data for Simone Biles from 2004 till now. It can be seen that every four years, the olympiad span, her name gets more relevance.
Fig. 23.1 Google Trends Tool result for “Simone Biles”.#
23.3. Methodology for analyzing Google Trends- Moroccos’s 2023 Earthquake Response#
The following code will show you how to access and analyze Google Trends data using the methodology the Data Lab used for Morocco’s 2023 earthquake response. The analysis was performed days after September 2023 Morocco’s earthquake happened. It tries to answer two questions:
Will the earthquake have an impact on the tourism industry?
Is there a change in search patterns for basic need services after the earthquake?
23.3.1. Impact on the Tourism Industry#
This part of the analysis focused on 5 countries in specific, the ones that make up for the highest number of visitors in Morocco: France, Germany, Spain, United Kingdom, United States. Thus, a list of commonly used terms that people tend to use to plan their holidays was created. The kinds of searches that users perform can be a good proxy for the public’s interests, concerns or intentions, but these searches do not necessarily represent users’ opinions. Therefore, the analysis uses statistical methods to come up with a probabilistic analysis of impact on Morocco’s tourism industry.
The following code uses Classes and Methods. If you are not familiar with this programming methodology, you can find more by searching for Object Oriented Programming (OOP).
The Class Google, as stated in its documentation, is used for handling the authentication and requests to Google API. You will need to save your Google API key as an environmental variable named GOOGLE_TOKEN in order to be correctly authenticated.
23.3.1.1. Lists of commonly used travel search terms in different languages catering to United States, United Kingdom, France, Germany and Spain.#
eng_terms = ['morocco flights', 'morocco hotels', 'marrakesh', 'marrakesh hotels', 'casablanca morocco','casablanca hotels',
'casablanca flights', 'casablanca travel', 'morocco tourist attractions', 'things to do in morocco', 'things to do in casablanca',
"moroccan cuisine", "beaches in morocco", "Safety in Morocco", "transportation in Morocco"]
french_terms = ['séjour à marrakech', "vacance maroc", "voyages marrakech", "billet avion Maroc", "casablanca hotels", "marrakech hotels",
"Cuisine marocaine", "Sécurité au Maroc"]
spanish_terms = ['Vuelos a Marruecos', 'Hoteles en Marruecos', 'Casablanca Marruecos', 'Hoteles en Casablanca', 'Vuelos a Casablanca', 'Viaje a Marruecos', 'Que hacer en Marruecos',
"Rutas por Marruecos", "Playas en Marruecos", "Seguridad en Marruecos"]
german_terms = ['Marokko Hotels', 'Hotels in Marrakesch', "things to do in morocco", "Pauschalreisen Marokko", "Marokko Sehenswürdigkeiten",
"Marokko Rundreise", "Marokko Strände", "Marokko Sicherheit", "Marokko Transport"]
con_dict = {"US" : "United States", "GB" : "United Kingdom", "ES" : "Spain", "FR" : "France", "DE" : "Germany"}
This example downloads the data from January 2023 till the 12th of November 2023 when the analysis was performed.
ax = df2023.reset_index().pivot(index = 'date', columns = 'country', values = 'value').plot(figsize = (15, 8))
ax.set_title("Google Search Trends for the year 2023", fontsize = 12);
23.3.1.2. Comparing the trends for the past 3 years for the same time period (Jan - Sep)#
def download_several_countries(list_countries, period_start, period_end):
df = pd.DataFrame()
for country in list_countries:
df_ = get_country_df(country, period_start, period_end)
df_['country'] = con_dict[country]
df = pd.concat([df, df_])
df['month'] = [dt.strptime(i, "%Y-%m-%d").strftime("%b-%d") for i in df.index]
return df
# Download the data with automation function
list_countries = ['US', 'GB', 'FR', 'ES', 'DE']
df2022 = download_several_countries(list_countries, "2022-01", "2022-09")
df2021 = download_several_countries(list_countries, "2021-01", "2021-09")
for abbr, name in con_dict.items():
plot_charts(abbr, df2023[df2023['country']==name], df2022[df2022['country']==name], df2021[df2021['country']==name])
2022 was one of the busiest years for travel in a decade, and the trends in the plots above show that. For 2023, the trends went slightly down, but some months show spikes in searches for travel terms.
23.3.1.3. Prediction#
To predict if tourism will be impacted using Google Trends data, data points for set query terms from 2019 to latest available date (removing the time periods when the country was under COVID lockdown) were collected.
Specific terms related to flight/hotel cancellation were used, but Google Trends did not have enough data to show that. This prediction is solely based on yearly trends and seasonality.
RMS error for United States's data is 7.810
Downward trend: -0.01134
RMS error for Spain's data is 10.025
Upward trend: 0.00197
RMS error for France's data is 7.894
Downward trend: -0.01797
RMS error for Germany's data is 9.968
Downward trend: -0.01918
RMS error for United Kingdom's data is 7.174
Downward trend: -0.00320
23.3.2. Needs Assessment#
This part of the analysis uses Google Trends data within Morocco to see if there was a change in the trend of some of the key search terms categories. The search terms comprise of English, French, and Arabic terms that could’ve been used by the local communities or the search and rescue operations staff. The analysis also takes a deeper dive at the admin regions most affected by the earthquake to see if there’s any considerable trend for the search terms.
water_terms = ['water', 'drinking water', 'يشرب الماء', 'ماء', 'eau']
food_terms = ['grocery store', 'souk', 'خضروات', 'restaurant near me', 'store' ]
ph_terms = ['toilette', 'الحمام', 'toilet']
hosp_terms = ['مستشفى', 'hôpital', 'clinique', 'عيادة', 'hospital', 'clinic', 'medical supply', 'medicines', 'الأدوية']
First, we analyze the trends of search keywords for before and after the earthquake period for Morocco. At the national level, we do not observe a significant change in the search trends.
Trends for before earthquake period of water terms
Downward trend: -0.009045352673062848
Trends for after earthquake period of water terms
Downward trend: -0.03261401557285964
Trends for before earthquake period of food terms
Upward trend: 0.019455566467728967
Trends for after earthquake period of food terms
Downward trend: -0.1265406006674088
Trends for before earthquake period of personal hygiene terms
Upward trend: 0.0461836770668073
Trends for after earthquake period of personal hygiene terms
Upward trend: 0.10967741935483752
Trends for before earthquake period of hospital terms
Downward trend: -0.12651406411321814
Trends for after earthquake period of hospital terms
Upward trend: 0.028896304535903285
23.3.2.1. Analyzing data at subnational regions for impacted areas#
Béni Mellal-Khénifra Region
Trends for Béni Mellal-Khénifra before earthquake period of water terms
Downward trend: -0.06971871773564
Trends for Béni Mellal-Khénifra after earthquake period of water terms
Upward trend: 0.1005116796440489
============*****=================
Trends for Béni Mellal-Khénifra before earthquake period of food terms
Downward trend: -0.030329127949434608
Trends for Béni Mellal-Khénifra after earthquake period of food terms
Upward trend: 0.2434705228031145
============*****=================
Trends for Béni Mellal-Khénifra before earthquake period of personal hygiene terms
No trend
Trends for Béni Mellal-Khénifra after earthquake period of personal hygiene terms
Downward trend: -0.1853911753800521
============*****=================
Trends for Béni Mellal-Khénifra before earthquake period of hospital terms
Downward trend: -0.06382782268556994
Trends for Béni Mellal-Khénifra after earthquake period of hospital terms
Upward trend: 0.08130021010999876
============*****=================
Marrakech-Safi Region
needs_terms = {"water" : water_terms, "food" : food_terms, "personal hygiene" : ph_terms, "hospital" : hosp_terms}
for terms in needs_terms.keys():
bef = get_df(needs_terms[terms], "MA-07", start_date='2023-07', end_date = "2023-08")
aft = get_df(needs_terms[terms], "MA-07", start_date='2023-09', end_date = '2023-09')
print(f"Trends for Marrakech-Safi Region before earthquake period of {terms} terms")
get_needs_trends(bef)
print(f"Trends for Marrakech-Safi Region after earthquake period of {terms} terms")
get_needs_trends(aft)
plot_needs_trends(bef, aft, terms, 'Marrakech-Safi Region')
print("============*****=================")
Trends for Marrakech-Safi Region before earthquake period of water terms
Downward trend: -0.04599733071441194
Trends for Marrakech-Safi Region after earthquake period of water terms
Upward trend: 0.0025806451612901617
============*****=================
Trends for Marrakech-Safi Region before earthquake period of food terms
Downward trend: -0.03227317367983696
Trends for Marrakech-Safi Region after earthquake period of food terms
Upward trend: 0.10095661846496075
============*****=================
Trends for Marrakech-Safi Region before earthquake period of personal hygiene terms
Upward trend: 0.005464480874316922
Trends for Marrakech-Safi Region after earthquake period of personal hygiene terms
Downward trend: -0.02224694104560624
============*****=================
Trends for Marrakech-Safi Region before earthquake period of hospital terms
Downward trend: -0.03399561834252485
Trends for Marrakech-Safi Region after earthquake period of hospital terms
Downward trend: -0.1000370782350763
============*****=================
Souss-Massa Region
needs_terms = {"water" : water_terms, "food" : food_terms, "personal hygiene" : ph_terms, "hospital" : hosp_terms}
for terms in needs_terms.keys():
bef = get_df(needs_terms[terms], "MA-09", start_date='2023-07', end_date = "2023-08")
aft = get_df(needs_terms[terms], "MA-09", start_date='2023-09', end_date = '2023-09')
print(f"Trends for Souss-Massa Region before earthquake period of {terms} terms")
get_needs_trends(bef)
print(f"Trends for Souss-Massa Region after earthquake period of {terms} terms")
get_needs_trends(aft)
plot_needs_trends(bef, aft, terms, 'Souss-Massa Region')
print("============*****=================")
Trends for Souss-Massa Region before earthquake period of water terms
Upward trend: 0.028712447432701255
Trends for Souss-Massa Region after earthquake period of water terms
Upward trend: 0.27083426028921026
============*****=================
Trends for Souss-Massa Region before earthquake period of food terms
Downward trend: -0.05153735740726758
Trends for Souss-Massa Region after earthquake period of food terms
Downward trend: -0.05477196885428272
============*****=================
Trends for Souss-Massa Region before earthquake period of personal hygiene terms
Upward trend: 0.034415317334407766
Trends for Souss-Massa Region after earthquake period of personal hygiene terms
Upward trend: 0.27771598071931786
============*****=================
Trends for Souss-Massa Region before earthquake period of hospital terms
Upward trend: 0.004110252127174388
Trends for Souss-Massa Region after earthquake period of hospital terms
Downward trend: -0.0620195278704734
============*****=================
23.4. Practice#
Using Google Trends Free Tool:
Search the trend for Gel Alcohol worldwide since 2004 till today.
Which is the country with the highest interest in the term?