More plotting with pandas

Things we will cover:

• scatter plots
• histograms
• grouping data
• box-and-whisker plots
• plotting on a map

First, the usual imports…

import matplotlib.pyplot as plt
import pandas as pd

Scatter plots

Let’s use the box office dataset

movie_data = pd.read_csv('all-time-worldwide-box-office.csv')
movie_data.head()

Remember, we need to clean those dollar amounts up. This is the same thing we did last time, except we’ll also conver the dollar amounts to billions at the same time.

def clean_money(entry):
    if isinstance(entry, float):
        return 0

    return int(entry.strip('$').replace(',', ''))

movie_data['Worldwide'] = movie_data['WorldwideBox Office'].apply(clean_money)/1e9
movie_data['Domestic'] = movie_data['DomesticBox Office'].apply(clean_money)/1e9
movie_data['International'] = movie_data['InternationalBox Office'].apply(clean_money)/1e9
movie_data.head()

To make a scatter plot, use:

kind='scatter'

Scatter plots work well when the values the data can take are granular. For example, no two movies likely have the exact same box office numbers. So let’s scatter year versus worldwide box office. This is a nice way to see what is in the dataset.

movie_data.plot(x='Year', y='Worldwide', kind='scatter',
                ylabel='WorldWide Box Office (billions)', xlabel='Year')

    <AxesSubplot:xlabel='Year', ylabel='WorldWide Box Office (billions)'>

Scatter plots are also useful when you want to see if there is a relationship between two columns in your dataset. For example, if a movie has a high domestic box office, is it likely to also have a high international box office? Let’s do a scatter plot of these two columns.

movie_data.plot(x='Domestic', y='International', kind='scatter',
                xlabel='Domestic Box Office (billions)',
                ylabel='International Box Office (billions)')

    <AxesSubplot:xlabel='Domestic Box Office (billions)', ylabel='International Box Office (billions)'>

One of the useful things about a scatter plot is we can look for and then examine outliers. What are these movies that have very low International box office numbers?

low_international = movie_data['International'] < 0.1
movie_data[low_international]

What about those movies that have very high Domestic box office numbers (all of them also doing quite well internationally)?

outliers = movie_data['Domestic'] > 0.6
movie_data[outliers]

Histograms

A histogram let’s us visualize how many things meet certain criteria. For examle, how many movies manage to make a billion at the box office? We can do a histogram on the worldwide box office to see this. To create a histogram, you can use hist() on a single column or an entire data frame:

movie_data.hist(figsize=(10,10))

    array([[<AxesSubplot:title={'center':'Rank'}>,
            <AxesSubplot:title={'center':'Year'}>],
           [<AxesSubplot:title={'center':'Worldwide'}>,
            <AxesSubplot:title={'center':'Domestic'}>],
           [<AxesSubplot:title={'center':'International'}>, <AxesSubplot:>]],
          dtype=object)

movie_data.hist(column='Worldwide')

    array([[<AxesSubplot:title={'center':'Worldwide'}>]], dtype=object)

To put labels on the axes of this plot, we need to use matplotlib because they don’t work inside of hist():

movie_data.hist(column='Worldwide')
plt.xlabel('Worldwide Box Office (billions)')
plt.ylabel('Number of Movies')
plt.title('Histogram of Worldwide Box Office')

    Text(0.5, 1.0, 'Histogram of Worldwide Box Office')

We can specify the number of bins to use in the histogram

movie_data.hist(column='Worldwide',bins=20)
plt.xlabel('Worldwide Box Office (billions)')
plt.ylabel('Number of Movies')
plt.title('Histogram of Worldwide Box Office')

    Text(0.5, 1.0, 'Histogram of Worldwide Box Office')

We can specify the exact bins to use:

movie_data.hist(column='Worldwide', bins=[0,0.5,1,1.5,2,2.5,3])
plt.xlabel('Worldwide Box Office (billions)')
plt.ylabel('Number of Movies')
plt.title('Histogram of Worldwide Box Office')

    Text(0.5, 1.0, 'Histogram of Worldwide Box Office')

Grouping Data

When you have data that is categorical — it takes a discrete set of values — you may want to group the data according to these values.

What are you going to do with the data once it is grouped? Maybe you want to look at the mean or max or min money earned by all the movies released each year.

Let’s group all of our movies by the year they were released and look at their mean box office:

means_per_year = movie_data.groupby('Year').mean().reset_index()
means_per_year.head()

Notice that we used reset_index() to get Year added as a column.

Now we can plot this:

recent = means_per_year['Year'] > 2000
means_per_year[recent].plot(x='Year', y='Worldwide', ylabel='Mean Worldwide Box Office (billions)', )

    <AxesSubplot:xlabel='Year', ylabel='Mean Worldwide Box Office (billions)'>

We can also look at the total box office of all movies released each year, using sum():

total_per_year = movie_data.groupby('Year').sum().reset_index()
total_per_year.head()

Now we can plot this:

total_per_year.plot(x='Year', y='Worldwide', ylabel='Total Box Office')

    <AxesSubplot:xlabel='Year', ylabel='Total Box Office'>

Box-and-whisker plots

A box-and-whisker plot is useful for exploring multiple statistics that characterize a distribution of values.

We can divide all the data into quartiles — four segments that each contain 25% of the data:

• Q1 is the 25th-percentile. 25% of the data is below that point.
• Q2 is the 50th-percentile — the median. Half of the data is at or below that point, and half above it.
• Q3 is the 75th percentile. 75% of the data is below that point.

A box plot shows all three of these.

Let’s create a box-and-whisker plot that shows statistics about the worldwide box office for movies since the year 2000.

recent = movie_data['Year'] > 2000
means = movie_data[recent].groupby('Year').mean()
means.boxplot(column=['Worldwide','International','Domestic'])

    <AxesSubplot:>

• The bottom of the box is Q1, the top of the box is Q3.

• The middle line is the Q2, the median.

• The distance between Q3 and Q1 is the interquartile range (IQR).

• Each whisker extends to the furthest data point that is within 1.5 times the IQR.

• Dots are outliers

Use box plots carefully. These two histograms have the same box-and-whiskers plot.

You really should use a violin plot instead of a box-and-whiskers plot. But for that you should use seaborn.

import seaborn as sns

recent_movies = movie_data[recent]
sns.violinplot(data=recent_movies, y="Worldwide")

    <AxesSubplot:ylabel='Worldwide'>

In the world of Python plotting:

• matplotlib — does everything, low level
• pandas — uses dataframes, built on matplotlib, can always use matplotlib to tweak
• seaborn — tries to do a small number of things well, and make them look pretty

We won’t spend time on violin plots here. Go here if you want to learn more about violin plots.

You can plot lots of box plots at once using by inside of boxplot():

recent = movie_data['Year'] > 2010
movie_data[recent].boxplot(by="Year", column='Worldwide', figsize=(8,5))
# Because we are plotting a bunch of things, we have  both 'suptitle' and 'title' available
plt.suptitle('Worldwide Box Office')
plt.title('')
plt.ylabel('Worldwide Box Office (billions)')

    Text(0, 0.5, 'Worldwide Box Office (billions)')

import seaborn as sns

recent_movies = movie_data[recent]
fig, ax = plt.subplots(figsize=(12, 5))
sns.violinplot(data=recent_movies, x="Year", y="Worldwide", ax=ax)

    <AxesSubplot:xlabel='Year', ylabel='Worldwide'>

Plotting on a map of the world

Let’s use a new data set — one that includes oil spills and their locations.

Let’s take a look at the dataset:

! head incidents.csv

It has lots of empty data points!

oil_spills = pd.read_csv('incidents.csv')
oil_spills.head()

Let’s create a simple scatter plot using latitude and longitude.

oil_spills.plot(x="lon", y="lat", kind="scatter", xlabel='Longitude', ylabel='Latitude', title='Oil Spills')

    <AxesSubplot:title={'center':'Oil Spills'}, xlabel='Longitude', ylabel='Latitude'>

This looks a little bit like the United States. What if we put a real map behind it?

🌎🐼

pip install geopandas

import geopandas as gpd

countries = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres"))

# plot the countries first
countries.plot(color="lightgrey")
# now plot the oil spills
# use ax=plt.gca() to get it on the same plot
# use figsize to get a good shape
oil_spills.plot(x="lon", y="lat", kind="scatter", ax=plt.gca(),figsize=(100,70))

    <AxesSubplot:xlabel='lon', ylabel='lat'>

That’s pretty good, but what if we want to only plot North and South America?

subset = countries.continent.isin(['North America','South America'])

countries = gpd.read_file(
               gpd.datasets.get_path("naturalearth_lowres"))
# get just the countries we want to plot
subset = countries.continent.isin(['North America','South America'])
countries[subset].plot(color="lightgrey")
oil_spills.plot(x="lon", y="lat", kind="scatter", ax=plt.gca(),figsize=(20,15), s=1, color='firebrick')

    <AxesSubplot:xlabel='lon', ylabel='lat'>

What if we want to zoom in on just the continental United States? Use these parameters in plot()

ylim=(start,end)
xlim=(start,end)

countries = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres"))
subset = countries.continent.isin(['North America'])
countries[subset].plot(color="lightgrey")
oil_spills.plot(x="lon", y="lat", kind="scatter", ax=plt.gca(),figsize=(20,15),
                s=1, color='firebrick', ylim=(10,55), xlim=(-150,-50))

    <AxesSubplot:xlabel='lon', ylabel='lat'>

Let’s do it all over again

We’ll read in the Chicago beach water monitoring data

chicago = pd.read_csv("bwq.csv", parse_dates=['Measurement Timestamp'])
chicago.head()

Then we need to drop any invalid measurements:

chicago = chicago.dropna(subset=['Measurement Timestamp'])

and add a column for the year:

chicago = chicago.dropna(subset=['Measurement Timestamp'])

def get_year(date):
    return int(date.year)

chicago['Year'] = chicago['Measurement Timestamp'].apply(get_year)
chicago.head()

Let’s try doing a scatter plot of wave height and wave period to see if they are related:

chicago.plot(x='Wave Period', y='Wave Height', kind='scatter')

    <AxesSubplot:xlabel='Wave Period', ylabel='Wave Height'>

Oops! Looks like we have some negative wave heights and periods! Let’s clean our data:

clean_chicago = chicago[(chicago['Wave Height'] > 0) & (chicago['Wave Period'] > 0)]
clean_chicago.plot(x='Wave Period', y='Wave Height', kind='scatter')

    <AxesSubplot:xlabel='Wave Period', ylabel='Wave Height'>

We can do the same thing with wave period and Turbidity.

clean_chicago.plot(x='Wave Period', y='Turbidity', kind='scatter')

    <AxesSubplot:xlabel='Wave Period', ylabel='Turbidity'>

This is a line plot, but for fun, let’s look at wave height over time at a certain beach:

recent = clean_chicago['Year'] == 2016
montrose = clean_chicago['Beach Name'] == 'Montrose Beach'
clean_chicago[recent & montrose].plot(x='Measurement Timestamp', y='Wave Height')

    <AxesSubplot:xlabel='Measurement Timestamp'>

How about a histogram so we can see how common various wave heights are?

clean_chicago[recent & montrose].hist(column='Wave Height')

    array([[<AxesSubplot:title={'center':'Wave Height'}>]], dtype=object)

A box plot would give us some statistical data about wave heights:

earlier = clean_chicago['Year'] == 2014
clean_chicago[earlier].groupby('Beach Name').boxplot(column='Wave Height', figsize=(6,18))

    63rd Street Beach         AxesSubplot(0.1,0.679412;0.363636x0.220588)
    Calumet Beach        AxesSubplot(0.536364,0.679412;0.363636x0.220588)
    Montrose Beach            AxesSubplot(0.1,0.414706;0.363636x0.220588)
    Ohio Street Beach    AxesSubplot(0.536364,0.414706;0.363636x0.220588)
    Osterman Beach                AxesSubplot(0.1,0.15;0.363636x0.220588)
    Rainbow Beach            AxesSubplot(0.536364,0.15;0.363636x0.220588)
    dtype: object

Or we could put those all on one graph:

clean_chicago[earlier].boxplot(by='Beach Name',column='Wave Height', figsize=(10,5))

    <AxesSubplot:title={'center':'Wave Height'}, xlabel='Beach Name'>

Finally, we can use groupby to compute max wave heights and see which beach has the highest wave:

grouped = clean_chicago[earlier].groupby('Beach Name').max().sort_values('Wave Height').reset_index()
grouped.plot(x = 'Beach Name', y='Wave Height', kind='bar')

    <AxesSubplot:xlabel='Beach Name'>