Airbnb data for New York City 2019
Group Dol_R
8/20/2020
Introduction
Airbnb is an online marketplace for accommodations and it has more than 150 million users worldwide. In this report, the listing activity and metrics in New York City for 2019 will be analyzed according to New York City Airbnb Open Data data from Kaggle.
Loading Libraries and Data
library(readr)
library(tidyverse)
library(lubridate)
library(tinytex)
data <- read.csv("https://raw.githubusercontent.com/pjournal/boun01g-dol-r/gh-pages/NYC_assignment/AB_NYC_2019.csv", header = TRUE, check.names=TRUE)Overview and Manipulation of Data
Variables
First of all we start with checking the structure of the variables by glimpse.
data %>% glimpse()## Rows: 48,895
## Columns: 16
## $ id <int> 2539, 2595, 3647, 3831, 5022, 5099, 51…
## $ name <chr> "Clean & quiet apt home by the park", …
## $ host_id <int> 2787, 2845, 4632, 4869, 7192, 7322, 73…
## $ host_name <chr> "John", "Jennifer", "Elisabeth", "Lisa…
## $ neighbourhood_group <chr> "Brooklyn", "Manhattan", "Manhattan", …
## $ neighbourhood <chr> "Kensington", "Midtown", "Harlem", "Cl…
## $ latitude <dbl> 40.64749, 40.75362, 40.80902, 40.68514…
## $ longitude <dbl> -73.97237, -73.98377, -73.94190, -73.9…
## $ room_type <chr> "Private room", "Entire home/apt", "Pr…
## $ price <int> 149, 225, 150, 89, 80, 200, 60, 79, 79…
## $ minimum_nights <int> 1, 1, 3, 1, 10, 3, 45, 2, 2, 1, 5, 2, …
## $ number_of_reviews <int> 9, 45, 0, 270, 9, 74, 49, 430, 118, 16…
## $ last_review <chr> "2018-10-19", "2019-05-21", "", "2019-…
## $ reviews_per_month <dbl> 0.21, 0.38, NA, 4.64, 0.10, 0.59, 0.40…
## $ calculated_host_listings_count <int> 6, 2, 1, 1, 1, 1, 1, 1, 1, 4, 1, 1, 3,…
## $ availability_365 <int> 365, 355, 365, 194, 0, 129, 0, 220, 0,…Here;
id : Listing ID
name : Name of the Listing
host_id : Host ID
host_name : Host Name
neighbourhood_group : Location
neighbourhood : Area
latitude : Latitude Coordinates
longitude : Longitude Coordinates
room_type : Listing Space Type
price : price (in US dollars)
minimum_nights : Minimum Nights
number_of_reviews : Total Number of Reviews
last_review : Date of the Last Review
reviews_per_month :
calculated_host_listings_count :
availability_365 :
Numerical Variables: id, host_id, latitude, longitude, price, minimum_nights, number_of_reviews, last_review, reviews_per_month, calculated_host_listings_count, availability_365.All the numerical values can be considered as discrete.
Categorical Variables: name, host_name, neighbourhood_group, neighbourhood, room_type
Since host_id identifies the host, there is no need for host_name and the respective column can be removed.
The types of the variables seems appropiate except last_review. We change its type to date.
data$last_review <- ymd(data$last_review)
data <- data %>% select(-host_name)NA values
There may be NA values missing in the data and these NA values may affect our analysis. So we check the variables having NA.
data %>% select(everything()) %>% summarise_all(funs(sum(is.na(.))))## id name host_id neighbourhood_group neighbourhood latitude longitude
## 1 0 0 0 0 0 0 0
## room_type price minimum_nights number_of_reviews last_review
## 1 0 0 0 0 10052
## reviews_per_month calculated_host_listings_count availability_365
## 1 10052 0 0We realized that there are 10052 NA values in reviews_per_month and last_review columns. We assume that these are the places with no reviews and check:
data %>% filter(number_of_reviews==0)%>%count()## n
## 1 10052We see that there are no data with reviews_per_month = 0 so we convert the na values in reviews_per_month column to 0.
data <- data %>% mutate(reviews_per_month=ifelse(is.na(reviews_per_month), 0, reviews_per_month))Analysis
Average price per night is 152.72.
Summary of Dataset
There are 48895 data points in total.
data %>% summary()## id name host_id neighbourhood_group
## Min. : 2539 Length:48895 Min. : 2438 Length:48895
## 1st Qu.: 9471945 Class :character 1st Qu.: 7822033 Class :character
## Median :19677284 Mode :character Median : 30793816 Mode :character
## Mean :19017143 Mean : 67620011
## 3rd Qu.:29152178 3rd Qu.:107434423
## Max. :36487245 Max. :274321313
##
## neighbourhood latitude longitude room_type
## Length:48895 Min. :40.50 Min. :-74.24 Length:48895
## Class :character 1st Qu.:40.69 1st Qu.:-73.98 Class :character
## Mode :character Median :40.72 Median :-73.96 Mode :character
## Mean :40.73 Mean :-73.95
## 3rd Qu.:40.76 3rd Qu.:-73.94
## Max. :40.91 Max. :-73.71
##
## price minimum_nights number_of_reviews last_review
## Min. : 0.0 Min. : 1.00 Min. : 0.00 Min. :2011-03-28
## 1st Qu.: 69.0 1st Qu.: 1.00 1st Qu.: 1.00 1st Qu.:2018-07-08
## Median : 106.0 Median : 3.00 Median : 5.00 Median :2019-05-19
## Mean : 152.7 Mean : 7.03 Mean : 23.27 Mean :2018-10-04
## 3rd Qu.: 175.0 3rd Qu.: 5.00 3rd Qu.: 24.00 3rd Qu.:2019-06-23
## Max. :10000.0 Max. :1250.00 Max. :629.00 Max. :2019-07-08
## NA's :10052
## reviews_per_month calculated_host_listings_count availability_365
## Min. : 0.000 Min. : 1.000 Min. : 0.0
## 1st Qu.: 0.040 1st Qu.: 1.000 1st Qu.: 0.0
## Median : 0.370 Median : 1.000 Median : 45.0
## Mean : 1.091 Mean : 7.144 Mean :112.8
## 3rd Qu.: 1.580 3rd Qu.: 2.000 3rd Qu.:227.0
## Max. :58.500 Max. :327.000 Max. :365.0
## top_hosts <- sort(table(data$host_id), decreasing = T)
top_hosts <- head(top_hosts, 10)
top_hosts <- data.frame(top_hosts)
bar_p <- ggplot(top_hosts, aes(x=Var1, y = Freq, fill = Var1)) + geom_bar(stat = "identity") + labs(x="Host IDs", y="Count of Host Listings", color="Host IDs", title = "Hosts with the most listing in NYC") + theme(axis.text.x = element_text(angle=45,hjust=1,vjust=1), legend.position = "none")
bar_p
Neigbourhoods
Now we check the number of the neighborhood groups and dispersion among listings.
data %>% select(neighbourhood_group) %>% unique() %>% count()## n
## 1 5df_neigbour <- data.frame(data$neighbourhood_group)
df_neigbour <- unlist(df_neigbour)
df_neigbour <- data.frame(df_neigbour)
pie_chart <- ggplot(df_neigbour) + geom_bar(mapping = aes(x = "", fill=df_neigbour ), width = 1) + coord_polar("y") +
theme_void() # remove background, grid, numeric labels
pie_chart <- pie_chart + labs(x="", fill="Neigbours", title = "Dispersion of Neigbour Groups")
pie_chart
There are 5 groups (boroughs of NYC)
And now we check the number of neighborhoods in each group and top 10 frequent neigbourhoods over all listings.
neighbor_count <- data %>% select(neighbourhood_group, neighbourhood) %>% group_by(neighbourhood_group) %>% unique() %>% count()
neighbor_count## # A tibble: 5 x 2
## # Groups: neighbourhood_group [5]
## neighbourhood_group n
## <chr> <int>
## 1 Bronx 48
## 2 Brooklyn 47
## 3 Manhattan 32
## 4 Queens 51
## 5 Staten Island 43top_neig <- sort(table(data$neighbourhood), decreasing = T)
top_neig <- head(top_neig, 10)
top_neig <- data.frame(top_neig)
bar_p2 <- ggplot(top_neig, aes(x=Var1, y = Freq, fill = Var1)) + geom_bar(stat = "identity") + labs(x="Neigbours", y="Frequency",color= "Neigbours", title = "Top 10 Frequent Neigbours") + theme(axis.text.x = element_text(angle=45,hjust=1,vjust=1), legend.position = "none")
bar_p2
We can see that there are 221 neighbourhoods in NYC.
Lets see the average price in every borough and their distributions with box-plot.
data %>% group_by(neighbourhood_group) %>% transmute(mean_price = mean(price)) %>% unique()## # A tibble: 5 x 2
## # Groups: neighbourhood_group [5]
## neighbourhood_group mean_price
## <chr> <dbl>
## 1 Brooklyn 124.
## 2 Manhattan 197.
## 3 Queens 99.5
## 4 Staten Island 115.
## 5 Bronx 87.5box_df <- data %>% select(id, neighbourhood_group, price)
box_plot1 <- ggplot(box_df, aes(x=neighbourhood_group, y=price, color=neighbourhood_group)) + geom_boxplot() + scale_y_log10() +labs(x="Neigbourhood Groups", y="Price", title = "Price Ranges across Neigbourhood Groups") + theme(legend.title= element_blank())
box_plot1
There seems a huge gap between neighbourhoods. There could be several reasons for this. So we check the effect of the room types.
Room Types
There are 3 different types of rooms which are Private room, Entire home/apt, Shared room . Mean prices per room type are :
data %>% select(price, room_type) %>% group_by(room_type)%>% summarise(mean(price))## # A tibble: 3 x 2
## room_type `mean(price)`
## <chr> <dbl>
## 1 Entire home/apt 212.
## 2 Private room 89.8
## 3 Shared room 70.1There seems a huge gap between entire home / apartment type and the others. Also it’s useful to see the ratio of the room types over the region.
df_rt <- data.frame(data$room_type)
df_rt <- unlist(df_rt)
df_rt <- data.frame(df_rt)
pie_chart2 <- ggplot(df_rt) + geom_bar(mapping = aes(x = "", fill=df_rt ), width = 1) + coord_polar("y") +
theme_void() # remove background, grid, numeric labels
pie_chart2 <- pie_chart2 + labs(x="", fill="Room Types", title = "Room Types Ratio")
pie_chart2
Hosts with the Most Listings in NYC
Speaking of hosts, it can give clues about availability througout the year. They can be listed as descending order based on listings.
top_hosts <- sort(table(data$host_id), decreasing = T)
top_hosts <- head(top_hosts, 10)
top_hosts <- data.frame(top_hosts)
bar_p <- ggplot(top_hosts, aes(x=Var1, y = Freq, fill = Var1)) + geom_bar(stat = "identity") + labs(x="Host IDs", y="Count of Host Listings", color="Host IDs", title = "Hosts with the most listing in NYC") + theme(axis.text.x = element_text(angle=45,hjust=1,vjust=1),legend.position = "none")
bar_p
Number of Reviews
We check the effect of the number of reviews.
review_0_15_mean <- mean((data %>% filter (number_of_reviews <=15) %>% select(price))$price) %>% round()
review_15_25_mean <- mean((data %>% filter (number_of_reviews >15 & number_of_reviews <=25) %>% select(price))$price) %>% round()
review_25_50_mean <- mean((data %>% filter (number_of_reviews >25 & number_of_reviews <=50) %>% select(price))$price) %>% round()
review_50_100_mean <- mean((data %>% filter (number_of_reviews >50 & number_of_reviews <=100) %>% select(price))$price) %>% round()
review_100_200_mean <- mean((data %>% filter (number_of_reviews >100 & number_of_reviews <=200) %>% select(price))$price) %>% round()
review_200plus_mean <- mean((data %>% filter (number_of_reviews >200) %>% select(price))$price) %>% round()Average price of places with number of reviews:
0 to 15 : 161 USD
15 to 25 : 141 USD Around the average number of reviews
25 to 50 : 137 USD
50 to 100 : 130 USD
100 to 200 : 130 USD
200 and above : 114 USD
We can understand that number of reviews has no positive effect on price since it is possible that customers prefer to choose places with low price.
Furthermore we check the house types in boroughs. It may explain the price gap. From the data we see, we can assume that there are more entire house / apartment offers in manhattan than the others.
data %>% select(neighbourhood_group,room_type) %>% group_by(neighbourhood_group,room_type) %>% count() %>% group_by(neighbourhood_group) %>% mutate(total_in_borough = sum(n)) %>% mutate (percentage = paste0(as.character(round(n / total_in_borough*100)),"%")) %>% select(-total_in_borough)## # A tibble: 15 x 4
## # Groups: neighbourhood_group [5]
## neighbourhood_group room_type n percentage
## <chr> <chr> <int> <chr>
## 1 Bronx Entire home/apt 379 35%
## 2 Bronx Private room 652 60%
## 3 Bronx Shared room 60 5%
## 4 Brooklyn Entire home/apt 9559 48%
## 5 Brooklyn Private room 10132 50%
## 6 Brooklyn Shared room 413 2%
## 7 Manhattan Entire home/apt 13199 61%
## 8 Manhattan Private room 7982 37%
## 9 Manhattan Shared room 480 2%
## 10 Queens Entire home/apt 2096 37%
## 11 Queens Private room 3372 60%
## 12 Queens Shared room 198 3%
## 13 Staten Island Entire home/apt 176 47%
## 14 Staten Island Private room 188 50%
## 15 Staten Island Shared room 9 2%Manhattan has higher percentage in having entire home apartment offers. This can explain the higher prices. But there is only 10% difference between Brooklyn and Staten Island percentages therefore we cannot say that this price gap is only caused by the room type.
Landmarks
AirBnb is used by tourists and travellers mostly. A tourist usually prefers to stay close to the places where they plan to visit which are commonly the landmarks of the city. We select several landmarks of NYC from here. This list shows the most popular tourist attractions. And we find the coordinates from here, a coordinate search engine.
Landmarks <- read_csv2("https://raw.githubusercontent.com/pjournal/boun01g-dol-r/gh-pages/NYC_assignment/landmarks_NYC.csv")
Landmarks <- as.tibble(Landmarks)
Landmarks%>%glimpse()## Rows: 9
## Columns: 3
## $ `Tourist Attraction` <chr> "Rockefeller Center", "Metropolitan Museum of Ar…
## $ Latitude <dbl> 40.75874, 40.77896, 40.74882, 40.71008, 40.75890…
## $ Longitude <dbl> -73.97867, -73.96231, -73.98543, -74.01241, -73.…We load our landmarks. These landmarks we choose are:
-Rockefeller Center
-Metropolitan Museum of Art
-Empire State Building
-9/11 Memorial and Museum
-Times Square
-Grand Central Terminal
-One World Observatory
-New York Public Library
-St. Patrick’s Cathedral
And now we will calculate the average distance of every single house to these landmarks. We use a package called geosphere with which we can calculate the distance between two coordinates.
library(geosphere)## Warning: package 'geosphere' was built under R version 4.0.2data <- data%>% rowwise() %>% mutate(rock_dist = distm (cbind(Landmarks$Longitude[1], Landmarks$Latitude[1]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(Met_dist = distm (cbind(Landmarks$Longitude[2], Landmarks$Latitude[2]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(Emp_State_dist = distm (cbind(Landmarks$Longitude[3], Landmarks$Latitude[3]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(NineElevenMemo_dist = distm (cbind(Landmarks$Longitude[4], Landmarks$Latitude[4]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(TimesSq_dist = distm (cbind(Landmarks$Longitude[5], Landmarks$Latitude[5]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(Terminal_dist = distm (cbind(Landmarks$Longitude[6], Landmarks$Latitude[6]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(OneWorld_dist = distm (cbind(Landmarks$Longitude[7], Landmarks$Latitude[7]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(Library_dist = distm (cbind(Landmarks$Longitude[8], Landmarks$Latitude[8]), cbind(longitude,latitude), fun = distHaversine))
data <- data%>% rowwise() %>% mutate(StPatrick_dist = distm (cbind(Landmarks$Longitude[9], Landmarks$Latitude[9]), cbind(longitude,latitude), fun = distHaversine))
data <- data %>% mutate(Mean_Distance = mean(rock_dist,Met_dist,Emp_State_dist,NineElevenMemo_dist,TimesSq_dist, Terminal_dist,OneWorld_dist,Library_dist,StPatrick_dist))Now we will check the possible effect of location to the prices. We will start with plotting. (We filter prices < 750 since there are outliers above and the graph do not mean anything.)
data %>% filter(price<750) %>% ggplot(.,aes(x=price,y=Mean_Distance)) + geom_point(color = "black") + labs(x="Price",y="Average Distance") + ggtitle("Price vs Distance to Landmarks Scatterplot")
We can see that high proportion of the places with higher price has shorter distance. But there are around 50K observations and this makes us harder to understand the graphics. Therefore we calculate the mean distance and mean prices of neighborhood.
Mean_Price_Dist_data <- data %>% select(neighbourhood_group,neighbourhood,Mean_Distance,price) %>% group_by(neighbourhood_group,neighbourhood) %>% mutate(mean_price_neig = mean(price)) %>% mutate(mean_dist_neig = mean(Mean_Distance)) %>% select(-Mean_Distance, -price) %>% group_by(neighbourhood_group) %>% unique() %>% arrange(neighbourhood_group)
Mean_Price_Dist_data %>% filter (mean_price_neig <300) %>% ggplot(.,aes(mean_price_neig,mean_dist_neig,color=neighbourhood_group)) +geom_point() + labs(x="Average Price",y="Average Distance") + ggtitle("Price vs Distance to Landmarks Scatterplot by Neighbourhoods")
References
Dataset: New York City Airbnb Open Data from Kaggle.
20 Top-Rated Tourist Attractions in New York City from
On Kaggle, there are some useful notebooks we benefited from:
To see Group Dol_R’s Progress Journal, visit Progress Journal website.