PreprocessingAviationData.Rmd

---
title: "Preprocessing Brazilian aviation data for RGraphSpace vignette"
subtitle: "ANAC flights, airports, spatial data, and RGraphSpace"
author: 
- name: Flávio Gabriel Carazza-Kessler
- name: Sysbiolab Team
date: "`r Sys.Date()`"
bibliography: bibliography.bib
output:
  html_document:
    toc: true
    toc_float: true
    toc_depth: 2
    self_contained: true
    includes:
      in_header: seo.html
---

```{r setup, include=FALSE, purl=FALSE}
knitr::opts_chunk$set(
  echo = TRUE,
  collapse = TRUE,
  comment = "#>",
  fig.align = "center",
  fig.width = 7,
  fig.height = 5
  )
```

# Overview

This vignette demonstrates how to preprocess Brazilian flight and airport data in R for spatial network analysis with RGraphSpace. It downloads **2024** flight records from Brazil’s National Civil Aviation Agency using the *flightsbr* [@Pereira2022] package, harmonizes IATA and ICAO airport codes, filters domestic routes, and prepares sf objects for spatial visualization.

The resulting objects contain Brazilian geographic boundaries, active airport locations, and aggregated origin–destination flight counts. These objects are saved for use in the RGraphSpace spatial-data vignette.

If you are not familiar to *RGraphSpace*, please see the available workflows: [Get Started](https://sysbiolab.github.io/RGraphSpace/articles/get-started.html){target="_blank" rel="noopener"}

# Before you start

This vignette retrieves data from external databases, including flight records from ANAC. Therefore, some calls may take a few minutes to run.

**Computational requirement:**

- Hardware: RAM \>= 12 GB

- Software: R (\>=4.5) and RStudio

# Required Packages

```{r check-required-packages, eval=FALSE, message=FALSE}
# Check required packages for this vignette
if(!require("data.table", quietly = TRUE)){install.packages("data.table")}
if(!require("ggplot2", quietly = TRUE)){install.packages("ggplot2")}
if(!require("airportr", quietly = TRUE)){install.packages("airportr")}
if(!require("dplyr", quietly = TRUE)){install.packages("dplyr")}
if(!require("sf", quietly = TRUE)){install.packages("sf")}
if(!require("rnaturalearth", quietly = TRUE)){install.packages("rnaturalearth")}
if(!require("maps", quietly = TRUE)){install.packages("maps")}
if(!require("ggpubr", quietly = TRUE)){install.packages("ggpubr")}
if(!require("ggrepel", quietly = TRUE)){install.packages("ggrepel")}
if(!require("stringr", quietly = TRUE)){install.packages("stringr")}
if(!require("flightsbr", quietly = TRUE)){
  remotes::install_github("ipeaGIT/flightsbr")}
if(!require("rnaturalearthhires", quietly = TRUE)){
  remotes::install_github("ropensci/rnaturalearthhires")}
```

```{r Load packages, eval=TRUE, message=FALSE}
# Load packages
library("flightsbr")
library("data.table")
library("ggplot2")
library("airportr")
library("dplyr")
library("sf")
library("rnaturalearth")
library("rnaturalearthhires")
```

# Data sources

First, we need to load three datasets: (i) flight records, (ii) airport metadata, and (iii) geographic boundary data.

## Defining a helper function

The helper function below keeps only flights for which both the origin and destination airports are present in a given airport dataset.

```{r, eval=TRUE, message=FALSE, results='hide'}
only_from_airports <- function(flights,airports){
  for(i in 1:length(flights[[1]])){
    flights$accessory[i] <- flights[[1]][i] %in% airports$codigo_oaci &&
      flights[[2]][i] %in% airports$codigo_oaci
  }
  flights_clean <- flights %>%
    filter(accessory == TRUE) %>%
    select(-c("accessory"))
  
  return(flights_clean)
}
```

## Flight records

In this vignette, we are interested in all domestic flights recorded in **2024**, that is, flights for which both departure and arrival airports are located in Brazil. In this section, we first load all flights from 2024, including the international ones. The flight data are filtered in later steps.

**Note:** This may take some time.

```{r, eval=TRUE, message=FALSE}
year <- 2024
df_year <- flightsbr::read_flights(
  date = year,
  showProgress = FALSE,
  select = c('id_empresa', 'nr_voo', 'dt_partida_real',
             'sg_iata_origem' , 'sg_iata_destino')
)

head(df_year)
```

The departure and arrival airports in `df_year` are stored in the `sg_iata_origem` and `sg_iata_destino` columns, respectively. These airports are identified using the International Air Transport Association (IATA) codes, which are unique three-letter location codes. Airports are also identified by four-letter International Civil Aviation Organization (ICAO) codes.

IATA codes are primarily used for passenger-facing purposes, such as baggage ticketing and flight booking. ICAO codes, however, are used for air traffic control and flight operations. After loading the data, code conversion is necessary to integrate flight and airport datasets.

## Airport metadata

```{r, eval=TRUE, message=FALSE}
airports_all <- flightsbr::read_airports(
  type = 'all', 
  showProgress = FALSE
)

head(airports_all)
```

Note that the first column, `codigo_oaci`, contains the ICAO code for each airport. In the `airports_all` dataset, IATA codes are not available. The airport metadata also include the city, state, name, latitude, longitude, altitude, and type (private or public) of each airport.

### Formatting airport data

Although the `latitude` and `longitude` columns of `airport_all` contain several `NA` values, we can extract geographic information from the `longeopoint` and `latgeopoint` columns. These latter columns are stored as character vectors, so they must be to convert to double vectors. The airport metadata is then converted to an *sf* object, in which the geographic coordinates are stored as `POINT` geometries.

```{r, eval=TRUE, message=FALSE}
airports_all$latgeopoint <- as.double(gsub(",",".",airports_all$latgeopoint))
airports_all$longeopoint <- as.double(gsub(",",".",airports_all$longeopoint))
airports_all$longitude <- airports_all$longeopoint
airports_all$latitude <- airports_all$latgeopoint
sf_air_all <- sf::st_as_sf(airports_all, 
                           coords = c("longeopoint", "latgeopoint"),
                           crs = 4674)

head(sf_air_all)
```

## Geographical boundaries

Here, we load the Brazilian map and a subset of the Southeast region of Brazil. The Southeast region comprises four states: São Paulo (SP), Minas Gerais (MG), Rio de Janeiro (RJ) and Espírito Santo (ES).

```{r, eval=TRUE, message=FALSE}
mapa_brasil <- ne_states(country = "brazil", returnclass = "sf") %>%
  select(c("name","postal","latitude","longitude","geometry"))
mapa_brasil <- st_transform(mapa_brasil)

head(mapa_brasil)

se_states <- mapa_brasil %>% filter(postal %in% c('RJ','SP','MG','ES'))
```

Plot Brazilian map:

```{r, eval=TRUE, message=FALSE, results='hide'}
ggplot() +
  geom_sf(data=mapa_brasil, fill="light grey", color="dark grey", size=.15, show.legend = FALSE) +
  labs(subtitle="Brazilian States", size=8) +
  theme_minimal() 
```

Plot states from Brazil's southeast region:

```{r, eval=TRUE, message=FALSE, results='hide'}
ggplot() +
  geom_sf(data=se_states, fill="light grey", color="dark grey", size=.15, show.legend = FALSE) +
  labs(subtitle="Southeast Region", size=8) +
  theme_minimal()
```

Plot airports:

```{r, eval=TRUE, message=FALSE, results='hide'}
ggplot() +
  geom_sf(data=mapa_brasil, fill="light grey", color="dark grey", size=.15, show.legend = FALSE) +
  labs(subtitle="Brazilian Airports") +
  geom_sf(data = sf_air_all,
          aes(geometry = geometry,
              color = type),
          size = .5) +
  theme_minimal() 
```

# Preparing the data

## Harmonizing IATA and ICAO airport codes

First, we collect all airports present in flights data (`df_year`), including both departure and arrival airports, and store them in a data frame named `IATA_to_ICAO`.

```{r, eval=TRUE, message=FALSE, results='hide'}
vec<-sort(unique(c(df_year$sg_iata_origem,df_year$sg_iata_destino)))
vec<-tail(vec,-2)

IATA_to_ICAO <- data.frame(IATA = vec, ICAO = NA, row.names = vec)
rm(vec)
```

Number of unique airports in flights data:

```{r, eval=TRUE, message=FALSE}
length(IATA_to_ICAO$IATA)
```

Now, we assign the ICAO codes to each airport present in `IATA_to_ICAO`. The `for` loop shown below skips iterations thar return errors during the `airport_lookup()` search.

```{r, eval=TRUE, message=FALSE}
for (i in 1:length(IATA_to_ICAO$IATA)) {
  # Wrap the failing code in tryCatch
  result <- tryCatch({
    airportr::airport_lookup(IATA_to_ICAO$IATA[i],
                   input_type = "IATA", output_type = "ICAO") # Your function that might fail
  }, error = function(e) {
    # If error occurs, return a special value or just continue
    return(NULL) 
  })
  
  # Skip to the next iteration if an error occurred
  if (is.null(result)) {
    next
  }
  # Process successful result here
  IATA_to_ICAO$ICAO[i] <- result
}

head(IATA_to_ICAO)
```

Assigning ICAO codes to the departure and destination airports in the flights data.

```{r, eval=TRUE, message=FALSE}
# Ensure that each IATA code has only one corresponding ICAO code
stopifnot(!anyDuplicated(IATA_to_ICAO$IATA))

# Find the corresponding row in IATA_to_ICAO for each airport
origin_match <- match(
  df_year$sg_iata_origem,
  IATA_to_ICAO$IATA
)

destination_match <- match(
  df_year$sg_iata_destino,
  IATA_to_ICAO$IATA
)

# Assign the ICAO codes
df_year$sg_icao_origem <- IATA_to_ICAO$ICAO[origin_match]
df_year$sg_icao_destino <- IATA_to_ICAO$ICAO[destination_match]

rm(origin_match, destination_match)

head(df_year)
```

## Cleaning flight records

Once we have converted the airport codes in the flight data, we filter and reshape the data to remove `NA` values and duplicate departure-arrivals pairs.

```{r, eval=TRUE, message=FALSE, results='hide'}
flights_clean <- na.omit(df_year) %>%
  dplyr::select(sg_icao_origem,sg_icao_destino) %>%
  add_count(across(everything()),name = "flights_count") %>%
  unique()
```

Number of unique departures-arrivals pairs:

```{r, eval=TRUE, message=FALSE}
length(flights_clean$sg_icao_origem)
```

Total number of flights after selections and code translation:

```{r, eval=TRUE, message=FALSE}
sum(flights_clean$flights_count)
```

In `flights_clean`, each row represent a unique departure-arrival pair from the annual flight records. The `flights_count` column indicates the number of flights of each pair.

**Note 1:** Flights are directional: A → B is different than B → A.\
**Note 2:** This dataset still contains international flights, that is, flights with either an arrival or departure airport located outside Brazil.

```{r, eval=TRUE, message=FALSE}
head(flights_clean)
```

Removing flights with 100 or fewer counts:

```{r, eval=TRUE, message=FALSE, results='hide'}
flights_cf <- flights_clean %>% filter(flights_count > 100)
```

## Selecting active airports

Not all airports present in `sf_air_all` were active during the year. Therefore, we only keep in airports that are present in the filtered flight data.

```{r, eval=TRUE, message=FALSE, results='hide'}
active_airports <- unique(c(flights_cf$sg_icao_origem,
                            flights_cf$sg_icao_destino))
```

Number of active airports (domestic and international):

```{r, eval=TRUE, message=FALSE}
length(active_airports)
```

The `flights_cf` dataset still contains both domestic and international flights. For this reason, we need to remove international airports from `active_airports`. This can be done by checking which active airport are present in `sf_air_all`, our database of Brazilian airports.

```{r, eval=TRUE, message=FALSE, results='hide'}
active_airports <- sf_air_all %>% filter(codigo_oaci %in% active_airports)

rownames(active_airports) <- active_airports$codigo_oaci
active_airports$latitude <- st_coordinates(active_airports$geometry)[,2]
active_airports$longitude <- st_coordinates(active_airports$geometry)[,1]
```

Number of active airports (only domestic):

```{r, eval=TRUE, message=FALSE}
length(active_airports$codigo_oaci)
```

Plot active airports:

```{r, eval=TRUE, message=FALSE}
ggplot() +
  geom_sf(data=mapa_brasil, fill="light grey", color="dark grey", size=.15, show.legend = FALSE) +
  labs(subtitle=paste("Brazilian active airports in",year)) +
  geom_sf(data = active_airports,
          aes(geometry = geometry,
              color = type),
          size = 1) +
  theme_minimal() 
```

## Filtering domestic flights

After airport filtering, the `active_airports` contains only airports that meet two criteria: (i) they had more than 100 flights during the year, and (ii) they are located in Brazil. Therefore, we can use `active_airports` to remove international flights from `flights_cf`. This task is performed by the helper function `only_from_airports()`.

```{r, eval=TRUE, message=FALSE, results='hide'}
flights_cf <- only_from_airports(flights_cf,active_airports)

colnames(flights_cf)[3] <- "weight"
```

## Saving Preprocessed data

Now all data from Brazilian flights recors were loaded and filtered, so we will save them for RGraphSpace vignette:

```{r, eval=TRUE, message=FALSE, results='hide'}
save(mapa_brasil, flights_cf, active_airports, file = "data/sf_brazilflights.RData")
tools::resaveRdaFiles(paths = "data/sf_brazilflights.RData")
```

# Southeast regional network

Next, we create a regional subset focused on airports in Brazil’s Southeast Region, which includes some of the country’s busiest airports. The objective is to examine the relationship between the number of flights and the population of the city served by each airport. The preprocessing workflow comprises the following steps:

1. Select active airports and flight records from Brazil’s Southeast Region.
2. Identify the city associated with each airport.
3. Retrieve the population of each city.
4. Retain only flights with both origin and destination in the Southeast Region.
5. Save the preprocessed flight data for Brazil’s Southeast Region.

**1) Select active airports and flight records from Brazil’s Southeast Region.**

```{r, eval=TRUE, message=FALSE}
se_airports <- active_airports %>% 
  filter(uf %in% c("Espírito Santo","Minas Gerais","Rio de Janeiro","São Paulo"))
```

**2) Identify the city associated with each airport.**

```{r, eval=TRUE, message=FALSE}
cities <- se_airports %>% select(c(municipio))
cities <- as.data.frame(cities) %>% select(-c("geometry"))
```

After identifying the cities served  by all active airports in Brazil's Southeast Region, special characters must br removed and the capitalization of city names standardized. These adjustments are required to matching the city names with those in the population database.

```{r, eval=TRUE, message=FALSE}
cities <- cities |>
  mutate(
    name = municipio |>
      iconv(to = "ASCII//TRANSLIT") |>
      stringr::str_to_title() |>
      stringr::str_replace_all(
        "\\b(?:Do|Da|De|Dos|Das)\\b",
        tolower
      )
  )
```

**3) Retrieve the population of each city.**

```{r, eval=TRUE, message=FALSE}
data(world.cities, package = "maps")
cities2 <- subset(world.cities, country.etc == "Brazil" & 
                   name %in% cities$name) %>%
  select(c("name","pop"))
cities <- left_join(cities,cities2,by= "name")

se_airports$pop <- cities[match(se_airports$municipio,cities$municipio),3]
rm(cities,cities2,world.cities)
se_airports$pop[is.na(se_airports$pop)] <- 0
```

Plot active airports in Brazil's Southeast Region:

```{r, eval=TRUE, message=FALSE}
ggplot() +
  geom_sf(data=se_states, fill="light grey", color="dark grey", size=.15, show.legend = FALSE) +
  geom_sf(data = se_airports,
          aes(geometry = geometry,
              color = type, 
              size = pop)) +
  labs(subtitle=paste("Brazil's Southeast Region active airports in",year),
       y = "Latitude",x= "Longitude",color="Type",size="Population") +
  theme_minimal() 
```

**4) Retain only flights with both origin and destination in the Southeast Region.**

```{r, eval=TRUE, message=FALSE}
se_flights <- only_from_airports(flights_cf,se_airports)

colnames(se_flights)[3] <- "weight"
```

**5) Save the preprocessed flight data for Brazil’s Southeast Region.**

```{r, eval=TRUE, message=FALSE, results='hide'}
save(se_states, se_flights, se_airports, file = "data/sf_SoutheastFlights.RData")
tools::resaveRdaFiles(paths = "data/sf_SoutheastFlights.RData")
```
# Citation

If you use *RGraphSpace*, please cite:

- Sysbiolab Team (2026). RGraphSpace: A lightweight interface between 'igraph' and 'ggplot2' graphics. R package version 1.2.4. DOI: 10.32614/CRAN.package.RGraphSpace

# Session information

```{r label='Session information', eval=TRUE, echo=FALSE}
sessionInfo()
```

# References