Planetscale recently announced their new $5 Postgres instance (PS-5) and I wanted to give it a test.

Since I’m working on ADBC, my first question was whether Planetscale $5 Postgres would work with the ADBC PostgreSQL driver.

Here’s what I did:

After creating a new $5 instance, I clicked Connect and created a role with the following permissions:

• pg_read_all_data (Read data from all tables, views, and sequences.)
• pg_write_all_data (Write data to all tables, views, and sequences.)
• postgres (Create, modify, and drop databases, users, roles, tables, schemas, and all other objects.)

Note: That last role (postgres) will be key since I want to test ingesting data into my instance.

When the Connect wizard asked me how I was connecting, I selected Python. At this point, the instructions show how to use the psycopg2-binary and they provide this code:

import psycopg2
from dotenv import load_dotenv

# Load environment variables from the .env file
load_dotenv()

conn = psycopg2.connect(
  host=os.getenv("DATABASE_HOST"),
  port=os.getenv("DATABASE_PORT"),
  user=os.getenv("DATABASE_USERNAME"),
  password=os.getenv("DATABASE_PASSWORD"),
  dbname=os.getenv("DATABASE"),
)

cur = conn.cursor()
cur.execute("SELECT version();")
print(cur.fetchone())

cur.close()
conn.close()

Because ADBC drivers use the same underlying protocols as the database their targetting,

1. We can swap psycopg2-binary out for the ADBC PostgreSQL driver and it should just work
2. We can use mostly the same code and exactly the same SQL

I installed the ADBC PostgreSQL driver using dbc, a new command line tool we’re building to make working with ADBC drivers easier. It’s also available on PyPI. I also did this in a venv to keep it contained (using uv):

$ uv venv
$ source .venv/bin/activate
$ dbc install postgresql
[✓] searching
[✓] downloading
[✓] installing
[✓] verifying signature

Installed postgresql 1.8.0 to /Users/bryce/planetscale-adbc/.venv/etc/adbc/drivers

This installed the driver into my new virtual environment as you can see above.

I then installed a few more packages for my test:

$ uv pip install adbc-driver-manager pyarrow

In their wizard, Planetscale gave me a set of environment variables for the connection so I stored those in a .env file and loaded them in my shell so they were available to Python below. For this I use direnv.

To do my test, I connected:

import os
from adbc_driver_manager import dbapi

URI=f"postgresql://{os.getenv('DATABASE_USERNAME')}:{os.getenv('DATABASE_PASSWORD')}@{os.getenv('DATABASE_HOST')}:{os.getenv('DATABASE_PORT')}/{os.getenv('DATABASE')}"

con = dbapi.connect(driver="postgresql", uri=URI)
cur = con.cursor()
cur.execute("SELECT version();").fetchone()
# => ('PostgreSQL 17.5 (Debian 17.5-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit',)

Ingested a Parquet file (Palmer Penguins, of course):

import pyarrow.parquet as pq

tbl = pq.read_table("./penguins.parquet")
cur.adbc_ingest("penguins", tbl, mode="create")
# => 344

And then read it back:

tbl = cur.execute("select * from penguins").fetch_arrow_table()
tbl.num_rows
# => 344
tbl

Which prints:

pyarrow.Table
species: string
island: string
bill_len: double
bill_dep: double
flipper_len: int64
body_mass: int64
sex: string
year: int64
----
species: [["Adelie","Adelie","Adelie","Adelie","Adelie",...,"Chinstrap","Chinstrap","Chinstrap","Chinstrap","Chinstrap"]]
island: [["Torgersen","Torgersen","Torgersen","Torgersen","Torgersen",...,"Dream","Dream","Dream","Dream","Dream"]]
bill_len: [[39.1,39.5,40.3,null,36.7,...,55.8,43.5,49.6,50.8,50.2]]
bill_dep: [[18.7,17.4,18,null,19.3,...,19.8,18.1,18.2,19,18.7]]
flipper_len: [[181,186,195,null,193,...,207,202,193,210,198]]
body_mass: [[3750,3800,3250,null,3450,...,4000,3400,3775,4100,3775]]
sex: [["male","female","female","NA","female",...,"male","female","male","male","female"]]
year: [[2007,2007,2007,2007,2007,...,2009,2009,2009,2009,2009]]

I’d say that was a successful test.

Now, connecting to a $5 PostgreSQL instance may not be the most realistic demonstration of how to use ADBC but I hope the above shows the value of interfaces. Notably, here are the things I didn’t have to do:

1. Learn a new Python database API (both psycogp2 and ADBC speak PEP 249)
2. Figure out how to get my Parquet data converted into whatever format PostgreSQL needs

And now, also because of interfaces, here’s what I can now easily do:

1. Work with it directly with PyArrow
2. Work with this data in DuckDB without copying
3. Work with this data in Polars without copying

And the list can go on because ADBC speaks Arrow and increasing amounts of the data engineering stack are speaking Arrow.

If you’re not familiar with Mermaid, they have great docs.

Gantt Diagrams

Gantt diagrams are typically used for scheduling multiple tasks along a shared timeline. In hindsight, it makes total sense to reach for a Gantt diagrams for diagraming a distributed trace.

The Mermaid syntax for a pretty typical Gantt looks like:

gantt
    title A Gantt Diagram
    dateFormat  YYYY-MM-DD
    section Section
    A task             : a1, 2014-01-01, 30d
    Another task       : after a1  , 20d
    section Another
    Task in sec        : 2014-01-12  , 12d
    another task       : 24d

and, when rendered, looks like:

A Basic Trace Diagram

The tweet I mentioned previously shows code for a Gantt diagrams of a simple trace:

The code for which is:

gantt
    title Trace Showing Attached and Detached Spans
    dateFormat x
    axisFormat %S.%L

    section Frontend
    /checkout               :crit, 0, 500ms
    App                     :300, 180ms
    POST /api/analytics     :done, 450, 70ms
    GET /assistant/poll     :done, 450, 120ms
    POST /api/analytics     :done, 580, 70ms

To do this, the code above uses just a few features of Mermaid’s Gantt syntax to make the diagram look less like a typical Gantt diagrams and more like an OpenTelemetry trace:

1. To show everything on a time scale instead of a calender date scale:
   • Specify a dateFormat of x (milliseconds) instead of the usual YYYY-MM-DD
   • Specify an axisFormat of %S.%L which makes the chart use seconds with milliseconds instead of dates
2. Separate each service into its own section
3. Visually distinguish spans using tags like :crit, :done which apply styling by default

A More Realistic Example

@mitsuhiko also linked to a Sentry RFC that’s in the works with a more representative example:

which has the following code:

gantt
    title Example Starfish Trace
    dateFormat x
    axisFormat %S.%L

    section Frontend
    /checkout                                        :crit, 0, 1500ms
    GET /api/session                                 :150, 170ms
    POST /api/analytics                              :190, 70ms
    GET /api/checkout/state                          :200, 500ms
    GET /api/checkout/cart                           :1100, 140ms
    App                                              :1300, 180ms
    POST /api/analytics                              :done, 1450, 70ms
    GET /assistant/poll                              :done, 1450, 120ms
    POST /api/analytics                              :done, 1580, 70ms

    section API Service
    /api/checkout/state                              :crit, 240, 440ms
    cache.get session#58;[redacted]                  :360, 10ms
    db.query select from session                     :370, 20ms
    db.query select from user                        :390, 20ms
    db.query select from checkout                    :410, 20ms
    http.request GET http#58;//payments/poll  :450, 210ms
    thread.spawn refresh-checkout-cache              :done, 670, 220ms

    section Payment Service
    /poll                                            :crit, 470, 180ms
    db.query select from payment                     :490, 30ms
    db.query update payment                          :530, 60ms

In his post, he describes the C++ Arrow ecosystem as being somewhat fractured and suggests this may be primarily out of the need for other teams to move fast but points out it may have something to do with libarrow’s attractiveness as a dependency.

One quote jumped out at me as particularly insightful is this one:

Yet those are all the same challenges our users experience; would it not be better if we felt those pains ourselves and had incentive to address them? I tend to think we would design better public APIs if we had to use them ourselves for our own query engine. #

This immediately reminded me of something I think Jenny Bryan said (which I cannot currently find) about doing the hard things often so they aren’t hard anymore. If integrating parts of the Arrow ecosystem with each is hard for members of the Arrow project, it’s likely to be considerably harder for those outside of it and I look forward to watching work on this front progress in 2023.

My first questions were which programming language I’d have to use and how I could write as little new code as possible.

I first tried to vendor libarrow and link against that but ran into issues making clang happy with the C++17 stdlib (which libarrow targets). I have a feeling it could be made to work but I went back to the web and found a neat project that used Go for the plugin code. The Go Arrow implementation happens to be one of the few that is written natively (rather than implementing as a binding to libarrow) so I gave that a shot.

Making a New XCode Project

To start out, I wasn’t able to figure out how to make XCode create a new QuickLook plugin from scratch so I ended up adapting from QLMarkdown. The important files seemed to be:

• main.c: Entrypoint for the plugin. Mostly boilerplate aside from the GUID.
• GeneratePreviewForURL.m: Definition and implementation of code for generating our QuickLook preview. This is what I cared about most.
• GenerateThumbnailForURL.m: Definition and implementation of code for generating thumbnails for our files. Not used here.

The core bit on the XCode side is essentially the implementation in GeneratePreviewForURL.h which implememnts what looks like a fairly reasonable interface in order to get data back to macOS for displaying the preview:

OSStatus GeneratePreviewForURL(void *thisInterface, QLPreviewRequestRef preview, CFURLRef url,
                               CFStringRef contentTypeUTI, CFDictionaryRef options) {

  NSString *content = MyFun((__bridge NSURL *)url);

  CFDictionaryRef previewProperties = (__bridge CFDictionaryRef) @{
    (__bridge NSString *)kQLPreviewPropertyTextEncodingNameKey : @"UTF-8",
    (__bridge NSString *)kQLPreviewPropertyMIMETypeKey : @"text/html",
  };

  QLPreviewRequestSetDataRepresentation(preview, (__bridge CFDataRef)[content dataUsingEncoding:NSUTF8StringEncoding],
                                        kUTTypeHTML, previewProperties);

  return noErr;
}

So basically I needed to write a function that:

1. Takes a filepath (URL)
2. Returns a string

Writing the Go Portion

Thanks to CGo, we can easily work with Go and C code at the same time which (I think) is why all of this works so well.

A basic skeleton for the Go code looks like this:

package internal

import ( //... your packages here )

import "C"

//export MyFun
func MyFun(cpath *C.char) (code C.int, outData unsafe.Pointer, outLen C.long) {
	path := C.GoString(cpath)

	var buf bytes.Buffer

    // Now just write data into `buf`

	return 0, C.CBytes(buf.Bytes()), C.long(buf.Len())
}

A couple of things to note:

1. The import "C" is key here
2. I’m not sure if the //export MyFun is required here but I left it in
3. The function you write just needs to write into buf which is pretty straightforward in Go

Last, to compile our Go module into something we can tell XCode to link against, we do something I’d never done before with Go:

go build -buildmode=c-archive -o internal.a ./internal

The above produces internal.a which is critical for the next step.

Bringing Both Sides Together

To tell XCode to compile our Objective-C code and to link against internal.a, we need to add it under XCode under Build Phases > Link Binary With Libraries.

Depending on what Go code you end up writing, you may need to also add various .frameworks until linking succeeds. One surprising thing I ran into was that using Go’s template/html package required Security.framework. In total, I ended up linking against:

Wrapping Up

Once built, you can move the result into ~/Library/QuickLook. You may have to run qlmanage -r and even preview other files to get the new previews to be picked up. Overally, this is a bit finicky and I wish double-clicking on a *.qlgenerator file just prompted you to install it and handled caches for you.

In the end, my preview for Parquet ended up looking like this:

I put the full source code for my plugin at QLArrow

The key of that demo — and something that takes advantage of how the Flatgeobuf format is designed — is the ReadableStream. Instead of having to read the entire file (14.1MB in the above demo) before starting to render our map, we can begin rendering essentially as soon as the first feature is read over the stream and continue rendering, feature-by-feature, as more features are loaded.

That’s accomplished with the following async code:

const response = await fetch('https://flatgeobuf.org/test/data/UScounties.fgb')

for await (let feature of flatgeobuf.deserialize(response.body)) {
  // Do stuff with `feature` here
}

The above snippet relies on your browser’s support for streaming which is surprisingly a sorta new thing. Streams have been around in non-web programming languages for ages and even Node.js made them a key language feature but support in browsers more or less just landed and I somehow missed it.

All of this got me thinking about a problem we ran into at $DAYJOB where we wanted to be able to show hundreds of thousands of points on a 2D map or 3D globe, all on the client-side. Since our data doesn’t change very often, we could just build custom 2D/3D tiles and send those but I wondered how fast doing this with a Flatgeobuf would be.

To test this out, I wrote a small Python scripts to generate a GeoJSON FeatureCollection of a million random points:

from collections import OrderedDict
import json
from numpy.random import default_rng

rng = default_rng()

n = 1000000
lons = rng.uniform(-180, 180, n)
lats = rng.uniform(-90, 90, n)
pairs = zip(lons, lats)

def create_feature(lon, lat):
    return {
        "type": "Feature",
        "properties": OrderedDict(),
        "geometry": {"type": "Point", "coordinates": (lon, lat)},
    }

features = [create_feature(pair[0], pair[1]) for pair in pairs]

geodata = {
    "type": "FeatureCollection",
    "features": features,
}

with open("./output.geojson", "w") as f:
    f.write(json.dumps(geodata))

And then ran that through ogr2ogr to generate a Flatgeobuf file:

ogr2ogr -f FlatGeobuf output.fgb output.geojson

While Flatgeobuf may be new to me, it seems like it already has pretty broad support in tools such as GDAL, Fiona, QGIS, and more.

The above ogr2ogr command created a 106.7MB Flatgeobuf file. I slapped together a quick demo using JS similar to the above but swapped out d3 for the Canvas API and promptly ran into my first hiccup with the format: My points didn’t start drawing until about 40MB of the file were streamed which is not what I was going for.

I asked about this on the developers’ Discord and they quickly set me straight: Flatgeobuf’s index is stored at the beginning of the file and uses about 40 bytes per node. So for my 1,000,000 points, roughly 40MB is just for the index.

Since my demo didn’t need the spatial index (I didn’t need to subset by a bounding box, which is another feature of the format), I could omit the index by passing the -lco SPATIAL_INDEX=NO flag to ogr2ogr:

ogr2ogr -f FlatGeobuf -lco SPATIAL_INDEX=NO output.fgb output.geojson

The resulting file ended up coming in at 64MB which matches the above estimate. The impact of not having the index is that drawing happens in the same order as the features were written out in the GeoJSON file, rather than following a Hilbert R-Tree For my use case, this is totally fine.

With this smaller, index-less file, the drawing begins immediately. And on my 10Mbit home connection, it actually takes about 20 seconds to stream which is (accidentally) a great way to showcase how well this works:

Check out the demo for yourself at https://amoeba-flatgeobuf-experiments.netlify.app or check out the code. The demo only really works on Chrome, probably due to differences in Canvas API implementations. Safari seems to delay doing any painting until all draw commands are done and Firefox seems to batch them.

1. I wanted only unit tests to run when CRAN checks the package. A package with tests that run on CRAN and depend on web services such as APIs are bound to cause your package to fail CRAN’s checks eventually which is a pain for both CRAN and you.
2. I wanted to check the package across a variety of platforms and R versions in a typical build matrix fashion.
3. I wanted to run a full integration test suite somewhere other than my machine in order to ensure the integration tests work in a clean environment.

I settled on GitHub Actions because it’s integrated with GitHub itself (which is really nice) and there are already great resources such as Jim Hester’s talk and helpful utilities such as usethis:use_github_actions() which makes it easy to get started.

The setup requires creating two GitHub Actions workflows:

1. One that runs R CMD CHECK across a build matrix of platforms and R versions to ensure the package works for others. This runs just unit tests (i.e., those that don’t depend on external access to an API).
2. Another that runs the full integration test suite. This will use a Docker container to spin up a fresh instance of the API I’m testing which is super easy with GitHub Actions.

Before setting up both workflows, I needed a way to skip a test if it’s an integration test (i.e., depended on having access to the API). I use testthat for my tests so I defined a helper in ./tests/setup-rt.R (rt is my package name here) which makes my helper available to all tests:

# Skip helper to control whether integration tests are run or not
skip_unless_integration <- function() {
  if (Sys.getenv("RT_INTEGRATION") != TRUE) {
    skip("Skipping integration test. Set RT_INTEGRATION to TRUE to run all tests.")
  }
}

This is the basis for a convention in my package where the full test suite is only run when the environmental variable RT_INTEGRATION is set to TRUE which I can control with GitHub Actions. With this setup, any test which requires access to the API gets skipped both on CRAN and when running the test suite locally when I prepend the following two lines to a test:

test_that("we can get properties of a ticket", {
  testthat::skip_on_cran()
  skip_unless_integration()

  # The rest of the test
})

With this test helper and testthat::skip_on_cran(), I can control which tests are run on CRAN and which tests are run when I have GitHub Actions run the full test suite depending on whether I include both, one, or none of them.

Now we need to pair this with the two workflows I mentioned above. These go in a .github folder at the top level of the package:

.github
└── workflows
    ├── ci.yml      # Build matrix
    └── tests.yml   # Integration tests

1 directory, 2 files

The first, ci.yml is a workflow that effectively runs R CMD CHECK on a variety of platforms and R versions (a build matrix):

on: [push, pull_request]

name: CI

jobs:
  CI:
    runs-on: ${{ matrix.config.os }}

    strategy:
      fail-fast: false
      matrix:
        config:
          - { os: windows-latest, r: "3.6", args: "--no-manual" }
          - { os: windows-latest, r: "4.0", args: "--no-manual" }
          - { os: macOS-latest, r: "3.6" }
          - { os: macOS-latest, r: "4.0" }
          - { os: macOS-latest, r: "devel", args: "--no-manual" }
          - { os: ubuntu-18.04, r: "3.5", args: "--no-manual" }
          - { os: ubuntu-18.04, r: "3.6", args: "--no-manual" }
          - { os: ubuntu-18.04, r: "4.0", args: "--no-manual" }
    env:
      R_REMOTES_NO_ERRORS_FROM_WARNINGS: true

    steps:
      - uses: actions/checkout@v1

      - uses: r-lib/actions/setup-r@master
        with:
          r-version: ${{ matrix.config.r }}

      - uses: r-lib/actions/setup-pandoc@master

      - uses: r-lib/actions/setup-tinytex@master
        if: contains(matrix.config.args, 'no-manual') == false

      - name: Cache R packages
        uses: actions/cache@v1
        if: runner.os != 'Windows'
        with:
          path: ${{ env.R_LIBS_USER }}
          key: ${{ runner.os }}-r-${{ matrix.config.r }}-${{ hashFiles('**/DESCRIPTION') }}

      - name: Install system dependencies
        if: runner.os == 'Linux'
        env:
          RHUB_PLATFORM: linux-x86_64-ubuntu-gcc
        run: |
          Rscript -e "install.packages('remotes')" -e "remotes::install_github('r-hub/sysreqs')"
          sysreqs=$(Rscript -e "cat(sysreqs::sysreq_commands('DESCRIPTION'))")
          sudo -s eval "$sysreqs"

      - name: Install dependencies
        run: |
          install.packages("remotes")
          remotes::install_deps(dependencies = TRUE)
          remotes::install_cran('rcmdcheck')
        shell: Rscript {0}

      - name: Check
        run: Rscript -e "rcmdcheck::rcmdcheck(args = '${{ matrix.config.args }}', error_on = 'warning', check_dir = 'check')"

      - name: Upload check results
        if: failure()
        uses: actions/upload-artifact@master
        with:
          name: ${{ runner.os }}-r${{ matrix.config.r }}-results
          path: check

The second, tests.yml runs the full test suite, which includes integration tests:

on: [push, pull_request]

name: Tests

jobs:
  CI:
    services:
      rt:
        image: netsandbox/request-tracker
        ports:
          - 80:80

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v1

      - uses: r-lib/actions/setup-r@master

      - name: Cache R packages
        uses: actions/cache@v1
        with:
          path: ${{ env.R_LIBS_USER }}
          key: ${{ hashFiles('**/DESCRIPTION') }}

      - name: Install system dependencies
        env:
          RHUB_PLATFORM: linux-x86_64-ubuntu-gcc
        run: |
          Rscript -e "install.packages('remotes')" -e "remotes::install_github('r-hub/sysreqs')"
          sysreqs=$(Rscript -e "cat(sysreqs::sysreq_commands('DESCRIPTION'))")
          sudo -s eval "$sysreqs"

      - name: Install dependencies
        run: |
          install.packages("remotes")
          remotes::install_deps(dependencies = TRUE)
          remotes::install_cran('rcmdcheck')
        shell: Rscript {0}

      - name: Check
        run: Rscript -e "rcmdcheck::rcmdcheck(args = \"--no-manual\", error_on = 'warning', check_dir = 'check')"

      - name: Upload check results
        if: failure()
        uses: actions/upload-artifact@master
        with:
          name: results
          path: check

Hopefully this pattern is useful to others. So far, I’ve found this setup works well and the hosting all of this on GitHub Actions also works well.

But the cultural tides are strong. Building a company on Django in 2020 seems like the equivalent of driving a PT Cruiser and blasting Faith Hill’s “Breathe” on a CD while your friends are listening to The Weeknd in their Teslas. Swimming against this current isn’t easy, and not in a trendy contrarian way.

The four problematic areas Tom mentions, (bundle splitting, SSR, APIs, and data fetching) come out of his experience building really awesome things (e.g., at Mapbox and Observable) so he knows at least a bit about this. To build a “modern” JS application, I find myself having to write considerably more code and configure a multitude of additional libraries to get what I feel I used to get for free with, say, Ruby. And then I’ve still got problems I can’t find good solutions for.

(Related are DHH ‘s thoughts about building for the web he delivered at RailsConf last week.)

Again from Tom’s article, this gem is kind of hidden near the end:

And it’s beneficial for companies to shift computing requirements from their servers to their customers browsers: it’s a real win for reducing their spend on infrastructure.

Complexity has to live somewhere. If you embrace it, give it the place it deserves, design your system and organisation knowing it exists, and focus on adapting, it might just become a strength.

Great take on managing complexity. Instantly reminds me of Rich Hickey’s excellent talk, Simple Made Easy where he discusses simplicity and complexity in software and how complexity does not necessarily mean something is complicated (to understand). Rich’s talk has stuck with me and I imagine this post will too. The entire post (and Rich’s talk) are well worth a read/listen.

For this demonstration, I started with the Natural Earth Data 1:10m Physical Vectors Land

shapefile and will convert it to a raster using the raster package.

A bit of searching around on the web led me to Amy Whitehead’s page on Converting shapefiles to rasters in R . The code listed there wasn’t quite what I needed but gave me the head start on figuring out what I needed to do.

Load required packages

The maptools package is used to import the shapefile to a SpatialPolygonsDataFrame and the raster package is for rasterizing the shapefile.

library(maptools)

## Loading required package: sp

## Checking rgeos availability: FALSE
##      Note: when rgeos is not available, polygon geometry     computations in maptools depend on gpclib,
##      which has a restricted licence. It is disabled by default;
##      to enable gpclib, type gpclibPermit()

library(raster)

Load the shapefile we’ll be rasterizing

Use the readShapePoly function from package maptools to read our shapefile in as a SpatialPolygonsDataFrame.

# Retrieved from http//www.naturalearthdata.com/download/110m/physical/ne_110m_land.zip
shape_url <- "http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_land.zip"
shape_folder <- tempdir()
shape_path <- tempfile(tmpdir = shape_folder, fileext = ".zip")
download.file(shape_url, shape_path)
unzip(shape_path, exdir = shape_folder)
dir(shape_folder)

##  [1] "devtools4bf4152fc28f"
##  [2] "devtools4bf431284c83"
##  [3] "devtools4bf4334c9928"
##  [4] "devtools4bf4355aa26c"
##  [5] "devtools4bf46278c6fd"
##  [6] "devtools4bf468d5df12"
##  [7] "devtools4bf47558b59"
##  [8] "devtools4bf48790d1c"
##  [9] "downloaded_packages"
## [10] "file4bf429c8b448"
## [11] "file4bf43b4c949f"
## [12] "file4bf4411676aa"
## [13] "file4bf4434bad9c"
## [14] "file4bf45ea45e1b.zip"
## [15] "file4bf4617ac199"
## [16] "file4bf4d7ffb4b"
## [17] "libloc_213_722ad5f9e07c7fe1.rds"
## [18] "ne_110m_land.cpg"
## [19] "ne_110m_land.dbf"
## [20] "ne_110m_land.prj"
## [21] "ne_110m_land.README.html"
## [22] "ne_110m_land.shp"
## [23] "ne_110m_land.shx"
## [24] "ne_110m_land.VERSION.txt"
## [25] "repos_https%3A%2F%2Fcran.rstudio.com%2Fbin%2Fmacosx%2Fel-capitan%2Fcontrib%2F3.5.rds"
## [26] "repos_https%3A%2F%2Fcran.rstudio.com%2Fsrc%2Fcontrib.rds"
## [27] "Rprofile-devtools"
## [28] "rs-graphics-0cee3d3c-f7dd-4e3f-b7e7-548de668efab"

land <- readShapePoly(file.path(shape_folder, "ne_110m_land.shp"))

## Warning: readShapePoly is deprecated; use rgdal::readOGR or sf::st_read

Create a blank raster

Before we can rasterize the shapefile, we need to create a blank raster. We specifiy the number of rows and columns to control the spatial resolution of our raster. We also have to specify the spatial extent, which will determine how much of the area of our shapefile we’re rasterizing. Here, I use the extent function from the raster package to grab the extent of the shapefile and pass that to the raster function. You can also specify extent manually.

blank_raster <- raster(nrow = 100, ncol = 100, extent(land))

If we were to plot this now, we would get an error because, by default, new rasters are created without any values. Rasters are fundamentally just like a 2-dimensional matrix (though they are stored as 1-d vectors). To get at this 2-d matrix, rasters have a slot

called data, which you can access by calling blank_raster@data. You can get (and set) the values of the raster using the values function.

values(blank_raster) <- 1
plot(blank_raster)

Above, we’ve set all those values to 1 and the result is just a raster of one value (color). The values inside the raster are stored in a vector of length nrow * ncol, so let’s set the values equal to the sequence of numbers from 1:nrow*ncol.

values(blank_raster) <- 1:(100*100)
plot(blank_raster, col=rainbow(50))

Pretty cool! So we can see that raster’s values are stored going from the top-left to the bottom-right.

Rasterize the shapefile

The first line here is pretty straightforward but the second line requires some explanation. The raster function tries to assign values to the resulting raster cells. Cells inside any of the polygons of the shapefile will have some value, while cells not included in any of the polygons of the shapefile will have the value NA. For some shapefiles, values are picked up from the attributes embedded in the shapefile and will result in variation in values (colors). In my case, I want to force all cells corresponding to land to be equal to 1 and leave the rest NA.

land_raster <- rasterize(land, blank_raster)
land_raster[!(is.na(land_raster))] <- 1

Plot the result

Rasters can be plotted directly with the plot function because the raster package implements its own plot method. Here, we also add the shapefile back on top of the raster to see how we did.

plot(land_raster, legend=FALSE)
plot(land, add=TRUE)

Looks pretty good! The filled-in yellow area is the new raster. Let’s zoom in to see what’s going on.

plot(land_raster, legend=FALSE, xlim=c(-170, -120), ylim=c(30, 65))
plot(land, add=TRUE,  xlim=c(-160, -120), ylim=c(30, 90))

Not great. The raster cells are roughly outlining the land but end up being pretty poor approximations in places. Clearly, we might like to improve the spatial resolution of our raster but this is a pretty good start.

Let’s the spatial resolution up to see if we can do better.

blank_raster <- raster(nrow = 1000, ncol = 2000, extent(land))
land_raster <- rasterize(land, blank_raster)
land_raster[!(is.na(land_raster))] <- 1
plot(land_raster, legend=FALSE, xlim=c(-170, -120), ylim=c(30, 65))
plot(land, add=TRUE,  xlim=c(-160, -120), ylim=c(30, 90))

Much better!

Bonus: Effect of different raster resolutions

In the above example, I used a 100x100 grid for the raster. Below, I test 20x20, 100x100, 500x500, and 1000x1000 grids together to explore the effect of that part of the process. What resolution do we need to adequately describe the shape of the land?

layout(matrix(1:4, ncol=2, byrow=TRUE))

resolutions <- c(20, 100, 500, 1000)

for(r in resolutions)
{
  blank_raster <- raster(nrow = r, ncol = r, extent(land))
  land_raster <- rasterize(land, blank_raster)
  land_raster[!(is.na(land_raster))] <- 1

  plot(land_raster, legend=FALSE, xlim=c(-170, -120), ylim=c(30, 65), main=paste0("Resolution: ", r, "x", r))
  plot(land, add=TRUE,  xlim=c(-160, -120), ylim=c(30, 90))
}

You can see that, by 500x500, we’re getting a raster that looks pretty close to the underlying shapefile.

Step 1: Acquire the NetCDF library

Before we can open a NetCDF file in R, we need to install the NetCDF library on our system. I’m using a Mac running OS 10.9 and I use homebrew as my package manager.

If you’re using homebrew, install the netcdf library with:

brew install netcdf

If you’re on Windows, you can find pre-built binaries at the developer’s download page. For other systems, consult the documentation.

Step 2: Install and load the ncdf package in R

install.packages("ncdf4")

And load it:

library(ncdf4)

Step 3: Load your NetCDF file

For this tutorial, I’ll be working with the NOAA Optimum Interpolation (OI) Sea Surface Temperature (SST) V2 data series of monthly means from December 1981 – current. These data are produced on a 1° grid for the entire globe.

sst_url <- "ftp://ftp.cdc.noaa.gov/Datasets/noaa.oisst.v2/sst.mnmean.nc"
sst_path <- tempfile()
download.file(sst_url, sst_path) # 53.0 MB
cdf <- nc_open(sst_path)

I am interested in the following variables: latitude, longitude, time, and SST:

lat <- ncdf4::ncvar_get(cdf, varid="lat")
lon <- ncdf4::ncvar_get(cdf, varid="lon")
time <- ncdf4::ncvar_get(cdf, varid="time")
sst <- ncdf4::ncvar_get(cdf, varid="sst")

The lat and lon variables are just vectors containing the range of latitudes (89.5S to 89.5N) and longitudes (0.5°E to 359.5°E, starting from the prime meridian).

The time variable is a little more complex. Let’s take a look:

head(time)

## [1] 66443 66474 66505 66533 66564 66594

Those don’t look like times at all. If you’re used to working with dates and times on computers, you may already be way ahead of me. If, however, if you are not, I’ll explain. Dates and times are usually stored by computers as a number of time units since we started counting time. The time units may be milliseconds, seconds, or even months – whatever suits the purpose. Computers commonly store dates and times as the number of seconds since January 1, 1970 (See UNIX time ). In the case of our data, time is being counted as days since January 1, 1800. This is a little weird but it makes sense for our data. If you’re using a different NetCDF file than me, you’ll want to consult the documentation that goes along with the data to figure out how they’re counting time.

Now that we know how time is being stored, we can make it human-readable:

time_d <- as.Date(time, format="%j", origin=as.Date("1800-01-01"))
time_years <- format(time_d, "%Y")
time_months <- format(time_d, "%m")
time_year_months <- format(time_d, "%Y-%m")

Here, I set the origin to January 1, 1800 and then convert the original variable time into a vector of R Date objects, passing the parameters format="%j" and origin=as.Date("1800-01-01"). The time variable is now much more easier to understand:

head(time_d)

## [1] "1981-12-01" "1982-01-01" "1982-02-01" "1982-03-01" "1982-04-01"
## [6] "1982-05-01"

The other variables I created, time_years, time_months, time_year_months are for added utility. I can now reference SST data for just a set of years

time_years %in% c("1990", "1991")

or all SST values from June

time_months %in% c("06")

Our last, but most important variable is SST.

dim(sst)

## [1] 360 180 441

sst is three dimensional, and is indexed by longitude, latitude, and time (respectively). To extract a single SST value from it, we’ll need to specify an index or range of indices for all three dimensions:

sst[lon==220.5, lat==50.5, time_d==as.Date("1990-06-01")]

## [1] 10.91

We can use our utility variables to, for example, extract all the observations for June from this same grid cell:

sst[lon==220.5, lat==50.5, time_months=="06"]

##  [1]  9.27 10.25  9.68  8.81  9.75  8.96  8.82  9.79 10.91 10.04  9.55
## [12] 11.01 10.55  9.46  9.39 10.21 10.00  7.94  9.39  8.78 10.08 10.05
## [23] 10.56 10.67  9.78  8.65  8.33 10.04  8.88  9.11  8.21 10.19 11.51
## [34] 12.05 10.27  9.76 10.32

Depending on your goals, this may be as far as you need to get. But maybe you want to display these data visually. Let’s plot the SSTs for a range of grid cells onto a map.

Step 4: Convert the SST data to a data.frame

Our NetCDF file has a lot of observations:

prod(dim(sst))

## [1] 28576800

To reduce the amount of computation, let’s subset the data to a range of latitudes and longitudes and also focus in on a particular month in the data set so we can plot this in two dimensions:

lat_range <- seq(55.5, 60.5)
lon_range <- seq(190.5, 195.5)

lat_indices <- lat %in% lat_range
lon_indices <- lon %in% lon_range
time_indices <- time_year_months=="1990-06" # June of 1990

Notice how I constructed the _indices variables. They are vectors of the same length as their corresponding variable but they contain TRUEs and FALSEs where the corresponding variable is equal to the desired ranges. This allows instant subsetting:

sst[lon_indices, lat_indices, time_indices]

##      [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 4.20 4.12 4.63 5.47 6.19 6.63
## [2,] 4.39 4.05 4.40 5.18 5.99 6.64
## [3,] 4.71 4.04 4.11 4.73 5.58 6.55
## [4,] 5.21 4.39 4.32 4.82 5.63 6.61
## [5,] 5.57 4.91 4.80 5.17 5.87 6.70
## [6,] 5.53 5.36 5.29 5.55 6.12 6.81

Which gives us the SST values for the 5x5 grid for June of 1990. To save them in a more convenient format, we’ll convert it to a data.frame.

cdf_df <- expand.grid(lat_range, lon_range)
names(cdf_df) <- c("lat", "lon")
cdf_df$sst <- NA
head(cdf_df)

##    lat   lon sst
## 1 55.5 190.5  NA
## 2 56.5 190.5  NA
## 3 57.5 190.5  NA
## 4 58.5 190.5  NA
## 5 59.5 190.5  NA
## 6 60.5 190.5  NA

Now each row of cdf_df will correspond to one SST value, one row for every grid cell combination (25 in total). Let’s populate it with SST values:

for(i in 1:nrow(cdf_df))
{
  lat_ind <- which(lat == cdf_df[i,"lat"])
  lon_ind <- which(lon == cdf_df[i,"lon"])

  cdf_df[i,"sst"] <- sst[lon_ind, lat_ind, time_indices]
}

head(cdf_df)

##    lat   lon  sst
## 1 55.5 190.5 6.63
## 2 56.5 190.5 6.19
## 3 57.5 190.5 5.47
## 4 58.5 190.5 4.63
## 5 59.5 190.5 4.12
## 6 60.5 190.5 4.20

Looks good! Let’s make a map to display these data on. Luckily, this is pretty straightforward in R using the maps and mapdata packages.

library(maps)
library(mapdata)
map('world2Hires', xlim=range(lon_range) + c(-10, 10), ylim=range(lat_range) + c(-5, 5))
box()
points(cdf_df$lon, cdf_df$lat)

The above map is a good start. I’ve used the world2Hires map from mapdata which lets me create a map centered on the Eastern Bering Sea. I’ve specific the x- and y-limits to show just the area where we have SST values. Let’s change the points to rectangles and make the background color of the rectangles correlate with the corresponding SST value for that grid.

We’ll first make the color scale:

ncolors <- 5
cols <- cut(cdf_df$sst, ncolors)
palette <- colorRampPalette((c("blue", "red")))(ncolors)

To plot rectangles instead of points, we can use the rect() function instead of points(). rect() draws rectangles on the graphics device and needs the user to specify the coordinates of each corner (xleft, ybottom, xright, ytop).

map('world2Hires', xlim=range(lon_range) + c(-10, 10), ylim=range(lat_range) + c(-5, 5))
box()

grid_hw <- 0.5 # Grid half width
rect(cdf_df$lon - grid_hw, cdf_df$lat - grid_hw, cdf_df$lon + grid_hw, cdf_df$lat + grid_hw, col=palette[cols])

map.axes()
title(main="June SST Values", xlab="Longitude", ylab="Lattitude")
legend("topright", legend=levels(cols), fill=palette)

Looks great! Hopefully this post was a useful introduction to working with and displaying NetCDF data.