Carson's blog on R, RStudio, plotly, shiny, data visualization, statistics, etc

plotly 4.9.0 now on CRAN

Thu, 11 Apr 2019 00:00:00 +0000

I’m excited to announce plotly 4.9.0 is now on CRAN! Go ahead and upgrade via:

install.packages("plotly")

This release is about 8 months in the making and includes many bug fixes, some new features, and a couple slightly breaking changes. Over this time, I’ve mainly focused on getting a first draft of my new book, https://plotly-r.com, and some of these additions to plotly are in support of that new content. This post highlights the most important release features, but I highly recommend going over the full release report as well.

New shiny events

This release exposes a handful of new plotly events in shiny. See here for the full list of new events, this section of my book to learn about how to leverage plotly events in shiny, and my RStudio webinar for a tutorial on how to create responsive and scalable crossfiltering applications with these new events (like in the video below).

Also, see the advanced applications section of the plotly book for some new drill down examples.

New version of plotly.js

This new version of the R package upgrades the underlying plotly.js library from v1.42.3 to v1.46.1. This includes many bug fixes and new features, some of which I don’t mention here or in the R package’s NEWS, but you can see plotly.js’ release page for more info. That being said, here’s a breakdown of the most important items:

New trace types

This release includes 3 new trace types: isosurface, sunburst, and waterfall. To get idea of what possible, you can search the plotly.js tests via https://rreusser.github.io/plotly-mock-viewer (the corresponding JSON source behind these example can be found here).

Isosurface

Sunburst

Waterfall

plot_ly() %>%
  add_trace(
    type = "waterfall",
    x = c(0, 1, 2, 3, 4, 5),
    y = c(1, 0.5, 0.7, -1.2, 0.3, 0.4)
  )

I suspect soon https://plot.ly/r will have more R examples of these traces, but for now you can see some examples here

https://github.com/ropensci/plotly/blob/master/tests/testthat/test-plotly-isosurface.R https://github.com/ropensci/plotly/blob/master/tests/testthat/test-plotly-sunburst.R https://github.com/ropensci/plotly/blob/master/tests/testthat/test-plotly-waterfall.R

New hovertemplate attribute

One of the more difficult things about plotly and plotly.js is about the various ways to control tooltips. This new special topic in my book attempts to cover all these approaches, as well as the new hovertemplate approach. This new attribute allows you to reference supplied (e.g., x in the Figure below) or computed variables in the tooltip (e.g., y in the Figure below). It also allows you to control the formatting (using d3-format’s syntax %{variable:d3-format}) and the <extra> bit of the tooltip that usually displays the trace name.

plot_ly(x = rnorm(100, 0, 1e5)) %>%
    add_histogram(
        histnorm = "density",
        hovertemplate = "The density between <br> (%{x}) is %{y:.1e} <extra>That's very <br> small!</extra>"
    )

New title API

Previously plot title’s were created by providing a string to layout.title. This will continue to work, but layout.title now accepts a list object, which adds the ability to move the title’s location.

library(plotly)
plot_ly(x = 1, y = 1) %>%
  layout(title = 
    list(
      text = "My title", 
      x = 1, 
      font = list(
        family = "Roboto", 
        color = "#1E86FF", 
        size = 18
      )
    )
  )

Since plotly.js now supports this feature, ggplotly() will now automatically pick up on the title alignment

library(plotly)
p <- qplot(1:10) + 
  ggtitle("My amazing plot!") +
  theme(
    plot.title = element_text(
      family = "Roboto", 
      color = "red", 
      size = 16
    )
  )
ggplotly(p)

Announcing my new book: Interactive web-based data visualization with R, plotly, and shiny

Mon, 01 Apr 2019 00:00:00 +0000

My online book (previously known as plotly for R, now titled Interactive web-based data visualization with R, plotly, and shiny) has received a major update with heaps of new examples and content! A print/pdf version of the book is now under review and should become available in the coming months. In the meantime, please do send me feedback or any comments as I would appreciate your input during the review process!

For those already somewhat familiar with the book, here is some highlights of the biggest additions and changes:

A new top-level Introductory chapter helping to explain what’s in and not in the book.
Chapters are now organized into 6 parts: Creating views, Publishing views, Combining multiple views, Linking multiple view, Custom behavior with JavaScript, and Various special topics.
The ‘plotly cookbook’ chapter has expanded into several chapters (Part I: Creating Views), complete with a new Overview Chapter, a much more complete Chapter on scatter traces, and much more on mapping functionality.
Lots of new examples and documentation of plotly’s ability to link views client-side as well as server-side with shiny
Lots of new examples and documentation about writing custom event handlers in JavaScript (see Part V: Custom behavior with JavaScript).
Updated examples and writing in the Chapters Arranging views and Animating views
Part VI includes a handful of new topics addressing functionality that currently isn’t well documented elsewhere, including everything you need to know about working with tooltips, colors, symbols, the modebar, and more!
A new RStudio cloud project for running the code examples in the book.

Some of this content assumes you are running the development version of a few packages, so if you want to run the code examples locally, you can install all the packages (and their correct versions) with:

remotes::install_github("cpsievert/plotly_book")

plotly 4.8.0 now on CRAN

Wed, 18 Jul 2018 00:00:00 +0000

Need help with R, plotly, data viz, and/or stats? Work with me!

I’m super excited to announce plotly 4.8.0 is now on CRAN! Go ahead and upgrade via:

install.packages("plotly")

This release is almost a year in the making and includes a huge amount of improvements, new features, and some slightly breaking changes. This post highlights the biggest and most recent features; but if you already use plotly, I highly recommend going over the full release report as well.

New trace types

Upgrading plotly from 4.7.1 to 4.8.0 implies an upgrade of plotly.js from 1.29.2 to 1.39.2. This means, amongst many other things, seven new trace types. Here’s a list with a link to their relevant online https://plot.ly/r documentation:

Violin
3D cones and streamtubes
New and improved SVG & WebGL polar charts
Scatterplot matrices (NOTE: click “Box select” in the modebar of these examples for linked brushing!)
Tables (NOTE: the new add_table() function makes it easier to map a data frame to this trace type)

Improved static image export

The new orca() function wraps the new orca project for killer exporting of plotly graphs, dashboards, etc to various formats (most notably, svg and pdf!). Simply give orca() a plotly object and a filename with the extension that you desire. Here’s an example of exporting to SVG and post-processing the result in Adobe Illustrator:

p <- plot_ly(z = ~volcano) %>% add_surface()
orca(p, "surface-plot.svg")

Auto margins

Axis objects now default to automargin = TRUE. This helps the axis labels of any plotly graph to be more readable by default:

library(plotly)
library(dplyr)

# plot the number of missing values by city
txhousing %>% 
  group_by(city) %>% 
  summarise(missing = sum(is.na(median))) %>% 
  filter(missing > 0) %>%
  plot_ly(
    x = ~missing, 
    y = ~forcats::fct_reorder(city, missing)
  )

To get the old behavior back, you can set automargin = FALSE. This hopefully won’t be needed except in extreme edge cases:

layout(.Last.value, yaxis = list(automargin = FALSE))

Localization support

The new locale argument in the config() function makes it easy to switch the language used for date axes and on-graph text (e.g., modebar buttons). This example demonstrates Japanese ("ja") – see ?config for more examples and documentation of other locales.

today <- Sys.Date()
x <- seq.Date(today, today + 360, by = "day")
plot_ly(x = x, y = rnorm(length(x))) %>%
  add_lines() %>%
  config(locale = "ja")

Partial plotly.js bundles

By default, the R package ships with a full bundle of plotly.js, which is quite large (about 2.6 MB in this release). In many cases, graphs can be rendered with a partial plotly.js bundle, which can lead to a 3x reduction in file size! To take advantage, just apply the partial_bundle() function to a plotly object, which (by default) will try and find a suitable bundle.

library(plotly)
p <- plot_ly(x = 1:10, y = 1:10) %>% add_markers()
save_widget <- function(p, f) {
  owd <- setwd(dirname(f))
  on.exit(setwd(owd))
  htmlwidgets::saveWidget(p, f)
  mb <- round(file.info(f)$size / 1e6, 3)
  message("File is: ", mb," MB")
}
f1 <- tempfile(fileext = ".html")
save_widget(p, f1)

#> File is: 2.94 MB

f2 <- tempfile(fileext = ".html")
save_widget(partial_bundle(p), f2)

#> File is: 0.973 MB

New and improved WebGL renderer

For a while now, we’ve had the ability to switch from SVG to WebGL via the function toWebGL(), which can improve performance when rendering lots of graphical elements. Now there is even more reason to do so as the new WebGL renderer is now much closer to being feature complete with the SVG renderer.

plot_ly(x = rnorm(1e6), y = rnorm(1e6)) %>%
  add_markers(color = I("transparent"), stroke = I("black"), span = I(1)) %>%
  toWebGL()

MathJax support

LaTeX rendering via MathJax is now possible via the new TeX() function may be used to flag a character vector as LaTeX. To load MathJaX externally (meaning an internet connection is needed for TeX rendering), set the new mathjax argument in config() to "cdn":

library(plotly)

plot_ly() %>%
  add_lines(x = zoo::index(co2), y = co2) %>%
  layout(
    title = TeX("CO_2 \\text{measured in } \\frac{parts}{million}"),
    xaxis = list(title = "Time"),
    yaxis = list(title = TeX("\\text{Atmospheric concentraion of CO}_2"))
  ) %>%
  config(mathjax = "cdn")

To use a local version of MathJax (so that your graphs will render without an internet connection), you need to inform plotly where it’s located. If you don’t already have MathJax locally, I recommend downloading the official MathJax git repo. Here’s how to do that using terminal commands:

$ git clone https://github.com/mathjax/MathJax.git
$ cd MathJax

Now set the PLOTLY_MATHJAX_PATH environment variable so that plotly knows where that MathJax folder lives. I recommend setting this variable in you .Rprofile so you don’t have to reset it everytime you restart R:

$ export PLOTLY_MATHJAX_PATH=`pwd`
$ echo "Sys.setenv('PLOTLY_MATHJAX_PATH' = '$PLOTLY_MATHJAX_PATH')" >> ~/.Rprofile

Once PLOTLY_MATHJAX_PATH is set, specify mathjax="local" in config():

library(plotly)

plotly_empty() %>%
  add_trace(x = 1, y = 1, text = TeX("\\alpha"), mode = "text", size = I(100), hoverinfo = "text") %>%
  config(mathjax = "local")

MathJax caveats

At least currently, plotly.js requires SVG-based rendering which doesn’t play nicely with HTML-based rendering (e.g. rmarkdown documents and shiny apps) . If you need both the SVG and HTML rendering, consider <iframe>-ing your plotly graph(s) into the larger document (see here for an example).
Due to the size and nature of MathJax, using htmlwidget::saveWidget() with selfcontained = TRUE won’t work. At least for now, when you need to save a plotly graph (p) with local MathJax, do htmlwidget::saveWidget(p, selfcontained = FALSE)

Other features discussed previously

There are numerous other exciting features that come with this release that I’ve written about previously:

sf support via ggplotly(), plot_ly(), plot_geo(), and plot_mapbox().
Better control over the stroke (i.e., outline) appearance of various filled graphical marks via the new “special arguments” (stroke, strokes, alpha_stroke, span, and spans).

Visualizing geo-spatial data with sf and plotly

Fri, 30 Mar 2018 00:00:00 +0000

Need help with R, plotly, data viz, and/or stats? Work with me!

In my last post, we explored interactive visualizations of simple features (i.e., interactive maps) via ggplot2’s geom_sf() and plotly’s ggplotly(). This time we’ll make similar visualizations using plotly’s “non-ggplot2” mapping interfaces (namely plot_ly(), plot_geo(), and plot_mapbox()). Here’s a quick example of reading a shape file into R as simple features via st_read(), then plotting those features (in this case, North Carolina counties) using each one of the four mapping approaches plotly provides. Notice how all of these options auto-magically know how to render simple features:

# to replicate these examples, you currently need the dev version
# devtools::install_github('ropensci/plotly')
library(plotly)

nc <- sf::st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)
class(nc)

#> [1] "sf"         "data.frame"

subplot(nrows = 2,
  ggplot(nc) + geom_sf(),
  plot_ly(nc),
  plot_geo(nc),
  plot_mapbox(nc)
) %>% hide_legend()

You might be wondering, “What can plotly offer over other interactive mapping packages such as leaflet, mapview, mapedit, etc?”. One big feature is the linked brushing framework, which works best when linking plotly together with other plotly graphs (i.e., only a subset of brushing features are supported when linking to other crosstalk-compatible htmlwidgets). Another is the ability to leverage the plotly.js API to make efficient updates in shiny apps via plotlyProxy(). Speaking of efficiency, plotly.js keeps on improving the performance of their WebGL-based rendering, so I recommend trying plot_ly() (with toWebGL()) and/or plot_mapbox() if you have lots of graphical elements to render. Also, by having a consistent interface between these various mapping approaches, it’s much quicker and easier to switch from one approach to another when you need to leverage a different set of strengths and weaknesses.

Speaking of strengths, plotly.js also has 🌎-class 3D rendering support. When used in combination with plotly’s linking framework, we can do some nifty things – all inside self-contained HTML! For example, here are linked 3D & 2D views of tropical storm paths which is useful for querying anomalies and provides some insight into the relationship between distance traveled, altitude, latitude, and longitude:¹

# To see the actual R code that generates the self-contained HTML result,
# enter this in your R console or visit https://github.com/ropensci/plotly/tree/master/demo
demo("sf-plotly-3D-globe", package = "plotly")

As you’ll see in the NEWS of the development version, I’ve also added new stroke and span arguments which make it easier to control the outline color and width of filled polygons/markers/bars/etc. These new arguments work in a similar way to existing arguments like color and linetype. In particular, constant values can be specified via I():

plot_ly(nc, stroke = I("#119dff"), span = I(5), color = I("#00cc96"), linetype = I("dash"))

Furthermore, values not wrapped in I() are mapped to a visual range defined by the plural form of the argument (i.e., strokes and spans). However, to effectively map data to these sorts of visuals, we might want to generate mutliple traces.

One trace to rule them all?

One of the trickiest things about mastering plotly and/or plotly.js is knowing what can and can not be done with just one trace. As I elude to in the plotly for R book, if something can be implemented with a single trace, then it should, because traces don’t scale very well (i.e., can easily lead to a sluggish plot). That’s why, by default, the maps above are implemented with just one trace.² However, when you need certain scalar (i.e., non-data-array) trace properties (e.g. fillcolor) to vary, you might want to use split to create a trace for every level in the split variable. For example, in this map of territories in Franconia, we ensure one trace per territory (via the split argument), which leads to two fairly obvious features:

The ability to hide/show territories by clicking (or double-clicking) legend entries. As we’ll discuss later, there are actually ways to do this sort of thing with just one trace via the crosstalk (i.e. linked views) framework.
A different color for each territory. In this case, we’ve used split without specifying color, so plotly.js will impose it’s default coloring rules, but you could easily set a constant color across traces (e.g., color = I("black")) or, as we’ll see shortly, use a custom color scheme.

# load trails data (an sf object bundled with the mapview package)
# install.packages('mapview')
data(franconia, package = "mapview")

# Compare this result with: `plot_ly(franconia, split = ~NAME_ASCI, color = I("black"))`
plot_ly(franconia, split = ~NAME_ASCI)

Having a different trace for each territory opens the door for further territory-level customization, such as having a custom color, linetype, fill-specific tooltip, etc. When color is numeric, and you want it to set a different fill-color for certain polygon(s), you’ll need split to ensure there is no more than one color per trace:

plot_ly(
  franconia, 
  split = ~NUTS_ID,
  color = ~SHAPE_AREA,
  alpha = 1,
  showlegend = FALSE
)

On the other hand, if color is a discrete variable, plot_ly() always produces one trace per level. Since, in this case, there are over 30 districts, I’m going to use a color palette that does a fairly good job in distinguishing between a large number of groups:

cols <- c("#89C5DA", "#DA5724", "#74D944", "#CE50CA", "#3F4921", "#C0717C", "#CBD588", "#5F7FC7", "#673770", "#D3D93E", "#38333E", "#508578", "#D7C1B1", "#689030", "#AD6F3B", "#CD9BCD", "#D14285", "#6DDE88", "#652926", "#7FDCC0", "#C84248", "#8569D5", "#5E738F", "#D1A33D", "#8A7C64", "#599861")
plot_ly(franconia, color = ~NAME_ASCI, colors = cols, alpha = 1)

Note that you can still color multiple traces the same color, but if you want to have a different tooltip to appear upon hovering a specific set of polygons (as opposed to along each point), you’ll want to still use split in combination with hoveron = "fill":

plot_ly(
  franconia,
  split = ~NUTS_ID,
  color = ~district,
  stroke = I("black"),
  text = ~paste(NAME_ASCI, "\n is in", district), 
  hoverinfo = "text",
  hoveron = "fill"
)

Leveraging mapbox basemaps

Among the four mapping options, the one that excites me the most is plot_mapbox()³. The primary reason to use plot_mapbox() (or plot_geo()) over geom_sf() (or plot_ly()) is that these approaches include a base-map layer which updates dynamically on zoom and can quite helpful for providing geographic context. The base-map styling can be easily customized – especially for plot_mapbox() via the layout.mapbox.style attribute. In addition to a URL to a custom style, you can provide this attribute with a pre-packaged style name. To get a list of those pre-packaged styles, reach into the plotly.js plot schema():

styles <- schema()$layout$layoutAttributes$mapbox$style$values
styles

[1] "basic"             "streets"           "outdoors"          "light"            
[5] "dark"              "satellite"         "satellite-streets"

A nice feature to include with these maps is a dropdown to dynamically change the base-map styling. To do so, use the layout.updatemenus.buttons attribute to control the value of the layout.mapbox.style attribute. Since we wish to modify the layout.mapbox.style attribute, which is part of the plot’s layout, we’ll want each dropdown “button” to trigger a relayout event with the appropriate args. Here is one way to create a list of buttons, one for each of the default styles:

# generate plot.js buttons, one for every style 
style_buttons <- lapply(styles, function(s) {
  list(label = s, method = "relayout", args = list("mapbox.style", s))
})

With our list of style_buttons, we’re now ready to visualize something. Here I’ll leverage the trails data from the mapview package, which has a number of hiking trails of Franconia, Germany.

data(trails, package = 'mapview')

plot_mapbox(trails, color = I("black")) %>%
  layout(
    title = "Selected hiking trails in Franconia",
    mapbox = list(style = "satellite-streets"),
    updatemenus = list(list(y = 0.8, buttons = rev(style_buttons)))
  )

As a side note, you could also use shiny to modify the basemap layer without redrawing the entire plot, but the result here is self-contained HTML/SVG, which is much easier to share, host, scale, and maintain.

Linking views without shiny

In terms of linked brushing in self-contained HTML, plotly is definitely the most advanced crosstalk-compatible htmlwidget. You’ll be able to do more when linking plotly to plotly, but since persistent selection was recently added to DT, let’s demonstrate linking plotly with DT. To link views via crosstalk, you’ll want to supply a data frame of interest to the SharedData$new() method and route the resulting object to any plots that you want to link. By default, the row index (which, in this case, is a simple feature) is used to define the graphical queries, but you can also reference a (discrete) variable to achieve generalized selections (e.g., you could query all trails in a district via SharedData$new(trails, ~district)).⁴ Linking plotly with DT in this way gives us a pretty powerful way to identify simple features both directly and indirectly.

library(crosstalk)
tsd <- SharedData$new(trails)

bscols(
  plot_mapbox(tsd, text = ~FKN, hoverinfo = "text"),
  DT::datatable(tsd)
)

To provide a sneak peak into the power of the linking framework in plotly, let’s leverage a fairly recent feature: brushing of aggregated traces. This example demonstrates how to brush a histogram, but a similar approach could be used to brush other aggregated traces (e.g., add_histogram2d(), add_boxplot(), etc). In fact, one could replicate this example with add_bars() (instead of add_histogram()) by pre-computing bars heights and using a list-column key to assign multiple counties to each bar. In either case (add_bars() or add_histogram()), it is usually a good idea to set layout.barmode = "overlay" so that newly added bars don’t use the plotly.js default of dodging the existing bars. In the case where we let plotly.js dynamically compute aggregates (i.e., add_histogram()) it’s also a good idea to also define xbins (or ybins) so that binning of new bars use the same rules as the existing (i.e. initial) bars.

ncsd <- SharedData$new(nc)

bscols(
  plot_mapbox(ncsd) %>%
    highlight(dynamic = TRUE, persistent = TRUE),
  plot_ly(ncsd, x = ~AREA) %>% 
    add_histogram(xbins = list(start = 0, end = 0.3, size = 0.02)) %>%
    layout(barmode = "overlay") %>% 
    highlight("plotly_selected", persistent = TRUE)
)

If you’re interested in understanding the full power of the linking framework, my 2 day plotly for R workshop is the best way to learn it effectively. I also offer this workshop as an on-site training course, so please get in touch if you have any interest!

More learning resources

At least currently, the best examples of using sf with plotly are within the package demos. Any demo names that are prefixed with ‘sf’ when you look at the list provided by demo(package = "plotly") are relevant. For example, demo("sf-dt", package = "plotly") gives an example of querying simple feature data by linking plot_mapbox() with DT via crosstalk. Also be on the look-out for updates to the mapping section of the plotly for R book as well as examples in some of my more recent talks.

Future work

I’m excited to see the author of sf, Edzer Pebesma, starting work on stars – a tidy (and sf friendly) approach to working with geo-spatial arrays (e.g. raster data). Once that project becomes stable, I’m hoping to find the time and resources to build a similar bridge between plotly and stars. This effort might have to take a back seat for several months though unless someone would like to sponsor (or otherwise assist in) development.

Appendix: strengths and weaknesses

Below is a list of strengths (blue) and weaknesses (red) for each mapping approach in plotly. Note that plotly.js is still under development, so this list is likely change a bit (please let me know if I’m missing anything):

1. `plot_ly()` and `geom_sf()`

Render in SVG or WebGL (toWebGL() makes SVG -> WebGL easy) 🔵
- fill='toself' is still buggy in WebGL 🔴
Make 3D visuals by adding a z attribute 🔵
- With a bit of math, you can project any lat/lon data onto a sphere (i.e. globe) and color the globe’s surface according to some measure, like radiation 🔵
- Unlike 2D lines, 3D lines support a color gradient 🔵
- Using subplot() with 3D doesn’t currently rescale domains correctly 🔴
Works with any coordinate system 🔵
- Currently no way to update a graticule on zoom 🔴

2. `plot_mapbox()`

Full customization of base-maps that provide geo-graphic context 🔵
Simple features can be rendered as data or layout components 🔵
Numerous trace rendering limitations including no marker.line hoveron='fill' (i.e., stroke and span won’t do anything) 🔴
WebGL infrastructure allows one to render lots of graphical elements 🔵

3. `plot_geo()`:

Somewhat customization base-maps provide geo-graphic context 🔵
- Compared to plot_mapbox(), base-map styling is quite limited 🔴
Supports orthographic (3D) projections as well as a number of other 2D projections 🔵

I got this idea thanks to this post and plotly user Emilia.↩
Check for yourself by doing plotly_json(plot_ly(nc)).↩
The plot_mapbox() builds on mapbox-gl.js which requires an access token. Once you have one, inform plotly about your token via Sys.setenv('MAPBOX_TOKEN' = 'secret token') (or via an .Renviron file).↩
To learn more, read https://plotly-book.cpsievert.me/linking-views-without-shiny.html#the-shareddata-plot-pipeline ↩

Learning from and improving upon ggplotly conversions

Tue, 30 Jan 2018 00:00:00 +0000

Need help with R, plotly, data viz, and/or stats? Work with me!

As the maintainer of the R package plotly, I’m certainly aware that ggplotly() is not perfect.¹ And, even when conversion from ggplot2 to plotly ‘works’ it can leave some things to be desired. For example, it might take a while to render, it might not look exactly the way ggplot2 does, and/or the default interactive properties (e.g., tooltips) might not work the way you want them to. In almost every case, these are issues that, with a bit of knowledge about plotly and the underlying plotly.js library, you can fix. That’s because, as long as ggplotly() returns an object, we can inspect its underlying data structure (which, by the way, is useful for learning plotly.js!) and improve on it by modifying that data.² This post explores several ways to learn from, leverage, and improve ggplotly() conversions of ggplot2’s geom_sf() (which is still in development), but some of the same lessons can apply more generally to other ggplot2 geoms.

Basic intro to `geom_sf()`

For a quick demonstration of geom_sf(), I’m using albersusa to access the laea projected boundaries of the United States as a simple features (sf) data structure, but sf also makes it easy to read various file formats and even convert various spatial objects to sf. There are also a bunch of other R packages that, like albersusa, make it easy to query geo-spatial data as an sf data. The “Reverse dependencies” section of sf’s CRAN page is a good place to discover them, but just to name a few: tidycensus, rnaturalearth, and mapsapi. One awesome consequence of using sf is that, since the data structure contains all the geo-spatial information, both plot() and geom_sf() just work^TM.

# to replicate these examples, you currently need the dev version
# devtools::install_github('ropensci/plotly')
# devtools::install_github('hrbrmstr/albersusa')
library(plotly)
library(sf)
library(albersusa)

us_laea <- usa_sf("laea")
p <- ggplot(us_laea) + geom_sf()
ggplotly(p)

The most brilliant thing about sf is that it stores geo-spatial structures in a special list-column of a data frame. This allows each row to represent the real unit of observation/interest – whether be a polygon, multi-polygon, point, line, or even a collection of these features – and as a result, supports workflows that leverage tidy-data principles³.

In our case, the us_laea data contains 51 rows (50 states + DC), and the geometry column tracks info about the geo-spatial objects:

library(dplyr)
select(us_laea, name, pop_2014, geometry)

Simple feature collection with 51 features and 2 fields
geometry type:  MULTIPOLYGON
dimension:      XY
bbox:           xmin: -2100000 ymin: -2500000 xmax: 2516374 ymax: 732103.3
epsg (SRID):    NA
proj4string:    +proj=laea +lat_0=45 +lon_0=-100 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs
First 10 features:
                   name pop_2014                       geometry
1               Arizona  6731484 MULTIPOLYGON (((-1111066 -8...
2              Arkansas  2966369 MULTIPOLYGON (((557903.1 -1...
3            California 38802500 MULTIPOLYGON (((-1853480 -9...
4              Colorado  5355866 MULTIPOLYGON (((-613452.9 -...
5           Connecticut  3596677 MULTIPOLYGON (((2226838 519...
6  District of Columbia   658893 MULTIPOLYGON (((1960720 -41...
7               Georgia 10097343 MULTIPOLYGON (((1379893 -98...
8              Illinois 12880580 MULTIPOLYGON (((868942.5 -2...
9               Indiana  6596855 MULTIPOLYGON (((1279733 -39...
10            Louisiana  4649676 MULTIPOLYGON (((1080885 -16...

Compared to older workflows (e.g., fortify() + geom_polygon()), this is a way more convenient and intuitive data structure (especially if your real unit of interest is the state). Moreover, sf tracks additional information about the coordinate system and bounding box which ensures your aspect ratios are always correct and also makes it easy transform and simplify those features (more on this later). It’s worth noting that sf is still very new, and improvements are constantly being made – to keep updated, check out the r-spatial blog.

Tools for improving upon `ggplotly()` conversions

Yes, ggplotly() converts the map ‘correctly’, but it does take a while to print, and for some reason, no tooltip appears when hovering over the map. To investigate why, lets examine the JSON that plotly sends along to plotly.js to render the map. This can be done via the plotly_json() function, which is useful for seeing how any R plotly object is serialized as JSON. This JSON represents what we call a figure, which is comprised of numerous components – the most important of which are: layout (for controlling axes, labels, titles, etc) and data (a collection of traces, each of which defines a mapping from data to visuals).⁴

# prints an interactive htmlwidget if you have listviewer
# install.packages('listviewer')
plotly_json(p)

Inspecting the data component, we see that the map contains two 'scatter' traces, both with a mode of 'lines'. The first trace is the graticule behind the states and the second trace contains the state outlines which contain over 50,000+ x/y (i.e., lat/lon) positions! This is certainly not the most efficient way to create such a map and there are several ways we could improve upon it without abandoning the comfort of geom_sf().⁵ Before we dive into those improvements, lets first consider what happens when plotly generates a plot.

Whether you’re printing the result of ggplotly(), plot_ly(), or generally any R htmlwidget, there are two main steps that occur: a build (i.e. compile) and render step. Roughly speaking, the build step translates R code to an R list. That list is then serialized as JSON (via jsonlite::toJSON()) and should match a JSON specification (i.e. schema) defined by the JavaScript library (which uses the JSON to render the widget).

If you’ve ever found ggplotly() slow to print, chances are, the bulk of the time is spent building the R list and sending the JSON to plotly.js. For many htmlwidgets, the build time is negligible, but for more complex widgets like plotly, a lot of things need to happen, especially for ggplotly() since we call ggplot2::ggplot_build(), then crawl and map that data structure to plotly.js. In a shiny app, both the build and render stages are required on initial load, but the new plotlyProxy() interface provides a way to ‘cache’ expensive build (and render!) operation and update a graph by modifying just specific components of the figure (via plotly.js functions). Outside of a ‘reactive context’ like shiny, you could use htmlwidgets::saveWidget() to ‘cache’ the results of the build step to disk, send the file to someone else (or host it online somewhere), then only the render step is required to view the graph.

A quick and easy way to try and improve render performance is to use canvas-based rendering (instead of vector-based SVG) with toWebGL(p). Switching from vector to canvas is generally a good idea when dealing with >30,000 vectors, but in this case, we’re only dealing with a couple hundred vector paths, so switching from vector to canvas for our map won’t significantly improve rendering performance, and in fact, we’ll lose some nice SVG exclusive features (the plotly.js team is getting close to eliminating these limitations!). Instead, what we could (and should!) do is reduce the amount of points along to each path (technically speaking, we’ll reduce the complexity of the SVG d attribute).

Thankfully, the sf package provides st_simplify() function that may be used to simplify polygons while still preserving their shape. A bit of trial and error is involved, but the idea is that by increasing the value of the dTolerance argument, you’ll decrease the number of points required to draw the spatial features. For fun, before simplifying, lets leverage some more sf awesomeness, st_transform(). This function allows us to transform our features to any projection via the proj4 standard. Here I’ll transform from laea to lcc, and in this case, a dTolerance of 10000 simplifies from roughly 50,000 to 2,500 points. On my machine, that cuts the build and render time in half and reduces the HTML/JSON file size from 1.7MB to 125.9KB.

library(sf)
us_lcc <- usa_sf("laea") %>%
  st_transform("+proj=lcc +lat_1=33 +lat_2=45 +lat_0=39 +lon_0=-96 +x_0=0 +y_0=0 +datum=NAD83 +units=m +no_defs") %>%
  st_simplify(TRUE, dTolerance = 10000)
  
plotly_json(ggplot(us_lcc) + geom_sf())

Now that we’ve simplified the borders, lets look into why there is no hover information. Since hoverinfo='text', plotly.js just shows the text attribute in the tooltip, but that attribute is blank! It’s blank because, by default, ggplotly() puts all aesthetic mapping information in the text field, but we haven’t actually used an aesthetic mapping here (geom_sf() is weird in that way – it is the only geom that doesn’t require any aesthetics)! We’ll eventually use aesthetic mappings with geom_sf() to create a choropleth map, but before we do, lets find out what the other trace attributes are doing (and find other relevant ones).

If you’re ever wondering what a particular attribute means (e.g., text, fill, fillcolor, hoveron, etc), you can look up the description online by going to https://plot.ly/r/reference/#[type]-[attr] for a specific trace [type] and attribute [attr] (e.g. https://plot.ly/r/reference/#scatter-fill). The online reference is nice, but I prefer to use the schema() function for a few reasons:

schema() provides a bit more information than the online docs (e.g., value types, default values, acceptable ranges, etc).
The interface makes it a bit easier to traverse and discover new attributes.
You can be absolutely sure it matches the version used in the R package (the online docs might use a different – probably older – version).

schema()

Go ahead and have a look under “traces” -> “scatter” -> “attributes”. These are all the attributes that may be used to control the appearance and interactive properties of a scatter trace. I won’t bother going through descriptions (you can see those for yourself by expanding an attribute), but I will demonstrate how we can leverage the style() function to modify just the state border’s attributes (specifying traces=2 ensures these changes are only applies to the second trace).

gg <- ggplotly(ggplot(us_lcc) + geom_sf())
style(
  gg, traces = 2, 
  mode = "markers+lines",
  hoverinfo = "x+y", 
  fillcolor = "transparent",
  hoverlabel = list(bgcolor = "white")
)

Leveraging plotly’s interactive features

After that deep dive into how ggplotly() works under the hood, and tips for improving rendering performance, let’s explore some lesser known, yet powerful ggplotly() features. As a side note, anything that can be done via ggplotly(), you can also do via plot_ly() (but not necessarily the other way round!) – it just might be more effort to do so. And, sometimes this extra effort is worth it, because ggplotly() isn’t really designed to be re-sized without being re-printed (get in touch if you want to help us solve this and other big issues). In fact, a lot of times, I prototype plots with ggplotly(), then translate it plot_ly() when it’s time to put it into production.

Customizing tooltips

At this point, we’ve learned how to turn a tooltip on/off (via hoverinfo), but perhaps more useful is the ability to completely control what appears in the tooltip. By using ggplotly()’s ability to translate a special text aesthetic with tooltip='text', we can effectively supply any text we want and even style it with HTML tags – as is done in this choropleth map encoding population in 2014.

p <- ggplot(us_lcc) + 
  geom_sf(aes(fill = pop_2014, text = paste("The state of <b>", name, "</b> had \n", pop_2010, "people in 2010")))
p %>%
  ggplotly(tooltip = "text") %>%
  style(hoverlabel = list(bgcolor = "white"), hoveron = "fill")

Animation

A choropleth map of the population in 2014 is not that interesting, so let’s grab some other data. Here I download physical activity and obesity ‘prevalence’ between 2001-2011 in every US county from the Global Health Data Exchange. then finding the mean rate for every combination of state and year:

url <- "http://ghdx.healthdata.org/sites/default/files/record-attached-files/IHME_USA_OBESITY_PHYSICAL_ACTIVITY_2001_2011.csv"

library(tidyr)
# reshape and create a column tracking the year
dat_clean <- readr::read_csv(url) %>%
  gather(year, value, matches("prevalence [0-9]+ \\(%\\)")) %>%
  mutate(year = stringr::str_extract(year, "[0-9]+")) 

# compute mean outcome for every combination of state/year/outcome
dat_state <- dat_clean %>%
  rename(name = State, outcome = Outcome) %>%
  group_by(name, year, outcome) %>%
  summarise(value = mean(value, na.rm = TRUE))

# attach our (simplified) state features to our summaries 
dat_sf <- dat_state %>%
  left_join(us_lcc) %>%
  st_as_sf() %>%
  mutate(txt = paste(name, year, "\n", outcome, scales::percent(value / 100)))
  
select(dat_sf, name, year, outcome, value, geometry)

Simple feature collection with 1683 features and 4 fields
geometry type:  GEOMETRY
dimension:      XY
bbox:           xmin: -2553435 ymin: -1777696 xmax: 2258071 ymax: 1407299
epsg (SRID):    NA
proj4string:    +proj=lcc +lat_1=33 +lat_2=45 +lat_0=39 +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
# A tibble: 1,683 x 5
# Groups:   name, year [561]
   name    year  outcome       value                       geometry
   <chr>   <chr> <chr>         <dbl>              <sf_geometry [m]>
 1 Alabama 2001  Any PA         66.0 MULTIPOLYGON (((761196.5 -9...
 2 Alabama 2001  Obesity        34.5 MULTIPOLYGON (((761196.5 -9...
 3 Alabama 2001  Sufficient PA  42.7 MULTIPOLYGON (((761196.5 -9...
 4 Alabama 2002  Any PA         66.8 MULTIPOLYGON (((761196.5 -9...
 5 Alabama 2002  Obesity        35.4 MULTIPOLYGON (((761196.5 -9...
 6 Alabama 2002  Sufficient PA NaN   MULTIPOLYGON (((761196.5 -9...
 7 Alabama 2003  Any PA         67.5 MULTIPOLYGON (((761196.5 -9...
 8 Alabama 2003  Obesity        36.2 MULTIPOLYGON (((761196.5 -9...
 9 Alabama 2003  Sufficient PA  44.6 MULTIPOLYGON (((761196.5 -9...
10 Alabama 2004  Any PA         67.7 MULTIPOLYGON (((761196.5 -9...
# ... with 1,673 more rows

There’s a number of ways we could try to visualize how physical activity and/or obesity has evolved over the years. One way would be to create a small multiple display (one for every year via facet_wrap(~ year)), but it’s a bit less visual work to perceive differences via animation. Creating animations with ggplotly() is very similar to gganimate – we just add an aesthetic mapping of frame = year.

obesity <- filter(dat_sf, outcome == "Obesity")
p <- ggplot(obesity) + 
    geom_sf(aes(fill = value, frame = year, text = txt)) + 
    ggtitle("Obesity rates in the US over time") +
    ggthemes::theme_map()
plotly_json(p)

Having a look at the underlying JSON reveals a special frame component which adheres to the underlying plotly.js animation api. It turns out that the data supplied to each frame of the animation has a bunch of redundant information. Specifically, the x/y coordinates for the state boundaries are repeated on every frame – which we don’t really need to do in this case, so let’s remove that data before printing:

# ggplotly() won't (for good reason) "register" an animation
# until print time, so to acess/modify the frames component, 
# you'll need plotly_build() -- use this function when you 
# want to be *absolutely* sure you're accessing the full list/JSON
gg <- p %>%
  ggplotly(tooltip = "text") %>%
  style(hoveron = "fill") %>%
  plotly_build()

# remove x/y data from every trace
gg$x$frames <- lapply(
  gg$x$frames, function(f) { 
    f$data <- lapply(f$data, function(d) d[!names(d) %in% c("x", "y")])
    f 
  })
gg

Highlighting

In addition to animation, plotly has powerful framework for filtering, highlighting, and linking views without shiny. This framework leverages the SharedData class from crosstalk package where one essentially defines a “unit of interaction” using a column from the data to be visualized. In our case, we can set name (i.e, state) as the interaction unit, then when we trigger a selection (e.g., brush a time series of physical actively), the relevant graphical marks are highlighted in every view:

library(crosstalk)
dat_shared <- dat_sf %>%
  filter(!is.na(value)) %>%
  SharedData$new(~name, "A")

p <- ggplot(dat_shared) + 
  geom_line(aes(x = year, y = value, group = name)) + 
  facet_wrap(~outcome, scales = "free") +
  theme_bw()

lines <- p %>%
  ggplotly(tooltip = "name") %>%
  style(mode = "markers+lines") %>%
  layout(dragmode = "select") %>%
  highlight("plotly_selected")

# compute mean obesity for each state, define "state" as interaction unit (in group "A")
ob_mean <- obesity %>%
  group_by(name) %>%
  summarise(mean_obesity = mean(value)) %>%
  mutate(txt = paste(name, "mean \n obesity", scales::percent(mean_obesity / 100))) %>%
  SharedData$new(~name, "A")

map <- ggplot(ob_mean) + 
  geom_sf(aes(fill = mean_obesity, text = txt)) + 
  ggthemes::theme_map()

library(htmltools)
browsable(tagList(ggplotly(map, tooltip = "text"), lines))

For a gentle overview of the linking framework, checkout my webinar from when it was initially released. If you’re interested in understanding the full power of the linking framework, my 2 day plotly for R workshop is the best way to learn it effectively. I also offer this workshop as an on-site training course, so please get in touch if you have any interest!

Discussion

Compared to other interactive mapping approaches, using geom_sf() + ggplotly() has it’s pros and its cons. It terms of it’s ability to link multiple views, it seems to be the most advanced, but there are a number of other features you might want (e.g., dynamic zooming of different baselayers). I think a good number of these can be resolved by using sf with plot_mapbox() and better support for that should be coming soon, but it would also be nice to see leaflet officially support more of the highlighting features that plotly does.

As it turns out, plotly’s ‘highlighting’ framework can actually be used to do much more than just highlighting. It works by adding additional traces that reflect the selected data and the attributes of these trace(s) can be customized. When used in clever ways, that capability can be used to, for example, outline simple features on hover, or even click on a simple feature to label it:

Conclusion

I hope this post convinces folks that even when a ggplotly() conversion doesn’t quite work the way you want it to, we can still leverage the underlying plotly.js library to do powerful things quickly. My goal for the R package plotly has always been to make interactive web graphics practically useful for exploratory data analysis in R. If you’re interested in supporting a particular facet of this mission, please get in touch (it’s much easier for plotly to monetize plotly.js and dash, which is why R plotly development has dwindled recently).

I still believe supporting ~99% of the ggplot2 API is achievable (we’re currently at about 80%), but we don’t have the funding/bandwidth for it currently. If you have any interest at all in supporting this effort, please get in touch.↩
I’ve already written a bit about how this idea in the plotly book, but I thought I’d make a more detailed guide for this sort of thing.↩
Jenny Bryan has a superb talk on leveraging and embracing list-columns.↩
plotly_json() also gives one insight into R-specific components, such as highlight (for controlling highlighting options in multiple linked views) and attrs (for lazy evaluation of data -> visual mappings in plot_ly()).↩
plotly offers other “native” mapping options such as plot_geo() and plot_mapbox(), which generally speaking are more performant, but they only offer a limited set of projections and/or resolutions (with geom_sf() you won’t run into the same limitations).↩

Components of a successful R workshop

Mon, 04 Dec 2017 00:00:00 +0000

Need help with R, plotly, data viz, and/or stats? Work with me!

I had an absolute blast running a 2-day R workshop in NYC last week at plotcon. It was perhaps even more rewarding and fulfilling than my previous plotcon workshop – I saw more attendees grasping concepts, accomplishing exercises, asking great questions, and most importantly, modifying code examples to analyze their own data. There are a few things I think I improved upon this year which helped contribute to the success, so I thought I would share some ideas that can generalize to other R (or, even more generally, data scienc-y) workshops.

Inspire first 😲, mindful explanation later 💁

To spark genuine interest and enthusiasm, it’s important to first demonstrate the power of the tools covered in the workshop. For my ‘plotly for R’ workshop, which is more specifically about ‘creating & using interactive graphics for exploratory data analysis (EDA)’, I first focus on the power of using interactive graphics to perform analysis tasks. In an EDA context though, that power depends on the amount of friction involved with creating graphics, since during EDA we don’t immediately know which graphics are informative, and likely have to iterate through many uninteresting graphics to find interesting ones.

With this delicate balance in mind, I like taking a ‘solution first’ approach that focuses first and foremost on the visuals and analysis (i.e., using interactive graphics), and then move on to explain the programming details (i.e., creating interactive graphics). I also try to be mindful of what knowledge is required to understand the implementation when ordering the workshop content, so I can introduce programming concepts as I need them, but also have a clear motivation as to why they’re needed. I also think basing the implementation on tidy-data principles is almost always a good idea since you’ll likely be able to reuse those principles and concepts in several different contexts which should help convince your audience that they can do powerful things quickly.

Encourage curiosity about data 🤔 📊

A big reason why I like this “solution first” approach is that attendees are less likely to get overwhelmed with programming details and more likely to ask interesting data analysis related questions. It also encourages a more natural train of thought for analysts; where questions derive from curiosity about the data, not the software tool. As a result, attendees tend to ask more creative questions that are not necessarily bound to technology restrictions, and thus, I often get questions that I’ve never considered about data that I know quite well! These questions often take the form: “Given that we’ve seen X, I wonder if Y is true? Can plotly do Z [to answer question Y]?”. This leads to a much more fruitful discussion (for everyone!) than if I were just to explain that plotly can do Z.

Furthermore, by keeping the focus on asking questions about data, attendees are more likely to apply what they’ve learned to their own data. In fact, one attendee said the following:

“Carson was very good in answering questions about specific use cases related to our own projects..so that always helps when you are able to see that techniques being presented can be made to work in context of your own datasets.”

Understand your audience 🕵️‍♀️

On a more general level, it seems the reason why this “solution first” approach works so well is because learning a new language is hard, but it can be made easier by building upon existing language or knowledge. So, of course, before preparing workshop materials, I always take some time to understand the audience. I do this by asking attendees to fill out a survey about their background several weeks before the workshop, so I can focus on concepts that I think will be most widely useful.

Encourage collaboration 👩‍💻 👨‍💻

In addition to asking attendees to fill out a survey several weeks beforehand, I also invite them to a Slack group, mainly for the ease of communication and knowledge sharing during the workshop, but also to encourage interaction before, during, and after the workshop. In fact, my very first your turn exercise¹ requires everyone to send me and their neighbor (via Slack) three things they hope to take away from the workshop. This way I can make sure everyone is on Slack which seems to increase the amount of collaboration among everyone.

Bring your work environment to them 💼 💻

Before the workshop, I also ask attendees to download my cpsievert/workshops Docker image on the machine that they plan on bringing. This way, in theory, we don’t waste any time on setup, and attendees have immediate access to a computational environment with all the materials and software dependencies for following demos and performing exercises. Moreover, since this image builds off of rocker/rstudio, they can also easily run RStudio via their web browser which provides a nice and consistent user experience. In practice, I’ve found about 1 in 4 attendees actually try to get Docker setup before the workshop, but with a little time and help, almost everyone can get it working.

That being said, there are bound to be issues that are difficult to immediately resolve (e.g., not enough disk space on the host machine), so in that case, I provide a USB with the workshop materials, and 🙏 they can install the required software (in theory, if you include a DESCRIPTION file with all your R dependencies for the workshop, they can devtools::install() R packages, but you’ll still likely run into system dependency issues).

In the future, I may experiment with hosting workshop containers on the cloud, so that attendees can just visit a web link to follow along, but I’m not sure if the convenience is necessarily worth the cost.

Looking forward 🔮

I really enjoy running R workshops and hope to continue improving and developing more of them. This is especially true of the ‘plotly for R’ workshop – it’s incredibly fulfilling to have others find your work useful for accomplishing their own work. Longer term though I’d like to spend more time developing other workshops that dive deeper into different aspects of the tidyverse, shiny, statistical modeling, and more. In fact, I’ll be attending RStudio’s “train the trainer” workshop (and giving a talk!) during rstudio::conf this year which I hope will give me more and better ideas for running successful R workshops. Hope to see you there!

Your turn exercises are largely designed to reinforce important concepts and break up the lecture every 10-20 minutes.↩

plotly 4.7.1 now on CRAN

Thu, 13 Jul 2017 00:00:00 +0000

I'm excited to announce that plotly 4.7.1 is now on CRAN! Along with some important bug fixes and numerous improvements to the underlying plotly.js library, this release includes an exciting new R-specific feature -- the ability to modify (i.e., update without a full redraw) an existing plotly graph inside a shiny app via the new plotlyProxy() function. In other words, this proxy interface allows one to perform more efficient and responsive updates to a plotly graph within a shiny app. Before I provide an overview of the proxy interface, lets jump right into some of new plotly.js goodies.

New plotly.js goodies

Upgrading the R package from 4.7.0 to 4.7.1 upgrades the corresponding plotly.js version from 1.27.1 to 1.29.2. A number of exciting improvements have been added, just to name a few: select/lasso events for scattergl/scattermapbox trace types, 3D annotations, contour labelling, and even touch events on mobile. For quite a while now, R users have been able to link multiple views (without shiny) using nearly any (SVG) trace type (e.g., scatter, bar, heatmap, etc), but the new brush events on scattergl/scattermapbox trace types allow us to brush way more points and trigger selections via a map. Here is an example of highlighting earthquakes west of Fiji to compare the relative frequency of their magnitude and number of reporting stations (to the overall relative frequency):

And the R code to generate the (self-contained!) HTML:

library(crosstalk)
eqs <- SharedData$new(quakes)

# you need a mapbox API key to use plot_mapbox()
# https://www.mapbox.com/signup/?route-to=https://www.mapbox.com/studio/account/tokens/
map <- plot_mapbox(eqs, x = ~long, y = ~lat) %>%
  add_markers(color = ~depth) %>%
  layout(
    mapbox = list(
      zoom = 2,
      center = list(lon = ~mean(long), lat = ~mean(lat))
    )
  ) %>%
  highlight("plotly_selected")

# shared properties of the two histograms
hist_base <- plot_ly(eqs, color = I("black"), histnorm = "probability density") %>%
  layout(barmode = "overlay", showlegend = FALSE) %>%
  highlight(selected = attrs_selected(opacity = 0.5))

histograms <- subplot(
  add_histogram(hist_base, x = ~mag),
  add_histogram(hist_base, x = ~stations),
  nrows = 2, titleX = TRUE
)

bscols(histograms, map)

Another super cool and easy-to-use plotly.js feature is contour line labelling:

plot_ly(z = volcano, type = "contour", contours = list(showlabels = TRUE)) %>%
  colorbar(title = "Elevation \n in meters")

To learn more about all the new plotly.js improvements, read the plotly.js release notes.

Modifying plotly graphs via `plotlyProxy()`

The design of plotly's new proxy interface is inspired by similar interfaces in leaflet and DT (thanks Joe Cheng and Yihui Xie!). That is, plotlyProxy() initiates a proxy object just like leafletProxy()/dataTableProxy() by referencing a shiny output ID. However, at least for now, you must use the plotlyProxyInvoke() function to modify a plotlyProxy() object, which requires knowledge/use of a plotly.js method for the updating logic (among them, Plotly.restyle, Plotly.relayout, Plotly.addTraces, and Plotly.deleteTraces are the most widely useful). This simple shiny app uses Plotly.restyle to change the fillcolor of Canada (i.e., polygons) in response to the dropdown.

Notice how the map and the outline of Canada are not effected (i.e., are not redrawn) when the fill color changes. The code for the app is below. Notice how, instead of having renderPlotly() regenerate the map in response to a change in input$color (i.e., the "old" or "naive" way of updating a plotly graph which always uses Plotly.newPlot), it uses Plotly.restyle via plotlyProxyInvoke() to perform a more efficient update.

# you can also run this example via the new plotly_example() function!
# plotly_example("shiny", "proxy_restyle_canada")

library(shiny)
library(plotly)

ui <- fluidPage(
  selectInput("color", "Canada's fillcolor", colors(), selected = "black"),
  plotlyOutput("map")
)

server <- function(input, output, session) {
  
  output$map <- renderPlotly({
    map_data("world", "canada") %>%
      group_by(group) %>%
      plot_mapbox(x = ~long, y = ~lat, color = I("black")) %>%
      add_polygons() %>%
      layout(
        mapbox = list(center = list(lat = ~median(lat), lon = ~median(long)))
      )
  })
  
  observeEvent(input$color, {
    plotlyProxy("map", session) %>%
      plotlyProxyInvoke("restyle", list(fillcolor = toRGB(input$color)))
  })
  
}

shinyApp(ui, server)

It's worth noting that using plotlyProxyInvoke() requires knowledge of the figure reference and the plotly.js API. Thus, it can help to have some knowledge of how your plots are actually represented in JSON, which you can always obtain via plotly_json(), for example:

p <- plot_ly(economics, x = ~pce, y = ~psavert, z = ~unemploy, color = ~as.numeric(date), mode = "markers+lines")
plotly_json(p)

This gives us a glimpse into what the R package actually sends along to the newPlot method to generate the initial view on page load. In this case, the R package generates scatter3d marker and line objects. We could use a similar plotlyProxyInvoke() pattern as before to alter the value of certain attribute(s) of these objects (e.g., marker/line size/width):

# you can also run this example via the new plotly_example() function!
# plotly_example("shiny", "proxy_restyle_economics")

library(shiny)
library(plotly)

ui <- fluidPage(
  sliderInput("marker", "Marker size", min = 0, max = 20, value = 8),
  sliderInput("path", "Path size", min = 0, max = 30, value = 2),
  plotlyOutput("p")
)

server <- function(input, output, session) {
  
  output$p <- renderPlotly({
    plot_ly(
      economics, x = ~pce, y = ~psavert, z = ~unemploy, 
      color = ~as.numeric(date), mode = "markers+lines"
    )
  })
  
  observeEvent(input$marker, {
    plotlyProxy("p", session) %>%
      plotlyProxyInvoke(
        "restyle", 
        # could also do list(marker = list(size = input$marker))
        # but that overwrites the existing marker definition
        # https://github.com/plotly/plotly.js/issues/1866#issuecomment-314115744
        list(marker.size = input$marker)
      )
  })
  
  observeEvent(input$path, {
    plotlyProxy("p", session) %>%
      plotlyProxyInvoke(
        "restyle", list(line.width = input$path)
      )
  })
  
}

shinyApp(ui, server)

At this point, you might be wondering something along these lines of: "how on earth am I supposed to know/remember what attributes a scatter3d line object has?" One option is to search through the figure reference, but I prefer to use the schema() function since it allows me to more easily traverse the reference and get more information about acceptable/default attribute values. For example, when creating this app, I forgot what line attribute controlled its size/width, so I did the following:

I hope you find this new release useful and this post informative for creating more performant shiny apps! If you're interested in seeing more examples of plotlyProxy() in action, see here.

Developing tools for exploratory visualization

Tue, 13 Jun 2017 00:00:00 +0000

I find it incredibly fulfilling to work on projects that enhance the productivity of other data scientists. After all, most data scientists need to maximize time spent formulating, examining, and refining questions about data; and minimize time translating those thoughts into machine code. I’ve talked at great lengths about how my work on plotly fits into this context, which roughly speaking, allows one to leverage many benefits of interactive web graphics without any knowledge of web technologies.

If you keep up with online media, the benefits of interactive web graphics are pretty clear: they can help grab the audience’s attention, enchance knowledge transfer, and allow the audience to further investigate detailed information. And yes, assuming you’re a web developer, we already have awesome tools for creating web-based data visualizations; but even if you’re a polyglot data science unicorn, you still shouldn’t be writing JavaScript/JSON to do exploratory analysis (where the most useful view/question of the data is not yet known). That’s because data exploration requires non-linear iteration between data transformation/modeling/visualization (which a web browser was not designed to do). That being said, interactive graphics are certainly useful for exploration,¹ which is partially why we’ve seen a recent explosion in R/Python/Julia interfaces to JavaScript graphing libraries.

I feel very fortunate to have maintained the (completely free and open source) R interface to plotly.js for over 2 years now. I also feel very fortunate to be in a position where the plotly.js maintainers respond very well to my bug reports and feature requests.² Now that I’ve graduated and had some time to reflect on the past few years, I’ve been thinking a bit more generally about data science software development, and how I can keep sustaining this type of work through my consulting services.

Generally speaking, designing data science software requires making hard decisions about which abstractions to make, and perhaps more importantly, which abstractions not to make (especially with respect to visualization). Doing this well requires an intimate knowledge of the most common/frustating/difficult tasks within the given domain, which means data science software developers should:

Be familiar with the vast ecosystem of existing tools or else we risk re-inventing the wheel.³
Get our hands dirty analyzing data, and applying our tools to real data-driven problems, or else we risk working on insignificant problems.

This is why I spend a good chunk of my time researching the constantly expanding ecosystem of R packages and using some of those tools to do real data analysis. Doing so not only forces you to eat your own dog food, but helps you focus on the bigger picture, rather than drowning in a sea of issues that don’t effect most users. For example, most of the new linking framework in plotly was motivated and (indirectly) supported by work on eechidna/pedestrians/bcviz. I hope to continue doing this sort of work with the help of more clients that have interesting data-driven problems and need better ways to explore/present their data visually.

Another thing I’ve learned through this work is that even the best programming interfaces do about 80% of what you’d like them to do. That number is even smaller for general-purpose visualization software (e.g., plotly) since their scope is so impossibly large. Fortunately, working with open-source tools means that we are (usually) free to modify the software to fit a particular use case. That sounds great in theory, but in practice, it can be a time-consuming and ultimately futile effort without an intimate understanding of the software. Since I have an intimate understanding of plotly and many other R packages, I also offer my clients the ability to adapt/modify/fix existing open-source tools for a particular project.

Assuming that the time spent iterating from one visualization to the next is relatively small.↩
Having tracked development on similar R packages, like ggvis, I’m convinced that a strong relationship amongst development teams (i.e., R and JavaScript devs) is necessary to maintain such a project.↩
That isn’t to say that “re-inventing the wheel” can’t be a good thing, especially when it leads to a better wheel. In my opinion, especially in academia, there is too much of a focus on whether tool(s) exist, and not nearly enough attention is paid to whether they are usable.↩

plotly 4.7.0 now on CRAN

Tue, 30 May 2017 00:00:00 +0000

I’m super excited about plotly 4.7.0 – it includes numerous improvements and new features related to performance, mapping, and API requests. It also includes support for something folks have wanted for a very long time – fixed coordinates via ggplotly()! In other words, if you use coord_equal(), coord_fixed(), etc to specify a ratio between the visual represntation of data units on positional axes (a key property of maps) in ggplot2, it should just work^TM when converted via ggplotly()! It also has support for the stellar new geom_sf() – an official ggplot2 geom that is so new, it is currently only available through the development version of ggplot2. For this and other reasons, I recommend using the development version of ggplot2 if you can:

devtools::install_github("tidyverse/ggplot2")
install.packages("plotly")

At plotcon Oakland, I held a workshop where I touched on why the new sf package and geom_sf() are so awesome for mapping in R, and how to make these maps interactive via ggplotly(). There are lots of other goodies in those workshop slides, such as embedding raster images, converting ggmap plots, an overview of animation, highlighting, and filtering. Also, some of the “your turn” exercises show you how to combine these concepts to do neat stuff like outline geo-spatial polygons on hover or annotate geo-spatial polygons on click:

Updating to 4.7.0 may also yield big speed improvements at print time, especially when plotting a large number of lines, paths, or polygons in a single trace. For a bit of context, this fairly simple example with 46 lines now takes a third of the time to print (about 1 instead of 3 seconds).

plot_ly(txhousing) %>%
  group_by(city) %>%
  add_lines(x = ~date, y = ~median)

The speed-up is a direct result of making (the previously internal, but now exported) function group2NA() much more efficient (huge thanks to Matt Summersgill for the help here). This function is used on any ggplot2 geom which can be reduced to a path or a polygon, so the impact should be large and widespread! This improvement was made possible through the new data.table dependency, which I hope to leverage more down the road to make other plots faster to print. This release also updates the plotly.js version from 1.26.1 to 1.27.1, so you can leverage the latest plotly.js features, such as parallel coordinates and Sankey diagrams. Numerous other improvements and bug fixes (especially for api_create()) are coming with this release, please see the NEWS file for more info. Happy plotting!

plotly 4.6.0 now on CRAN

Tue, 25 Apr 2017 00:00:00 +0000

I'm super excited to announce that plotly 4.6.0 is now on CRAN! Go ahead and install it with:


install.packages("plotly")

This release brings a ton of new features and improvements -- some of which are over in the year making. I recently held a webinar introducing the biggest features, including linked views, animation, and crosstalk integration. In addition to the webinar recording, some good resources for getting up-to-speed include new demos (`demo(package = "plotly")`), help pages (`help(highlight)` and `help(animation)`), and the official NEWS file. I also recommend checking out my vimeo page for some more advanced applications of the linked views framework. My upcoming plotcon workshop will be a great opportunity to learn all these new features over 2 full days of lectures, demos, hands-on exercises, and one-on-one Q&A. PS. I can't wait to see what you do with these new features! And I always love hearing from the community; so tweet me if you run into issues/questions, want to share your work, and/or just want to say hi! :)

News and Updates Surrounding plotly for R

Wed, 12 Apr 2017 00:00:00 +0000

Today I held a free webinar to announce/present some new exciting features coming with plotly 4.6.0, including animations and multiple linked views. Here is a video of it:

Also, here is a link to the slides used during the presentation:

Slides from my webinar today --https://t.co/ZMJpaQfpfZ
Click on the image here to watch the video (skip to 3min) -- https://t.co/brdSPsntGw
— Carson Sievert (@cpsievert) April 12, 2017

Not Found

Tue, 31 Jan 2017 00:00:00 +0000

Page not found. Please notify me if a page should really exist here or go back to the home page.

Upgrading to plotly 4.0 (and above)

Mon, 26 Sep 2016 00:00:00 +0000

I’m excited to announce that plotly’s R package just sent its first CRAN update in nearly four months. To install the update, run install.packages("plotly").

This update has breaking changes, enables new features, fixes numerous bugs, and takes us from version 3.6.0 to 4.5.2. To see all the changes, I encourage you to read the NEWS file. In this post, I’ll highlight the most important changes, explain why they needed to happen, and provide some tips for fixing errors brought about by this update. As you’ll see, this update is mostly about improving the plot_ly() interface, so ggplotly() users won’t notice much (if any) change. I’ve also started a plotly for R book which provides more narrative than the documentation on https://plot.ly/r (which is now updated to 4.0), more recent examples, and features exclusive to the R package. The first three chapters are nearly finished and replace the package vignettes. The later chapters are still in their beginning stages – they discuss features that are still under development, but I plan adding stability, and more documentation in the coming months.

Formula mappings

In the past, you could use an expression to reference variable(s) in a data frame, but this no longer works. Consequently, you might see an error like this when you update:

library(plotly)
plot_ly(mtcars, x = mpg, y = sqrt(wt))

## Error in plot_ly(mtcars, x = mpg, y = sqrt(wt)): object 'wt' not found

plot_ly() now requires a formula (which is basically an expression, but with a ~ prefixed) when referencing variables. You do not have to use a formula to reference objects that exist in the namespace, but I recommend it, since it helps populate sensible axis/guide title defaults (e.g., compare the output of plot_ly(z = volcano) with plot_ly(z = ~volcano) ).

plot_ly(mtcars, x = ~mpg, y = ~sqrt(wt))

There are a number of technical reasons why imposing this change from expressions to formulas is a good idea. If you’re interested in the details, I recommend reading Hadley Wickham’s notes on non-standard evaluation, but here’s the gist of the situation:

Since formulas capture the environment in which they are created, we can be confident that evaluation rules are always correct, no matter the context.
Compared to expressions/symbols, formulas are easier to program with, which makes writing custom functions around plot_ly() easier.

myPlot <- function(x, y, ...) {
  plot_ly(mtcars, x = x, y = y, color = ~factor(cyl), ...)
}
myPlot(~mpg, ~disp, colors = "Dark2")

Also, it’s fairly easy to convert a string to a formula (e.g., as.formula("~sqrt(wt)")). This trick can be quite useful when programming in shiny (and a variable mapping depends on an input value).

Smarter defaults

Instead of always defaulting to a “scatter” trace, plot_ly() now infers a sensible trace type (and other attribute defaults) based on the information provided. These defaults are determined by inspecting the vector type (e.g., numeric/character/factor/etc) of positional attributes (e.g., x/y). For example, if we supply a discrete variable to x (or y), we get a vertical (or horizontal) bar chart:

subplot(
  plot_ly(diamonds, y = ~cut, color = ~clarity),
  plot_ly(diamonds, x = ~cut, color = ~clarity),
  margin = 0.07
) %>% hide_legend()

Or, if we supply two discrete variables to both x and y:

plot_ly(diamonds, x = ~cut, y = ~clarity)

Also, the order of categories on a discrete axis, by default, is now either alphabetical (for character strings) or matches the ordering of factor levels. This makes it easier to sort categories according to something meaningful, rather than the order in which the categories appear (the old default). If you prefer the old default, use layout(categoryorder = "trace")

library(dplyr)
# order the clarity levels by their median price
d <- diamonds %>%
  group_by(clarity) %>%
  summarise(m = median(price)) %>%
  arrange(m)
diamonds$clarity <- factor(diamonds$clarity, levels = d[["clarity"]])
plot_ly(diamonds, x = ~price, y = ~clarity, type = "box")

`plot_ly()` now initializes a plot

Previously plot_ly() always produced at least one trace, even when using add_trace() to add on more traces (if you’re familiar with ggplot2 lingo, a trace is similar to a layer). From now on, you’ll have to specify the type in plot_ly() if you want it to always produce a trace:

subplot(
  plot_ly(economics, x = ~date, y = ~psavert, type = "scatter") %>% 
    add_trace(y = ~uempmed) %>%
    layout(yaxis = list(title = "Two Traces")),
  plot_ly(economics, x = ~date, y = ~psavert) %>% 
    add_trace(y = ~uempmed) %>% 
    layout(yaxis = list(title = "One Trace")),
  titleY = TRUE, shareX = TRUE, nrows = 2
) %>% hide_legend()

Why enforce this change? Often times, when composing a plot with multiple traces, you have attributes that are shared across traces (i.e., global) and attributes that are not. By allowing plot_ly() to simply initialize the plot and define global attributes, it makes for a much more natural to describe such a plot. Consider the next example, where we declare x/y (longitude/latitude) attributes and alpha transparency globally, but alter trace specific attributes in add_trace()-like functions. This example also takes advantage of a few other new features:

The group_by() function which defines “groups” within a trace (described in more detail in the next section).
New add_*() functions which behave like add_trace(), but are higher-level since they assume a trace type, might set some attribute values (e.g., add_marker() set the scatter trace mode to marker), and might trigger other data processing (e.g., add_lines() is essentially the same as add_paths(), but guarantees values are sorted along the x-axis).
Scaling is avoided for “AsIs” values (i.e., values wrapped with I()) which makes it easier directly specify a constant value for a visual attribute(s) (as opposed to mapping data values to visuals).
More support for R’s graphical parameters such as pch for symbols and lty for linetypes.

map_data("world", "canada") %>%
  group_by(group) %>%
  plot_ly(x = ~long, y = ~lat, alpha = 0.1) %>%
  add_polygons(color = I("black"), hoverinfo = "none") %>%
  add_markers(color = I("red"), symbol = I(17),
              text = ~paste(name, "
", pop),
              hoverinfo = "text", data = maps::canada.cities) %>%
  hide_legend()

New interpretation of group

The group argument in plot_ly() has been removed in favor of the group_by() function. In the past, the group argument incorrectly created multiple traces. If you want that same behavior, use the new split argument, but groups are now used to define “gaps” within a trace. This is more consistent with how ggplot2’s group aesthetic is translated in ggplotly(), and is much more efficient than plotting a trace for each group.

txhousing %>%
  group_by(city) %>%
  plot_ly(x = ~date, y = ~median) %>%
  add_lines(alpha = 0.3)

The default hovermode (compare data on hover) isn’t super useful here since we have only 1 trace to compare, so you may want to add layout(hovermode = "closest") when using group_by(). If you’re group sizes aren’t that large, you may want to use split to generate one trace per group, then set a constant color (using the I() function to avoid scaling).

txhousing %>%
  plot_ly(x = ~date, y = ~median) %>%
  add_lines(split = ~city, color = I("steelblue"), alpha = 0.3)

In the coming months, we will have better ways to identify/highlight groups to help combat overplotting (see here for example). This same interface can be used to coordinate multiple linked plots, which is a powerful tool for exploring multivariate data and presenting multivariate results (see here and here for examples).

New plotly object representation

Prior to version 4.0, plotly functions returned a data frame with special attributes attached (needed to track the plot’s attributes). At the time, I thought this was the right way to enable a “data-plot-pipeline” where a plot is described as a sequence of visual mappings and data manipulations. For a number of reasons, I’ve been convinced otherwise, and decided the central plotly object should inherit from an htmlwidget object instead. This change does not destroy our ability to implement a “data-plot-pipeline”, but it does, in a sense, constrain the set manipulations we can perform on a plotly object. Below is a simple example of transforming the data underlying a plotly object using dplyr’s mutate() and filter() verbs (the plotly book has a whole section on the data-plot-pipeline, if you’d like to learn more).

library(dplyr)
economics %>%
  plot_ly(x = ~date, y = ~unemploy / pop, showlegend = F) %>%
  add_lines(linetype = I(22)) %>%
  mutate(rate = unemploy / pop) %>% 
  slice(which.max(rate)) %>%
  add_markers(symbol = I(10), size = I(50)) %>%
  add_annotations("peak")

In this context, I’ve often found it helpful to inspect the (most recent) data associated with a particular plot, which you can do via plotly_data()

diamonds %>%
  group_by(cut) %>%
  plot_ly(x = ~price) %>%
  plotly_data()

## Source: local data frame [53,940 x 10]
## Groups: cut [5]
## 
##    carat       cut color clarity depth table price     x     y     z
##                   
## 1   0.23     Ideal     E     SI2  61.5    55   326  3.95  3.98  2.43
## 2   0.21   Premium     E     SI1  59.8    61   326  3.89  3.84  2.31
## 3   0.23      Good     E     VS1  56.9    65   327  4.05  4.07  2.31
## 4   0.29   Premium     I     VS2  62.4    58   334  4.20  4.23  2.63
## 5   0.31      Good     J     SI2  63.3    58   335  4.34  4.35  2.75
## 6   0.24 Very Good     J    VVS2  62.8    57   336  3.94  3.96  2.48
## 7   0.24 Very Good     I    VVS1  62.3    57   336  3.95  3.98  2.47
## 8   0.26 Very Good     H     SI1  61.9    55   337  4.07  4.11  2.53
## 9   0.22      Fair     E     VS2  65.1    61   337  3.87  3.78  2.49
## 10  0.23 Very Good     H     VS1  59.4    61   338  4.00  4.05  2.39
## # ... with 53,930 more rows

To keep up to date with currently supported data manipulation verbs, please consult the help(reexports) page, and for more examples, check out the examples section under help(plotly_data).

This change in the representation of a plotly object also has important implications for folks using plotly_build() to “manually” access or modify a plot’s underlying spec. Previously, this function returned the JSON spec as an R list, but it now returns more “meta” information about the htmlwidget, so in order to access that same list, you have to grab the “x” element. The new as_widget() function (different from the now deprecated as.widget() function) is designed to turn a plotly spec into an htmlwidget object.

# the style() function provides a more elegant way to do this sort of thing,
# but I know some people like to work with the list object directly...
pl <- plotly_build(qplot(1:10))[["x"]]
pl$data[[1]]$hoverinfo <- "none"
as_widget(pl)

Conclusion

The latest CRAN release upgrades plotly’s R package from version 3.6.0 to 4.5.2. This upgrade includes a number of breaking changes, as well as a ton of new features and bug fixes. The time spent upgrading your code will be worth it as enables a ton of new features. It also provides a better foundation for advancing the plot_ly() interface (not to mention the linked highlighting stuff we have on tap). This post should provide the information necessary to fix these breaking changes, but if you have any trouble upgrading, please let us know on http://community.plot.ly. Happy plotting!

3D surface plots of the strikezone

Mon, 08 Jun 2015 00:00:00 +0000

Over the past month, I’ve been working on plotly’s R package; and in particular, a new interface for creating plotly visualizations from R. I’m really excited about this project and I think it’s one of the most elegant, straight-forward ways to create interactive graphics that are easy to share. In this post, I’ll show you just how easy it is to create 3D surface plots of the strikezone using plotly.

Kernel Densities

The MASS package in R has a function called kde2d() which performs 2D density estimation (with a bivariate normal kernel)

data(pitches, package = "pitchRx")
dens <- with(pitches, MASS::kde2d(px, pz))

# plotly isn't available on CRAN, but u can install from GitHub
# devtools::install_github("ropensci/plotly@carson-dsl")
library(plotly)
with(dens, plot_ly(x = x, y = y, z = z, type = "surface"))

Although this plot is cool, we can’t perform any interesting statistical inference with it. All we can see is an estimated frequency.

Probabilistic Surfaces

Brian Mills and I have a number of posts/papers on using generalized additive models (GAMs) to model event probabilities over the strikezone. To keep things simple, we’ll stick with the example data, and model the probablity of a called strike by allowing it to vary by location and batter stance.

# condition on umpire decisions
noswing <- subset(pitches, des %in% c("Ball", "Called Strike"))
noswing$strike <- as.numeric(noswing$des %in% "Called Strike")
library(mgcv)
m <- bam(strike ~ s(px, pz, by = factor(stand)) + factor(stand), 
         data = noswing, family = binomial(link = 'logit'))

## Warning in attr(pterms[tind[j]], "term.label"): partial match of
## 'term.label' to 'term.labels'

Now we use the predict.gam() method to fit response values (for right handers) over a strike-zone grid.

px <- round(seq(-2, 2, length.out = 20), 2)
pz <- round(seq(1, 4, length.out = 20), 2)
dat <- expand.grid(px = px, pz = pz, stand = "R")
dat$fit <- as.numeric(predict(m, dat, type = "response"))

plotly’s z argument likes numeric matrices, so we need change the data structure accordingly.

z <- Reduce(rbind, split(dat$fit, dat$px))
plot_ly(x = px, y = pz, z = z, type = "surface")

It’s more interesting to look at the difference in fitted values for right/left handed batters:

dat2 <- expand.grid(px = px, pz = pz, stand = "L")
dat2$fit <- as.numeric(predict(m, dat2, type = "response"))
z <- Reduce(rbind, split(dat2$fit - dat$fit, dat2$px))
plot_ly(x = px, y = pz, z = z, type = "surface")

Touring PITCHf/x Data

Wed, 15 Apr 2015 00:00:00 +0000

If you haven’t read them already, go check out Brian’s great posts on pitch classification and cluster analysis. I particularly like his argument that our goal should be to “Identify all the pitches that can reasonably be classified as ‘different’ from one another”. Grouping observations into similar sets is a very common task in unsupervised settings where a response variable of interest is either unknown or doesn’t exist. There are a mind-boggling number of different methods for “learning” the “best” grouping and Brian has gracefully covered two of them (k-means and model based clustering). Here I’ll present yet another approach that relies on human perception and our ability to recognize patterns.

In Brian’s post on model based clustering, he visually inspected the fit of the model in the data space with mclust::coordProj(). This approach is limited in the sense that we can only verify how the model fits onto a scatterplot(s) between two numeric variables, but the whole data space is a higher dimensional beast. Although we are visually limited to 3 dimensions, we can still dynamically explore projections of data in high dimensions in such a way that allows us to construct a mental model of the entire space (commonly referred to as touring multidimensional data).

There are many ways to tour multidimensional data. The most common are the grand tour and the guided tour. The grand tour interpolates through random projections of the data whereas the guided tour progresses through “interesting” projections. In this post, I’ll keep things simple, and we’ll take a grand tour of Mariano Rivera’s pitches from 2011.

data(pitches, package = "pitchRx")
rivera <- subset(pitches, pitcher_name == "Mariano Rivera")

When I say a grand tour walks through random projections, I really mean that it goes through random linear combinations of all the variables that we give it (subject to some constraints). This is important since this implies that we can’t include categorical variables in the tour. Let’s use the same numeric variables that Brian did when examining Mark Buerhle’s pitches:

vars <- c("start_speed", "break_y", "break_angle", "break_length")
rivera_num <- rivera[names(rivera) %in% vars]
# ensure these variables are numeric
rivera_num[] <- lapply(rivera_num, as.numeric)

With the tourr package, it’s really easy to perform the grand tour (although, I’ve had problems running this code inside of RStudio, so you may want to run this outside of RStudio):

library(tourr)
animate(rivera_num, grand_tour(), display_xy())

We can actually mimic this functionality and make the results easier to share using animint. In fact, I’ve provided a convenience function for creating tours of PITCHf/x data called pitch_tour(). By default, pitch_tour() will use MLBAM’s pitch type classification for the point color.

devtools::source_gist("4673815529b7a1e6c1aa")
pitch_tour(rivera)

If you view the result here (best viewed with Chrome), you’ll notice that the projection begins with break_y on the y-axis and start_speed on the x-axis. There appears to be 4-5 very distinct groups/clusters of pitches along the break_y axis (I’ll return to this shortly). As the tour progresses through the random projections, you should also notice that MLBAM’s classification is almost entirely determined by the break_angle (that is, we could draw a line orthogonal to the break_angle axis that separates red points from blue points):

library(ggplot2)
rivera$break_angle <- as.numeric(rivera$break_angle)
ggplot(rivera, aes(x = break_angle, fill = pitch_type)) + 
  geom_density(alpha = 0.2)

rivera$mlbam_type <- cut(rivera$break_angle, c(-100, 0, 100))
with(rivera, table(mlbam_type, pitch_type))

          pitch_type
mlbam_type    FC  FF
  (-100,0]   775   4
  (0,100]     14 119

However, given what we’ve seen in the tour, splitting on break_y is clearly better if our goal is to produce the most “distinct” grouping (that is, most separated from one another in the data space). In fact, a model based clustering mostly agrees with our visual intuition:

m <- mclust::Mclust(rivera_num)
rivera_num$classification <- m$classification
pitch_tour(rivera_num, color_by = "classification", out_dir = "rivera-mbc")

Of course, Rivera doesn’t have anywhere near 8 different pitch types. The point here is that the cluster analysis is fooled by the “discreteness” of break_y. Rivera’s break_y only varies from 23.6 to 24.0 and the precision of break_y measurements only goes to the tenth place. Whenever the ratio of precision to range for a numerical variable is this large, it is bound to cause problems, so we should to remove it from consideration for Rivera. Interestingly, Mark Buerhle’s break_y data is similar to Rivera’s, but the model based clustering approach doesn’t get fooled as badly. I suppose that is due to that fact that Mark has a higher number of actual types are more spread out in the data space.

mark <- read.csv("http://www.brianmmills.com/uploads/2/3/9/3/23936510/markb2013.csv")
table(mark$break_y)

23.6 23.7 23.8 23.9   24 
  24  521 2128  668   27

mark_num <- mark[names(mark) %in% vars]
mark_num[] <- lapply(mark_num, as.numeric)
m2 <- mclust::Mclust(mark_num)
mark_num$classification <- m2$classification
pitch_tour(mark_num, color_by = "classification", out_dir = "mark-mbc")

Carson's blog on R, RStudio, plotly, shiny, data visualization, statistics, etc

plotly 4.9.0 now on CRAN

New shiny events

New version of plotly.js

New trace types

New hovertemplate attribute

New title API

Announcing my new book: Interactive web-based data visualization with R, plotly, and shiny

plotly 4.8.0 now on CRAN

New trace types

Improved static image export

Auto margins

Localization support

Partial plotly.js bundles

New and improved WebGL renderer

MathJax support

MathJax caveats

Other features discussed previously

Visualizing geo-spatial data with sf and plotly

One trace to rule them all?

Leveraging mapbox basemaps

Linking views without shiny

More learning resources

Future work

Appendix: strengths and weaknesses

1. plot_ly() and geom_sf()

2. plot_mapbox()

3. plot_geo():

Learning from and improving upon ggplotly conversions

Basic intro to geom_sf()

Tools for improving upon ggplotly() conversions

Leveraging plotly’s interactive features

Customizing tooltips

Animation

Highlighting

Discussion

Conclusion

Components of a successful R workshop

Inspire first 😲, mindful explanation later 💁

Encourage curiosity about data 🤔 📊

Understand your audience 🕵️‍♀️

Encourage collaboration 👩‍💻 👨‍💻

Bring your work environment to them 💼 💻

Looking forward 🔮

plotly 4.7.1 now on CRAN

New plotly.js goodies

Modifying plotly graphs via plotlyProxy()

Developing tools for exploratory visualization

plotly 4.7.0 now on CRAN

plotly 4.6.0 now on CRAN

News and Updates Surrounding plotly for R

Not Found

Upgrading to plotly 4.0 (and above)

Formula mappings

Smarter defaults

plot_ly() now initializes a plot

New interpretation of group

New plotly object representation

Conclusion

3D surface plots of the strikezone

Kernel Densities

Probabilistic Surfaces

Touring PITCHf/x Data

1. `plot_ly()` and `geom_sf()`

2. `plot_mapbox()`

3. `plot_geo()`:

Basic intro to `geom_sf()`

Tools for improving upon `ggplotly()` conversions

Modifying plotly graphs via `plotlyProxy()`

`plot_ly()` now initializes a plot