Shiny Dev and Software Release Cycles and Rayshader, Oh My!

Inspiration for the Shiny Application

You can read as much theory as you want but the best way to learn something is to pull-up your sleeves and get your hands dirty by implementing a project.

I’ve been wanting to develop a Shiny application for sometime but the opportunity has never arisen. In my current role as a R developer, I provide client-side implementations via Java with the help of the RServe 📦 - so Shiny isn’t needed. Well, if the opportunity doesn’t present itself on the job it’s time for a side-project!

I wasn’t exactly sure what kind of project I should do until I saw a blog post by Timo Grossenbacher on bivariate maps with ggplot2 and sf.

The map shows income (in-)equality in Switzerland at the municipality level by visualizing two variables at the same time. What instantly attracted me was the beautiful relief of the mountainous Swiss landscape. It’s not something I’ve seen much from ggplot2 maps - something usually reserved for GIS programs. I thought creating an application for geography would be a fun and engaging use of a Shiny application. My current role as a developer for a web-application that enables complex meta-analysis and visualization caters to biologists rather than specilazied bioinformaticians- NetworkAnalyst.ca. I thought it might be cool to do a similar thing for people who aren’t GIS specialist (or even familiar with R) and enable them to quickly create maps and save them (either as PNGs or PDFs)?

One of the intial challenges was finding a good dataset. I had wanted to do a project on my home province of Alberta but eventually settled on Hawaii instead for a couple reasons. One is that I could find relief’s for the State of Hawaii in two levels of detail: 100m and 200m. The second was the State of Hawaii also provides a large number of shapefiles which I could lay overtop of the relief. The amount of open data available to Hawaii attractive, and the fact that they are islands presented certain opportunities… and challenges, as well.

I wanted to provide users with a range maps. I wanted maps at different scales and with different contexts such as focusing on the marine environment or the terrestrial enviornment. I wanted maps covering the whole state of Hawaii, others showing individual islands (or groups of islands), and others giving the user a view of the city level. It was here that I got inspiration from Erin Davis’s beautiful street maps in “The Beatiful Hidden Logic of Cities - Worldwide”.

Erin’s legend looked great! Unfortunately she didn’t have a programatic way to generate these labels, she did them in Adobe Photoshop (personal correspondence). I tried to write some code to generate labels programattically using key_glyph = "point" to set sf object glyphs as points and then use the override.aes trick to get circles (like in this SO post post) but I didn’t like the look. To be honest lables are something that could still use work in the ggplot2 world IMO 🤷

Jake Kaupp did a #TidyTuesday submission for Week #40: Pizza Party! showing all of the pizza spots in the five-boroughs and came up with a nice way of constructing a color legend with ggplotGrob() as well as a neat use of colorspace::darken():

For Shiny UI inspiration I looked towards some of the winners of the 1st ever Shiny Contest (something I hope RStudio brings back next year!). I can’t express how much I learn from reading other peoples code and for this particular application I was heavily inspired by the David Smale’s, winner of “Best Design”, 69 Love Songs: A Lyrical Analysis. What was really cool was it was hosted on the RStudio cloud so I could go in and change things and see how it affected which for me was one of the coolest ways to grok Shiny.

Building the application

Getting back to elevation reliefs for a second. Before I learned that you could easily get Digital Elevation Model (DEM) data from mapzen via the {`geoviz’} 📦, I used QGIS to subset the reliefs. Here’s a screencast of the process:

For the shapefiles, sometimes it was simple to subset the geom_sf() object via a column with island information (Hawaii, Maui, Oahu, Kauai) using dplyr::filter(). However, when there was no information I also used QGIS to subset the shapefiles.

Note: I’m positive there is a way to do this via a bounding-box in R I just never learned how… please send me a message and I’ll update this if you know a way! Here’s a video of how I did it with QGIS:

This project was a great learning experience not only for learning Shiny & Spatial Mapping, but also learning a bit more about the development cycle of a “production” applcation is like. It’s kind of like the ol’ aphorism “Perfect is the eney of good” - you want to get your appltion out to the public ASAP. I released the app as soon as I had tested for bugs and thought the UI was good-enough.

After its release, I wanted to add a new feature - Rayshader!, Rayshader is one of the hot-new tools in The Landscape of Spatial Data Analysis in R. Historically geographic information had always been presented in the form of two-dimensional maps, all the way from cave walls, parchment to computer screens. Because spatial information is inherently 3D, it makes sense that the tools available today are allowing 3D depictions of geographical data to escape the realm of mere novelty. With Rayshader you can control things like the lighting, camera tilt, and angle of view which provides an engaging experience for the audience.

I was signed up for Tyler Morgan-Wall’s (the developer of Rayshader) Penn State MUSA Masterclass: 3D Mapping and Visualization with R and Rayshader the next month (the awesome talk is here and the Git repo is here) and wanted to spend a bit of time to demo the package before taking the class.

I came up with some very plain “desert”-y looking maps of Hawaii:

Although I had wanted to allow the user to interact with the 3D maps, I ran into issues where the application attempted to use more memory then was avialable under the free-tier plan of Shinyapps.io. The RStudio documentation on debugging your application’s section on Memory wa useful for figuring this out as it suggests looking in the application’s logs for 2019-11-18T15:12:16.088451+00:00 shinyapps[system]: Out of memory!. I settled on providing the “fly-by” videos innstead.

After attending Tyler’s class I wanted to update my application again by including a satellite overlay; I spent some time figuring out the best way to do this, modifying the easing function and rayshader::render_movie() parameters to make longer videos, testing it out, and then using rsconnect::deployApp() to re-deploy my application.

I decided to merge all four videos together using VSCD Video Editor and overlaid some a musical track. Although Tyler promised Rayshader would have VR/AR capabilities in the near-future, I simply couldn’t wait so I also made another version of the video with a steroscopic 3D effect for viewing with red and cyan anaglyph 3D glasses:

Brief thought on the future of Rayshader

There are a number of companies today trying to map out city streets to create high-quality 3D maps for self-driving cars and projection projects (see: The Race to Map the World in 3D):

Image: Carmera

Getting access to this kind of data would be awesome because it would mean you could overlay high-quality images over LIDAR like this one produced by Tyler of Philadelphia:

If you missed it at the top you can CLICK HERE TO GO TO MY SHINY APPLICATION DEPLOYMENT!!!