In my last post I talked about the strategy I used to figure out what should go into a LEMRS launch script. A major theme of this post is that I now know more about Linux than I ever have before and ever will again, so I’m trying to document what I’ve done and otherwise take care of future me when I have to get back in and update all this stuff. [NOTE: There is now an updated version of this script.]
The launch script I’ve developed has these major parts:
- update yum
- create a folder for the files the script will download
- install missing Linux packages
- install ER (nginx and R)
- add needed packages to R
- install Shiny Server
- add ec2-user to shiny and nginx groups
- build web server root directories in /srv
- build a configuration folder in /srv
- build a logs folder in /srv
- add some descriptive names to a few Linux commands
The actual script is at the very end of this post, but first let’s take a closer look at the section that installs R packages, because you may need different packages than I do. The way this works is by creating a file with three install.packages() commands in it (each ending with a semicolon, which R understands as a command separator), then having the launch script (running as root) execute the file in R. We create the file with a Linux command that simplifies to printf “R code” > filename and we run it with a command that simplifies to Rscript filename. Here’s what this part of the script looks like:
# add packages to R
# first create a script and save it
printf "install.packages('Rcpp', repos='http://XXX'); install.packages('devtools', repos='http://cran.rstudio.com/'); install.packages(c('shiny', 'shinythemes', 'shinyBS', 'V8', 'shinyjs', 'stringr', 'DBI', 'dplyr', 'dbplyr', 'pool', 'tibble', 'purrr', 'magrittr', 'lubridate', 'rmarkdown', 'ggplot2', 'xml2', 'bcrypt', 'mailR'), repos='http://XXX')\n" > /home/ec2-user/instance-setup-files/install.R.packages
# next run the script
Rscript /home/ec2-user/instance-setup-files/install.R.packages
You’ll need to specify the repository you want to use. The complete script below uses the one at Carnegie Mellon in Pittsburgh for everything except devtools, which is available from the repository at cran.rstudio.com. The first install.packages() loads only the Rcpp package, which I had difficulty getting to install on a 512 MB RAM instance, followed by the devtools package. If you can get those two to install, you can put everything else in one install.packages() command, assuming they’re all available at the same repository.
Next, how do you pass the script to Amazon Lightsail? As shown in the next figure, when you create a new Amazon Lightsail instance, first click on the OS Only button in the Select a blueprint section. This script has only been tested with the default, Amazon Linux (EDIT: in early November 2017 the version of Amazon Linux offered by Lightsail changed from the 2017.03.1 version shown below to the 2017.09.0 version; this didn’t require any changes to this script, however.) Then, under the grey box, click on + Add launch script. A box will open and you can paste the launch script into that box. Finally, you select a plan (this script doesn’t work well with the 512 MB RAM plan, pick something larger), a name, and click the Create button.
The script takes about 15 minutes to run. No kidding. After waiting patiently, click the square box on the image of your instance to open a terminal connection to it. The very first thing to do is enter the command I’ve added called yum.history. Here is what you should see:
This is a list of everything yum has installed, with the last installation at the top. Note that things went very quickly until we installed R, then 15 minutes passed before the script did its final installation, which is Shiny Server. In between those two yum installations, R installed all of its packages. That’s what takes so long. Importantly, if the top line in the history is the install R -y line (line 10 in this example), it probably means you haven’t waited long enough, although it could also mean the script failed while doing the R package installations.
If this looks good, next enter R (just R this time, not sudo R) and from the R console enter installed.packages()[,1:3] to make sure everything you expected to be installed was actually installed.
Next, enter cd /srv, which moves you to the folder where everything we’ll need will be. Enter ls -la to see the files. Now ender cd configs to get into the /srv/configs folder. Do another ls -la and you’ll see a bunch of files that do things we need done. So, next, enter this one: start.nginx. Now go to a browser window and enter the IP number that’s in the lower right corner of your terminal window. You should get back Welcome to nginx on the Amazon Linux AMI!
If you do, we’re rolling. Before we start Shiny Server, however, we have to give both nginx and shiny-server new configuration files, restart nginx, and start shiny-server. I’ll tell you how to configure a LEMRS stack in my next post.
Ok, here’s the script:
# LEMRS launch script
# For Amazon Lightsail, Amazon Linux instance
# update yum
yum update -y
# folder for storing files needed to setup this instance
mkdir /home/ec2-user/instance-setup-files
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files
chmod 755 /home/ec2-user/instance-setup-files
# install missing Linux packages we'll need
yum install git.x86_64 -y
yum install libcurl-devel.x86_64 -y
yum install openssl-devel.x86_64 -y
yum install libxml2-devel.x86_64 -y
wget -O /home/ec2-user/instance-setup-files/v8-3.14.5.10-25.el7.x86_64.rpm "http://dl.fedoraproject.org/pub/epel/7Server/x86_64/Packages/v/v8-3.14.5.10-25.el7.x86_64.rpm"
yum install /home/ec2-user/instance-setup-files/v8-3.14.5.10-25.el7.x86_64.rpm -y
wget -O /home/ec2-user/instance-setup-files/v8-devel-3.14.5.10-25.el7.x86_64.rpm "http://dl.fedoraproject.org/pub/epel/7Server/x86_64/Packages/v/v8-devel-3.14.5.10-25.el7.x86_64.rpm"
yum install /home/ec2-user/instance-setup-files/v8-devel-3.14.5.10-25.el7.x86_64.rpm -y
# install E-R part of the stack
# we'll install M in a later episode
yum install nginx -y
yum install R -y
# start nginx at reboot
chkconfig nginx on
# add packages to R
# first create a script and save it
printf "install.packages('Rcpp', repos='http://lib.stat.cmu.edu/R/CRAN/'); install.packages('devtools', repos='http://cran.rstudio.com/'); install.packages(c('shiny', 'shinythemes', 'shinyBS', 'V8', 'shinyjs', 'stringr', 'DBI', 'dplyr', 'dbplyr', 'pool', 'tibble', 'purrr', 'magrittr', 'lubridate', 'rmarkdown', 'ggplot2', 'xml2', 'bcrypt', 'mailR'), repos='http://lib.stat.cmu.edu/R/CRAN/')\n" > /home/ec2-user/instance-setup-files/install.R.packages
# next run the script
Rscript /home/ec2-user/instance-setup-files/install.R.packages
# install Shiny Server
wget -O /home/ec2-user/instance-setup-files/shiny-server.rpm https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.5.5.872-rh5-x86_64.rpm
yum install --nogpgcheck /home/ec2-user/instance-setup-files/shiny-server.rpm -y
# install turns it on; turn it back off
stop shiny-server
# fix some folder ownerships for shiny
chown shiny:shiny /var/lib/shiny-server
chown shiny:shiny /opt/shiny-server
# Users!
# Add ec2-user to the shiny and nginx groups and vice-versa
usermod -a -G shiny ec2-user
usermod -a -G nginx ec2-user
usermod -a -G ec2-user shiny
usermod -a -G ec2-user nginx
# Web server root directories!
# permissions should be 775 = rwx rwx r-x so we can ftp in files
# fix /srv owner and permissions
chown ec2-user:ec2-user /srv
chmod 775 /srv
# this is the root for standard shiny apps
mkdir /srv/shiny-site-dir
chown shiny:shiny /srv/shiny-site-dir
chmod 775 /srv/shiny-site-dir
# move what shiny-server auto installs
mv /srv/shiny-server/sample-apps /srv/shiny-site-dir
mv /srv/shiny-server/index.html /srv/shiny-site-dir
rm -r /srv/shiny-server # -r because it's a directory
# this is the root for the shiny app that routes requests
mkdir /srv/shiny-app-dir
chown shiny:shiny /srv/shiny-app-dir
chmod 775 /srv/shiny-app-dir
# this is the nginx root
mkdir /srv/nginx/
chown nginx:nginx /srv/nginx
chmod 775 /srv/nginx
# copy default nginx files from std nginx root folder to new root at /srv/nginx
cp /usr/share/nginx/html/404.html /srv/nginx/
cp /usr/share/nginx/html/50x.html /srv/nginx/
cp /usr/share/nginx/html/index.html /srv/nginx/
cp /usr/share/nginx/html/nginx-logo.png /srv/nginx/
cp /usr/share/nginx/html/poweredby.png /srv/nginx/
rm -rf /usr/share/nginx/html
# put a symbolic link in the old location
ln -s /srv/nginx/ /usr/share/nginx/html
# configs!
# add folder in /srv for configs
mkdir /srv/configs
chown ec2-user:ec2-user /srv/configs
chmod 755 /srv/configs
# "Move" configuration files to /srv/configs
chown nginx:nginx /etc/nginx/ # update nginx folder owner
chmod 775 /etc/nginx/ # and permissions
cp /etc/nginx/nginx.conf /srv/configs/original-nginx-conf # save old configuation file
mv /etc/nginx/nginx.conf /srv/configs/nginx.conf # put nginx configuration file here
chown ec2-user:ec2-user /srv/configs/nginx.conf # take ownership
chmod 644 /srv/configs/nginx.conf # make writable
ln -s /srv/configs/nginx.conf /etc/nginx/nginx.conf # replace with a link to the new location
chown ec2-user:ec2user /etc/nginx/nginx.conf # clarify how this link happened
# repeat for shiny configuration folder
chown shiny:shiny /etc/shiny-server
chmod 775 /etc/shiny-server
cp /etc/shiny-server/shiny-server.conf /srv/configs/original-shiny-conf
mv /etc/shiny-server/shiny-server.conf /srv/configs/shiny-server.conf
chown ec2-user:ec2-user /srv/configs/shiny-server.conf
chmomd 644 /srv/configs/shiny-server.conf
ln -s /srv/configs/shiny-server.conf /etc/shiny-server/shiny-server.conf
chown ec2-user:ec2user /etc/shiny-server/shiny-server.conf
# shiny init configuration is a little different
chmod 775 /etc/init
cp /etc/init/shiny-server.conf /srv/configs/original-shiny-init ### shiny won't start if you delete this
chmod 775 /etc/init/shiny-server.conf ### even with a symbolic link replacement
# logs!
# add folder in /srv for logs
mkdir /srv/logs
chown ec2-user:ec2-user /srv/logs
chmod 777 /srv/logs
# make the REAL nginx log folders readable (shiny's are ok)
# 755 because programs need to write to log folders and I don't care who reads them
chown ec2-user:ec2-user /var/log
chmod 755 /var/log
chown nginx:nginx /var/log/nginx
chmod 755 /var/log/nginx
# symbolic links to actual log folders from /srv/logs
ln -s /var/log/nginx /srv/logs/var-log-nginx
ln -s /var/log/shiny-server /srv/logs/var-log-shiny
ln -s /var/log/ /srv/logs/var-log
# and vice-versa
ln -s /srv/logs/ /var/log/srv-logs
ln -s /srv/logs/ /var/log/nginx/srv-logs
ln -s /srv/logs/ /var/log/shiny-server/srv-logs
# Commands!
# set up ~/bin folder
mkdir /home/ec2-user/bin
chown ec2-user:ec2-user /home/ec2-user/bin
chmod 755 /home/ec2-user/bin
# this adds the file "add-cmds" to the ~/instance-setup-files folder
printf "printf 'top' > /home/ec2-user/bin/see.mem.space\nprintf 'df' > /home/ec2-user/bin/see.disk.space\nprintf 'sudo service nginx restart' > /home/ec2-user/bin/restart.nginx\nprintf 'sudo service nginx start' > /home/ec2-user/bin/start.nginx\nprintf 'sudo service nginx stop' > /home/ec2-user/bin/stop.nginx\nprintf 'sudo stop shiny-server\nsudo start shiny-server' > /home/ec2-user/bin/restart.shiny\nprintf 'sudo start shiny-server' > /home/ec2-user/bin/start.shiny\nprintf 'sudo stop shiny-server' > /home/ec2-user/bin/stop.shiny\nprintf 'sudo ps -ely | more' > /home/ec2-user/bin/whats.running\nprintf 'sudo chkconfig --list | more' > /home/ec2-user/bin/whats.autostart\nprintf 'sudo yum history' > /home/ec2-user/bin/yum.history\nprintf 'sudo yum list | more' > /home/ec2-user/bin/yum.installed\nprintf 'sudo yum search \0441' > /home/ec2-user/bin/yum.search\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-debug.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-debug\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-info.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-info\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-warn.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-warn\nchown -R ec2-user:ec2-user /home/ec2-user/bin\nchmod -R 755 /home/ec2-user/bin # change on all files in the directory\n" > /home/ec2-user/instance-setup-files/add-cmds
# This runs the "add-cmds" file
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files/add-cmds
chmod 755 /home/ec2-user/instance-setup-files/add-cmds
cd /home/ec2-user/instance-setup-files
./add-cmds
# create links to these commands from /srv/configs
ln -s /home/ec2-user/bin/see.disk.space /srv/configs/see.disk.space
ln -s /home/ec2-user/bin/see.mem.space /srv/configs/see.mem.space
ln -s /home/ec2-user/bin/restart.nginx /srv/configs/restart.nginx
ln -s /home/ec2-user/bin/restart.shiny /srv/configs/restart.shiny
ln -s /home/ec2-user/bin/start.nginx /srv/configs/start.nginx
ln -s /home/ec2-user/bin/start.shiny /srv/configs/start.shiny
ln -s /home/ec2-user/bin/stop.nginx /srv/configs/stop.nginx
ln -s /home/ec2-user/bin/stop.shiny /srv/configs/stop.shiny
ln -s /home/ec2-user/bin/switch-shiny-logs-to-debug /srv/configs/switch-shiny-logs-to-debug
ln -s /home/ec2-user/bin/switch-shiny-logs-to-info /srv/configs/switch-shiny-logs-to-info
ln -s /home/ec2-user/bin/switch-shiny-logs-to-warn /srv/configs/switch-shiny-logs-to-warn
ln -s /home/ec2-user/bin/whats.autostart /srv/configs/whats.autostart
ln -s /home/ec2-user/bin/whats.running /srv/configs/whats.running
Next we need to provide configuration files for nginx and Shiny-Server.