This week my fellow doctoral students see the first live demonstration (and alpha-test) of the Open-Meta app. Before loading the code onto AWS, I decided to start a new instance and update the installation scripts.
The primary changes to the Amazon Lightsail launch script involve:
- Incorporating the commands for loading MySQL into the main script
- Removing R packages and associated Linux packages that I’m no longer using.
- Adding the DT package for R.
- Using newer versions of R and Shiny Server.
- The mailR package depends on the rJava package, which now requires finding some files by hand that the gcc compiler could no longer find and creating symbolic links to them.
# LEMRS launch script
# For Amazon Lightsail, Amazon Linux instance
# update yum
yum update -y
# folder for storing files needed to setup this instance
mkdir /home/ec2-user/instance-setup-files
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files
chmod 755 /home/ec2-user/instance-setup-files
# install missing Linux packages we'll need
yum install git.x86_64 -y
yum install libcurl-devel.x86_64 -y
yum install openssl-devel.x86_64 -y
yum install libxml2-devel.x86_64 -y
# install E-R-M part of the stack
yum install nginx -y
yum install R -y
yum install mysql57-server.x86_64 -y
yum install mysql57-devel.x86_64 -y
# start nginx and MySQL at reboot
chkconfig nginx on
chkconfig mysqld on
# symbolic links needed to install the rJava package required by mailR
ln -s /usr/lib/gcc/x86_64-amazon-linux/6.4.1/libgomp.spec /usr/lib64/libgomp.spec
ln -s /usr/lib/gcc/x86_64-amazon-linux/6.4.1/libgomp.a /usr/lib64/libgomp.a
ln -s /usr/lib64/libgomp.so.1.0.0 /usr/lib64/libgomp.so
# symbolic link needed to install the RMariaDB package
ln -s /usr/lib64/mysql57/libmysqlclient.so.1020 /usr/lib64/libmysqlclient.so.1020
# add packages to R
# first create a script and save it
printf "install.packages('Rcpp', repos='http://lib.stat.cmu.edu/R/CRAN/'); install.packages('devtools', repos='http://cran.rstudio.com/'); install.packages(c('shiny','htmltools','dplyr','stringr','lubridate','DT','DBI','pool','RMariaDB','bcrypt', 'mailR'), repos='http://lib.stat.cmu.edu/R/CRAN/')\n" > /home/ec2-user/instance-setup-files/install.R.packages
# next run the script
Rscript /home/ec2-user/instance-setup-files/install.R.packages
# install Shiny Server
wget -O /home/ec2-user/instance-setup-files/shiny-server.rpm https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.5.6.875-rh5-x86_64.rpm
yum install --nogpgcheck /home/ec2-user/instance-setup-files/shiny-server.rpm -y
# install turns it on; turn it back off
stop shiny-server
# fix some folder ownerships for shiny
chown shiny:shiny /var/lib/shiny-server
chown shiny:shiny /opt/shiny-server
# Users!
# Add ec2-user to the shiny and nginx groups and vice-versa
usermod -a -G shiny ec2-user
usermod -a -G nginx ec2-user
usermod -a -G ec2-user shiny
usermod -a -G ec2-user nginx
# Web server root directories!
# permissions should be 775 = rwx rwx r-x so we can ftp in files
# fix /srv owner and permissions
chown ec2-user:ec2-user /srv
chmod 775 /srv
# this is the root for standard shiny apps
mkdir /srv/shiny-site-dir
chown shiny:shiny /srv/shiny-site-dir
chmod 775 /srv/shiny-site-dir
# move what shiny-server auto installs
mv /srv/shiny-server/sample-apps /srv/shiny-site-dir
mv /srv/shiny-server/index.html /srv/shiny-site-dir
rm -r /srv/shiny-server # -r because it's a directory
# this is the root for the shiny app that routes requests
mkdir /srv/shiny-app-dir
chown shiny:shiny /srv/shiny-app-dir
chmod 775 /srv/shiny-app-dir
# this is the nginx root
mkdir /srv/nginx/
chown nginx:ec2-user /srv/nginx
chmod 775 /srv/nginx
# copy default nginx files from std nginx root folder to new root at /srv/nginx
cp /usr/share/nginx/html/404.html /srv/nginx/
cp /usr/share/nginx/html/50x.html /srv/nginx/
cp /usr/share/nginx/html/index.html /srv/nginx/
cp /usr/share/nginx/html/nginx-logo.png /srv/nginx/
cp /usr/share/nginx/html/poweredby.png /srv/nginx/
rm -rf /usr/share/nginx/html
# put a symbolic link in the old location
ln -s /srv/nginx/ /usr/share/nginx/html
# configs!
# add folder in /srv for configs
mkdir /srv/configs
chown ec2-user:ec2-user /srv/configs
chmod 755 /srv/configs
# "Move" configuration files to /srv/configs
chown nginx:nginx /etc/nginx/ # update nginx folder owner
chmod 775 /etc/nginx/ # and permissions
cp /etc/nginx/nginx.conf /srv/configs/original-nginx-conf # save old configuation file
mv /etc/nginx/nginx.conf /srv/configs/nginx.conf # put nginx configuration file here
chown ec2-user:ec2-user /srv/configs/nginx.conf # take ownership
chmod 644 /srv/configs/nginx.conf # make writable
ln -s /srv/configs/nginx.conf /etc/nginx/nginx.conf # replace with a link to the new location
chown ec2-user:ec2user /etc/nginx/nginx.conf # clarify how this link happened
# repeat for shiny configuration folder
chown shiny:shiny /etc/shiny-server
chmod 775 /etc/shiny-server
cp /etc/shiny-server/shiny-server.conf /srv/configs/original-shiny-conf
mv /etc/shiny-server/shiny-server.conf /srv/configs/shiny-server.conf
chown ec2-user:ec2-user /srv/configs/shiny-server.conf
chmomd 644 /srv/configs/shiny-server.conf
ln -s /srv/configs/shiny-server.conf /etc/shiny-server/shiny-server.conf
chown ec2-user:ec2user /etc/shiny-server/shiny-server.conf
# shiny init configuration is a little different
chmod 775 /etc/init
cp /etc/init/shiny-server.conf /srv/configs/original-shiny-init ### shiny won't start if you delete this
chmod 775 /etc/init/shiny-server.conf ### even with a symbolic link replacement
# logs!
# add folder in /srv for logs
mkdir /srv/logs
chown ec2-user:ec2-user /srv/logs
chmod 777 /srv/logs
# make the REAL nginx log folders readable (shiny's are ok)
# 755 because programs need to write to log folders and I don't care who reads them
chown ec2-user:ec2-user /var/log
chmod 755 /var/log
chown nginx:nginx /var/log/nginx
chmod 755 /var/log/nginx
# symbolic links to actual log folders from /srv/logs
ln -s /var/log/nginx /srv/logs/var-log-nginx
ln -s /var/log/shiny-server /srv/logs/var-log-shiny
ln -s /var/log/ /srv/logs/var-log
# and vice-versa
ln -s /srv/logs/ /var/log/srv-logs
ln -s /srv/logs/ /var/log/nginx/srv-logs
ln -s /srv/logs/ /var/log/shiny-server/srv-logs
# Commands!
# set up ~/bin folder
mkdir /home/ec2-user/bin
chown ec2-user:ec2-user /home/ec2-user/bin
chmod 755 /home/ec2-user/bin
# this adds the file "add-cmds" to the ~/instance-setup-files folder
printf "printf 'top' > /home/ec2-user/bin/see.mem.space\nprintf 'df' > /home/ec2-user/bin/see.disk.space\nprintf 'sudo service nginx restart' > /home/ec2-user/bin/restart.nginx\nprintf 'sudo service nginx start' > /home/ec2-user/bin/start.nginx\nprintf 'sudo service nginx stop' > /home/ec2-user/bin/stop.nginx\nprintf 'sudo stop shiny-server\nsudo start shiny-server' > /home/ec2-user/bin/restart.shiny\nprintf 'sudo start shiny-server' > /home/ec2-user/bin/start.shiny\nprintf 'sudo stop shiny-server' > /home/ec2-user/bin/stop.shiny\nprintf 'sudo ps -ely | more' > /home/ec2-user/bin/whats.running\nprintf 'sudo chkconfig --list | more' > /home/ec2-user/bin/whats.autostart\nprintf 'sudo yum history' > /home/ec2-user/bin/yum.history\nprintf 'sudo yum list | more' > /home/ec2-user/bin/yum.installed\nprintf 'sudo yum search \0441' > /home/ec2-user/bin/yum.search\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-debug.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-debug\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-info.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-info\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-warn.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-warn\nchown -R ec2-user:ec2-user /home/ec2-user/bin\nchmod -R 755 /home/ec2-user/bin # change on all files in the directory\n" > /home/ec2-user/instance-setup-files/add-cmds
# This runs the "add-cmds" file
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files/add-cmds
chmod 755 /home/ec2-user/instance-setup-files/add-cmds
cd /home/ec2-user/instance-setup-files
./add-cmds
# Additional commands for MySQL
printf 'sudo service mysqld restart' > /home/ec2-user/bin/restart.mysql
printf 'sudo service mysqld start' > /home/ec2-user/bin/start.mysql
printf 'sudo service mysqld stop' > /home/ec2-user/bin/stop.mysql
chown -R ec2-user:ec2-user /home/ec2-user/bin
chmod -R 755 /home/ec2-user/bin # change on all files in the directory
# create links to these commands from /srv/configs
ln -s /home/ec2-user/bin/see.disk.space /srv/configs/see.disk.space
ln -s /home/ec2-user/bin/see.mem.space /srv/configs/see.mem.space
ln -s /home/ec2-user/bin/restart.nginx /srv/configs/restart.nginx
ln -s /home/ec2-user/bin/restart.shiny /srv/configs/restart.shiny
ln -s /home/ec2-user/bin/start.nginx /srv/configs/start.nginx
ln -s /home/ec2-user/bin/start.shiny /srv/configs/start.shiny
ln -s /home/ec2-user/bin/stop.nginx /srv/configs/stop.nginx
ln -s /home/ec2-user/bin/stop.shiny /srv/configs/stop.shiny
ln -s /home/ec2-user/bin/switch-shiny-logs-to-debug /srv/configs/switch-shiny-logs-to-debug
ln -s /home/ec2-user/bin/switch-shiny-logs-to-info /srv/configs/switch-shiny-logs-to-info
ln -s /home/ec2-user/bin/switch-shiny-logs-to-warn /srv/configs/switch-shiny-logs-to-warn
ln -s /home/ec2-user/bin/whats.autostart /srv/configs/whats.autostart
ln -s /home/ec2-user/bin/whats.running /srv/configs/whats.running
ln -s /home/ec2-user/bin/ /srv/configs/drop-d-here-and-chmod-755-it
ln -s /home/ec2-user/bin/restart.mysql /srv/configs/restart.mysql
ln -s /home/ec2-user/bin/start.mysql /srv/configs/start.mysql
ln -s /home/ec2-user/bin/stop.mysql /srv/configs/stop.mysql
Next we have to secure MySQL. Complete instructions on how to do this are in this previous post. Start where it says, Enter mysql_secure_installation to begin. However, before that, enter start.mysql or you’ll get an error about not being able to connect to MySQL though some socket.
This previous post has the configuration files for nginx and Shiny Server. I made one tiny change to the configuration file for Shiny Server: I changed preserve_logs true; to preserve_logs false; preserving the Shiny logs ended up to be more trouble than it was worth.
After the MySQL installation and installing the configuration files, enter the following commands using PuTTY:
restart.nginx restart.mysql restart.shiny
The final step is to transfer the files for your apps to the new instance.