LEMRS v0.5 – Adding rvest and aws.signature

I’ve been having problems with the web version of the Open-Meta app crashing with a C stack usage error. I entered an issue on the Shiny Server Github page and with the help of others who have had this problem, now suspect that the error is caused by the rJava package, which is required by mailR, the package I’ve been using to send email.

I’ve been wanting to switch to directly using the Amazon Web Services api for its Simple Email Service rather than using mailR and its SMTP connection to Amazon’s email service anyhow, because mailR takes about 10 seconds to send an email. It turns out this delay is mostly related to the time it takes to load and initialize the rJava package! My difficulty has been figuring out how to build the encrypted signature Amazon requires to use its api interface.

It turns out there’s a package for that, aws.signature! In order to switch over to the AWS SES api and away from mailR and rJava, I’ve created a new instance, v0.5. Here’s the script:

# LEMRS launch script
#    For Amazon Lightsail, Amazon Linux instance
# v0.3 May 16, 2018 - Amazon Linux 2017.09.01
# v0.4 Jul 11, 2018 - Amazon Linux 2018.03.0
# v0.5 Nov 07, 2018 - Amazon Linux 2018.03.0

   # update yum
yum update -y
yum install epel-release  # Gets more recent version of R

   # folder for storing files needed to setup this instance
mkdir /home/ec2-user/instance-setup-files
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files
chmod 755 /home/ec2-user/instance-setup-files

   # install missing Linux packages we'll need
#yum install git.x86_64 -y
yum install libcurl-devel.x86_64 -y
yum install openssl-devel.x86_64 -y
yum install libxml2-devel.x86_64 -y
yum install libgfortran.x86_64 -y
yum install libquadmath.x86_64 -y

   # install E-R-M part of the stack
yum install nginx -y
yum install R -y
yum install mysql57-server.x86_64 -y
yum install mysql57-devel.x86_64 -y

   # start nginx and MySQL at reboot
chkconfig nginx on
chkconfig mysqld on

   # symbolic link needed to install the RMariaDB package
ln -s /usr/lib64/mysql57/libmysqlclient.so.1020 /usr/lib64/libmysqlclient.so.1020

   # symbolic links needed to install Tidyverse package
ln -s /usr/lib/gcc/x86_64-amazon-linux/6.4.1/libgfortran.so /usr/lib64/
ln -s /usr/lib/gcc/x86_64-amazon-linux/6.4.1/libquadmath.so /usr/lib64/

   # add packages to R
      # v0.2: Rcpp, devtools, 'shiny', 'htmltools', 'dplyr', 'stringr', 'lubridate', 'DT', 'DBI', 'pool', 'RMariaDB', 'bcrypt', 'mailR'
      # v0.4: Rcpp, devtools, 'shiny', 'tidyverse', 'pool', 'RMariaDB', 'bcrypt', 'DT', 'mailR', 'RefManageR', 'RCurl'
      # v0.5: Rcpp, 'shiny', 'tidyverse', 'pool', 'RMariaDB', 'bcrypt', 'DT', 'RefManageR', 'RCurl', 'rvest', 'aws.signature'

      # first create a script and save it
printf "install.packages('Rcpp', repos='http://lib.stat.cmu.edu/R/CRAN/'); install.packages(c('shiny','tidyverse','pool','RMariaDB','bcrypt','DT','RefManageR','RCurl','rvest','aws.signature'), repos='http://lib.stat.cmu.edu/R/CRAN/')\n" > /home/ec2-user/instance-setup-files/install.R.packages
      # next run the script
Rscript /home/ec2-user/instance-setup-files/install.R.packages

   # install Shiny Server
wget -O /home/ec2-user/instance-setup-files/shiny-server.rpm https://download3.rstudio.org/centos6.3/x86_64/shiny-server-1.5.9.923-x86_64.rpm
yum install --nogpgcheck /home/ec2-user/instance-setup-files/shiny-server.rpm  -y

      # install turns it on; turn it back off
stop shiny-server
      # fix some folder ownerships for shiny
chown shiny:shiny /var/lib/shiny-server
chown shiny:shiny /opt/shiny-server

   # Users!
      # Add ec2-user to the shiny and nginx groups and vice-versa
usermod -a -G shiny ec2-user
usermod -a -G nginx ec2-user
usermod -a -G ec2-user shiny
usermod -a -G ec2-user nginx

   # Web server root directories!
      # permissions should be 775 = rwx rwx r-x so we can ftp in files
      # fix /srv owner and permissions
chown ec2-user:ec2-user /srv
chmod 775 /srv
      # this is the root for standard shiny apps
mkdir /srv/shiny-site-dir
chown shiny:shiny /srv/shiny-site-dir
chmod 775 /srv/shiny-site-dir
      # move what shiny-server auto installs
mv /srv/shiny-server/sample-apps /srv/shiny-site-dir
mv /srv/shiny-server/index.html /srv/shiny-site-dir
rm -r /srv/shiny-server                      # -r because it's a directory
      # this is the root for the shiny app that routes requests
mkdir /srv/shiny-app-dir
chown shiny:shiny /srv/shiny-app-dir
chmod 775 /srv/shiny-app-dir
      # this is the nginx root
mkdir /srv/nginx/
chown nginx:ec2-user /srv/nginx
chmod 775 /srv/nginx
      # copy default nginx files from std nginx root folder to new root at /srv/nginx
cp /usr/share/nginx/html/404.html /srv/nginx/
cp /usr/share/nginx/html/50x.html /srv/nginx/
cp /usr/share/nginx/html/index.html /srv/nginx/
cp /usr/share/nginx/html/nginx-logo.png /srv/nginx/
cp /usr/share/nginx/html/poweredby.png /srv/nginx/
rm -rf /usr/share/nginx/html
         # put a symbolic link in the old location
ln -s /srv/nginx/ /usr/share/nginx/html

   # configs!
      # add folder in /srv for configs
mkdir /srv/configs
chown ec2-user:ec2-user /srv/configs
chmod 755 /srv/configs
      # "Move" configuration files to /srv/configs
chown nginx:nginx /etc/nginx/                              # update nginx folder owner
chmod 775 /etc/nginx/                                      #    and permissions
cp /etc/nginx/nginx.conf /srv/configs/original-nginx-conf  # save old configuation file
mv /etc/nginx/nginx.conf /srv/configs/nginx.conf           # put nginx configuration file here
chown ec2-user:ec2-user /srv/configs/nginx.conf            #   take ownership
chmod 644  /srv/configs/nginx.conf                         #   make writable
ln -s /srv/configs/nginx.conf /etc/nginx/nginx.conf        # replace with a link to the new location
chown ec2-user:ec2user /etc/nginx/nginx.conf               # clarify how this link happened
      # repeat for shiny configuration folder
chown shiny:shiny /etc/shiny-server
chmod 775 /etc/shiny-server
cp /etc/shiny-server/shiny-server.conf /srv/configs/original-shiny-conf
mv /etc/shiny-server/shiny-server.conf /srv/configs/shiny-server.conf
chown ec2-user:ec2-user /srv/configs/shiny-server.conf
chmomd 644 /srv/configs/shiny-server.conf
ln -s  /srv/configs/shiny-server.conf /etc/shiny-server/shiny-server.conf
chown ec2-user:ec2user /etc/shiny-server/shiny-server.conf
      # shiny init configuration is a little different
chmod 775 /etc/init
cp /etc/init/shiny-server.conf /srv/configs/original-shiny-init ### shiny won't start if you delete this
chmod 775 /etc/init/shiny-server.conf                           ###   even with a symbolic link replacement

   # logs!
      # add folder in /srv for logs
mkdir /srv/logs
chown ec2-user:ec2-user /srv/logs
chmod 777 /srv/logs
      # make the REAL nginx log folders readable (shiny's are ok)
         # 755 because programs need to write to log folders and I don't care who reads them
chown ec2-user:ec2-user /var/log
chmod 755 /var/log
chown nginx:nginx /var/log/nginx
chmod 755 /var/log/nginx
      # symbolic links to actual log folders from /srv/logs
ln -s /var/log/nginx /srv/logs/var-log-nginx
ln -s /var/log/shiny-server /srv/logs/var-log-shiny
ln -s /var/log/ /srv/logs/var-log
      # and vice-versa
ln -s /srv/logs/ /var/log/srv-logs
ln -s /srv/logs/ /var/log/nginx/srv-logs
ln -s /srv/logs/ /var/log/shiny-server/srv-logs

   # Commands!
      # set up ~/bin folder
mkdir /home/ec2-user/bin
chown ec2-user:ec2-user /home/ec2-user/bin
chmod 755 /home/ec2-user/bin
      # this adds the file "add-cmds" to the ~/instance-setup-files folder
printf "printf 'top' > /home/ec2-user/bin/see.mem.space\nprintf 'df' > /home/ec2-user/bin/see.disk.space\nprintf 'sudo service nginx restart' > /home/ec2-user/bin/restart.nginx\nprintf 'sudo service nginx start' > /home/ec2-user/bin/start.nginx\nprintf 'sudo service nginx stop' > /home/ec2-user/bin/stop.nginx\nprintf 'sudo stop shiny-server\nsudo start shiny-server' > /home/ec2-user/bin/restart.shiny\nprintf 'sudo start shiny-server' > /home/ec2-user/bin/start.shiny\nprintf 'sudo stop shiny-server' > /home/ec2-user/bin/stop.shiny\nprintf 'sudo ps -ely | more' > /home/ec2-user/bin/whats.running\nprintf 'sudo chkconfig --list | more' > /home/ec2-user/bin/whats.autostart\nprintf 'sudo yum history' > /home/ec2-user/bin/yum.history\nprintf 'sudo yum list | more' > /home/ec2-user/bin/yum.installed\nprintf 'sudo yum search \0441' > /home/ec2-user/bin/yum.search\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-debug.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-debug\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-info.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-info\nprintf 'sudo stop shiny-server\nsudo cp /srv/configs/shiny-logs-warn.init /etc/init/shiny-server.conf\nsudo start shiny-server' > /home/ec2-user/bin/switch-shiny-logs-to-warn\nchown -R ec2-user:ec2-user /home/ec2-user/bin\nchmod -R 755 /home/ec2-user/bin  # change on all files in the directory\n" > /home/ec2-user/instance-setup-files/add-cmds
      # This runs the "add-cmds" file
chown ec2-user:ec2-user /home/ec2-user/instance-setup-files/add-cmds
chmod 755 /home/ec2-user/instance-setup-files/add-cmds
cd /home/ec2-user/instance-setup-files
./add-cmds
      # Additional commands for MySQL
printf 'sudo service mysqld restart' > /home/ec2-user/bin/restart.mysql
printf 'sudo service mysqld start' > /home/ec2-user/bin/start.mysql
printf 'sudo service mysqld stop' > /home/ec2-user/bin/stop.mysql
chown -R ec2-user:ec2-user /home/ec2-user/bin
chmod -R 755 /home/ec2-user/bin # change on all files in the directory

      # create links to these commands from /srv/configs
ln -s /home/ec2-user/bin/see.disk.space /srv/configs/see.disk.space
ln -s /home/ec2-user/bin/see.mem.space /srv/configs/see.mem.space
ln -s /home/ec2-user/bin/restart.nginx /srv/configs/restart.nginx
ln -s /home/ec2-user/bin/restart.shiny /srv/configs/restart.shiny
ln -s /home/ec2-user/bin/start.nginx /srv/configs/start.nginx
ln -s /home/ec2-user/bin/start.shiny /srv/configs/start.shiny
ln -s /home/ec2-user/bin/stop.nginx /srv/configs/stop.nginx
ln -s /home/ec2-user/bin/stop.shiny /srv/configs/stop.shiny
ln -s /home/ec2-user/bin/switch-shiny-logs-to-debug /srv/configs/switch-shiny-logs-to-debug
ln -s /home/ec2-user/bin/switch-shiny-logs-to-info /srv/configs/switch-shiny-logs-to-info
ln -s /home/ec2-user/bin/switch-shiny-logs-to-warn /srv/configs/switch-shiny-logs-to-warn
ln -s /home/ec2-user/bin/whats.autostart /srv/configs/whats.autostart
ln -s /home/ec2-user/bin/whats.running /srv/configs/whats.running
ln -s /home/ec2-user/bin/ /srv/configs/drop-d-here-and-chmod-755-it
ln -s /home/ec2-user/bin/restart.mysql /srv/configs/restart.mysql
ln -s /home/ec2-user/bin/start.mysql /srv/configs/start.mysql
ln -s /home/ec2-user/bin/stop.mysql /srv/configs/stop.mysql

As before, I couldn’t get the script to run on a 1 gigabyte instance, so I used the next size up, 2GB. After the script runs, you need to follow this process:

  • Use PuTTY to log in to the new instance
    • run yum.history to make sure all the programs installed correctly
    • run R, note version number, then enter installed.packages()[,1:3] and make sure all the R packages installed correctly. quit() R.
  • Use an FTP program to copy over non-redundant files in /srv/configs, /srv/nginx/, /srv/shiny-app-dir, and /srv/shiny-site-dir to the same folders on the new instance.
  • Run the following commands:
    • start.mysql
    • mysql_secure_installation
    • restart.nginx
    • restart.mysql
    • restart.shiny

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.