My last post included an Amazon Lightsail launch script to set up an instance with a LEMRS (Linux / EngineX (nginx) / MySQL / R / Shiny-Server) stack. This time we’re going transfer files into the instance, including new configuration files for nginx and shiny-server, start both of them up, and see how they work.
Before we start, however, notice that next to the Create instance button we’ve been using in Amazon Lightsail, there’s another button labeled Create static IP. Static IP numbers are used to link together computing resources and domain names. So if you have a domain name, you can simplify things for yourself by clicking that button to get a static IP, connecting it to your Amazon instance, then using your domain name registrar’s DNS records to point your domain name or a new subdomain (eg, dev.domainname.com) to your static IP.
Next, I need to admit that you can create and edit files directly in the terminal window using your favorite Linux editor. If you don’t have a favorite, I suggest you try nano, which is already installed in Amazon Linux. To look at or edit an existing file, use nano filename; for a new file, it’s just nano. You use your control and arrow keys to interact with nano. There’s a menu of options visible at the bottom of the nano window, so it’s pretty easy to use. Still, without standard copy-paste commands, which no Linux editor has (in Linux, control-c kills the current terminal activity!), I can’t imagine using nano or any other Linux editor for anything real.
For me the best option should be to use RStudio to create and edit files, but there’s one gotcha because I’m using a Microsoft system – line endings. On Windows systems, editors like RStudio typically end each line with <CR><LF> (control-M + control-N), while on Linux and on the newer Mac systems based on Linux, the line-end character is just <LF> (control-N). This gives birth to litters of mysterious bugs. However, RStudio projects have a setting for line endings at Tools / Project Options… / Code Editing / Line ending conversion that I’m trying to use to terminate this problem by setting it to Posix (LF). Now I just need to remember to use that particular RStudio project when creating and editing configuration files destined for Linux.
Once you’ve decided to create and edit files on your own computer, you need to locate an FTP (File Transfer Protocol) client that supports logging in with ssh keys rather than with a password. The one I’ve been using is called FileZilla. To connect to your Amazon Lightsail instance, you enter the Host – use either the IP number of the instance (which will be your static IP if you’ve linked them) or the domain name you’ve set up; the User – always ec2-user; and the Key file – which is the ssh key file you used to set up your instance. Since I’m using RStudio to deal with line endings, I also set its Transfer / Transfer type menu to Binary, which means FileZilla shouldn’t try to do any conversions on its own.
The LEMRS file system was designed to support future me, who will forget most of what he knows right now about Linux. Today’s me decided future me should almost never have to leave the /srv folder. Here’s the overall plan for that folder’s structure:
/srv
/configs
nginx.conf # the nginx configuration file
shiny-server.conf # the shiny-server configuration
...additional shiny configuration options
...available commands with descriptive names
/logs
nginx-access.log
nginx-error.log
shiny-access.log
...logs for individual shiny applications
/nginx
/css # folders for specific kinds of files
/images # nginx will take care of all static
/js # files (the ones that don't change)
/pdf
404.html # file not found error
50x.html # server error
index.html # the site's home page
robots.txt # Google this one
/shiny-app-dir
app.R # the folder's startup file
...other files used by app.R
/shiny-site-dir
index.html # this folder's startup file
sample-apps # used by the default index.html
/shiny-app-name # folder with an app's files
/shiny-app-name2 # any number of these...
So inside /srv there’s a folder for configuration files (me talking to the servers), a folder for log files (the servers talking to me), and three root directories (/srv/nginx, /srv/shiny-app-dir, and /srv/shiny-site-dir). To understand how we can have three different root directories, let’s look at how URLs relate to server folder structures.
In the simplest case, the URL domain.com/xxx/yyy/home.html means I want the home.html file in the directory /srv/nginx/xxx/yyy. All static (unchanging / non-reactive) files come straight from nginx without bothering shiny-server. So images, for example, will have a URL like domain.com/images/xxx.jpg. and will be on the server at /srv/nginx/images/xxx.jpg. And so on for style sheets (css) and javascript files (js). You can have an entire static web site in /srv/nginx if you want.
So how do we form a URL that comes from shiny-server? For that we use a fake folder name in the URL. For example, how about the URL domain.com/app/xxx? Nginx gets the first shot at the URL. But there’s no folder in /srv/nginx called app. So (based on what we’ve told it to do via the configuration file) nginx will pass the URL to shiny-server.
When shiny-server gets the URL, it’s going to first look to see if the URL begins with domain.com/app. If so, it will use the /srv/shiny-app-dir folder, running the app found there and passing the rest of the URL to the app. This is the kind of setup we’d use with our multi-page, multi-user shiny web site.
If instead the URL begins with domain.com/site, shiny-server will use the /srv/shiny-site-dir root and look for a file or folder that matches the rest of the URL. If it finds one, it uses the file or the app in that folder. If it doesn’t find a match, it will return a 404 file-not-found error. This is the kind of setup we’d use with more traditional shiny projects where what’s important is the name of the folder. You can also put either static or reactive R Markdown files here.
The one problem with this structure – and the reason you don’t usually see web servers set up with the configuration and log folders inside the same folder as the server root – is that if you misconfigure one of the servers – that is, if you set the root as /srv rather than a folder inside /srv – it’s possible to make your configuration and log files public. On the other hand, if a server is misconfigured, neither the configuration or log files will be of much interest. This is a risk I’m willing to take to simplify things for future me.
So now let’s take a look at the actual configuration files. First the one for nginx. I have no idea what most of this means; it’s just copied from the default configuration file. The important parts are the two location sections in bold. The first one says to see if there’s a file matching the URL and, if not, to pass the URL to @shiny. The second location tells nginx where @shiny is. I’ve also used bold on a line where you should enter your own domain name or instance/static IP number.
user nginx;
worker_processes auto;
error_log /srv/logs/nginx-error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /srv/logs/nginx-access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80;
server_name www.domain.com domain.com;
location / {
root /srv/nginx/;
index index.html index.htm;
try_files $uri $uri/ @shiny;
}
location @shiny {
proxy_pass http://localhost:3838;
proxy_buffers 16 16K;
}
error_page 404 /404.html
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
(EDIT: After a few days of using this configuration, there were warning messages in my nginx-error.log that an upstream response is buffered to a temporary file. Upon Googling that message, I found a suggestion to add the proxy_buffers 16 16K; line to the configuration file, which increases the size of some memory buffers so that responses don’t have to be temporarily stored on disk.)
And here’s the shiny-server configuration file. Again there are two location sections. The first one is for URLs that begin domain.com/app and the second one is for URLs that begin domain.com/site.
run_as shiny;
access_log /srv/logs/shiny-access.log dev;
preserve_logs true;
server {
listen 3838;
location /app {
app_dir /srv/shiny-app-dir;
app_idle_timeout 86400; # 24 hr in sec
log_dir /srv/logs;
directory_index off;
sanitize_errors off;
}
location /site {
site_dir /srv/shiny-site-dir;
log_dir /srv/logs;
directory_index off;
sanitize_errors off;
}
}
I should emphasize that these configuration files are meant to be used for app development! On a production server, you’d want to make preserve_logs false; otherwise you’ll quickly fill your instance’s disk space with logs. And you’ll want to use sanitize_errors on; the current setting displays R errors to the app’s user.
After transferring these files into your instance (they go in /srv/configs and are named nginx.conf and shiny-server.conf), use the terminal to run the restart.nginx and restart.shiny added commands. Now, in a browser, enter the IP number of your instance or your domain name, if you’ve managed to get that set up. You should see the nginx default page again, or whatever you’ve put at /srv/nginx/index.html. Add /site to the URL and you should see shiny-server’s default start page, unless you’ve changed it. Add /app to the URL and you should get a 404 error unless you’ve put something in /srv/shiny-app-dir.
That should get you started. However, notice we haven’t yet installed, much less turned on or configured, a database engine like MySQL. That’s coming up, but first, let’s add the ability to send email from our server.