So, how do you actually setup a web service that isn't bound by the performance of single machine? Like... how do you really do it? Not with docker, not with the cloud, not with automation. How do you actually setup a webserver? It starts with knowing what most sites are. The vast majority of websites are really simple things. They are some code, poorly written, in a language called PHP that reads and writes to a MySQL/MariaDB database. The code is formatted in a pretty way by CSS, and that's about it. The sender of the data is also the receiver of the request. This is a webserver. The two most common are NGiNX and Apache.
So far, we have four components: webserver, code, code interpreter, database.
The thing is, putting all of these components on a single server isn't usually a good idea. Even if the server is very powerful, that server will pay a penalty in the number of context switches that the CPU is required to do to keep up with everything. It's generally advisable to use multiple servers. Those with the most CPU power and the most RAM should be the PHP interpeters. The one with the most and fastest disk storage should be the one hosting the code. The one that is high CPU and decent disk storage and RAM should be the database host, and the one that is puny in all counts should host the webservice. The main requirement for the web service is just the network interfaces. Those need to be amazing on your webserver.
The exception to the above is if you're intending to use caching in your webservice. In that case, you will also want a very fast but small storage device. This is where Intel Optane or high end PCIe NVME storage would come into play.
Specifically, what we really want to be able to do is arbitrarily add more PHP FPM hosts to churn through that crappily written PHP code. You could also replace the MySQL/MariaDB server with something like TiDB to get more performance out of your database. For most people, this should not be problem however. A properly configured MySQL server can handle quite a bit of traffic.
For our purposes I will assume that the NGiNX server is 10.0.0.30, the PHP nodes are 10.0.0.51..53, the storage server is 10.0.0.40, and the database server is 10.0.0.20. As the image there suggests, we will start with NGiNX.
sudo apt install nginx -y
sudo vim /etc/nginx/sites-available/example.com.conf
If you do not know how to use vim, I have a guide available for that too. Within this file, we need to add some basic configuration information.
server {
listen 80;
listen [::]:80;
server_name example.com www.example.com;
root /var/www/example.com;
location ~ /.well-known { default_type text/plain; allow all; }
location ^~ / { return 301 https://$host$request_uri; }
access_log off; error_log off; log_not_found off;
}
So, the first line just tells NGiNX that we are dealing with a virtual host. The next two lines are saying that we should listen on all interfaces at port 80, which is the common HTTP traffic port. The server_name line is just telling NGiNX what the domain(s) is. The line that starts root is setting the document root, which is the place NGiNX will look for files for this website. The first location line is telling NGiNX that everyone should be able to access the /var/www/example.com/.well-known directory, and that that directory is holding plain text content. The second location line is telling NGiNX to redirect all requests for HTTP to HTTPS. Finally, on the last line, we turn off logging for the HTTP virtual host, because we will be using SSL/TLS and we don't care to have a log of thousands of redirects.
So, having saved and exited vim, we can now test and reload the configuration.
sudo ln -s /etc/nginx/sites-available/example.com.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo nginx -s reload
Now, before we can make the TLS configuration, we need to do a few different things. First, let's make sure we can reach the PHP servers. On the PHP servers, you should go ahead and install PHP FPM.
sudo apt install php-fpm -y
Then back on the NGiNX server you should do something like the following (change the IP address to your backend network's address for the FPM servers in question).
telnet 10.0.0.51 9000
Then, we need to add those servers to the NGiNX configuration at /etc/nginx/sites-available/
upstream fastcgis {
server 10.0.0.51:9000;
server 10.0.0.52:9000;
server 10.0.0.53:9000;
}
Now, on your storage server, you need to install NFS and set it up.
sudo apt install nfs-kernel-server -y
Then you need to edit the /etc/exports file.
/mnt/yourNfsDir 10.0.0.51(rw,sync,no_subtree_check)
/mnt/yourNfsDir 10.0.0.52(rw,sync,no_subtree_check)
/mnt/yourNfsDir 10.0.0.53(rw,sync,no_subtree_check)
/mnt/yourNfsDir 10.0.0.30(rw,sync,no_subtree_check)
Then you need to publish the exports.
sudo exportfs -a && sudo systemctl restart nfs-kernel-server
Then, on each of your PHP servers and the NGiNX server, you need to add the NFS share in /etc/fstab.
10.0.0.40:/mnt/yourNfsDir /var/www/example.com nfs auto,nofail,noatime,nolock,intr,tcp 0 0
If you added the NFS storage correctly, you should be able to go to any of your PHP servers and do touch /var/www/example.com/test and the see that new file on your NGiNX server with ls /var/www/example.com/. Next, on each of the PHP servers, you need to make sure that the FPM process is configured in /etc/php/7.4/fpm/pool.d/www.conf to look for connections on the backend IP at port 9000.
[www]
user = www-data
group = www-data
listen = 10.0.0.51:9000
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 2048
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 20
pm.max_requests = 0
listen.backlog = -1
request_terminate_timeout = 10s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = no
env[HOSTNAME] = $HOSTNAME
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
At this point, we have the shared storage setup, FPM servers set up, and we have a disable NGiNX configuration. We need to get a TLS certificate. So, on your NGiNX server you need to install and run letsencrypt (this assumes that DNS is already set).
sudo apt install letsencrypt -y
Then, we can fetch the certificate.
sudo certbot --certonly --webroot -w /var/www/example.com -d example.com -d www.example.com --dry-run
Assuming that this completes successfully, you can do the following.
sudo certbot --certonly --webroot -w /var/www/example.com -d example.com -d www.example.com
Then, you need to make sure that you have a cronjob in place to renew that certificate, you can add this via sudo crontab -e.
0 0 * * * certbot renew; nginx -t && nginx -s reload
This makes it possible for us to add the following configuration to our previously created virtual host configuration in /etc/nginx/sites-available/example.com.conf.
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name example.com www.example.com;
root /var/www/example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM;
index index.html index.htm index.php;
charset utf-8;
location ~ /\.(?!well-known).* { deny all; }
location ~* /favicon.ico { access_log off; log_not_found off; }
location ~* /robots.txt { access_log off; log_not_found off; }
location ~* \.(eot|ttf|woff|woff2)$ { add_header Access-Control-Allow-Origin *; }
location ~* \.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf|html)$ { expires 2d; add_header Cache-Control "public, no-transform"; }
error_page 404 /index.php;
location / { try_files $uri $uri/ /index.php?$query_string; }
location ~ \.php$ {
fastcgi_pass fastcgis;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_buffers 256 16k;
fastcgi_buffer_size 128k;
fastcgi_connect_timeout 5s;
fastcgi_send_timeout 10s;
fastcgi_read_timeout 10s;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
}
log_not_found off;
access_log /var/log/nginx/example.com-access.log;
error_log /var/log/nginx/example.com-error.log warn;
}
Assuming that everything was done correctly, you should now be able to to check and reload the configuration again.
sudo nginx -t && sudo nginx -s reload
We now need to setup the database server. So, connect to it, and install MariaDB.
sudo apt install mariadb-server -y
We then need to run the secure setup command.
mysql_secure_installation
Remove anonymous users? [Y/n] Y
Disallow root login remotely? [Y/n] Y
Remove test database and access to it? [Y/n] Y
Reload privilege tables now? [Y/n] Y
You should then make sure that you can access the MariaDB server from your PHP FPM nodes.
telnet 10.0.0.20 3306
You will then want to make certain that you have a decent default configuration for the database server in /etc/mysql/mariadb.conf.d/50-server.conf. For most people, you will start with a default configuration, and add two lines and alter one.
bind = 10.0.0.20
skip-external-locking
skip-name-resolve
You then need to upload your PHP content to your storage server, and import your DB. You then can start your performance tuning on everything. This tuning will depend upon what the code is and how well it's running, and what resources you're running out of with mostly default configurations. Then, you can add more PHP nodes as needed, or take some away to save money.