Supervisor & Nginx¶
Introduction¶
This guide talks about how to setup Python related projects in production. Before getting into details ensure the following:
1. Make sure you’ve configured Sentry into your application 1. Make sure you’ve configured Nagios on the server you’re deploying on
Doing application and server level logging and monitoring is important.
Nginx is the load balancer, that a we’re using. Our main intention of using this is Reverse Proxy.
NginxConfig.io is a great tool for making online Nginx configurations.
Supervisord is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. In our case we’re using Supervisor for controlling Django and Flask applications.
uWSGI is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C.
Flow of Request¶
Web Request
Nginx
uWSGI
Django/Flask
Assumptions¶
While writing this guide we’ve assumed:
You’re deploying application on Ubuntu
We’re using Python 2.7
We also assume you’ve installed your DB of Choice
You will be using ubuntu as username from which you will be running your application
Reference Supervisor & Nginx Configurations¶
default.conf for Nginx looks like follows
upstream my_webapp {
server unix://tmp/my_webapp_uwsgi.sock;
}
server {
listen 80;
server_name my_webapp.com;
root /home/ubuntu/Code/my_webapp;
server_tokens off;
add_header Server 'my_webapp.com';
client_max_body_size 5m;
location / {
uwsgi_pass my_webapp;
include /etc/nginx/uwsgi_params;
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
real_ip_recursive on;
real_ip_header X-Forwarded-For;
set_real_ip_from 0.0.0.0/0;
}
location /static {
alias /home/ubuntu/Code/my_webapp/static/;
expires 30d;
}
}
Contents of nginx.conf looks like follows
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 19000;
}
worker_rlimit_nofile 20000;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'$request_time $upstream_response_time $pipe';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
proxy_buffering on;
proxy_buffers 8 64k;
proxy_buffer_size 16k;
proxy_busy_buffers_size 64k;
# proxy_read_timeout 300s;
open_file_cache max=10000 inactive=5m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
#send_timeout 3;
###ADDED BY RANVIJAY NEXT 4
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
sendfile_max_chunk 512k;
gzip on;
gzip_comp_level 2;
gzip_http_version 1.0;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
gzip_disable "MSIE [1-6].(?!.*SV1)";
gzip_vary on;
include /etc/nginx/conf.d/*.conf;
}
Contents of supervisord.conf looks like follows
[inet_http_server]
port=127.0.0.1:9001
[supervisord]
logfile=/var/log/supervisor/supervisord.log
logfile_maxbytes=20MB
logfile_backups=4
loglevel=debug
pidfile=/var/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=http://127.0.0.1:9001
[program:recharge]
directory = /home/ubuntu/Code/my_webapp
user = ubuntu
command = /home/ubuntu/Code/my_webapp/my_command
autostart=true
autorestart=true
startretries=3
stopsignal=TERM
stdout_logfile=/var/log/supervisor/my_webapp.stdout.log
stderr_logfile=/var/log/supervisor/my_webapp.stderr.log
Postgres Setup¶
Refer: Digital Ocean article on Postgres for more detail.
sudo apt-get update
sudo apt-get install postgresql postgresql-contrib
Check if installation is fine:
sudo -i -u postgres
psql
If you see a prompt everything is working fine at this point
MySQL Setup¶
Refer: Digital Ocean article on MySQL for more details
sudo apt-get update
sudo apt-get install mysql-server
mysql_secure_installation
Steps for Production Setup
Check if installation is fine:
systemctl status mysql.service
mysqladmin -p -u root version
Django Production Setup¶
Ensure all the dependencies are installed and make sure you run the application in development mode on port 8000
When running Django in production please ensure the following:
Right DATABASE_NAME and other related settings are working fine
DEBUG is set to False
# Update package listing
sudo apt-get update
# Install python dependencies
sudo apt-get install python-pip
sudo pip install virtualenv virtualenvwrapper
mkdir ~/Installs ~/Code
virtualenv ~/Installs/<project_name>
source ~/Installs/<project_name>/bin/activate
# Make sure to install supervisor and uwsgi
pip install supervisor uwsgi
cd ~/Code
git clone <project_repo>
cd ~/Code/<project_repo>/webapp
pip install -r requirements.txt
# Before running this make sure right DB strings are configured in your settings files
# Now run migrations
python manage.py migrate
python manage.py runserver 0.0.0.0:8000
At this point in time we will have the development server working on server.
Note:
Django doesn’t serve static in production, so we need to make that working with Nginx.
If you’re using Flask, only thing that really changes in listing above are last two steps. Instead of running manage.py command, you would run something along the lines python server.py
uWSGI setup using Supervisord¶
We need to create supervisord.conf contents of which look something like following
[inet_http_server]
port=127.0.0.1:9001
[supervisord]
logfile=/var/log/supervisor/supervisord.log
logfile_maxbytes=20MB
logfile_backups=4
loglevel=debug
pidfile=/var/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=http://127.0.0.1:9001
[program:my_webapp]
directory = /home/ubuntu/Code/my_webapp
user = ubuntu
command = /home/ubuntu/Code/my_webapp/my_command
autostart=true
autorestart=true
startretries=3
stopsignal=TERM
stdout_logfile=/var/log/supervisor/my_webapp.stdout.log
stderr_logfile=/var/log/supervisor/my_webapp.stderr.log
Note:
You will need to create supervisor directory in /var/log
Make sure you replace my_webapp with the values that are applicable
- You will also need to create log file using touch command. Eg:
sudo touch /var/log/supervisor/my_webapp.stdout.log
sudo touch /var/log/supervisor/my_webapp.stderr.log
Make sure /etc/supervisor directory is created, if not create it using mkdir command. Ensure right username and group are applicable onto this directory
Once this is done:
Make sure to copy files to /etc/supervisor
Start supervisor using following command sudo supervisord -c /etc/supervisor/supervisord.conf
To check if your application has started use supevisorctl status to get listing of application and respective statues
Nginx Setup¶
sudo apt-get install nginx
Check if installation is fine:
systemctl status nginx
Update Nginx configurations
There are two files that we care about, which are usually located in /etc/nginx directory:
nginx.conf: This is the main Nginx configuration file
sites-enabled: This directory contains per site configuration file. By default it can be file called default
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 19000;
}
worker_rlimit_nofile 20000;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'
'$request_time $upstream_response_time $pipe';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
proxy_buffering on;
proxy_buffers 8 64k;
proxy_buffer_size 16k;
proxy_busy_buffers_size 64k;
open_file_cache max=10000 inactive=5m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
sendfile_max_chunk 512k;
gzip on;
gzip_comp_level 2;
gzip_http_version 1.0;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
gzip_disable "MSIE [1-6].(?!.*SV1)";
gzip_vary on;
include /etc/nginx/conf.d/*.conf;
}
default.conf should look like
upstream my_webapp {
server unix://tmp/my_webapp_uwsgi.sock;
}
server {
listen 80;
server_name my_webapp.com;
root /home/ubuntu/Code/my_webapp;
server_tokens off;
add_header Server 'my_webapp.com';
# Increase this value if you're uploading a file greater than this defined value
client_max_body_size 5m;
location / {
uwsgi_pass my_webapp;
include /etc/nginx/uwsgi_params;
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
real_ip_recursive on;
real_ip_header X-Forwarded-For;
set_real_ip_from 0.0.0.0/0;
}
location /static {
alias /home/ubuntu/Code/my_webapp/static/;
expires 30d;
}
}
Note:
Take note of /static section, you may need to re-map the directory based on where your JS and CSS assets are located
Take note of client_max_body_size you may need to change this settings if you’re uploading a file that is larger in size
Replace my_webapp with applicable values
Once both these changes are done:
Ensure configuration is correct /etc/init.d/nginx configtest
Restart nginx using /etc/init.d/nginx restart
At this point fireup the browser and check if everything is working as per expectation
