Supervisor & Nginx

Introduction

This guide talks about how to setup Python related projects in production. Before getting into details ensure the following:

1. Make sure you’ve configured Sentry into your application 1. Make sure you’ve configured Nagios on the server you’re deploying on

Doing application and server level logging and monitoring is important.

  1. Nginx is the load balancer, that a we’re using. Our main intention of using this is Reverse Proxy.
  2. NginxConfig.io is a great tool for making online Nginx configurations.
  3. Supervisord is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. In our case we’re using Supervisor for controlling Django and Flask applications.
  4. uWSGI is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C.

Flow of Request

  1. Web Request
  2. Nginx
  3. uWSGI
  4. Django/Flask

Assumptions

While writing this guide we’ve assumed:

  1. You’re deploying application on Ubuntu
  2. We’re using Python 2.7
  3. We also assume you’ve installed your DB of Choice
  4. You will be using ubuntu as username from which you will be running your application

Reference Supervisor & Nginx Configurations

default.conf for Nginx looks like follows

upstream my_webapp {
    server unix://tmp/my_webapp_uwsgi.sock;
}

server {
    listen 80;
    server_name my_webapp.com;
    root /home/ubuntu/Code/my_webapp;
    server_tokens off;
    add_header Server 'my_webapp.com';
    client_max_body_size 5m;

    location / {
        uwsgi_pass my_webapp;
        include /etc/nginx/uwsgi_params;
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        real_ip_recursive on;
        real_ip_header X-Forwarded-For;
        set_real_ip_from 0.0.0.0/0;

    }

    location /static {
        alias /home/ubuntu/Code/my_webapp/static/;
        expires 30d;
    }

}

Contents of nginx.conf looks like follows

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  19000;
}
worker_rlimit_nofile    20000;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"'
                    '$request_time $upstream_response_time $pipe';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    proxy_buffering on;
    proxy_buffers 8 64k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 64k;
    #  proxy_read_timeout 300s;

    open_file_cache          max=10000 inactive=5m;
    open_file_cache_valid    2m;
    open_file_cache_min_uses 1;
    open_file_cache_errors   on;
    #send_timeout 3;

    ###ADDED BY RANVIJAY NEXT 4
    proxy_connect_timeout   600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;
    sendfile_max_chunk 512k;

    gzip on;
    gzip_comp_level 2;
    gzip_http_version 1.0;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;
}

Contents of supervisord.conf looks like follows

[inet_http_server]
port=127.0.0.1:9001

[supervisord]
logfile=/var/log/supervisor/supervisord.log
logfile_maxbytes=20MB
logfile_backups=4
loglevel=debug
pidfile=/var/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:9001

[program:recharge]
directory = /home/ubuntu/Code/my_webapp
user = ubuntu
command = /home/ubuntu/Code/my_webapp/my_command
autostart=true
autorestart=true
startretries=3
stopsignal=TERM
stdout_logfile=/var/log/supervisor/my_webapp.stdout.log
stderr_logfile=/var/log/supervisor/my_webapp.stderr.log

Postgres Setup

Refer: Digital Ocean article on Postgres for more detail.

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

Check if installation is fine:

sudo -i -u postgres
psql

If you see a prompt everything is working fine at this point

MySQL Setup

Refer: Digital Ocean article on MySQL for more details

sudo apt-get update
sudo apt-get install mysql-server
mysql_secure_installation

Steps for Production Setup

Check if installation is fine:

systemctl status mysql.service
mysqladmin -p -u root version

Django Production Setup

Ensure all the dependencies are installed and make sure you run the application in development mode on port 8000

When running Django in production please ensure the following:

  1. Right DATABASE_NAME and other related settings are working fine
  2. DEBUG is set to False
# Update package listing
sudo apt-get update

# Install python dependencies
sudo apt-get install python-pip
sudo pip install virtualenv virtualenvwrapper
mkdir ~/Installs ~/Code
virtualenv ~/Installs/<project_name>
source ~/Installs/<project_name>/bin/activate

# Make sure to install supervisor and uwsgi
pip install supervisor uwsgi

cd ~/Code
git clone <project_repo>
cd ~/Code/<project_repo>/webapp
pip install -r requirements.txt

# Before running this make sure right DB strings are configured in your settings files

# Now run migrations
python manage.py migrate
python manage.py runserver 0.0.0.0:8000

At this point in time we will have the development server working on server.

Note:

  1. Django doesn’t serve static in production, so we need to make that working with Nginx.
  2. If you’re using Flask, only thing that really changes in listing above are last two steps. Instead of running manage.py command, you would run something along the lines python server.py

uWSGI setup using Supervisord

We need to create supervisord.conf contents of which look something like following

[inet_http_server]
port=127.0.0.1:9001

[supervisord]
logfile=/var/log/supervisor/supervisord.log
logfile_maxbytes=20MB
logfile_backups=4
loglevel=debug
pidfile=/var/run/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:9001

[program:my_webapp]
directory = /home/ubuntu/Code/my_webapp
user = ubuntu
command = /home/ubuntu/Code/my_webapp/my_command
autostart=true
autorestart=true
startretries=3
stopsignal=TERM
stdout_logfile=/var/log/supervisor/my_webapp.stdout.log
stderr_logfile=/var/log/supervisor/my_webapp.stderr.log

Note:

  1. You will need to create supervisor directory in /var/log
  2. Make sure you replace my_webapp with the values that are applicable
  3. You will also need to create log file using touch command. Eg:
    • sudo touch /var/log/supervisor/my_webapp.stdout.log
    • sudo touch /var/log/supervisor/my_webapp.stderr.log
  4. Make sure /etc/supervisor directory is created, if not create it using mkdir command. Ensure right username and group are applicable onto this directory

Once this is done:

  1. Make sure to copy files to /etc/supervisor
  2. Start supervisor using following command sudo supervisord -c /etc/supervisor/supervisord.conf

To check if your application has started use supevisorctl status to get listing of application and respective statues

Nginx Setup

sudo apt-get install nginx

Check if installation is fine:

systemctl status nginx

Update Nginx configurations

There are two files that we care about, which are usually located in /etc/nginx directory:

  1. nginx.conf: This is the main Nginx configuration file
  2. sites-enabled: This directory contains per site configuration file. By default it can be file called default
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  19000;
}
worker_rlimit_nofile    20000;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"'
                    '$request_time $upstream_response_time $pipe';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    proxy_buffering on;
    proxy_buffers 8 64k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 64k;

    open_file_cache          max=10000 inactive=5m;
    open_file_cache_valid    2m;
    open_file_cache_min_uses 1;
    open_file_cache_errors   on;

    proxy_connect_timeout   600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;
    sendfile_max_chunk 512k;

    gzip on;
    gzip_comp_level 2;
    gzip_http_version 1.0;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;
}

default.conf should look like

upstream my_webapp {
    server unix://tmp/my_webapp_uwsgi.sock;
}

server {
    listen 80;
    server_name my_webapp.com;
    root /home/ubuntu/Code/my_webapp;
    server_tokens off;
    add_header Server 'my_webapp.com';

    # Increase this value if you're uploading a file greater than this defined value
    client_max_body_size 5m;

    location / {
        uwsgi_pass my_webapp;
        include /etc/nginx/uwsgi_params;
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        real_ip_recursive on;
        real_ip_header X-Forwarded-For;
        set_real_ip_from 0.0.0.0/0;

    }

    location /static {
        alias /home/ubuntu/Code/my_webapp/static/;
        expires 30d;
    }

}

Note:

  1. Take note of /static section, you may need to re-map the directory based on where your JS and CSS assets are located
  2. Take note of client_max_body_size you may need to change this settings if you’re uploading a file that is larger in size
  3. Replace my_webapp with applicable values

Once both these changes are done:

  1. Ensure configuration is correct /etc/init.d/nginx configtest
  2. Restart nginx using /etc/init.d/nginx restart

At this point fireup the browser and check if everything is working as per expectation