Supervisor & Nginx ================== Introduction ````````````` This guide talks about how to setup Python related projects in production. Before getting into details ensure the following: 1. Make sure you've configured Sentry into your application 1. Make sure you've configured Nagios on the server you're deploying on Doing application and server level logging and monitoring is important. 1. `Nginx `_ is the load balancer, that a we're using. Our main intention of using this is `Reverse Proxy`. 2. `NginxConfig.io `_ is a great tool for making online Nginx configurations. 3. `Supervisord `_ is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. In our case we're using Supervisor for controlling Django and Flask applications. 4. `uWSGI `_ is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C. Flow of Request ``````````````` 1. Web Request 2. Nginx 3. uWSGI 4. Django/Flask Assumptions ```````````` While writing this guide we've assumed: 1. You're deploying application on Ubuntu 2. We're using Python 2.7 3. We also assume you've installed your DB of Choice 4. You will be using `ubuntu` as username from which you will be running your application Reference Supervisor & Nginx Configurations ````````````````````````````````````````````` `default.conf` for Nginx looks like follows .. code-block:: nginx upstream my_webapp { server unix://tmp/my_webapp_uwsgi.sock; } server { listen 80; server_name my_webapp.com; root /home/ubuntu/Code/my_webapp; server_tokens off; add_header Server 'my_webapp.com'; client_max_body_size 5m; location / { uwsgi_pass my_webapp; include /etc/nginx/uwsgi_params; proxy_pass_header Server; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; real_ip_recursive on; real_ip_header X-Forwarded-For; set_real_ip_from 0.0.0.0/0; } location /static { alias /home/ubuntu/Code/my_webapp/static/; expires 30d; } } Contents of `nginx.conf` looks like follows .. code-block:: nginx user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 19000; } worker_rlimit_nofile 20000; http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' '$request_time $upstream_response_time $pipe'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; proxy_buffering on; proxy_buffers 8 64k; proxy_buffer_size 16k; proxy_busy_buffers_size 64k; # proxy_read_timeout 300s; open_file_cache max=10000 inactive=5m; open_file_cache_valid 2m; open_file_cache_min_uses 1; open_file_cache_errors on; #send_timeout 3; ###ADDED BY RANVIJAY NEXT 4 proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; sendfile_max_chunk 512k; gzip on; gzip_comp_level 2; gzip_http_version 1.0; gzip_min_length 1100; gzip_buffers 16 8k; gzip_proxied any; gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon; gzip_disable "MSIE [1-6].(?!.*SV1)"; gzip_vary on; include /etc/nginx/conf.d/*.conf; } Contents of `supervisord.conf` looks like follows .. code-block:: ini [inet_http_server] port=127.0.0.1:9001 [supervisord] logfile=/var/log/supervisor/supervisord.log logfile_maxbytes=20MB logfile_backups=4 loglevel=debug pidfile=/var/run/supervisord.pid nodaemon=false minfds=1024 minprocs=200 [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=http://127.0.0.1:9001 [program:recharge] directory = /home/ubuntu/Code/my_webapp user = ubuntu command = /home/ubuntu/Code/my_webapp/my_command autostart=true autorestart=true startretries=3 stopsignal=TERM stdout_logfile=/var/log/supervisor/my_webapp.stdout.log stderr_logfile=/var/log/supervisor/my_webapp.stderr.log Postgres Setup `````````````` Refer: `Digital Ocean article on Postgres `_ for more detail. .. code-block:: sh sudo apt-get update sudo apt-get install postgresql postgresql-contrib Check if installation is fine: .. code-block:: sh sudo -i -u postgres psql If you see a prompt everything is working fine at this point MySQL Setup `````````````` Refer: `Digital Ocean article on MySQL `_ for more details .. code-block:: sh sudo apt-get update sudo apt-get install mysql-server mysql_secure_installation Steps for Production Setup Check if installation is fine: .. code-block:: sh systemctl status mysql.service mysqladmin -p -u root version Django Production Setup ```````````````````````` Ensure all the dependencies are installed and make sure you run the application in development mode on port `8000` When running Django in production please ensure the following: 1. Right `DATABASE_NAME` and other related settings are working fine 2. `DEBUG` is set to `False` .. code-block:: sh # Update package listing sudo apt-get update # Install python dependencies sudo apt-get install python-pip sudo pip install virtualenv virtualenvwrapper mkdir ~/Installs ~/Code virtualenv ~/Installs/ source ~/Installs//bin/activate # Make sure to install supervisor and uwsgi pip install supervisor uwsgi cd ~/Code git clone cd ~/Code//webapp pip install -r requirements.txt # Before running this make sure right DB strings are configured in your settings files # Now run migrations python manage.py migrate python manage.py runserver 0.0.0.0:8000 At this point in time we will have the development server working on server. Note: 1. Django doesn't serve static in production, so we need to make that working with Nginx. 2. If you're using Flask, only thing that really changes in listing above are last two steps. Instead of running manage.py command, you would run something along the lines `python server.py` uWSGI setup using Supervisord `````````````````````````````` We need to create `supervisord.conf` contents of which look something like following .. code-block:: ini [inet_http_server] port=127.0.0.1:9001 [supervisord] logfile=/var/log/supervisor/supervisord.log logfile_maxbytes=20MB logfile_backups=4 loglevel=debug pidfile=/var/run/supervisord.pid nodaemon=false minfds=1024 minprocs=200 [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=http://127.0.0.1:9001 [program:my_webapp] directory = /home/ubuntu/Code/my_webapp user = ubuntu command = /home/ubuntu/Code/my_webapp/my_command autostart=true autorestart=true startretries=3 stopsignal=TERM stdout_logfile=/var/log/supervisor/my_webapp.stdout.log stderr_logfile=/var/log/supervisor/my_webapp.stderr.log Note: 1. You will need to create `supervisor` directory in `/var/log` 2. Make sure you replace `my_webapp` with the values that are applicable 3. You will also need to create log file using `touch` command. Eg: * `sudo touch /var/log/supervisor/my_webapp.stdout.log` * `sudo touch /var/log/supervisor/my_webapp.stderr.log` 4. Make sure `/etc/supervisor` directory is created, if not create it using `mkdir` command. Ensure right username and group are applicable onto this directory Once this is done: 1. Make sure to copy files to `/etc/supervisor` 2. Start supervisor using following command `sudo supervisord -c /etc/supervisor/supervisord.conf` To check if your application has started use `supevisorctl status` to get listing of application and respective statues Nginx Setup ``````````` .. code-block:: sh sudo apt-get install nginx Check if installation is fine: .. code-block:: sh systemctl status nginx Update Nginx configurations There are two files that we care about, which are usually located in `/etc/nginx` directory: 1. `nginx.conf`: This is the main Nginx configuration file 2. `sites-enabled`: This directory contains per site configuration file. By default it can be file called `default` .. code-block:: nginx user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 19000; } worker_rlimit_nofile 20000; http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' '$request_time $upstream_response_time $pipe'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; proxy_buffering on; proxy_buffers 8 64k; proxy_buffer_size 16k; proxy_busy_buffers_size 64k; open_file_cache max=10000 inactive=5m; open_file_cache_valid 2m; open_file_cache_min_uses 1; open_file_cache_errors on; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; sendfile_max_chunk 512k; gzip on; gzip_comp_level 2; gzip_http_version 1.0; gzip_min_length 1100; gzip_buffers 16 8k; gzip_proxied any; gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon; gzip_disable "MSIE [1-6].(?!.*SV1)"; gzip_vary on; include /etc/nginx/conf.d/*.conf; } `default.conf` should look like .. code-block:: nginx upstream my_webapp { server unix://tmp/my_webapp_uwsgi.sock; } server { listen 80; server_name my_webapp.com; root /home/ubuntu/Code/my_webapp; server_tokens off; add_header Server 'my_webapp.com'; # Increase this value if you're uploading a file greater than this defined value client_max_body_size 5m; location / { uwsgi_pass my_webapp; include /etc/nginx/uwsgi_params; proxy_pass_header Server; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; real_ip_recursive on; real_ip_header X-Forwarded-For; set_real_ip_from 0.0.0.0/0; } location /static { alias /home/ubuntu/Code/my_webapp/static/; expires 30d; } } Note: 1. Take note of `/static` section, you may need to re-map the directory based on where your JS and CSS assets are located 2. Take note of `client_max_body_size` you may need to change this settings if you're uploading a file that is larger in size 3. Replace `my_webapp` with applicable values Once both these changes are done: 1. Ensure configuration is correct `/etc/init.d/nginx configtest` 2. Restart nginx using `/etc/init.d/nginx restart` At this point fireup the browser and check if everything is working as per expectation