nginx – setup as reverse proxy

Previously to take down this wordpress site all you needed to do was hold down F5 for about 20 seconds then the site would take about 5 mins to recover.

There were a few factors causing this and quite a few different methods to solving the problem. WordPress itself is run on php / apache, apache has an evasive mod which can block certain IPs depending on the defined abusive behavior (typical DDOS attack). Since i like to run a few websites behind a single IP i looked at fixing the issue closer to the perimeter….

Enter nginx (engine x) as a reverse proxy, the site now typically caches all content and serves it straight out of memory. No longer does mysql / apache kill itself under high load on the backend…

nginx

You will need to create the nginx directories if they dont already exist. Check /var/log/nginx/error.log (default ubuntu) if any issues starting the service.

sudo aptitude install nginx
sudo service nginx start

The following added to http {}
(located in /etc/nginx/nginx.conf)

log_format cache '***$time_local '
'$remote_addr '
'$upstream_cache_status '
'Cache-Control: $upstream_http_cache_control '
'Expires: $upstream_http_expires '
'"$request" ($status) ';
access_log /var/log/nginx/access.log cache;
error_log /var/log/nginx/error.log;
server_names_hash_bucket_size 64;
proxy_cache_path /var/www/nginx_cache levels=1:2
keys_zone=one:10m
max_size=1g inactive=30m;
proxy_temp_path /var/www/nginx_temp;

the following added to location / {}
(located in /etc/nginx/sites-enabled/default)

proxy_pass http://sigtar;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_cache one;
proxy_cache_valid 200 302 304 10m;
proxy_cache_valid 301 1h;
proxy_cache_valid any 1m;
client_body_buffer_size 128k;
proxy_busy_buffers_size 64k;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header Set-Cookie;

Note if you have problems with wordpress redirect issues… check this post;

http://tommcfarlin.com/resolving-the-wordpress-multisite-redirect-loop/

Also confirm you have this line in http {}

server_names_hash_bucket_size 64;

Very basic load test, user load time is reasonably consistent as user count increases.
http://loadimpact.com/load-test/sigtar.com-ad07b7870a75c854a935752b0a032c53

load test

zfs – playing with various configs

If you dont have the disks available to build up a zpool and have a play with zfs you can actually just use files created with the mkfile command… The commands are exactly the same.

mkfile 64m disk1

mkfile 64m disk2

mkfile 64m disk3

mkfile 10m disk4

mkfile 100m disk5

mkfile 100m disk6

Now you can create a zpool using the above files… (i’m using raidz for this setup)

zpool create test raidz /fullpath/disk1 /fullpath/disk2 /fullpath/disk3

if you now want to expand this pool using another three drives (files) you can run this command

zpool add test raidz /fullpath/disk4 /fullpath/disk5 /fullpath/disk6

Check the status of the zpool

zpool status test

NAME STATE READ WRITE CKSUM

test ONLINE 0 0 0
raidz1 ONLINE 0 0 0
/export/home/daz/disk1 ONLINE 0 0 0
/export/home/daz/disk2 ONLINE 0 0 0
/export/home/daz/disk3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
/export/home/daz/disk4 ONLINE 0 0 0
/export/home/daz/disk5 ONLINE 0 0 0
/export/home/daz/disk6 ONLINE 0 0 0

errors: No known data errors


Now time to replace a drive (perhaps you wish to slowly increase your space) Note: all drives in that particular raidz pool need to be replaced with larger drives before the additional space is shown.

mkfile 200m disk7

mkfile 200m disk8

mkfile 200m disk9

Check the size of the zpool first;

zpool list test

NAME SIZE USED AVAIL CAP HEALTH ALTROOT
test 464M 349M 115M 75% ONLINE –

Now replace all of the smaller drives with the larger ones…

zpool replace test /export/home/daz/disk1 /export/home/daz/disk7
zpool replace test /export/home/daz/disk2 /export/home/daz/disk8
zpool replace test /export/home/daz/disk3 /export/home/daz/disk9

The space will show up if you bounce the box, i’ve heard that sometimes you may need to export and import but i’ve never had to do that.