nginx – setup as reverse proxy

 

Previously to take down this wordpress site all you needed to do was hold down F5 for about 20 seconds then the site would take about 5 mins to recover.

There were a few factors causing this and quite a few different methods to solving the problem. WordPress itself is run on php / apache, apache has an evasive mod which can block certain IPs depending on the defined abusive behavior (typical DDOS attack). Since i like to run a few websites behind a single IP i looked at fixing the issue closer to the perimeter….

Enter nginx (engine x) as a reverse proxy, the site now typically caches all content and serves it straight out of memory. No longer does mysql / apache kill itself under high load on the backend…

nginx

You will need to create the nginx directories if they dont already exist. Check /var/log/nginx/error.log (default ubuntu) if any issues starting the service.

sudo aptitude install nginx
sudo service nginx start

The following added to http {}
(located in /etc/nginx/nginx.conf)


log_format cache '***$time_local '
'$remote_addr '
'$upstream_cache_status '
'Cache-Control: $upstream_http_cache_control '
'Expires: $upstream_http_expires '
'"$request" ($status) ';
access_log /var/log/nginx/access.log cache;
error_log /var/log/nginx/error.log;
server_names_hash_bucket_size 64;
proxy_cache_path /var/www/nginx_cache levels=1:2
keys_zone=one:10m
max_size=1g inactive=30m;
proxy_temp_path /var/www/nginx_temp;

the following added to location / {}
(located in /etc/nginx/sites-enabled/default)


proxy_pass http://sigtar;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_cache one;
proxy_cache_valid 200 302 304 10m;
proxy_cache_valid 301 1h;
proxy_cache_valid any 1m;
client_body_buffer_size 128k;
proxy_busy_buffers_size 64k;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header Set-Cookie;

Note if you have problems with wordpress redirect issues… check this post;

http://tommcfarlin.com/resolving-the-wordpress-multisite-redirect-loop/

Also confirm you have this line in http {}

server_names_hash_bucket_size 64;

Very basic load test, user load time is reasonably consistent as user count increases.
http://loadimpact.com/load-test/sigtar.com-ad07b7870a75c854a935752b0a032c53

load test

use nmap to port scan / find free IPS

I use angry IP scanner in windows, linux has a good util for port scanning an ip range called nmap;

Grab nmap from usual repositories

sudo nmap -sP 192.168.9.0/24 (will show hosts up and resolve mac addresses to vendors)

sudo nmap -v -sT 192.168.9.0/24 (will show hosts up and the various open ports)

sudo nmap -v -sT 192.168.9.104 (will show particular host and the various open ports)

Starting Nmap 6.00 ( http://nmap.org ) at 2013-07-12 11:48 NZST
Initiating ARP Ping Scan at 11:48
Scanning 192.168.9.104 [1 port]
Completed ARP Ping Scan at 11:48, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 11:48
Completed Parallel DNS resolution of 1 host. at 11:48, 0.00s elapsed
Initiating Connect Scan at 11:48
Scanning win2012-veeam.fritz.box (192.168.9.104) [1000 ports]
Discovered open port 111/tcp on 192.168.9.104
Discovered open port 445/tcp on 192.168.9.104
Discovered open port 3389/tcp on 192.168.9.104
Discovered open port 135/tcp on 192.168.9.104
Discovered open port 139/tcp on 192.168.9.104
Discovered open port 49155/tcp on 192.168.9.104
Discovered open port 1063/tcp on 192.168.9.104
Discovered open port 2049/tcp on 192.168.9.104
Discovered open port 3260/tcp on 192.168.9.104
Completed Connect Scan at 11:48, 4.21s elapsed (1000 total ports)
Nmap scan report for win2012-veeam.fritz.box (192.168.9.104)
Host is up (0.00059s latency).
Not shown: 991 filtered ports
PORT STATE SERVICE
111/tcp open rpcbind
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
1063/tcp open kyoceranetdev
2049/tcp open nfs
3260/tcp open iscsi
3389/tcp open ms-wbt-server
49155/tcp open unknown
MAC Address: 00:50:56:9C:68:FA (VMware)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 4.25 seconds
Raw packets sent: 1 (28B) | Rcvd: 1 (28B)

Linux – install vmware tools onto guest

fire up the vm, then run the following after initiating a vmware tools install…

mount /dev/cdrom /mnt/cdrom
cd /tmp
tar zxf /mnt/cdrom/VMwareTools-x.x.x.gz
cd vmware-tools-disstrib
./vmware-install.pl

Then just follow the prompts through to the end.

If your running fedora or similar make sure your’ve got gcc and kernel headers…. (you’ll probably have to update kernel too)

yum update
shutdown -r now
yum install -y gcc make kernel-devel perl

Ubuntu 12.x

apt-get install open-vm-tools

some notes from fedora 13…

Did you also copy the missing/misplaced include file?

(Having just updated the kernel I am getting the original messages, so have copied them below as I workaround the problem)

= = = First I get:

What is the location of the directory of C header files that match your running
kernel? [/usr/src/linux/include] /usr/src/kernels/2.6.33.5-112.fc13.x86_64/include

The directory of kernel headers (version @@VMWARE@@ UTS_RELEASE) does not match
your running kernel (version 2.6.33.5-112.fc13.x86_64). Even if the module
were to compile successfully, it would not load into the running kernel.

= = = Then over in another session at
/usr/src/kernels/2.6.33.5-112.fc13.x86_64/include

[Tom@tlsf13a include]$ find . -iname ‘*relea*’
./config/kernel.release
./generated/utsrelease.h
[Tom@tlsf13a include]$ sudo cp -p generated/utsrelease.h linux/

= = = Then back in first session:

What is the location of the directory of C header files that match your running
kernel? [/usr/src/linux/include] /usr/src/kernels/2.6.33.5-112.fc13.x86_64/include

Extracting the sources of the vmmemctl module.

= = = and the vmware-config-tools.pl runs ….
(well, all but vmci builds … :-/ )

VirtualBox – crashing / freezing

I’ve had some problems since my upgrade to virtualbox 2.2.0 on OpenSolaris. After some time all of my linux boxes seem to just die. The virtual machine just stops responding. Strangely there was no problem with my windows vms after the update.

From what i can tell it looks like the upgrade turned off “IO APIC” – this is the bit that seemed to cause the problem. Re-enabling this on all of my linux boxes seems to have fixed the problem. I’ll continue testing for another week and update this post if any problems re-occur.

Updated : 01/09/2009

Here is a bit more on IO APIC from the virtualbox wiki…  (from a windows perspective)
http://www.virtualbox.org/wiki/Migrate_Windows

The hardware dependent portion of the Windows kernel is dubbed “Hardware Abstraction Layer” (HAL). While hardware vendor specific HALs have become very rare, there are still a number of HALs shipped by Microsoft. Here are the most common HALs (for more information, refer to this article: http://support.microsoft.com/kb/309283):

Hal.dll (Standard PC)
Halacpi.dll (ACPI HAL)
Halaacpi.dll (ACPI HAL with IO APIC)

If you perform a Windows installation with default settings in VirtualBox, Halacpi.dll will be chosen as VirtualBox enables ACPI by default but disables the IO APIC by default. A standard installation on a modern physical PC or VMware will usually result in Halaacpi.dll being chosen as most systems nowadays have an IO APIC and VMware chose to virtualize it by default (VirtualBox disables the IO APIC because it is more expensive to virtualize than a standard PIC). So as a first step, you either have to enable IO APIC support in VirtualBox or replace the HAL. Replacing the HAL can be done by booting the VM from the Windows CD and performing a repair installation.

Updated : 5/09/2009

I’ve had even more problems with opensolaris crashing completely after upgrading to the newer versions of virtualbox (3.0.4), and have since reverted back to 2.2.0 which has fixed alot of the hanging issues i have encountered

Linux – Add DHCP and DNS

How to install DHCP and DNS on Fedora / Centos / Redhat box;

yum -y install dhcp.i386 bind.i386

Set services to auto start using ntsysv — services are called “dhcpd” and “named”.

How to configure dhcp;

nano /etc/dhcpd.conf — configure as per sample provided. The sample is usually located in the same directory.
service dhcpd restart – should start without problem if done correctly

Note: dhcp range has to be on the same network as your adapter. You can run dhcpd from the command line to diagnose any issues.

How to configure bind (DNS);

I usually just forward DNS requests to my internal router. To do this just edit /etc/resolve.conf and place the following line in the file;

nameserver 192.168.9.1

Use your ip above then all should be good. Test by pinging a domain like www.yahoo.com etc… You should get an ip back even if you don’t get successful pings (due to firewall etc). nslookup is also another quick and easy way to check DNS requests.

Linux – Installing VirtualBox additions

The following works on CentOs / Fedora / redhat etc… 

Logon to your distro. The packages you need are the kernel-devel.i686, kernel.i686 (if not the latest), and gcc.i386 (complier)

yum -y install kernel.i686 gcc.i386

reboot as the kernel needs to load, then logon again install the kernel dev; 

yum -y install kernel-devel.i686

Now you want to present the virtualbox additions to the cdrom drive on your distro. Click “install additions” on the virtualbox guest console. 

then back on the box mount the cdrom to a directory (make one if you dont have one avail) 

mkdir /mnt/cdrom

mount /dev/cdrom /mnt/cdrom 

Now go to the newly mounted cdrom drive and find the installation;

cd /mnt/cdrom 

Execute the installer, and if all goes well reboot. Services should auto start and sync your time etc. 

./VBoxLinuxAdditions-x86.run

reboot

logon and check your time; 

date

Done.

Linux – crontab

Time to schedule some tasks!

First you need to make sure your in the /etc/cron.d/cron.allow file. If you are not SU to root and add yourself into it.

Now to create your new crontab file;

crontab -e

You are in vi in your newly created crontab file (note if you do not have access to do this it will say so)

Now, i’m no expert at using vi, but if you press i you will go into insert mode. Do this, then type the cron job details. A typical cron job will look like this

0 4 * * * /export/home/user/backup.sh

so there are five slots before the command you want to run. They are: min, hour, day, month, day-week. So my example above will run at 4am every day. See below for more detail.

once you have entered the line press ESC then type :wq to save and quit.

To confirm the job has been saved successfully in cron type this command;

crontab -l  this will show you your current scheduled tasks, it should output the above job.

In OpenSolaris the crontab files are located at /var/spool/cron/crontabs/ they are named after the specific user. Sometimes its easier to go there than to use vi, but make sure your permissions are all set correctly before editing any system files.

In some distros there are folders like /etc/cron.daily/ which run any scripts inside them on a schedule matching the folder name.

This is a bit more on the format of the 5 timings within the crontab file;

*     *   *   *    *  command to be executed

|     |     |     |     |
|     |     |     |     +—– day of week (0 – 6) (Sunday=0)
|     |     |     +——- month (1 – 12)
|     |     +——— day of month (1 – 31)
|     +———– hour (0 – 23)
+————- min (0 – 59)

Update

@reboot is also another very handy cron modifier. I use the following quite a lot…
@reboot /bin/sleep 600 ; /path/to/your/your_program@reboot /bin/sleep 600 ; /path/to/your/your_program

Linux – Hard Drive Performance

Or lack of.

You must insure that your drives are using the sata driver (if they are sata disks), quickest way to check is that they will be called sda sdb etc…. and not hda hdb, as this is the default IDE driver and will slow down the performance majorly.

How to check performance;

hdparm -tT /dev/hda

Modern drives should be getting over 50MB/s easily. If your getting about 5MB/s you have the problem.

How to Fix;

I had to change the sata mode to Enhanced in the bios and disable any of the on-board IDE controllers. When rebooting all your device names will change, and you will need to edit the /etc/fstab file as appropriate.

Apparently this is due to a conflict in the drivers, and them confusing your SATA drives as standard IDE disks.