pc engine – pfsense as router / firewall

download (1)Just built a pfsense router on  a pc engine to replace my aging 7390 fritzbox…. Very very awesome and fast!

Perfect for UFB (ultra fast broadband) here in New Zealand – FTTH has arrived :)


I’m currently running these services on it (2 x 1GHZ cpu, 4GB RAM, 16GB msata SSD);

  • dhcpd – DHCP server
  • miniupnpd – UPnP server
  • ntpd – NTP server
  • squid – Transparent Proxy & Reverse Proxy
  • snort – IDS (Intrusion Detection System) / IPS (Intrusion Prevention System)
  • ssd – SSH server
  • unbound – DNS Server
  • ipsec – IPsec VPN (site to site VPN)
  • openvpn – Open VPN (client VPN)

Check these links for some great advise…




Grab hardware here – http://www.pcengines.ch/apu.htm


Creating a .pem with the Private Key and Entire Trust Chain

download (3)You may find you install a SSL cert, but certain browsers show the connection as unsafe (i.e. mobile browser has issues but desktop browsers are happy)

This generally points to not having the cert chain correct. You can check your cert chain at sites like –




  1. Download your Intermediate and Primary Certificates.
  2. Open a text editor (such as notepad) and paste the entire body of each certificate into one text file in the following order:
    1. The Private Key – your_domain_name.key
    2. The Primary Certificate – your_domain_name.crt
    3. The Intermediate Certificate – gd_bundle_g2_g1.crt
    4. The Root Certificate – TrustedRoot.crt

    Make sure to include the beginning and end tags on each certificate. The result should look like this:

    (Your Private Key: your_domain_name.key)
    (Your Primary SSL certificate: your_domain_name.crt)
    (Your Intermediate certificate: DigiCertCA.crt)
    (Your Root certificate: TrustedRoot.crt)

    Save the combined file as your_domain_name.pem. The .pem file is now ready to use.

ERROR: No forward-proxy ports configured

I had the above issue after a squid upgrade and after changing from http_port 3128 transparent to http_port 3128 intercept.

Add “http_port 8080” line to squid.config to avoid this message, if you are not already using that port.

The changes in security require that a separate port be setup for forwarding proxy requests

nginx – and gzip

i only have vdsl at home and host my website via it… the following graphs shows when nginx started serving faster than the vdsl connection could handle (which is about 9Mbits)

ngnix with cache enabled, gzip enabled and set to 6


ngnix with cache enabled, gzip enabled and set to 9 (maximum)


ngnix with cache disabled, gzip enabled and set to 9 (maximum)



nginx – setup as reverse proxy


Previously to take down this wordpress site all you needed to do was hold down F5 for about 20 seconds then the site would take about 5 mins to recover.

There were a few factors causing this and quite a few different methods to solving the problem. WordPress itself is run on php / apache, apache has an evasive mod which can block certain IPs depending on the defined abusive behavior (typical DDOS attack). Since i like to run a few websites behind a single IP i looked at fixing the issue closer to the perimeter….

Enter nginx (engine x) as a reverse proxy, the site now typically caches all content and serves it straight out of memory. No longer does mysql / apache kill itself under high load on the backend…


You will need to create the nginx directories if they dont already exist. Check /var/log/nginx/error.log (default ubuntu) if any issues starting the service.

sudo aptitude install nginx
sudo service nginx start

The following added to http {}
(located in /etc/nginx/nginx.conf)

log_format cache '***$time_local '
'$remote_addr '
'$upstream_cache_status '
'Cache-Control: $upstream_http_cache_control '
'Expires: $upstream_http_expires '
'"$request" ($status) ';
access_log /var/log/nginx/access.log cache;
error_log /var/log/nginx/error.log;
server_names_hash_bucket_size 64;
proxy_cache_path /var/www/nginx_cache levels=1:2
max_size=1g inactive=30m;
proxy_temp_path /var/www/nginx_temp;

the following added to location / {}
(located in /etc/nginx/sites-enabled/default)

proxy_pass http://sigtar;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_cache one;
proxy_cache_valid 200 302 304 10m;
proxy_cache_valid 301 1h;
proxy_cache_valid any 1m;
client_body_buffer_size 128k;
proxy_busy_buffers_size 64k;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header Set-Cookie;

Note if you have problems with wordpress redirect issues… check this post;


Also confirm you have this line in http {}

server_names_hash_bucket_size 64;

Very basic load test, user load time is reasonably consistent as user count increases.

load test

use nmap to port scan / find free IPS

I use angry IP scanner in windows, linux has a good util for port scanning an ip range called nmap;

Grab nmap from usual repositories

sudo nmap -sP (will show hosts up and resolve mac addresses to vendors)

sudo nmap -v -sT (will show hosts up and the various open ports)

sudo nmap -v -sT (will show particular host and the various open ports)

Starting Nmap 6.00 ( http://nmap.org ) at 2013-07-12 11:48 NZST
Initiating ARP Ping Scan at 11:48
Scanning [1 port]
Completed ARP Ping Scan at 11:48, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 11:48
Completed Parallel DNS resolution of 1 host. at 11:48, 0.00s elapsed
Initiating Connect Scan at 11:48
Scanning win2012-veeam.fritz.box ( [1000 ports]
Discovered open port 111/tcp on
Discovered open port 445/tcp on
Discovered open port 3389/tcp on
Discovered open port 135/tcp on
Discovered open port 139/tcp on
Discovered open port 49155/tcp on
Discovered open port 1063/tcp on
Discovered open port 2049/tcp on
Discovered open port 3260/tcp on
Completed Connect Scan at 11:48, 4.21s elapsed (1000 total ports)
Nmap scan report for win2012-veeam.fritz.box (
Host is up (0.00059s latency).
Not shown: 991 filtered ports
111/tcp open rpcbind
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
1063/tcp open kyoceranetdev
2049/tcp open nfs
3260/tcp open iscsi
3389/tcp open ms-wbt-server
49155/tcp open unknown
MAC Address: 00:50:56:9C:68:FA (VMware)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 4.25 seconds
Raw packets sent: 1 (28B) | Rcvd: 1 (28B)

squid – reverse proxy

Great guide here – http://www.classhelper.org/articles/reverse-proxy-server-squid-debian/installing-squid-proxy-server.shtml

I’ve modified slightly for Ubuntu, but almost identical.

apt-get install squid
cp /etc/squid3/squid.conf /etc/squid3/squid.bak
rm /etc/squid3/squid.conf
nano /etc/squid/squid.conf

http_port 80 defaultsite=www.yoursite.com vhost
forwarded_for on
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
cache_peer parent 80 0 no-query no-digest originserver name=web1
acl sites_web1 dstdomain www.yoursite.com yoursite.com
cache_peer_access web1 allow sites_web1
cache_peer parent 80 0 no-query no-digest originserver name=web2
acl sites_web2 dstdomain www.anothersite.com anothersite.com
cache_peer_access web2 allow sites_web2
acl all src
acl manager proto cache_object
acl localhost src
acl to_localhost dst
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl Safe_ports port 1025-65535 # unregistered ports
http_access allow sites_web1
http_access allow sites_web2
http_access allow manager all
http_access allow manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
#http_access deny all
access_log /var/log/squid3/access.log
cache_mgr support@yoursite.com


Modify your hosts file to include web1 and web2

nano /etc/hosts web1 web2

Another good article about benefits of reverse proxy


4k sector hard drives and zfs

I hit this as a problem recently. One of my disks died in my raidz so i ran down to the store and grabbed me a replacement WD10EARS (Western Digital 1Tb Green) drive.


The one thing the store didn’t mention to me is the new 4K cluster sizing on the drive. I guess they assume most people run windows (though the issues are also present in XP). See these posts…


Unfortunately they really don’t work as well as you’d like in a server :(

  • They come with a 5 second head spin down setting that causes them to park their heads if they have been left idle for more than 5 seconds. As it takes a second or two to spin back up this can result in a very laggy experience during interactive sessions.
  • They do not have NCQ or any form of command queing/optimisation. This means that (on FreeBSD at least) you are stuck in the LOOK elevator. In particular this was noticed when doing sequential read & write (think dump|restore tar|untar etc) and interactive tasks simultaneously
  • They really suck with FreeBSD and ZFS…


While the other 512-byte sector HDDs were reading/writing at 30MB/s sustained, this EARS model did not exceeded the 1MB/s barrier.

I know for sure that this is related to the 512-byte sector firmware emulation, because the disk works perfectly well if I partition it in a 4k-sector alignment.

The thing is that even in that way, using it in a ZFS RAIDZ configuration the performance is very poor because RAIDZ uses a dynamic stripe size.

The bottom line here is that folks like me, that use different versions of Unix, need the firmware to present the disk as a 4K-sector disk to unleash the full potential of the technology. The OS is already prepared to support that sector size, no need for emulation here.


Some preliminary testing that I have done…the WD20EARS (2TB advanced format drives) actually presents emulated 512byte sectors to the host o/s.

The drive documentation indicates that jumpers 7-8 should be enabled if the o/s does not support advanced format drives – the drive still present 512 bytes sectors.

I have attempted to raise a support ticket querying this, and how one can disable 512byte sector emulation in the drive (perhaps through a firmware upgrade) but I have not received any response to date.

Hopefully is enough people raise support tickets, WD may release firmware that allows the drive to natively present 4k blocks. Other doco indicates several other jumper combinations – all do not seem to make the drive present 4k byte blocks.

Perhaps someone internal to sun that has a relationship with WD may be able to shed some light on this? It would be fantastic to find out that I was just doing something wrong -> then I can get the drives to be seen on 32bit systems (ie – our embedded kit for osol, velitium)

Tested using b133 (64bit intel).

Try to avoid the green drives in ZFS for now. Remember to do your research before you buy a bunch of disks. I was caught off guard by this small change (works fine in win7 etc) which kills performance in ZFS. Ouch.