Drupal 7 + HTTPS + Nginx + Varnish + Apache + Boost + APC + Securepages + Drupal

If you happen to develop large sites in Drupal, you might fall upon a case like this one, where different servers (namely at least one reverse proxy and one web server) interact, causing a series of chain reactions every time you change something.

It might be frustrating, at times, to try and boost a coordinated system like this, and end up getting your users frustrated because part of it doen’t work, when the rest (the part that *does* work) is super-fast. Come on, you deserve some praise!

Let’s explain a little bit about what requirements might lead to this system…

First of all, you want a website. You don’t want to play with php-fpm yet, so you decide to use Apache + mod-PHP. Great.

Then you want to boost the page interpretation time a little bit, so you decide to use APC. Nothing wrong there.
You now have Apache + PHP + APC + Drupal in line.

Then you want to make sure anonymous users get pre-generated pages to go even faster. You add the Boost module to the loop.
You now have Apache + Boost + PHP + APC + Drupal.

Then you want to speed up loading small static resources, like icons and stuff. You decide to add Varnish in front of the queue.
You now have Vanish + Apache + Boost + PHP + APC + Drupal.

And then comes the late requirement for HTTPS… Damn… Varnish doesn’t support HTTPS!? Man, and now what? Well… you can use Nginx to decipher HTTPS and handle things over to Varnish to follow in the normal queue.
You now have HTTPS + Nginx + Varnish + Apache + Boost + PHP + APC + Drupal.

And finally, your customer tells you that, because of the extra load (between 20% and 400%) generated by the ciphering to HTTPS, he only wants specific pages to be HTTPS. Apart from the mess in redirecting pages to HTTP when they shouldn’t be HTTPS (which can be done in Nginx’s config), you will also need to define rules to send HTTP pages to HTTPS when they should be secure. You can do that with Drupal’s Securepages module.
You finally have your (rather) complete schema of HTTPS + Nginx + Varnish + Apache + Boost + PHP + APC + Securepages + Drupal.

Here is a list of things to think about when doing this:

1. the Boost module had a bug in early 2012, which didn’t define the DRUPAL_UID cookie very consistently (see previous article on this blog). Because of that, users might loose their session “from time to time” (which is more than frustrating for both you and your customer)
2. in the specific case of requesting pages to be HTTPS without being logged in (which has the effect of not generating  Drupal connection cookie), the user will first get to Varnish, which will pass on to Apache, which will pass on to Drupal (I’m skipping a few steps here), which will send a HTTP Redirect to Varnish and Varnish will (cache it and) pass it to the user. The user (the user’s browser, in practice) then calls the same URL as HTTPS. Now Nginx takes over for deciphering HTTPS into HTTP before passing it on to Varnish. When Varnish rceives the call, whether it is cached or not, the return value it sends is a 301 Redirect, to the same URL as HTTPS. And so you now are in a loop.

Ref: Minimum RSA public key lengths

Just a reference to an interesting article on SSL and guidelines for the length of public keys in RSA before and after December 2013:

http://news.netcraft.com/archives/2012/09/10/minimum-rsa-public-key-lengths-guidelines-or-rules.html

Munin nginx_status fails on HTTPS

This must appear in one opportunity in 1000, but it happened to me, so I bet it might have just happened to you…

Munin is great, Nginx is great, and SSL is great, but when you mix all of them together, you might get some frustrating behaviour.

If you don’t know it already, you can test the results of a Munin plugin on Debian-based systems with the command

sudo munin-run [plugin]

For example, if your nginx_status graph in the Munin web interface is empty, you can try

sudo munin-run nginx_status

The name of the plugin will auto-complete with the TAB key, with any plugin present in /etc/munin/plugins/

In my case, the command gave the following result:

total.value U
reading.value U
writing.value U
waiting.value U

“U” in the result above is the equivalent of “Undefined” or “Unavailable”. Translation: the plugin can’t get its data.

If you’ve got Munin running only for HTTPS (the rest is handled by Varnish, for example), then the first thing is to make sure that the plugin is effectively querying an HTTPS URL managed by Munin, and that the URL, even if tested on the same machine, is the one that matches your SSL certificate. For example: https://www.example.com/nginx_status if your domain is example.com and you have the certificate for that domain. Make sure you have a block like this in your /etc/munin/plugin-conf.d/munin-node file (you can add it at the end):

[nginx*]
env.url https://www.example.com/nginx_status

Then reload your munin configuration with

sudo /etc/init.d/munin-node restart

Second, you want to make sure the URL nginx_status is actually managed by Nginx. To do this, check the section “location / { … }” in your nginx config file (/etc/nginx/sites-available/default, for example) and make sure there is another location block like this one:

location /nginx_status {
  stub_status on;

  access_log off;

  allow 127.0.0.1;

  deny all;
}

(change – or copy over – the “allow” property if needed).
In particular, it happens that you need to put the public IP in an “allow” tag in order for munin to work.
This will ensure the URL https://www.example.com/nginx_status actually answers something like this:

Active connections: 1
server accepts handled requests
8723 8723 10497
Reading: 0 Writing: 1 Waiting: 0

… Which you can test with lynx or links2 from the same server: lynx https://www.example.com/nginx_status

Now (we’re not finished yet) you might have all this running, and still an empty munin graph.
This is the most tricky part, because it’s difficult to get to that conclusion without hacking the code a little bit.

Without you noticing, the munin plugin might now be failing because you don’t have the IO::Socket::SSL library installed on your computer. I only realized that by adding a die() call in the Perl script of the plugin when getting the answer from Nginx, to see that it was still in encrypted form and contained a message saying I should install this module in order to make SSL readable by Munin.
Well, once you know this, it’s fairly easy to do:

sudo apt-get install libio-socket-ssl-perl

sudo /etc/init.d/munin-node restart

You can try it out with munin-run:

$ sudo munin-run nginx_status
total.value 1
reading.value 0
writing.value 1
waiting.value 0

Done! If this helped you, please share it.

Renew expired self-signed SSL certificate

For some reason, it might be very difficult to find information on how to renew a self-signed certificate. This is a nice (and short) explanation: http://linux.togaware.com/survivor/Renew_SSL.html.

Please note that a .pem file is in fact (as you can guess from the small guide) a combined .key and .crt.

In short and only for the purpose of not loosing this reference (as has happened many times before with sites referenced on this blog), here is the procedure (just adapt to your case, i.e. replace togaware.com and all locality details with yours). The “Common name” requested by the openssl command is the domain name. Leave top level domain name without prefix for multiple domains certificates. Note that the filenames (togaware.com.key, etc) do not have any importance apart that the extension (.pem, .key, .crt) might be helpful later on when wondering which file does what:

  # cd /etc/apache2/ssl
  # openssl genrsa -out togaware.com.key 1024
  # chmod 600 togaware.com.key
  # openssl req -new -key togaware.com.key -out togaware.com.csr
    AU
    ACT
    Canberra
    Togaware
    Data Mining
    Kayon Toga
    Kayon.Toga@togaware.com
    (no challenge password)
  # openssl x509 -req -days 365 -in togaware.com.csr \
            -signkey togaware.com.key -out togaware.com.crt
  # mv apache.pem apache.pem.old
  # cp togaware.com.key apache.pem
  # cat togaware.com.crt >> apache.pem
  # chmod 600 apache.pem
  # service apache2 restart

If you are only replacing an old certificate, make sure you save the old file and generate the new files using the previous names. If you have several virtual hosts, this will save you a whole lot of time.

 

Creating multi-domain SSL certificates

The post is in French, but is certainly worth the effort of translation for people looking on how to do this:
http://howto.landure.fr/gnu-linux/debian-4-0-etch/creer-un-certificat-ssl-multi-domaines

If enough people ask for a good translation, I’ll consider doing that here.