Magento_nginx_varnish

Magento online stores need to be fast. There are many ways in which you can implement a high performance web server for your Magento Store. Let’s discuss a few ways here. The first approach uses a Nginx Web Server and the second approach uses a Varnish proxy.

Nginx Web Server

This is a webserver similar to Apache. It is however known to give better performance results than Apache. Nginx is known for it’s stability when handling lots of requests. This ofcourse results in high CPU usage. Apache on the other hand does not scale up it requires a large amount of knowhow. Nginx however requires the installation of PHP – FPM. PHP FPM is a FastCGI Process Manager for PHP.

Varnish Cache

Varnish Cache is a HTTP proxy that is also known as the HTTP accelerator. IT sits between you and the web server and whenever a web page is requested, it looks into it’s cache. If a cached copy of the page is available, it is immediately made available to the user. If not, the request is passed on to the web server. The plus point is that Varnish makes use of the physical memory on the server as opposed to the CPU this greatly increases speed without slowing down the server.

Both these methods provide good performance results.

Let’s check how to configure this on Cent OS. Cent OS 6 or higher would be fine. Using a dedicated server is recommended. Here’s how to begin:

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
yum install mysql mysql-server
/etc/init.d/mysqld start
/usr/bin/mysql_secure_installation

It is recommended to set up a root password for MySQL.

yum install nginx

/etc/init.d/nginx start
yum --enablerepo=remi install php-fpm php-mysql php-pdo php-common php-mcrypt php-gd php-curl php-soap
vi /etc/php.ini

Next, open the php.ini, find cgi.fix_pathinfo and set the value to 0.

Now create a virtual host in Nginx.

vi /etc/nginx/conf.d/magentosite.conf

You can modify the details below to match your needs:

server {
    listen      8080;
    server_name www.magentosite.co.uk;
    root        /var/www/sites/magentosite/;
    index index.html index.php;

    access_log /var/log/nginx/www.magentosite.co.uk-access_log;
    error_log /var/log/nginx/www.magentosite.co.uk-error_log;

    location / {
        try_files $uri $uri/ @handler;
        expires 30d;
    }
    location /app/                       { deny all; }
    location /includes/                  { deny all; }
    location /lib/                       { deny all; }
    location /media/downloadable/        { deny all; }
    location /pkginfo/                   { deny all; }
    location /report/config.xml          { deny all; }
    location /var/                       { deny all; }

    location /var/export/ {
        auth_basic              "Restricted";
        auth_basic_user_file    htpasswd;
        autoindex               on;
    }
    location  /. {
        return 404;
    }

    location @handler {
        rewrite / /index.php;
    }

    location ~ .php/ {
        rewrite ^(.*.php)/ $1 last;
    }

    location ~ \.php$ {
        try_files $uri =404;
        expires off;
        fastcgi_read_timeout 900s;
        fastcgi_index index.php;
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }
    rewrite ^/minify/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;
    rewrite ^/skin/m/([0-9]+)(/.*.(js|css))$ /lib/minify/m.php?f=$2&d=$1 last;

    location /lib/minify/ {
        allow all;
    }
    gzip on;
    #gzip_comp_level 9;
    gzip_min_length  1000;
    gzip_proxied any;
    gzip_types       text/plain application/xml text/html text/css text/js application/x-javascript;
}

This creates a virtual host, assigns a server name and root directory where your store is set. Certain other tweaks will also be required to pass PHP requests to PHP-FPM and allow friendly URLs.

We will require to set Nginx to run on port 8080 rather than the usual 80. This is done to enable Varnish to run on port 80.

Now, let’s configure PHP FPM to work with Nginx. For this open

/etc/php-fpm.d/www.conf and find the User & Group settings, and set both the user and group to ‘nginx’.

Then start php-fpm:

Then run a service nginx restart to restart to restart Nginx. Now you can check your site at http://example.com:8080/.

To install varnish simply run:

sudo yum install varnish

Now, let’s set up Varnish so that when a web page is requested it will go through the Varnish Cache first.
vi /etc/sysconfig/varnish
Find VARNISH_LISTEN_PORT=6081 and change it to port 80.
For Varnish 2.1 use the code below:

# This configuration is what is provided by PageCache by Varnish for Magento module:
# http://www.magentocommerce.com/magento-connect/pagecache-powered-by-varnish.html
# default backend definition. Set this to point to your content server.

backend default {
  .host = "127.0.0.1";
  .port = "80";
}
 
# admin backend with longer timeout values. Set this to the same IP & port as your default server.
backend admin {
  .host = "127.0.0.1";
  .port = "80";
  .first_byte_timeout = 18000s;
  .between_bytes_timeout = 18000s;
}
 
 
# add your Magento server IP to allow purges from the backend
acl purge {
  "localhost";
  "127.0.0.1";
}
 
 
sub vcl_recv {
    if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For ", " client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }
    
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE" &&
      req.request != "PURGE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    
    # purge request
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        purge("obj.http.X-Purge-Host ~ " req.http.X-Purge-Host " && obj.http.X-Purge-URL ~ " req.http.X-Purge-Regex " && obj.http.Content-Type ~ " req.http.X-Purge-Content-Type);
        error 200 "Purged.";
    }
    
    # switch to admin backend configuration
    if (req.http.cookie ~ "adminhtml=") {
        set req.backend = admin;
    }
 
    # we only deal with GET and HEAD by default    
    if (req.request != "GET" && req.request != "HEAD") {
        return (pass);
    }
    
    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://[^/]+", "");
    
    # static files are always cacheable. remove SSL flag and cookie
    if (req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$") {
        unset req.http.Https;
        unset req.http.Cookie;
    }
 
    # not cacheable by default
    if (req.http.Authorization || req.http.Https) {
        return (pass);
    }
 
    # do not cache any page from
    # - index files
    # - ...
    if (req.url ~ "^/(index)") {
        return (pass);
    }
 
    # as soon as we have a NO_CACHE cookie pass request
    if (req.http.cookie ~ "NO_CACHE=") {
        return (pass);
    }
 
    # normalize Aceept-Encoding header
    # http://varnish.projects.linpro.no/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }
    
    # remove Google gclid parameters
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
  
    return (lookup);
}
 
# sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set bereq.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
#     return (pipe);
# }
# 
# sub vcl_pass {
#     return (pass);
# }
# 
sub vcl_hash {
    set req.hash += req.url;
    if (req.http.host) {
         set req.hash += req.http.host;
    } else {
        set req.hash += server.ip;
    }
    if (!(req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$")) {
        call design_exception;
    }
    return (hash);
}
# 
# sub vcl_hit {
#     if (!obj.cacheable) {
#         return (pass);
#     }
#     return (deliver);
# }
# 
# sub vcl_miss {
#     return (fetch);
# }
 
sub vcl_fetch {
    if (beresp.status == 500) {
       set beresp.saintmode = 10s;
       restart;
    }
    set beresp.grace = 5m;
 
    # add ban-lurker tags to object
    set beresp.http.X-Purge-URL = req.url;
    set beresp.http.X-Purge-Host = req.http.host;
    
    if (beresp.status == 200 || beresp.status == 301 || beresp.status == 404) {
        if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") {
            if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
                set beresp.ttl = 0s;
                return (pass);
            }
 
            # marker for vcl_deliver to reset Age:
            set beresp.http.magicmarker = "1";
            
            # Don't cache cookies
            unset beresp.http.set-cookie;
        } else {
            # set default TTL value for static content
            set beresp.ttl = 4h;
        }
        return (deliver);
    }
    
    return (pass);
}
 
sub vcl_deliver {
    # debug info
    if (resp.http.X-Cache-Debug) {
        if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT";
            set resp.http.X-Cache-Hits = obj.hits;
        } else {
           set resp.http.X-Cache = "MISS";
        }
        set resp.http.X-Cache-Expires = resp.http.Expires;
    } else {
        # remove Varnish/proxy header
        remove resp.http.X-Varnish;
        remove resp.http.Via;
        remove resp.http.Age;
        remove resp.http.X-Purge-URL;
        remove resp.http.X-Purge-Host;
    }
    
    if (resp.http.magicmarker) {
        # Remove the magic marker
        unset resp.http.magicmarker;
 
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "Mon, 31 Mar 2008 10:00:00 GMT";
        set resp.http.Age = "0";
    }
}
 
# sub vcl_error {
#     set obj.http.Content-Type = "text/html; charset=utf-8";
#     synthetic {"
# 
# 
# 
#   
#     "} obj.status " " obj.response {"
#   
#   
#     

Error "} obj.status " " obj.response {"

#

"} obj.response {"

#

Guru Meditation:

#

XID: "} req.xid {"

#
#

Varnish cache server

# # # "}; # return (deliver); # } sub design_exception { }

For Varnish 3.0 use the following code

# This configuration is what is provided by PageCache by Varnish for Magento module:
# http://www.magentocommerce.com/magento-connect/pagecache-powered-by-varnish.html
# default backend definition. Set this to point to your content server.

backend default {
  .host = "127.0.0.1";
  .port = "80";
}
 
# admin backend with longer timeout values. Set this to the same IP & port as your default server.
backend admin {
  .host = "127.0.0.1";
  .port = "80";
  .first_byte_timeout = 18000s;
  .between_bytes_timeout = 18000s;
}
 
# add your Magento server IP to allow purges from the backend
acl purge {
  "localhost";
  "127.0.0.1";
}
 
 
sub vcl_recv {
    if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }
    
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&	
      req.request != "OPTIONS" &&
      req.request != "DELETE" &&
      req.request != "PURGE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    
    # purge request
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        ban("obj.http.X-Purge-Host ~ " + req.http.X-Purge-Host + " && obj.http.X-Purge-URL ~ " + req.http.X-Purge-Regex + " && obj.http.Content-Type ~ " + req.http.X-Purge-Content-Type);
        error 200 "Purged.";
    }
 
    # switch to admin backend configuration
    if (req.http.cookie ~ "adminhtml=") {
        set req.backend = admin;
    }
 
    # we only deal with GET and HEAD by default    
    if (req.request != "GET" && req.request != "HEAD") {
        return (pass);
    }
    
    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://[^/]+", "");
    
    # static files are always cacheable. remove SSL flag and cookie
    if (req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$") {
        unset req.http.Https;
        unset req.http.Cookie;
    }
 
    # not cacheable by default
    if (req.http.Authorization || req.http.Https) {
        return (pass);
    }
 
    # do not cache any page from
    # - index files
    # - ...
    if (req.url ~ "^/(index)") {
        return (pass);
    }
 
    # as soon as we have a NO_CACHE cookie pass request
    if (req.http.cookie ~ "NO_CACHE=") {
        return (pass);
    }
 
    # normalize Aceept-Encoding header
    # http://varnish.projects.linpro.no/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }
    
    # remove Google gclid parameters
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
  
    return (lookup);
}
 
# sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set bereq.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
#     return (pipe);
# }
# 
# sub vcl_pass {
#     return (pass);
# }
# 
sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    if (!(req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$")) {
        call design_exception;
    }
    return (hash);
}
# 
# sub vcl_hit {
#     return (deliver);
# }
# 
# sub vcl_miss {
#     return (fetch);
# }
 
sub vcl_fetch {
    if (beresp.status == 500) {
       set beresp.saintmode = 10s;
       return (restart);
    }
    set beresp.grace = 5m;
 
    # add ban-lurker tags to object
    set beresp.http.X-Purge-URL = req.url;
    set beresp.http.X-Purge-Host = req.http.host;
    
    if (beresp.status == 200 || beresp.status == 301 || beresp.status == 404) {
        if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") {
            if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
                set beresp.ttl = 0s;
                return (hit_for_pass);
            }
 
            # marker for vcl_deliver to reset Age:
            set beresp.http.magicmarker = "1";
            
            # Don't cache cookies
            unset beresp.http.set-cookie;
        } else {
            # set default TTL value for static content
            set beresp.ttl = 4h;
        }
        return (deliver);
    }
    
    return (hit_for_pass);
}
 
sub vcl_deliver {
    # debug info
    if (resp.http.X-Cache-Debug) {
        if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT";
            set resp.http.X-Cache-Hits = obj.hits;
        } else {
           set resp.http.X-Cache = "MISS";
        }
        set resp.http.X-Cache-Expires = resp.http.Expires;
    } else {
        # remove Varnish/proxy header
        remove resp.http.X-Varnish;
        remove resp.http.Via;
        remove resp.http.Age;
        remove resp.http.X-Purge-URL;
        remove resp.http.X-Purge-Host;
    }
    
    if (resp.http.magicmarker) {
        # Remove the magic marker
        unset resp.http.magicmarker;
 
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "Mon, 31 Mar 2008 10:00:00 GMT";
        set resp.http.Age = "0";
    }
}
 
# sub vcl_error {
#     set obj.http.Content-Type = "text/html; charset=utf-8";
#     set obj.http.Retry-After = "5";
#     synthetic {"
# 
# 
# 
#   
#     "} + obj.status + " " + obj.response + {"
#   
#   
#     

Error "} + obj.status + " " + obj.response + {"

#

"} + obj.response + {"

#

Guru Meditation:

#

XID: "} + req.xid + {"

#
#

Varnish cache server

# # # "}; # return (deliver); # } # # sub vcl_init { # return (ok); # } # # sub vcl_fini { # return (ok); # } sub design_exception { }

Now run varnish with the following command : service varnish start

And access your website without using the port 8080. Your site should run perfectly!

Share Button