Migrate Old URLs to New URL Structure Using Nginx and Redis.

While maintaining a website, webmasters may decide to move the whole website or parts of it to a new location. For example, you might have a URL structure which is not SEO or user friendly and you have to make it one. Changing URL structure can involve a bit of effort, but it’s worth doing it properly.

It’s very important to redirect all of your old URLs traffic to the new location using 301 redirects and make sure that it’s possible to navigate it without running into 404 error pages.

To start with, you will need to generate a list of old URLs and map them to their new destinations. However, this list can grow bigger and bigger depending on the size of your website. Storing this mapping also depends on your servers and size of the website URLs. You can use a database or configure some URL rewriting on your server or application for common redirect patterns.

The problem with database is that it is slow, while file based mapping (by nginx) can take long time just to reload or restart nginx (i.e. need reload or restart nginx in case you add more redirect rules) and also take significant amount of memory depending on size of the mapping file.

Nginx  Redis - Migrate Old URLs to New URL Structure

Nginx + Redis – Migrate Old URLs to New URL Structure

Fortunately, by using Redis and Nginx Lua Module you can make this transaction smooth and the overall migration process – painless.

Requirements:

1 – Install packages nginx-extras & redis-server (http://www.dotdeb.org/instructions/)
2 – Install http://openresty.org/download/ngx_openresty-1.2.4.14.tar.gz
3 – Configure nginx
+ Add the following line at the start of nginx file (replace path with the proper location where you installed openresty module).


lua_package_path "/usr/local/openresty/lualib/?.lua;;";

4 – Add following location block in nginx file :


location ~ "^/[\d]{4}/[\d]{2}/[\d]{2}/(?<slug>[\w-]+)/?$" {

content_by_lua '

local redis = require "resty.redis"
local red = redis:new()

red:set_timeout(1000) -- 1 sec
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.exit(503)
return
end

local key = ngx.var.slug
local res, err = red:get(key)

if not res then
ngx.exit(404)
return
end

if res == ngx.null then
ngx.exit(404)
return
end

ngx.redirect(res, 301)
';

}

 

How does it work?


lua_package_path "/usr/local/openresty/lualib/?.lua;;";

This line is to tell nginx to load lua module as you are intended to use lua script in you configuration.


location ~ "^/[\d]{4}/[\d]{2}/[\d]{2}/(?<slug>[\w-]+)/?$"

This line is to check all requests with old URL pattern to fall under this block with a lua variable (i.e. slug)


local redis = require "resty.redis"
local red = redis:new()

red:set_timeout(1000) -- 1 sec
local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
ngx.exit(503)
return
end

Above lua script will try to connect with Redis server (on host 127.0.01 and port 6397) with 1 second timeout.


local key = ngx.var.slug
local res, err = red:get(key)

Above lua script will get key from Redis (i.e. usgin lua variable slug which we got from regex)

Rest is quite self explanatory as it will will redirect with 301 if found or with 404 if not.

+ NOTE: In example above replace regex, redis server host & port according to your need
a. Above regex is for URL pattern /{year}/{month}/{day}/slug
b. Redis server path (i.e. host 127.0.0.1) and port (i.e. 6379)

Know another or perhaps a better way  to migrate old URLs to new URL structure? Or have used the same method for your website’s URL migration? Share your experience with us through comments. We are always happy to hear from you.