How to Make a Custom URL Shortener Using Golang and Redis

URL's shorteners are popular web services that create short "aliases" (short links) for lengthy URLs (Uniform Resource Locators). Users get redirected to the original URL after they follow a respective short link. Shortened URLs are easier to remember, and the probability that a user makes a mistake when typing a short URL is significantly lower.

URL shortening services have been widely used in social networks, mobile applications and websites since the 2000s. Today we have dozens of URL shorteners available including TinyURL, Bitly, Yourls, and more.

Here below we describe how to write your own URL shortener in several simple steps using Golang and Redis. This article will be helpful for experienced developers and enthusiasts who want to get started with the basics of Golang and Redis for fast http/https redirects.

URL shortener - the requirements

Functional: 

• The service should generate a unique alias for the provided address. This will be the short link and this link should be short enough to pass it comfortably. 
• The service should redirect users to the original URL when they access a short link.
• The short link should have a lifetime, that the user specifies on creation.
• The service should provide a statistic about the short link (visits count).

Non-functional:

• The service should be able to handle numerous requests. 
• Forwarding should be real-time with minimal delay. 
• The short link should be completely random (so that it cannot be predicted). 

Estimations 

The service will be read-heavy since there will be a huge number of redirects compared to creating new ones. Let’s assume that the ratio between reading and writing is 50:1. 

Traffic estimates: 

If we have 500k new short links every month, then we will expect 25 million (50 * 500k = 25 million) redirects for the same period. So we have 1 new link every 5 seconds: 

500k / (30 days * 24 hours * 3600 seconds) = ~ 1 link in 5 seconds. 

And 10 redirects every second: 

25 million / (30 days * 24 hours * 3600 seconds) = ~ 10 redirects per second. 

Memory estimates: 

Let’s say we store each address for a maximum - 1 year. Since we expect 500k new links every month than we will have near 6 million records in the database: 

Hire talented developers from LATAM, Canada, and Europe

500k record/month * 12 months = 6 million 

Let’s assume that each record in the database - approximately 1000 bytes. The recommended maximum size for a link is 2000 characters and according to the standard, the URL encodes with ASCII characters, which occupy 1 byte, i.e. the link can hold  2000 bytes by recommended maximum size). So we will use half of this value as average. Then we need 6 TB of memory to store records for 1 year: 

6 million record * 1000 bytes per record = 6 GB

There is a little summary of the nature of the model that we will be working with: 

• We need to store several million records
• Each record is small 
• The service is very read-heavy 

Technology stack 

Since we don’t need to store relations between different data models and need to store large amounts of data, it’s preferable to look at NoSQL databases. Also which, if necessary, will be much easier to scale. 

Anyway, as we found previously, this example should store relatively small amounts of data, and redirections should be processed as quickly as possible, we will use Redis.

We also will use Golang language and necessary libraries: 

• Fasthttp - fast implementations for Http server 
• HttpRouter - requests mapping 
• Redigo - working with Redis

For dependency management, we’re going to use Go Modules (Go version 1.12+).

Module initialization and adding dependencies 

Open Terminal and execute the next commands: 

Would you like to learn more

about talent solutions

• Create a new module

• Add required dependencies 

How to create a unique identifier for URL?

Now let’s figure out how we can get a short and unique key for the given URL. The most obvious solution that comes to mind - use a hash function for the original URL. However, this approach has a few significant disadvantages: 

• Length. Most of the standard hash functions (for example, the SHA or MD family of hash functions) produce a long string as output, which in turn contradicts the idea of short links. 
• Uniqueness. If we want to use the output as an identifier, it must be unique. But a hash function by its nature can give the same result for different inputs (it’s called a collision)
• Search. For most hash functions, getting input by its output is extremely difficult or impossible.  

First of all, let’s figure out what symbols we can potentially use. If we look at the RFC 3986 standard, we can see that the URL can contain the following characters: [a-z A-Z 0-9_.-~].

However, for better readability and appearance of links (to avoid links like https://example.com/~~~~~~ or https://example.com/~..-_~), we will use only alphanumeric characters. So we can say that we have 62 possible characters (Base62). So using 62 length alphabet and potential length of the resulting identifier n and combinatorial formula for permutations with repetition, we can calculate the number of different possible links we can store: 

A62n=62n

Try to use values n=[5-10]

625=916 132 832

626=56 800 235 584

Hire talented developers from LATAM, Canada, and Europe

627=3 521 614 606 208

628=218 340 105 584 896

629=13 537 086 546 263 552

6210=839 299 365 868 340 224

As we can see, using just 5 characters we can encode almost a billion possible options and using 10 characters - 839 quadrillions. Thus, using regular integer and Base62 encoding, we can create any unique link with a short address.

We will generate a random number from the interval [0, 18 446 744 073 709 551 615] and encode using Base62 to get a short link and decode to get the identifier from it.

package base62

import (
  "errors"
  "math"
  "strings"
)

const (
  alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
  length   = uint64(len(alphabet))
)

func Encode(number uint64) string {
  var encodedBuilder strings.Builder
  encodedBuilder.Grow(11)

  for ; number > 0; number = number / length {
     encodedBuilder.WriteByte(alphabet[(number % length)])
  }

  return encodedBuilder.String()
}

func Decode(encoded string) (uint64, error) {
  var number uint64

  for i, symbol := range encoded {
     alphabeticPosition := strings.IndexRune(alphabet, symbol)

     if alphabeticPosition == -1 {
        return uint64(alphabeticPosition), errors.New("invalid character: " + string(symbol))
     }
     number += uint64(alphabeticPosition) * uint64(math.Pow(float64(length), float64(i)))
  }

  return number, nil
}

After we have decided how to get a unique link, let’s move on tho the design of the data model that we will store. We need to store the following information for each link: 

• Id integer 
• Original string 
• Expiration date 
• Visits integer

So now out code should generate an identifier, check it for uniqueness, insert the link using that unique identifier along with the date when the link should be removed. Then encode this identifier with Base62. After that, we just need to implement redirection logic, that everyone who follows the short link, redirects to the original one. 

Let’s specify the configurable values of the service: 

Would you like to learn more

about talent solutions

• Port to listen to incoming connections 
• The schema for short links 
• Domain for short links 
• DB connection parameters

For simplicity, we will configure everything by using a JSON configuration file, that for local development purpose looks like: 

 
{
 "server": {
   "port": "8080"
 },
 "options": {
   "schema": "http",
   "prefix": "localhost:8080"
 },
 "redis": {
   "host": "127.0.0.1",
   "port": "6379"
 }
}

Let’s create a package that will contain the structure for configuration and will allow as to use these values everywhere in the easiest way. Also, we need a function that will read all values from the JSON file.

 
package config

import (
  "encoding/json"
  "io/ioutil"
)

type Config struct {
  Server struct {
     Port string `json:"port"`
  } `json:"server"`
  Redis struct {
     Host     string `json:"host"`
     Port     string `json:"port"`
  } `json:"redis"`
  Options struct {
     Schema string `json:"schema"`
     Prefix string `json:"prefix"`
  } `json:"options"`
}

func FromFile(path string) (*Config, error) {
  b, err := ioutil.ReadFile(path)
  if err != nil {
     return nil, err
  }

  var cfg Config
  if err := json.Unmarshal(b, &cfg); err != nil {
     return nil, err
  }

  return &cfg, nil
}
 

After that let’s create a package for the service that will be responsible for saving and getting data to/from DB, declare an interface with the required method: create a new link, get the original link, information for the short link, and close connection to the data source.

package storage

import "time"

type Service interface {
  Save(string, time.Time) (string, error)
  Load(string) (string, error)
  LoadInfo(string) (*Item, error)
  Close() error
}

type Item struct {
  Id      uint64 `json:"id" redis:"id"`
  URL     string `json:"url" redis:"url"`
  Expires string `json:"expires" redis:"expires"`
  Visits  int    `json:"visits" redis:"visits"`
}

And implement this interface to use Redis as a data source. For the first step, we will create a constructor that prepares the connection pool for Redis. 

type redis struct{ pool *redisClient.Pool }

func New(host, port, password string) (storage.Service, error) {
  pool := &redisClient.Pool{
     MaxIdle:     10,
     IdleTimeout: 240 * time.Second,
     Dial: func() (redisClient.Conn, error) {
        return redisClient.Dial("tcp", fmt.Sprintf("%s:%s", host, port))
     },
  }

  return &redis{pool}, nil
}

After that, we will implement saving short links to the database. As a short link identifier, we will use a pseudorandom number generated by the built-in Golang tool, check it for uniqueness, and save it to the DB. Due that we have a certain lifetime for the link, we must implement appropriate logic.

But Redis already supports this functionality to specify record lifetime and we will use it. At the end of the method, we will return the generated identifier encoded with Base62.

func (r *redis) isUsed(id uint64) bool {
  conn := r.pool.Get()
  defer conn.Close()

  exists, err := redisClient.Bool(conn.Do("EXISTS", "Shortener:"+strconv.FormatUint(id, 10)))
  if err != nil {
     return false
  }
  return exists
}

func (r *redis) Save(url string, expires time.Time) (string, error) {
  conn := r.pool.Get()
  defer conn.Close()

  var id uint64

  for used := true; used; used = r.isUsed(id) {
     id = rand.Uint64()
  }

  shortLink := storage.Item{id, url, expires.Format("2006-01-02 15:04:05.728046 +0300 EEST"), 0}

  _, err := conn.Do("HMSET", redisClient.Args{"Shortener:" + strconv.FormatUint(id, 10)}.AddFlat(shortLink)...)
  if err != nil {
     return "", err
  }

  _, err = conn.Do("EXPIREAT", "Shortener:"+strconv.FormatUint(id, 10), expires.Unix())
  if err != nil {
     return "", err
  }

  return base62.Encode(id), nil
}

The rest of the methods aren’t so interesting, there is simply getting data from DB.

func (r *redis) Load(code string) (string, error) {
  conn := r.pool.Get()
  defer conn.Close()

  decodedId, err := base62.Decode(code)
  if err != nil {
     return "", err
  }

  urlString, err := redisClient.String(conn.Do("HGET", "Shortener:"+strconv.FormatUint(decodedId, 10), "url"))
  if err != nil {
     return "", err
  } else if len(urlString) == 0 {
     return "", storage.ErrNoLink
  }

  _, err = conn.Do("HINCRBY", "Shortener:"+strconv.FormatUint(decodedId, 10), "visits", 1)

  return urlString, nil
}

func (r *redis) isAvailable(id uint64) bool {
  conn := r.pool.Get()
  defer conn.Close()

  exists, err := redisClient.Bool(conn.Do("EXISTS", "Shortener:"+strconv.FormatUint(id, 10)))
  if err != nil {
     return false
  }
  return !exists
}

func (r *redis) LoadInfo(code string) (*storage.Item, error) {
  conn := r.pool.Get()
  defer conn.Close()

  decodedId, err := base62.Decode(code)
  if err != nil {
     return nil, err
  }

  values, err := redisClient.Values(conn.Do("HGETALL", "Shortener:"+strconv.FormatUint(decodedId, 10)))
  if err != nil {
     return nil, err
  } else if len(values) == 0 {
     return nil, storage.ErrNoLink
  }
  var shortLink storage.Item
  err = redisClient.ScanStruct(values, &shortLink)
  if err != nil {
     return nil, err
  }

  return &shortLink, nil
}

func (r *redis) Close() error {
  return r.pool.Close()
}

Now we need to write a simple HTTP server that will handle required requests and call appropriate service methods. Let’s define two structures: 

• Wrapper for the server response
• An object that will store the data needed to create a short link (HTTP/HTTPS protocol, host) and as well as a storage service implementation for working with DB.

type response struct {
  Success bool        `json:"success"`
  Data    interface{} `json:"shortUrl"`
}

type handler struct {
  schema  string
  host    string
  storage storage.Service
}

Let’s create a constructor that specifies mappings for requests:

func New(schema string, host string, storage storage.Service) *router.Router {
  router := router.New()

  h := handler{schema, host, storage}
  router.POST("/encode/", responseHandler(h.encode))
  router.GET("/{shortLink}", h.redirect)
  router.GET("/{shortLink}/info", responseHandler(h.decode))
  return router
}

We will have 3 endpoints: 

• Create a new short link 
• Redirect to the origin URL on passing a short link 
• Get general information about the short link

Let’s implement them:

 
package handler

import (
  "encoding/json"
  "fmt"
  "github.com/fasthttp/router"
  "github.com/valyala/fasthttp"
  "log"
  "net/http"
  "net/url"
  "time"
  "urlShorter/storage"
)

func New(schema string, host string, storage storage.Service) *router.Router {
  router := router.New()

  h := handler{schema, host, storage}
  router.POST("/encode/", responseHandler(h.encode))
  router.GET("/{shortLink}", h.redirect)
  router.GET("/{shortLink}/info", responseHandler(h.decode))
  return router
}

type response struct {
  Success bool        `json:"success"`
  Data    interface{} `json:"shortUrl"`
}

type handler struct {
  schema  string
  host    string
  storage storage.Service
}

func responseHandler(h func(ctx *fasthttp.RequestCtx) (interface{}, int, error)) fasthttp.RequestHandler {
  return func(ctx *fasthttp.RequestCtx) {
     data, status, err := h(ctx)
     if err != nil {
        data = err.Error()
     }
     ctx.Response.Header.Set("Content-Type", "application/json")
     ctx.Response.SetStatusCode(status)
     err = json.NewEncoder(ctx.Response.BodyWriter()).Encode(response{Data: data, Success: err == nil})
     if err != nil {
        log.Printf("could not encode response to output: %v", err)
     }
  }
}

func (h handler) encode(ctx *fasthttp.RequestCtx) (interface{}, int, error) {
  var input struct {
     URL     string `json:"url"`
     Expires string `json:"expires"`
  }

  if err := json.Unmarshal(ctx.PostBody(), &input); err != nil {
     return nil, http.StatusBadRequest, fmt.Errorf("Unable to decode JSON request body: %v", err)
  }

  uri, err := url.ParseRequestURI(input.URL)

  if err != nil {
     return nil, http.StatusBadRequest, fmt.Errorf("Invalid url")
  }

  layoutISO := "2006-01-02 15:04:05"
  expires, err := time.Parse(layoutISO, input.Expires)
  if err != nil {
     return nil, http.StatusBadRequest, fmt.Errorf("Invalid expiration date")
  }

  c, err := h.storage.Save(uri.String(), expires)
  if err != nil {
     return nil, http.StatusInternalServerError, fmt.Errorf("Could not store in database: %v", err)
  }

  u := url.URL{
     Scheme: h.schema,
     Host:   h.host,
     Path:   c}

  fmt.Printf("Generated link: %v \n", u.String())

  return u.String(), http.StatusCreated, nil
}

func (h handler) decode(ctx *fasthttp.RequestCtx) (interface{}, int, error) {
  code := ctx.UserValue("shortLink").(string)

  model, err := h.storage.LoadInfo(code)
  if err != nil {
     return nil, http.StatusNotFound, fmt.Errorf("URL not found")
  }

  return model, http.StatusOK, nil
}

func (h handler) redirect(ctx *fasthttp.RequestCtx) {
  code := ctx.UserValue("shortLink").(string)

  uri, err := h.storage.Load(code)
  if err != nil {
     ctx.Response.Header.Set("Content-Type", "application/json")
     ctx.Response.SetStatusCode(http.StatusNotFound)
     return
  }

  ctx.Redirect(uri, http.StatusMovedPermanently)
}

After all the logic of the service has been implemented, we are going to create the main package, which will be the starting point for the service, will initialize all necessary services and start the web-server.

package main

import (
  "log"
  "urlShorter/config"
  "urlShorter/handler"
  "urlShorter/storage/redis"

  "github.com/valyala/fasthttp"
)

func main() {
  configuration, err := config.FromFile("./configuration.json")
  if err != nil {
     log.Fatal(err)
  }

  service, err := redis.New(configuration.Redis.Host, configuration.Redis.Port, configuration.Redis.Password)
  if err != nil {
     log.Fatal(err)
  }
  defer service.Close()

  router := handler.New(configuration.Options.Schema, configuration.Options.Prefix, service)

  log.Fatal(fasthttp.ListenAndServe(":" + configuration.Server.Port, router.Handler))
}

So after all these steps, the project’s root directory should look like:

.
├── base62
│   └── base62.go
├── config
│   └── configuration.go
├── configuration.json
├── go.mod
├── go.sum
├── handler
│   └── handler.go
├── main.go
└── storage
     ├── redis
     │   └── redis.go
     └── storage.go

Service run 

To run the service execute the next command from the project’s root directory: 

Service check

To check that service works as expected we will use CUrl and run the following command at the terminal: 

1. Create a short link 

curl -L -X POST 'localhost:8080/encode' \
-H 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.alexedwards.net/blog/working-with-rediss",
    "expires": "2020-10-04 17:18:00"
}'

And we will get a response like:

{
   "success":true,
   "shortUrl":"http://localhost:8080/OTv0FdGU8Ng"
}

2. Get detailed information for the short link

curl -L -X GET 'http://localhost:8080/info/OTv0FdGU8Ng'

3. Open the browser and follow the short link http://localhost:8080/OTv0FdGU8Ng

Creation a docker image 

Now when the service is ready and has been tested, it will be useful to create a Docker image for easy lunch in any environment. To speed up builds, we will use Multistage build. Let’s create Dockerfile with the following content: 

FROM golang:alpine as builder

WORKDIR /build

COPY go.mod .
COPY go.sum .
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o shortener .

WORKDIR /dist

RUN cp /build/shortener .

FROM scratch

COPY --from=builder /dist/shortener /app/

COPY configuration.json /app/

WORKDIR /app

CMD ["./shortener"]

As a basis, we use the alpine version of the image with preinstalled Golang.  After that to avoid downloading all dependencies again and again on each build, we copy module related files with dependencies and download them.

Later Docker will take these layers from the cache on each build. Because the compiled application will run in a completely empty image (where all libraries are missing), it’s necessary to compile all together with all necessary libraries, to do that we add the following parameters to compiling command: 

CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

After we will run in terminal command to build Docker image:

docker build . -t shortener

It should create Docker image with ready to use the application, that we can start just using the command:

docker run -p 8080:8080 shortener

Conclusion

In this article we described the implementation of a simple URL-shortening web-service with Golang and Redis, that is based on Fasthttp and has built-in functionality to create short links for any url.