Home About Blog

Can I A/B test with my Varnish Cache Setup?

Can I use section.io’s Varnish reverse proxy to A/B test my site?

Sure. There are a couple of ways. I’ll describe one way that allows you to make sure your A/B test is very clean, with dynamic content caching, so that your results aren’t skewed by performance problems.

One of the main reasons we are asked about A/B Testing (a.k.a Multivariate Testing) in Varnish Cache is when people move their site to a responsive design.

In order to do A/B testing with Varnish Cache, we need to make two (or more) buckets inside Varnish’s internal cache storage for each URL.

Varnish controls elements in its cache using the vcl_hash function. To A/B test, we add some additional metadata to the object in the cache - this says whether the object is for the desktop version of the site, or the responsive version.

The following model allows you to cache your dynamic content for maximum performance to make sure that performance of your two implementations isn’t affecting your results. You wouldn’t want to skew the results of your new responsive design negatively because it is slower since you haven’t had a long time to optimize it.

Let’s look at an example:

  1. A new customer comes to your site, so you randomly allocate them into a bucket. To do this, use the std vmod. Inside your vcl_recv function, generate a random number between 1 and 100. If the number is between 1 and 50, we’ll show the desktop version. If between 51 and 100, we’ll show the responsive version.

  2. When we make the request to the origin (your application server), we will send an additional HTTP request header. For example, set bereq.http.X-Bucket = Desktop

  3. Inside your origin application, when you receive the HTTP request, look at the X-Bucket header. Control your application to generate the right response for the bucket.

  4. When Varnish receives the backend response it puts it into the cache, but we want to add metadata to create two buckets inside Varnish’s memory. Include in the vcl_hash function an additional parameter indicating which bucket this response belongs to.

  5. Send the response to the client, and add a new cookie to the response, which is generated in Varnish. Get varnish to send a Set-cookie: Bucket=Desktop header.

  6. On the next request, the browser will send the Bucket=Desktop cookie. Varnish will look into the cache and see if there’s an object that matches both the URL and the bucket name. If there’s a match, you’ll get an almost instantaneous response.

When a second customer comes along, the same process is repeated. If they are allocated to the same bucket as a previous user and Varnish has something cached for them, they too will get an almost instantaneous response.

We have used this method many times to conduct a lot of A/B tests with Varnish Cache.

If you’d like any help doing this for yourself, just ask. Or, we could help set it up for you too.