A/B Testing ConvertKit Landing Pages with Cloudflare Workers

Over the past couple weeks, I've become obsessed with ConvertKit. I'm a long time Mailchimp user, but as I grow Bytesized Code into more of a brand (ew, I know), I want to be able to be smarter about how I share information with my subscribers. ConvertKit's approach to sequences, broadcasts, and user segments has really spoken to the way that I envision a healthy list for Bytesized Code, and it's been awesome to start to dig into actually building that vision out.

Real quick – if you don't have a ConvertKit account, but are interested in trying it, consider using my affiliate link! If you become a paying user of ConvertKit with my link, I get a little kickback which helps pay for Bytesized Code's operations – this stuff costs money! Sign up for ConvertKit with my affiliate link.

One of my favorite features of ConvertKit is subject line A/B testing. In short, it takes a scheduled broadcast, and tests two different versions of the subject line, sending the better-performing version to the full list. I documented how it worked in this tweet:

While that's handy, it didn't solve my more immediate problem: getting more people on my list. There's a lot of landing page tooling in ConvertKit for generating landing pages to capture new emails, but I'm not particularly great at copywriting, and I don't feel like I have good instincts around what performs better.

What I really wanted was A/B testing for landing pages! Unfortunately, ConvertKit doesn't support it natively, so I elected to build it myself. The result is a simple A/B test of two landing page variants, live at bytesized.xyz/newsletter. Make sure to visit the link and sign up for the newsletter, too :)

A/B Testing ConvertKit Landing Pages

When planning how to A/B test my two landing page variants, I opted to reach for a tool that I often use for neat URL-based trickery: Cloudflare Workers (disclosure: I work as the developer advocate for Cloudflare Workers). It's super straightforward to put a Workers script in front of any route on your website, which makes it a perfect fit for this kind of problem.

A landing page created in ConvertKit has a unique URL (at time of writing, hosted at ck.page) which ConvertKit users often will send their users to directly. Relatedly, this kind of approach has one glaring issue – if you change to a new landing page, any usage of the past URLs doesn't carry over, particularly if you delete a past landing page. I've been reading a lot of content from writers and bloggers using ConvertKit over the past few weeks, and I've been really surprised to see the number of people using this ck.page approach to hosting a landing page.

By using Cloudflare Workers, we can transparently serve the ConvertKit landing page to users, with the added benefit of owning the URL, allowing you to change the underlying landing pages, or even the landing page provider itself!

The code to implement this is pretty straightforward – it's about 20 lines of code, written in plain (ES6) JavaScript. In short, this code intercepts an incoming request, and randomly returns a landing page from a list of known URLs:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

const urls = [
  'https://bytesized.ck.page/abcdefghij',
  'https://bytesized.ck.page/1234567890'
]

const handleRequest = request => {
  const { headers } = request

  const random = Math.floor(Math.random() * urls.length)
  const url = urls[random % urls.length]
  return fetch(url, headers)
}

By hosting this at my newsletter page (bytesized.xyz/newsletter), users are automatically split between the two landing pages. In the ConvertKit UI, I can look at the two landing pages side-by-side to start to get a sense of which page is performing better:

As mentioned above, what's powerful about this approach is the ability to reconfigure what landing pages are presented to a user, such as adding more pages, or replacing with a new set of URLs, without it affecting any existing links or potential downtime for your landing page.

For instance, when I decide that one of the above variants is a clear winner, I can duplicate the winning result, build a new hypothesis to increase my conversion rate, and deploy a new version of my Workers script with a new set of URLs.

One additional change that could be made to this script is once a variant has been chosen for a client, we can continue to show that version of the page, using cookies. While it adds a slight bit of complexity to the code, the end result is still pretty readable (and is, in large part, derived from the A/B testing snippet in the Workers docs):

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

const urls = [
  'https://bytesized.ck.page/7228472d12',
  'https://bytesized.ck.page/01a42e7fc8',
]

const handleRequest = async request => {
  const { headers } = request

  const cookie = headers.get('cookie')
  if (cookie && cookie.includes('variant=')) {
    let re = /.*variant=(\d).*/gi
    let match = re.exec(cookie)
    return fetch(urls[match[1]])
  } else {
    const random = Math.floor(Math.random() * urls.length)
    const index = random % urls.length
    const url = urls[index]
    let response = await fetch(url)
    const respHeaders = { 'Set-Cookie': `variant=${index}; path=/` }
    return new Response(response, { headers: respHeaders })
  }
}

When a user hits my newsletter route, the Workers script checks for the presence of a a string variant=$index in the browser's cookies. If the string is found, the parsed index is used to send the same URL back again to the user. If the string isn't found in the cookie, a new URL is picked randomly (the original behavior), and on the response sent back to the user, the variant=$index string is added to the cookie.

I want to give a single disclaimer here, which is that while we are "A/B testing" these pages, the results are decidedly unscientific – this approach requires you to do the due-diligence of building useful A/B tests. I'm not great at that, and if I had to guess, I would say that my example picture above, where I tested almost two entirely different landing pages, is also pretty unscientific. This solution, for instance, isn't great for green versus blue button-style tests. That level of specificity is something I'd like to crack, but as ConvertKit landing pages are currently designed, it isn't really feasible (yet, hopefully)!

Conclusion

This implementation of A/B testing on ConvertKit landing pages is live on the Bytesized blog, and I'm really happy with the results so far. In addition, being able to swap out the underlying pages with new variants, without any downtime, makes experimentation feel fairly safe and easy – exciting!

The usage of Cloudflare Workers to solve this problem is another great reminder of how powerful serverless development can be as a multi-tool for all kinds of problems on the web. I wrote this solution within about thirty minutes, and deployed it in less than twenty seconds in total. The result is a stable URL that I can safely re-use everywhere in my projects (I've already begun linking to bytesized.xyz/newsletter on Twitter, YouTube, and a number of other places) without having to worry about my content going out of date.

Thanks to Avery Harnish and Stuart Sandine for feedback on this post!