How to set up A/B tests in Django

Feb 09, 2024

Posted by

On this page

A/B tests help you improve your Django app by enabling you to compare the impact of changes on key metrics. To show you how to set one up, we create a basic Django app, add PostHog, create an A/B test, and implement the code for it.

1. Create a basic Django app

First, ensure Python 3 and Django are installed. Then, create a new Django project called django_ab_tests with a basic app:

Terminal
django-admin startproject django_ab_tests
cd django_ab_tests
python3 manage.py startapp basic_app

Next, we create a simple view with a heading and paragraph. Replace the code in basic_app/views.py with the following:

basic_app/views.py
from django.http import HttpResponse


def home(request):
    paragraphText = 'Placeholder text'
    
    return HttpResponse(f"""
    <!DOCTYPE html>
   <html>
      <body>
        <h1>Django A/B Testing Tutorial</h1>
        <p>{paragraphText}</p>
      </body>
    </html>
    """)

Then, map a URL to your view by adding a path to urlpatterns in django_ab_tests/urls.py. Replace the code in that file with the following:

django_ab_tests/urls.py
from django.contrib import admin
from django.urls import path
from basic_app.views import home


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', home),
]

Run python3 manage.py migrate to migrate the database (you only need to run this once) and then python3 manage.py runserver to see our app in action at http://127.0.0.1:8000.

Basic Django app

2. Add PostHog to your app

With our app set up, it’s time to install and set up PostHog. If you don't have a PostHog instance, you can sign up for free.

To start, run pip install posthog to install PostHog’s Python SDK.

Then, set the PostHog API key and host in your AppConfig in basic_app/apps.py so that's it's available everywhere:

basic_app/apps.py
from django.apps import AppConfig
import posthog


class BasicAppConfig(AppConfig):
    default_auto_field = "django.db.models.BigAutoField"
    name = "basic_app"
    def ready(self):
        posthog.api_key = '<ph_project_api_key>'
        posthog.host = 'https://us.i.posthog.com'

You can find your project API key and instance address in your project settings.

Next, add your AppConfig to django_ab_tests/settings.py under INSTALLED_APPS:

django_ab_tests/settings.py
INSTALLED_APPS = [
    'basic_app.apps.BasicAppConfig',  # Add your app config
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]

Lastly, we import posthog into basic_app/views.py and capture a $pageview event using posthog.capture():

views.py
from django.http import HttpResponse
import posthog


def home(request):
    distinct_id = 'placeholder-user-id' 


    paragraphText = 'Placeholder text'
    
    posthog.capture(distinct_id, '$pageview')


    return HttpResponse(f"""
    <!DOCTYPE html>
    <html>
      <body>
        <h1>Django A/B Testing Tutorial</h1>
        <p>{paragraphText}</p>
      </body>
    </html>
    """)

With this set up, restart your app and then refresh your browser a few times. You should now see captured events in your PostHog activity tab.

Events captured in PostHog

3. Create an A/B test in PostHog

Next, go to the A/B testing tab and create an A/B test by clicking the New experiment button. Add the following details to your experiment:

  1. Name it "My cool experiment".
  2. Set "Feature flag key" to my-cool-experiment.
  3. Under the experiment goal, select the pageview event we captured in the previous step.
  4. Use the default values for all other fields.

Click "Save as draft" and then click "Launch".

Experiment setup in PostHog

4. Implement the A/B test code

To implement the A/B test, we:

  1. Fetch the my-cool-experiment flag using posthog.get_feature_flag().
  2. Update the paragraph text based on whether the user is in the control or test variant of the experiment.
views.py
from django.http import HttpResponse
from posthog import Posthog


def home(request):
    distinct_id = 'placeholder-user-id' 
    enabled_variant = posthog.get_feature_flag('my-cool-experiment', distinct_id)
    
    paragraphText = 'Placeholder text'
    if enabled_variant == "control":
        paragraphText = "Control variant!"
    elif enabled_variant == "test":
        paragraphText = "Test variant!"


    # rest of your code

When you restart your app and refresh the page, you should see the text updated to either Control variant! or Test variant!.

💡 Setting the correct distinctId:

You may notice that we set distinctId = 'placeholder-user-id' in our flag call above. In production apps, to ensure you fetch the correct flag value for your user, distinctId should be set to their unique ID.

For logged-in users, you typically use their email or user ID as their distinctId. For logged-out users, assuming they made their request from a browser, you can use values from their request cookies. See an example of this in our Nuxt feature flags tutorial.

5. Include the feature flag when capturing your event

To ensure our goal metric is correctly calculated for each experiment variant, we need to include our feature flag information when capturing our $pageview event.

To do this, we add the $feature/my-cool-experiment key to our event properties:

views.py
# rest of your code


posthog.capture(
  distinct_id, 
  '$pageview',
  {
    '$feature/my-cool-experiment': enabled_variant
  }
)


# rest of your code

Now PostHog is able to calculate our goal metric for our experiment results:

Experiment results in PostHog

Further reading

Comments