Experiments

An experiment runs on a flag you already have. It takes part of the flag’s traffic in one environment (the users one rule matches, or everyone who reaches the default), splits it at random between two or more variants, and measures which did better on the metrics you pick, such as completed checkouts or errors. featflags collects the data and computes the results; there’s no warehouse to connect.

Experiments are on every plan. Free runs one at a time, Team 20, and Business as many as you like, with bandits.

Your users stay anonymous

To measure an experiment, SDKs send featflags which variant each user saw and the events you track. They never send who the user is. Each user is reported under an ID computed with your environment’s experiment key, which you generate and keep, and which featflags never sees. Without it, nobody can work out which person an ID belongs to, and the same person has unrelated IDs in two experiments, so experiments can’t be joined together either.

An event is a name and an optional number (checkout.completed, 49.90), nothing else: no properties, no attributes, no page, device or IP address. Times are kept to the minute. Per-user rows are deleted 30 days after the experiment stops, and the results keep only totals per variant. Security and privacy has the details.

The other side of this: results can’t be broken down by country, plan or browser, since none of that is collected. To compare segments, run the experiment on a rule that matches only the segment.

1. Generate the experiment key

Each environment needs its own key before an experiment can start there. In the app, open the environment’s settings and choose Generate key: it’s created in your browser and shown once, as xk_ followed by 64 hex digits. Copy it into your servers’ SDK configuration, as a secret. featflags keeps only a short fingerprint of it, enough to check that SDKs use the right one.

Or generate it in a terminal with the CLI, which sends only the fingerprint too:

featflags experiment-key generate --project acme --env production

Where the key goes:

• Servers hold the key itself, as an SDK option (experimentKey, experiment_key or similar; see your SDK’s page).
• Browser and mobile apps of signed-in users get seeds instead: your backend computes the user’s seeds with a server SDK and hands them over at sign-in, as it does tokens for private segments, so the key never ships in your app.
• Anonymous visitors have no backend session to hand seeds over, so the app can hold the key itself. A key in an app is public: anyone can read it from the bundle. The app says so when you copy it.

If the key leaks, or you lose it, generate a new one. Every user gets new IDs with a new key, so rotating while experiments run in the environment stops them first; the app (or --stop-running on the CLI) asks you to confirm. SDKs still using the old key send nothing and log why.

2. Define metrics

Metrics are defined once per project, on the project’s Metrics page, and any experiment can use them.

Field	What it does
Event	The event name it counts, as your code tracks it: lowercase letters, digits and _ . : -, up to 64 characters. Several metrics can count one event.
Type	Conversion: whether a user had the event at least once. Count: how many times. Sum: the total of its values. Mean: the average value per event.
Window	How long after a user first sees the experiment their events count, from 1 hour to 90 days, or until the experiment stops (the default).
Direction	Whether more is better (the default) or less, such as for errors or load time.
Cap	For counts and sums, the most any one user can add, so one outlier doesn’t swamp the average.

3. Track events

Call track where the thing you measure happens. In a browser, for the signed-in user:

flags.track('checkout.completed', 49.9)

On a server, with the user it’s about:

flags.track('checkout.completed', order.total, { key: user.id })

An event name that isn’t valid, or a value that isn’t a finite number, is logged once and dropped; track never throws. SDKs send an event only when a running experiment counts it, so tracking freely costs nothing. A user’s conversion can happen on your server while they saw the variant in the browser: as long as both use the same user key, featflags joins them.

4. Run the experiment

Open the flag, pick the environment, and on the rule or the default whose traffic you want to split choose Run an experiment:

• Variants: two to ten of the flag’s variants, with any split (50/50, or 90/10 to limit the risk), and which one is the control.
• Primary metric: the one that decides the experiment, and up to ten guardrails, metrics you watch for harm, such as page errors.
• Analysis: Bayesian (the default) with the chance to beat control that calls a winner, from 90% to 99% (95% by default); or frequentist, with a significance level of 1%, 5% or 10%, sequential by default so you can look at results as often as you like.
• Maximum length: 1 to 365 days, 90 by default, after which it stops on its own.

Start writes the split into the flag’s targeting as an ordinary change, so the environment’s required comments and approvals apply: in an environment with approvals, the experiment starts once the change is approved. A flag can run one experiment per environment at a time. While it runs, the experiment owns its rule’s or default’s serve, and edits that would change it are refused until you stop it; everything else on the flag works as usual. Experiments can’t run on a flag that has a release plan in the same environment.

The same start, from the API:

curl -X PUT -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' "$API/projects/acme/metrics/checkout-completed" \
  -d '{"event":"checkout.completed","type":"conversion","window":"7d"}'
curl -X POST -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' "$API/projects/acme/experiments" \
  -d '{"key":"checkout-cta","flag":"new-checkout","environment":"production","audience":{"rule":"r1"},"primary":"checkout-completed"}'
curl -X POST -H "Authorization: Bearer $TOKEN" "$API/projects/acme/experiments/checkout-cta/start"

5. Read the results

Results are computed every hour while the experiment runs, and you can refresh them by hand at most every 5 minutes. For each metric and each variant against the control, the results page shows:

• The lift: how much the variant changed the metric, relative to the control, with its interval.
• The chance to beat control (Bayesian) or whether the interval excludes zero (frequentist).
• A verdict: better, worse, inconclusive, or too early.

A Bayesian verdict and its interval always agree: at the default 95% threshold the interval shown is the central 90%, so better means the whole interval is above zero. The page labels each interval with its level.

Before any verdict, featflags checks that the experiment is healthy:

• Too early: no verdict until every variant has 100 users, 10 of them with the metric’s event.
• Sample ratio: if the number of users per variant is too far from the split you set (p < 0.001), something is assigning users wrongly, any comparison would be biased, and no verdict is shown.
• Users in several variants: users who saw more than one variant (because their key changed, say) are left out and counted; above 1% the page warns.

Guardrails aren’t corrected for multiple comparisons, since they look for harm, and a worse guardrail is called out whatever the primary metric says. With several treatment variants, a frequentist primary metric is tested at a correspondingly stricter level.

The results page also lists every change to the flag in that environment while the experiment ran, since a change to who reaches the rule changes what’s measured.

6. Stop it

Stop ends the measurement at once and computes the final results. Choose a variant to serve the experiment’s traffic from then on, usually the winner (serving it is a change like any other, so it can wait for approval), or leave the split as it is. A stopped experiment can’t run again; Copy makes a new draft with the same setup.

Bandits

A bandit is an experiment that moves traffic toward the variant that’s winning its primary metric while it runs, instead of holding the split: useful when you’d rather lose less to the worse variants than measure them precisely, such as for a headline or a default sort order. Bandits are on Business.

• It starts with an even split and, after a burn-in period (a day by default) and once each variant has 100 users, looks again every hour, 6 hours or day.
• Each look gives every variant the probability that it’s the best, and serves it that share of the traffic, never less than a floor (1% by default) so no variant is starved. A look that would move nothing by a percentage point or more keeps the split.
• Its primary metric is a conversion, count or sum, and its results are Bayesian.

Moving the split moves some users from one variant to another. Results count every user for the variant they saw first, and weigh each period’s users alike, so a variant isn’t flattered by getting more traffic in a better week; the page shows how many users moved. A bandit suits what can change for a user from one visit to the next; for anything users would notice changing, run an ordinary experiment.