How Can I Use My Existing Analytics Stack on Data Generated by Context?
Context doesn't generate any new Segment events. However the hooks useActiveExperiment and useActiveVariant return the ids for the experiment and variant a user is currently seeing. You can use these to enrich your existing Segment events by passing them to anidentify event, or whatever works with your existing analytics implementation. A user will only be a part of one experiment and one variant at a time.
We recommend capturing both the variant and experiment ids for each user. That way you can, for example, explore your events by pivoting on which variant users saw in your visualization tool of choice, or build a statistical model around experiment treatments.
How are Results Calculated?
Our results are calculated using Bayesian Estimation. Essentially we use machine learning to create a model of user behavior for each variant in an experiment. Bayesian methods are less prone to error, generally require smaller sample sizes, and allow you to confidently know there is no difference between groups - all things traditional t-tests are bad at.
Practically speaking that means results are presented in terms of how much an event changed and how confident we are in that change, instead of having to use something less intuitive like p-values. For example - 95% confident the event Checkout increased by 10%. By default all hypotheses require 95% confidence to be appear in the results list.
How Do I Integrate with a Server Side Rendered Application or a Custom Segment Implementation?
The decision engine automatically fetches a JSON file of feature states (aka the dataFile), and accesses the window's Segment object, but these things are configurable. Here are some of most common optional parameters forcreateDecisionEnginereferenced in Integration Steps. You can see the full type annotation using any typescript IDE.
deviceId - if Segment is not accessible from the window you must pass in the users anonymous id
The deviceId must be the Segment anonymous id analytics.user().anonymousId()in order for results to calculated correctly.
fetch - if you are rendering the application using Node or another platform that doesn't have fetch globally available, you'll need to pass it in here.
dataFile - If you are server side rendering we recommend manually fetching and caching the data file. Pass it into this parameter as JSON. This should improve performance by removing the latency of your client refetching the file every time your app is rendered. Manually fetching the file doesn't effect billing, even if you make fewer API calls by reusing a data file multiple times.
How do I debug or introspect what the SDK is doing?
The createDecisionEngine function shown in Integration Steps takes a logLevel parameter that will make it print why each feature is turned on and off if it is set to debug. The logLevel parameter expects one of the fields from the LogLevel object, which is also exported from the core package.
How do I tell which version of a data file a user is seeing?
When you make changes to your dashboard, it takes about 5 minutes for the data file to fully propagate through the caching layer of our CDN. In that time some users will see the current data file, and some will see the previous. To debug which version of the data file a user is seeing, each data file has a revision field, which is a hash of the data file's contents. The useRevision hook returns this field.
How do I partially roll out a feature?
Many teams want to show a feature to only a small percentage of their users when it firsts launches, because they want to see if it effects metrics they care about. These partial rollouts are experiments in disguise. Context makes this explicit with ON_OFF experiments. That way you can leverage the statistical tools in Context to monitor your metrics and you can reason about your application state more easily. If a feature isn't part of an experiment it falls back to it's default state for all users.
How do I develop locally?
Treat it like any other environment. Pass the id for your development environment into the SDK.