Spinnaker – Using Spinnaker for Automated Canary Analysis

Docs: Canary Overview

Mon, 01 Jan 0001 00:00:00 +0000

Canary is a deployment process in which a change is partially rolled out, then evaluated against the current deployment (baseline) to ensure that the new deployment is operating at least as well as the old. This evaluation is done using key metrics that are chosen when the canary is configured.

Canaries are usually run against deployments containing changes to code, but they can also be used for operational changes, including changes to configuration.

The canary process is not a substitute for other forms of testing.

Prerequisites

Have metrics to evaluate

Your application might send performance metrics which are published and available by default. You can also install a monitoring agent to collect more comprehensive metrics, and you can instrument your code to generate further metrics for that agent. In any case, you need to have access to a set of metrics, using some telemetry provider, which Kayenta can then use to make the canary judgment.

Support is built in for Stackdriver , Datadog , Prometheus , Signalfx , and New Relic .

Set up your canary environment

Before you can configure canary analysis and create canary stages for your pipelines, your Spinnaker administrator needs to enable canary for your installation .

How to make Canary work in Spinnaker—the high-level process

In Spinnaker, create one or more canary configurations .

The configuration provides the set of metrics for use in all pipeline canary stages that reference it, plus default scoring thresholds and weights—defaults that can be overridden in a canary stage

You can configure each metric flexibly, to define its scope and whether it fails when it deviates upward or down. You can also group metrics logically and assign weights to groups to control their impact on the final score.

You can think of this configuration as a templated set of queries against your metric store.
In any deployment pipeline that will use canary, add one or more canary stages .

The canary stage includes information that scopes the templated query (canary config) to a specified set of resources and time boundaries.

Docs: Best practices for configuring canary

Mon, 01 Jan 0001 00:00:00 +0000

Don’t put too many metrics in one group

If you have many metrics in a group and one fails while the rest pass, the group gets a passing score overall.

For truly critical metrics (e.g., error rates), use the critical: true flag in the metric configuration. This causes the entire canary to fail immediately (score = 0) if that metric is classified as High or Low, regardless of group scores.

Alternatively, you can put an important metric in a group by itself to ensure that if it fails, the whole group fails.

Compare canary against baseline, not against production

You might be tempted to compare the canary deployment against your current production deployment. Instead always compare the canary against an equivalent baseline, deployed at the same time.

The baseline uses the same version and configuration that is currently running in production, but is otherwise identical to the canary:

Same time of deployment
Same size of deployment
Same type and amount of traffic

In this way, you control for version and configuration only, and you reduce factors that could affect the analysis, like the cache warmup time, the heap size, and so on.

Run the canary for enough time

You need at least 50 pieces of time series data per metric for the statistical analysis to produce accurate results. That is 50 data points per canary run, with potentially several runs per canary analysis. In the end, you should plan for canary analyses several hours long.

You will need to tune the time parameters to your particular application. A good starting point is to have a canary lifetime of 3 hours, an interval of 1 hour and no warm-up period (unless you already know your application needs one). This gives you 3 canary runs, each 1 hour long.

Carefully choose your thresholds

You need to configure two thresholds for a canary analysis:

marginal

If a canary run scores below this threshold, the whole canary fails immediately.
pass

The final canary run must score at or above this threshold for the analysis to be considered successful. Otherwise it fails.

Note: Threshold comparisons are inclusive (≥). A score of exactly 95 with a pass threshold of 95 results in a pass.

These thresholds are very important for the analysis to give an accurate result. You need to experiment with them, in the context of your own application, its traffic and its metrics.

Keep in mind that your configuration will be refined over time. Don’t think of it as something you set up once and never think about again.

Good starting points:

marginal threshold of 75
pass threshold of 95

Carefully choose the metrics to analyze

You can get started with a single metric, but in the long run, your canaries will use several.

Use a variety of metrics that reflect different aspects of the health of your application. Use these three out of the four “golden signals,” as defined in the SRE Book :

latency
errors
saturation

If you consider some other specific metrics critical, place them in their own group in the canary configuration. That allows you to fail the whole canary analysis if there is a problem with one of those specific metrics. You can also use the criticality flag on the individual metric.

Create a set of standard, reusable canary configs

Configuring a canary is difficult, and not every developer in your organization will be able to do so. Also, if you let all the teams in the org manage their own configs, you will likely end up with too many configs, too many metrics, nobody will know what is happening, and people will be afraid to change anything.

For these reasons, it’s a good idea to curate a set of configs that all the teams can reuse.

Use retrospective analysis to make debugging faster

It takes a long time to configure a canary analysis. It can take a long time to debug it too, partly because with a long-running canary analysis you have to wait a long time for the analysis to finish before you can refine it.

Fortunately, a Canary Analysis stage can be configured to use a retrospective analysis instead of a real-time analysis. This analysis is based on past monitoring data, without having to wait for the data points to be generated. With this mode, you can iterate more quickly on the development of the canary configuration.

Tip: Use the copy-to-clipboard icons in the canary stage execution details view to capture the needed timestamps.

Compare equivalent deployments

To compare the metrics between baseline and canary, Kayenta needs the exact same metrics for both. This means that metrics should have the same labels. Problems can arise if the metrics are labeled with their respective instance names.

If you see that Kayenta is not actually comparing the metrics, confirm that the queries that it executes against your monitoring system return metrics with the same labels.

If you’re using Stackdriver, you can use the Google APIs Explorer to debug such problems.

Some configuration values to start with

Although these values are not necessarily “best practices,” they are reasonable starting points for your canary configs:

Setting	Value
canary lifetime	3 hours
successful score	95
unhealthy score	75
warmup period	0 minutes
frequency	60 minutes
use lookback	no

Docs: How canary judgment works

Mon, 01 Jan 0001 00:00:00 +0000

Note: Automated canary analysis in Spinnaker is designed to support pluggable judges. This document describes how the default judge (NetflixACAJudge) works.

To assess the quality of a canary deployment against a baseline, metrics from both deployments are compared in order to check for significant degradation. This is done in two phases:

Metric collection (retrieval)

This phase retrieves the key metrics from the baseline and canary deployments. These metrics are typically stored in a time-series database, and include a set of tags or annotations that identify which deployment the data was collected from (canary or baseline).

This phase is performed by Kayenta, not by the judge. Besides the default judge, it is possible to plug in a custom judge, and metric collection is not the responsibility of the judge. The judge merely receives timeseries from Kayenta and analyzes those.
Judgment

In this phase Spinnaker compares those metrics and renders a decision to pass or fail the canary (that is, was there a significant degradation in the metrics?) The judgment can also be configured to continue on with a canary when the result is “marginal.”

The judgment consists of four main steps, and they’re described below.

Step 1: Data validation and NaN handling

Data validation ensures there’s data for the baseline and canary metrics before analysis begins. The behavior depends on the nanStrategy configuration:

NaN strategies

remove: NaN values are filtered out of the metric arrays. If this results in an empty array, the metric is classified based on other flags. This is the default behavior when nanStrategy is not specified or set to an unrecognized value.
replace: NaN values are replaced with 0.0. The metric proceeds to comparison even if all original values were NaN (the array will contain zeros).

Classification when data is missing

When either baseline or canary has no data after NaN handling:

`nanStrategy`	`mustHaveData`	`critical`	Classification
`remove`	`false`	`false`	`Nodata`
`remove`	`false`	`true`	`Nodata` (critical)
`remove`	`true`	`false`	`NodataFailMetric`
`replace`	any	any	`Pass` (arrays have zeros)

Nodata: Metric is excluded from group score calculations (doesn’t hurt or help the score), but counts toward the 50% NODATA threshold.
NodataFailMetric: Metric counts as a failure in group score calculations.

Tip: Use nanStrategy: replace for error metrics where no data means zero errors. Use mustHaveData: true for metrics that should always have data.

Step 2: Data cleaning (outlier removal)

After NaN handling, the judge can optionally remove outliers from the data.

Outlier detection

When outliers.strategy: remove is configured, the judge uses an IQR-based detector with reduced sensitivity:

Calculate Q1 (25th percentile) and Q3 (75th percentile)
Calculate IQR = Q3 - Q1
Calculate standard Tukey fences: [Q1 - factor×IQR, Q3 + factor×IQR]
Also calculate 1st and 99th percentiles
Final fences: [min(P1, lower_fence), max(P99, upper_fence)]

The default outlierFactor is 3.0. Values outside the final fences are removed.

Note: The reduced sensitivity approach makes outlier detection less aggressive than standard Tukey’s test, keeping more data points.

Step 3: Metric comparison (classification)

The judge compares canary and baseline data for each metric and assigns a classification indicating whether there’s a significant difference.

Classification labels

Label	Meaning
`Pass`	No significant difference detected
`High`	Canary is significantly higher than baseline
`Low`	Canary is significantly lower than baseline
`Nodata`	Insufficient data for comparison
`NodataFailMetric`	Missing data for a metric with `mustHaveData: true`
`Error`	Classification failed due to an internal error

Mann-Whitney U test

The judge compares the canary and baseline metric values to determine if there’s a real difference or just random noise. It uses a statistical test that answers: “Are these two sets of numbers meaningfully different?”

In plain terms:

The judge needs to be 98% confident there’s a real difference before flagging a metric
Small differences that could be random noise are ignored
The test doesn’t assume your data follows any particular pattern (like a bell curve)

Technical details:

The Mann-Whitney U test is a nonparametric test that compares two distributions. A metric is classified as High or Low only when:

The 98% confidence interval falls entirely outside a tolerance band
The effect size exceeds the configured threshold (allowedIncrease/allowedDecrease)
The metric’s direction allows that classification

The tolerance band is ±(0.25 × |Hodges-Lehmann estimate|), which creates a “dead zone” around zero where small differences are ignored.

Effect size and thresholds

The effectSize configuration controls classification sensitivity:

allowedIncrease: Minimum ratio for a High classification (default: 1.0)
allowedDecrease: Maximum ratio for a Low classification (default: 1.0)
criticalIncrease/criticalDecrease: Thresholds for critical failures

Important: Effect size thresholds are secondary gates. A metric must first show statistical significance before the effect size is checked. This means a metric with a 20% increase might still pass if the confidence interval is wide. If the effect size is NaN (e.g., when a mean is zero), the threshold check is skipped and only statistical significance determines the classification.

Critical metrics

When a metric has critical: true and is classified as High or Low with an effect size exceeding criticalIncrease/criticalDecrease, it becomes a critical failure. This immediately sets the canary score to 0.

Step 4: Score computation

After classification, the judge computes group scores and a summary score.

Group scores

For each metric group:

Group Score = (Pass count / Total count) × 100

Where Total count includes Pass, High, Low, and NodataFailMetric classifications. Metrics classified as Nodata are excluded from the count.

If all metrics in a group are Nodata, the group gets an effective score of 100 (it doesn’t penalize the canary).

Summary score

The summary score is a weighted average of group scores:

Groups with configured weights use those weights
Unweighted groups share the remaining weight equally: (100 - sum_of_configured_weights) / number_of_unweighted_groups

Caution: If configured weights sum to 100 or more, unweighted groups receive no positive remaining weight (they may effectively have zero or even negative influence). Avoid having group weights sum to 100 or more.

Automatic failures

The canary automatically fails (score = 0) in these cases:

Critical metric failure: Any metric with critical: true that is classified as High or Low and exceeds the critical effect size threshold (or has a NaN effect size, in which case the threshold check is skipped)
50% NODATA rule: If 50% or more of all metrics are classified as Nodata

Muted metrics

Metrics with muted: true are excluded from all scoring calculations but still appear in results for visibility.

Final classification

The summary score is compared against thresholds (comparisons are inclusive):

Score ≥ passThreshold → Pass
Score ≥ marginalThreshold → Marginal
Otherwise → Fail

Reference

Effect size measures

The judge supports two effect size measures:

Measure	Formula	Range	Default	Best for
`meanRatio`	canary_mean / baseline_mean	0 to ∞	1.0	Latency, throughput metrics
`cles`	Proportion of pairs where canary > baseline	0 to 1	0.5	When means can be zero

Note: meanRatio returns NaN if either mean is zero. Use cles for metrics like error counts where zero values are common.

Result metadata

Each metric result includes statistics for debugging:

controlMetadata.stats: {min, max, mean, std, count} for baseline
experimentMetadata.stats: {min, max, mean, std, count} for canary
resultMetadata.ratio: The mean ratio (may be NaN)

These statistics are calculated after NaN handling and outlier removal.

Hard-coded constants

These values are built into NetflixACAJudge and cannot be configured:

Constant	Value	Purpose
Confidence level	98%	Mann-Whitney U test confidence interval
Tolerance	0.25	Tolerance band = ±(0.25 × estimate)
NODATA threshold	50%	Canary fails if ≥50% metrics are Nodata
Reduce sensitivity	true	Outlier detection uses permissive fences

Special cases

Identical data: If baseline and canary arrays are identical after transformation, the metric is classified as Pass with ratio 1.0.
Degenerate distributions: If both arrays have only one unique value, tiny Gaussian noise is added to enable the Mann-Whitney test.

Worked examples

Example A: Error spike with NaN baseline

Scenario: An error count metric where the baseline has no errors (all NaN) but the canary has errors.

Setting	Value
`nanStrategy`	`replace`
`direction`	`increase`
`critical`	`true`

What happens:

Baseline: all NaN → replaced with zeros, mean = 0
Canary: positive values, mean > 0
Classification: High (canary significantly higher)
Ratio: NaN (because baseline mean is 0)
Result: Canary fails with score 0 (critical metric classified High)

Key insight: A metric can be classified High even when the ratio is NaN. The statistical test compares distributions, not just means.

Example B: No errors on either side

Scenario: Both baseline and canary have no errors (all NaN values).

Setting	Value
`nanStrategy`	`replace`

What happens:

Both arrays: all NaN → replaced with zeros
Arrays are identical after transformation
Classification: Pass with reason “data are identical”
Ratio: 1.0

Example C: Latency increase within tolerance

Scenario: Latency increased by 23%, but the canary passes.

Setting	Value
`direction`	`increase`
`allowedIncrease`	`1.2`
Observed ratio	`1.23`

What happens:

The ratio (1.23) exceeds allowedIncrease (1.2)
However, the 98% confidence interval is wide due to variance
The lower bound of the CI doesn’t exceed the tolerance band
Classification: Pass

Key insight: Effect size thresholds are secondary gates. Statistical significance must be established first. Noisy data can cause metrics to pass even when the observed ratio exceeds the threshold.

Example D: Critical latency failure

Scenario: Latency increased by 60% on a critical metric.

Setting	Value
`direction`	`increase`
`critical`	`true`
`criticalIncrease`	`1.5`
Observed ratio	`1.6`

What happens:

Statistical test shows significant increase (CI outside tolerance)
Ratio (1.6) exceeds criticalIncrease (1.5)
Metric is critical: true
Classification: High (Critical)
Result: Canary fails with score 0 regardless of other metrics

FAQ

Why is my metric High but its ratio is NaN?

The ratio is calculated as canary_mean / baseline_mean. If either mean is zero, the ratio is undefined (NaN). However, the Mann-Whitney test compares the full distributions, not just means, so it can still detect a significant difference.

Why did my canary Pass even though the ratio exceeds allowedIncrease?

The effect size threshold is a secondary check. The metric must first show statistical significance (98% CI outside the tolerance band). If the data is noisy, the confidence interval may be too wide to establish significance.

Why did my canary get a score of 0 when only one metric failed?

The metric likely has critical: true configured. Critical metrics that are classified as High or Low immediately set the canary score to 0.

What’s the difference between Nodata and NodataFailMetric?

Nodata: Metric had no data but wasn’t required. It’s excluded from scoring but counts toward the 50% NODATA threshold.
NodataFailMetric: Metric had mustHaveData: true but no data. It counts as a failure in group score calculations.

Other classifiers

NetflixACAJudge uses the Mann-Whitney classifier by default. Kayenta also includes two other classifiers that are not recommended for production:

MeanInequalityClassifier: Compares means directly without statistical testing. Primarily used for benchmarking.
RandomClassifier: Assigns random Pass/High/Low classifications. Intended for testing only.

These classifiers are not exposed in the standard configuration UI.

Docs: Configure a canary

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Add a canary stage to a pipeline

Mon, 01 Jan 0001 00:00:00 +0000

If you have enabled canary for your application and have one or more canary configs prepared , you can now add a canary stage to your pipeline and configure it to perform canary analysis for your deployment.

This stage type is for the canary analysis only. The canary stage doesn’t perform any provisioning or cleanup operations for you. Those must be configured elsewhere in your pipeline.

About the canary stage

The canary analysis can be performed over data points collected beginning from the moment of execution and into the future, or it can be performed over a specified time interval.

Real-time versus retrospective analysis

A real-time analysis means that the canary analysis is performed over a time interval beginning at the moment of execution. In a retrospective analysis the canary analysis is performed over an explicitly specified time interval (likely in the past).

Metric scope

Metric scope defines where, when, and on what the canary analysis occurs. It describes the specific baseline and canary server groups, the start and end times and interval, and the cloud resource on which the baseline and canary are running.

You can also refine the scope using extended params .

Define the canary stage

In the pipeline in which you will run the canary, click Add stage.

This pipeline needs to be in an application that has access to the canary configuration you want to use.
For Type select Canary.
Give the stage a name, and use the Depends On field to position the stage downstream of its dependencies.
Select the Analysis Type—either Real Time or Retrospective.
- Real Time
  
  The analysis happens for a specified time period, beginning when the stage executes (or after a specified Delay).
  
  For Real Time, also specify the number of hours to run (Lifetime).
- Retrospective
  
  Analysis occurs over some specified period. Typically, this is done for a time period in the past, against a baseline deployment and a canary deployment which have already been running before this canary analysis stage starts.
  
  Note that this analysis might analyze data for resources which no longer exist, for which there are still published time series.
Specify the analysis configuration:
- Choose the Config Name.
  
  This is the canary config you created here . That configuration must be visible to this application. By default, all configs are visible to all applications, but your canary might be set up so that each config is limited to the application in which it is created.
- Set a Delay.
  
  For real-time analyses, how many minutes to wait before starting the analysis. The gives both the baseline and canary a chance to get “warmed up” before they’re expected to provide meaningful metrics. Leave blank for a delay of zero minutes.
- Set the Interval.
  
  This is how frequently (in minutes) to capture and score the metrics.
- For Lookback Type, select Growing or Sliding.
  - In a growing analysis, a judgment is taken every [interval] minutes, but each judgment goes all the way back to the beginning of the Lifetime.
  - A sliding Lookback also makes a judgment every [interval], but each judgment only looks at the data from the most recent lookback duration. It would not be unusual for the Interval and the look-back duration to be the same, but they don’t have to be.
Describe the metric scope.

You can enter server groups and regions by name, but you can also click the magic wand here to automatically populate the fields with expressions that resolve to available resources. Those resources are a starting point, which you can edit to match the specific resources you need.
- Baseline
  
  The server group to treat as the “control” in the canary analysis—that is, the deployment against which to compare the canary deployment.
- Baseline Region
  
  The region in which the baseline server group is deployed.
- Canary
  
  The server group to treat as the experiment in the analysis.
- Canary Region
  
  The region in which that canary server group is deployed.
- Step
  
  The interval, in seconds, for the metric time series.
- Start Time and End Time (for retrospective)
  
  For a retrospective analysis, the specific time span over which to conduct the analysis.
- Extended Params
  
  Add any additional parameters, which are specific to the metric sources and which can be used to refine the scope of the analysis. These parameters can provide variable bindings for use in the expansion of custom filter templates specified in the canary config .
Adjust the Scoring Thresholds, if needed.

The thresholds are pre-populated based on those configured in the main canary config , but you can override them here.
Specify the accounts you’re using for metrics and storage.
- The Metrics Account points to the telemetry service provider account you configured here .
- The Storage Account points to the GCS or S3 account you configured here .