Spinnaker – Managed Delivery

Docs: Managed Delivery Overview

Mon, 01 Jan 0001 00:00:00 +0000

What is Managed Delivery?

Managed Delivery is an initiative with two key goals.

Provide native means for Spinnaker users to declaratively manage software delivery.
- With a declarative model, users describe the desired end-state of a deployed application. Spinnaker continually calculates how this desired state maps to cloud resources. When desired state diverges from actual cloud state, Spinnaker figures out how to safely bring the deployed state in-line with the desired.
- This is in stark contrast to the imperative pipeline model, which requires users to define every step of their delivery flow. Pipelines remain a powerful first class feature of Spinnaker that will offer greater flexibility than Managed Delivery for the foreseeable future. However, we feel the declarative approach is preferable for most delivery flows. Good pipeline design requires a depth of Spinnaker and cloud knowledge, while system failures impacting pipeline execution may leave environments in a transitional state without an automatic path back to normalcy. A declarative model solves both issues.
Provide Spinnaker Operators with powerful new means to raise the level of operational abstraction offered to their users.
- By informing how declarative models are resolved, operators can centrally maintain delivery best practices and automate change propagation without knowledge of how product teams have customized delivery pipelines.
- Examples range from offloading decision making about the regions a service should deploy into, to providing means to rapidly redeploy thousands of services running in hundreds of AWS accounts in order to address an OS level security vulnerability.

Contributing

If you are interested in contributing, the Managed Delivery initiative is under the auspices of the Spinnaker-as-Code SIG

Docs: Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

This means that we support only EC2 and Titus, and that we have limited feature and UI support. This guide walks through onboarding to Managed Delivery assuming that you are using EC2 or Titus.

Intro

To get started with Managed Delivery you’ll need an idea of what infrastructure you want to manage. You can pick a new piece of infrastructure that you want to create (a new load balancer, for example) or migrate an existing piece of infrastructure (an existing cluster, for example).

We recommend creating a new security group to test things out. This guide will walk you through creating a new managed security group.

Here are the steps you’ll complete:

First, you’ll create a YAML file to store your delivery config in.
Then, you’ll set up a Spinnaker pipeline to submit that YAML file to Spinnaker.
Finally, you’ll watch Spinnaker take actions to make your resource exist.

Background

Right now, most users manage infrastructure resources in the Spinnaker UI. These resources are security groups, load balancers, and clusters. To create these resources you manually define them by clicking the create button in the Spinnaker UI.

Managed Delivery changes this interaction. To define resources that Spinnaker manages, you’ll create a YAML-based “delivery config” file specifying the desired state of your infrastructure resources. “Desired state” is a new concept in Spinnaker that means you’re specifying what you want your infrastructure to always look like, regardless of how Spinnaker decides to make it look that way. In practice, this desired state will often look like the configuration options you’re used to defining through the Spinnaker UI.

To learn more about delivery configs, check out the Delivery Configs docs

Once you create a YAML delivery config, you’ll need to tell Spinnaker to manage it. This is done by submitting the file to Spinnaker’s API. You can submit a delivery config from your local machine, but that’s not very traceable or repeatable.

We recommend checking delivery configs into your git repository in a .spinnaker/ directory at the base level of your repository. In order to get Spinnaker to pick up your configuration (and monitor for changes) you should create a git-triggered pipeline that submits those files to Spinnaker.

An important note: Spinnaker is the source of truth for desired state of resources. You must submit your YAML files to Spinnaker every time they change.

All the instructions for this flow are captured in this doc.

Let’s get started!

Prerequisites

Spinnaker takes a lot of actions on your behalf, and we do that by using the permissions of a service account. Make sure you have one set up before you continue.

Create Your First Resource

We’re going to create a test security group. The new security group will be called YOUR_APP-md. For example, if the app that you’re using is keeldemo then the security group will be keeldemo-md.

We’re going to base this security group on your existing application security group. If you want to, locate your application security group in the Spinnaker UI and inspect the rules you have on it. This security group is usually auto-created for you and is identical to the name of your app.

To create a resource definition based on this security group, we’re going to use the Spinnaker API to export your existing application security group to a YAML definition (Note: this will have UI support in the future).

The base URL for exporting a security group is:

/managed/resources/export/aws/AWS_ACCOUNT_NAME/securitygroup/APP_NAME_OR_SG_NAME?serviceAccount=SERVICE_ACCOUNT

Replace AWS_ACCOUNT_NAME with the account your app is in. You can check the credentials endpoint in Spinnaker API for a list of valid account names.
Replace APP_NAME_OR_SG_NAME with either the name of your application or the name of the security group you want to export. You probably have a security group with the same name as your app, which is the one you should use here.
Replace SERVICE_ACCOUNT with the same service account you created or found in the prerequisites step.

If you’ve gotten all the information right a resource configuration will show up. See the API docs for more information on usage.

Save Your Resource In Your Git Repository

The next step in managing a resource is to check it into a git repository. Create a .spinnaker/ directory in the git repository of your application. This should be at the top level of your repo. In this directory, create a file called spinnaker.yml (your delivery config) and paste in this skeleton delivery config:

name: a-manifest-name # change me
application: yourapp # change me
artifacts: []
environments:
  - name: testing # this is a fine default but you can change it
    constraints: []
    notifications: []
    resources: []

Be sure to change the manifest name (a human readable name) and the application to your app. You can also change the environment name.

Next, we will add the security group to the list of resources.

Paste in the YAML output you got from the export endpoint in the previous step into the list of resources. It’ll look something like this:

name: a-manifest-name
application: yourapp
artifacts: []
environments:
  - name: testing
    constraints: []
    notifications: []
    resources: # you can add more resources here, this is a list
    - apiVersion: ec2.spinnaker.netflix.com/v1
      kind: security-group
      # ... the other info here from the security group you just exported

Next, modify the resource you just added to have a new name! Edit the moniker section of the resource you just pasted:

...
    moniker:
      app: YOUR_APP_NAME
      stack: md
...

This will name your security group YOUR_APP_NAME-md. For example, my sample security group for my app keeldemo will have a moniker section like:

...
    moniker:
      app: keeldemo
      stack: md
...

which will name the security group keeldemo-md.

:stop_sign: STOP! Don’t commit your delivery config yet! Just save it locally.

Repository Permissions

Make sure your git repository is accessible by Spinnaker.

Setting Up Resource Submission

Since Spinnaker is the source of truth (and not Git) you need to submit your managed resource configuration to Spinnaker every time there are changes. There’s a Spinnaker stage for this! See the docs on Git-based-workflows for detailed instructions.

Testing Out Resource Submission

Now that you have a pipeline in your application that’s triggered off of a git commit to your repository, you can commit your first resource configuration file. Commit the file you created earlier (.spinnaker/spinnaker.yml) and push it to the master branch of your repository.

Iterating on Managed Configurations

It’s hard to get YAML right every time. We have an endpoint where you can check if your YAML is right and see if there is a diff between your desired state (in your YAML) and the current state of the resource.

In the swagger api view of your Spinnaker instance, scroll to the managed-controller and find the /managed/delivery-configs/diff endpoint.

Paste in your delivery config YAML file, choose application/x-yaml for the content type, and hit execute. If the resource YAML has an invalid format, you’ll get an error back with the reason why. If the resource YAML is correct, you’ll see some information about whether or not there is a diff, and what that diff is.

For more help with this endpoint, see the the API doc.

Is Anything Happening?

You can see the actions Spinnaker is taking on your behalf in the Tasks view for your app.

If you navigate to Security Groups you should be able to see a small flag on the resource that will show you it’s being manged. If you click on a managed resource you will see the available managed actions.

Under the actions you can click to see the resource definition that Spinnaker has for that resource. You can also see the history of that resource. The history view shows every diff Spinnaker has seen, and every action taken to resolve that diff.

Find Your Resource ID

For some API calls you need the resource ID. You can find this by clicking on the resource in the UI, and then clicking on Resource Actions -> Raw Source (right side on the panel that pops out). The ID will be in the the metadata section and will be human readable, like ec2:cluster:test:keeldemo-main.

Pausing Management

Sometimes you need to take a break. Maybe there’s a problem you’re trying to diagnose and you need Spinnaker to stop taking actions on a resource. Maybe there’s a bug and it’s preventing you from using Spinnaker to manage resources. That’s ok! It’s easy to pause management.

Navigate to your application in Spinnaker and click on the CONFIG tab in the top right. If you have 1 or more managed resources, a section will appear called Managed Resources. That section contains a pause/resume button which you can use to stop and start management of your Resources.

Cleaning Up / Opting Out

If you don’t want Spinnaker to manage your resource declaratively anymore you can delete the declarative config from Spinnaker:

DELETE /managed/delivery-configs/{deliveryConfigName}

This endpoint is available in the Swagger UI as well.

This will not delete the underlying infrastructure that was previously in your delivery config. It will only delete the managed definition and stop Spinnaker from taking action. Use the Spinnaker UI to delete any resources you created.

Next Steps

Hooray! You created your first managed resource. If you’d like, you can follow these same steps to create clusters, load balancers, and security groups for your application. You can add resources to the environment you’ve created by added them to the list of resources.

Docs: API

Mon, 01 Jan 0001 00:00:00 +0000

The endpoints for managed delivery are located in the managed-controller. The endpoints are easily accessible from the swagger api page of your Spinnaker instance.

This doc provides samples for how to use the endpoints.

Export an Existing Resource

The export endpoint (under the “managed-controller”) allows you to take an existing resource and export it to a yaml config.

GET /managed/resources/export/{cloudProvider}/{account}/{type}/{name}?serviceAccount=yourServiceAccount@company.com

These definitions can be stored in your git repo.

Validating yaml

The ad-hoc diff endpoint allows you to see if a resource config is valid, and to see the difference between config (desired state) and reality.

POST /managed/resources/diff -d "{YOUR_RESOURCE_CONFIG}"

If the resource configuration is valid you’ll see information returned about the resource, like whether or not there is a diff, what the diff is, and what the resource name will be. If the resource configuration is invalid you’ll see an error and some text indicating what is wrong about the schema.

Viewing a Resource

The UI will show a flag on each resource that is declaratively managed. If you click on the resource, you can view its definition (raw source)

If you’d rather hit the api directly, you can refer to the UI for the resource name, or the logs for the publish stage.

GET /managed/resources/{name}

Viewing a Resource History

Spinnaker will take actions to make sure that your resource matches what you’ve defined. To view those actions in a list you can hit the history endpoint:

GET /history/{name}

This endpoint shows you why an action was taken on a resource, and what that action was. Soon there will be a nicer UI for this.

Docs: Managed Delivery Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

Managed Delivery supports two types of delivery artifacts: debians and Docker images.

Delivery artifacts are meant to provide Spinnaker information about:

Where to locate the available versions of an artifact you care about
How to choose the latest version from a list of versions

Spinnaker needs this information to make decisions about when and how to roll out new versions as they become available. Artifacts are defined with name, type, and version strategy information.

Debians

** Warning: this is Netflix specific. Interested in contributing? Reach out to us in the #sig-spinnaker-as-code slack channel. **

All debians at Neflix are named and compared in the same way. For example, a debian might look like this:

# package name is keel
keel_0.353.0-h290.ae13adb_all.deb

We compare the semantic version located after the packagename_ (keel_ in this example) and before the -h.

Debians have status information on them. A debian can have any of these statuses:

snapshot
candidate
release

which are parsed from the semantic version of the debian.

To define a debian artifact for use with your delivery config, you’ll do something like this:

artifacts:
- name: keeldemo
  type: deb
  statuses: ["RELEASE"] # This is optional
  reference: my-debian-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name

If you provide status information, you limit the possible artifacts that can be deployed to your environment. This is useful if you have a test environment that only snapshots should be deployed to, or a prod environment that only release artifacts should be deployed to.

Status information is optional. If you leave it out, versions with any status will be considered eligible for deployment into your environment.

Docker Images

Docker versioning is denoted by the tag value. At Netflix, there are two main ways versions are structured:

# branch-jenkinsJob.commitSha
master-h5.5a52206

# semver-jenkinsJob.commitSha
v1.12.1-h1159.b839a00

Basic Configuration

In order to support the different ways of doing versioning, docker artifacts have some more config options. The basic structure is like this:

artifacts:
- name: emburns/spin-titus-demo
  type: docker
  reference: my-docker-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name
  tagVersionStrategy: branch-job-commit-by-job

Note that just like with pipelines, we don’t support using a constantly updated latest tag.

The key difference here is the tagVersionStrategy field, which indicates how we should sort a list of tags to choose the most recent software version.

The tagVersionStrategy Options:

increasing-tag: your tags are all integers, and we should choose the highest integer as the latest
semver-tag: your tags are a single semantic version (with or without a v prefix), and we should choose the largest semantic version
branch-job-commit-by-job: your tags are of the format “branch-jenkinsJob.commitSha” (like master-h5.5a52206), and we should choose the latest based on the highest jenkins job number (h5 in this example)
semver-job-commit-by-job: your tags are of the format “semver-jenkinsJob.commitSha” (like v1.12.1-h1159.b839a00), and we should choose the latest based on the highest jenkins job number (h1159 in this example)
semver-job-commit-by-semver: your tags are of the format “semver-jenkinsJob.commitSha” (like v1.12.1-h1159.b839a00), and we should choose the latest based on the largest semver (v1.12.1 in this example)

Advanced Configuration

If none of these work for you, you can provide your own regex to capture something to compare (using a capture group ). You must create a regex that parses valid tags and captures a string from them which can be compared either as an increasing number or a semantic version. The capture group must only capture one thing (you can’t have multiple capture groups).

For example, let’s say your tags look like master-h5.5a52206 and we didn’t have a built in strategy for this style. You’d define an artifact like this:

artifacts:
- name: emburns/spin-titus-demo
  type: docker
  reference: my-docker-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name
  tagVersionStrategy: increasing-tag
  captureGroupRegex: ^master-h(\d+).*$

The regex capture group (the parens in the regex expression) indicates that out of the string tag we should grab any numbers after the h and before the .. The version strategy indicates that we will take the result of the capture group and compare it as an increasing number.

Docs: Delivery Configs

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisite: read and complete the Getting Started document!

What is a Delivery Config?

A delivery config allows you to group resources together in a single file (e.g. spinnaker.yml) and specify how these resources work together to promote an artifact between environments.

Artifacts

An artifact represents the package or docker image that you want to promote in your delivery flow. For example, if you develop the package keeldemo (which is built into a debian) and that is the only package that runs in your application, you’d have the following artifact section:

artifacts:
- name: keeldemo
  type: deb
  reference: my-artifact   # optional human-readable reference to be used elsewhere in the config, defaults to artifact name
  vmOptions:               # only required for Debian artifacts, this information is used to determine how to bake a virtual machine image
    baseOs: bionic-classic # the base operating system for the virtual machine image
    regions:               # the regions to bake the image in (this should at least correspond to the regions you will deploy to)
    - us-west-2
    - us-east-1
    baseLabel: RELEASE     # the operating system label, optional and defaults to "RELEASE"
    storeType: EBS         # the storage type for the virtual machine image, optional and defaults to "EBS"

You can have multiple artifacts in your delivery config. We will watch for new versions of every artifact you define. In order to deploy a cluster that is running an artifact you’ll need to use it within a resource (in an Environment).

For detailed artifact information, please refer to the Artifacts page.

Environments

An environment represents all the infrastructure and instructions that are needed to run your package. For example, we usually see environments like dev, testing, staging, integration, and production. Each of these potentially have different resources, but are likely pretty homogeneous.

A simple and supported workflow looks like this:

Every build of your package should be deployed into your testing environment. This environment might have a lower capacity, and a load balancer that only allows your team to access the running application.
Once your build has been deployed to testing and has come up as healthy, it should be promoted to your next environment (called staging). Staging has a different capacity than testing.
(Coming soon…) Each build that has been healthy in staging is a candidate for production. Before deploying to production you must manually approve each version.

Defining an Environment

An environment is made up of a list of declarative infrastructure resources (like the resource you created in the getting started guide ). An environment is configured as follows:

environments:
- name: testing
  resources:
  - <full resource definition>
  - <another resource definition>

That’s all the config you need for a simple environment.

Environment Notifications

You can add environment level notifications. They apply to all resources in the environment. The config looks like:

environments:
- name: testing
  resources: # omitted for brevity
  notifications:
  - type: slack
    address: "#managed-delivery"
    frequency: verbose

There are several frequency options:

verbose: notification on task starting, completing, failing
normal: notification on task completing or failing
quiet: notification only for failure

There are two type options:

email: the address is the email address to send to
slack: the address is the slack channel to send to (the spinnakerbot must be in the channel for you to receive notifications)

Environment Constraints

Constraints control how an artifact progresses through environments.

The config looks like:

environments:
- name: staging
  resources: # omitted for brevity
  notifications: # omitted for brevity
  constraints:
  - type: depends-on
    environment: testing

This constraint definition says that an artifact must have been deployed successfully into the testing environment before it can be deployed into the staging environment.

A growing set of constraints are available, see Environment Constraints for more information.

Creating a delivery config

Let’s pull it all together! Delivery configs must be stored in a file called spinnaker.yml. The rough structure is:

name: sample-delivery-config
application: keeldemo
artifacts:
- name: keeldemo
  type: deb
  reference: my-artifact
  vmOptions: # details omitted for brevity
environments:
- name: testing
  notifications: # omitted for brevity
  constraints: []
  resources: # details omitted for brevity
  - kind: ec2/cluster@v1
    # details
  - kind: ec2/classic-load-balancer@v1
    # details
- name: staging
  notifications: # omitted for brevity
  constraints:
  - type: depends-on
    environment: testing
  resources: # details omitted for brevity
  - kind: ec2/cluster@v1
    # details
  - kind: ec2/classic-load-balancer@v1
    # details

This shortened delivery config shows how to promote a debian artifact through two environments.

Note that the first environment has no constraints.

Example

TODO eb: add an example

Docs: Environment Constraints

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisite: understand the environment concept offered by Delivery Configs

What is an Environment Constraint?

Environment constraints control how an artifact version flows through environments. A constraint can be as simple as ensuring an artifact isn’t promoted into Environment B unless it has successfully deployed into Environment A. Constraints can also specify that an artifact needs to have a canary evaluated by Kayenta before it can be deployed into an environment, while also defining how the canary should be deployed and evaluated.

Defining Constraints

Every environment defined in a Delivery Config supports a top-level constraints list property.

The following example ensures that artifact versions are only promoted into the staging environment after successful deployment to the test environment, and only if the current time is within the days Monday thru Thursday and the hours of 8am and 4pm.

environments:
- name: staging
  resources: # omitted for brevity
  notifications: # omitted for brevity
  constraints:
  - type: depends-on
    environment: testing
  - type: allowed-times
    windows:
    - days: Monday-Thursday
      hours: 8-16

Constraint Types and Ordering

Constraints are currently bucketed into stateless and stateful types. Stateless constraints can always be fully evaluated in a single pass and do not interact with users or external systems. The current stateless constraint types are depends-on and allowed-times.

Stateless constraints are always evaluated first, and must be satisfied before any stateful constraints are evaluated. Stateful constraints are a powerful construct, with the ability to mutate cloud state during their evaluation. Examples include deploying a new artifact version into an environment for the sake of canary analysis, or launching a smoke-test pipeline against a prior environment, gating promotion on its success.

This implicit ordering by type ensures that a pipeline constraint gating promotion to a production environment on smoke-tests run against a prior environment will only invoke the test pipeline after the artifact version under evaluation is deployed to the prior environment.

All stateful constraints can be manually overridden, such as when a canary failure is deemed to be expected for a given change, or a false alarm. When an environment defines multiple constraints, all must pass before a new artifact is promoted into the environment.

Available Constraints

Depends On

The depends-on constraint ensures that an artifact version will not deploy into the environment it is attached to, unless it has successfully deployed into the dependent environment.

Parameters environment: The name of the dependent environment. It must be defined within the same delivery config as the target environment containing the constraint.

Example

  constraints:
  - type: depends-on
    environment: testing

Allowed Times

The allowed-times constraints provides a time based gate on promotion into an environment. It can be used to ensure promotion to an environment only occurs during business hours, or quiet times.

Parameters windows: A list of time windows; multiple windows can be specified on a single constraint. The constraint is satisfied if the current time matches any single window. tz: Optional The timezone to evaluate the constraint in. This is optional, by default the timezone specified via the default.time-zone spring configuration property is used, not the JVM or OS timezone. If your organization operates in a single timezone, this property should likely be set that timezone.

Window Parameters hours: Supports ranges (i.e. 10-20) and comma separated lists (i.e. 5,6,7,8) or a combination of the two. days: Supports aliases (weekdays and weekends), as well as ranges and comma separated lists of days in their java standard (java.util.Calendar) short (mon-thu) or full (monday-thursday) names.

Example

  constraints:
  - type: allowed-times
    windows:
      - days: Monday-Wednesday,Friday
        hours: 8-16
      - days: Thursday
        hours: 14-16

This allows artifacts to deploy into the environment between 8am-4pm on Monday, Tuesday, Wednesday, and Friday or between 2-4pm on Thursday.

Manual Judgement

The manual-judgement constraint prevents an artifact from deploying to an environment without explicit approval via an external API client. UI support for approving or rejecting judgements as well as an interactive slack bot are under development but direct use of Spinnaker’s REST API is currently required to answer a manual judgement. See Interacting with and Overriding Constraints for details.

Parameters

timeout: (Optional); Type: Duration; Default Value: PT1W

If judgement is not provided within this time (starting when the constraint is initially evaluated for a given environment and artifact version), it is considered rejected.

Example

  constraints:
  - type: manual-judgement

Pipeline

The pipeline constraint launches a pipeline, optionally passing along constraint provided trigger parameters. The pipeline execution is periodically monitored and the constraint status derived from the final status of the execution.

Parameters

pipelineId: Type: String

The configuration id of the pipeline to be executed during constraint evaluation. If unknown, edit the given pipeline from the Spinnaker UI. From the edit configuration view, the pipelineId is the final part of the URL. The pipeline can be a part of any application, but must be runnable via the service account defined in the target environment’s delivery config.

retries: (Optional); Type: Integer

If set, pipeline failures will be retried up-to this many times if pipeline execution goes terminal for any reason. The constraint evaluation will fail once retries are exhausted.

parameters: (Optional); Type: Map<String, Object>

If set, the contents of this map are passed as trigger parameters when executing the pipeline. Expressions are not currently supported.

timeout: (Optional); Type: Duration; Default Value: PT2H

If the pipeline execution does not complete within this time, it is considered a failure.

Example

  constraints:
  - type: pipeline
    pipelineId: fn0rd5d1-6217-4fcf-aa3d-ce7f59af9cac
    retries: 1
    parameters:
      foo: bar

Canary

The canary constraint supports deploying experiment and control clusters across multiple regions within the target environment. Experiment/control cluster health is evaluated via Kayenta and the clusters removed once evaluation is complete. By default, a multi-region constraint must pass in all regions for the constraint to pass but this behavior is configurable.

The canary constraint currently only supports EC2. Keel configuration properties may need to be set depending on how Kayenta accesses time-series data and persists state.

System Configuration Properties

defaults.constraint.canary.metrics-account: Default Value: atlas-global.prod

Unless your time series data is stored in Netflix Atlas, this should be set to the equivalent value used with the Orca Kayenta stage.

defaults.constraint.canary.storage-account: Default Value: s3-objects

This should be set to the value of Kayenta’s default-storage-account.

Parameters

canaryConfigId: Type: String

The configuration id of the canary rules used for evaluation. If unknown, select a specific configuration from Spinnaker’s Canary Config UI. The canaryConfigId is the final segment of the resulting url.

beginAnalysisAfter: (Optional); Type: Duration; Default Value: PT10M

After the deployment of canary clusters completes, Spinnaker will wait this amount of time before launching the first Kayenta analysis. This allows for the filtering or inclusion of service warmup behavior in the analysis.

canaryAnalysisInterval: (Optional); Type: Duration; Default Value: PT30M

After the completion of any beginAnalysisAfter delay, Kayenta analysis are launched at this recurring interval, until post-beginAnalysisAfter runtime exceeds the lifetime parameter.

lifetime: (Optional); Type: Duration; Default Value: PT30M

Maximum runtime for the canary, exempting deployment times and any initial delay configured via beginAnalysisAfter.

cleanupDelay: (Optional); Type: Duration; Default Value: 0

When a canary fails due to a failing Kayenta score, Spinnaker will wait this amount of time before deleting canary clusters. Useful if engineers desire an opportunity to manually inspect instances involved in the failure.

marginalScore: Type: Integer

If Kayenta generates a canary score below the marginalScore, it is considered an immediate failure.

passScore: Type: Integer

If Kayenta generates a final canary score >= passScore, the constraint will pass.

regions: Type: Set<String>

Each region will get its own independently deployed and analyzed control/experiment pair.

capacity: Type: Integer

Control and experiment clusters have their min/max/desired pinned to this value.

source: Type: CanarySource

Defined in yaml as a Map<String, String> with the following properties:

account
cloudProvider
cluster

This should refer to an active cluster in the target environment deployed in all of the regions the constraint is configured to run canaries in. Control/experiment clusters are configured based on the source cluster (i.e. account, subnet, instance type, firewall rules).

minSuccessfulRegions: (Optional); Type: Integer; Default Value: 0

If set to >0 on a multi-region canary, the constraint will pass if Kayenta analysis passes in this many regions. I.e. if set to 2 on a constraint that defines a 3-region canary, the failure of a single region is ignored.

failureCancelsRunningRegions: (Optional); Type: Boolean; Default Value: true

If true, the failure of any region that violates the minSuccessfulRegions parameter triggers the immediate cancellation of any canaries still running in other regions.

metricsAccount: (Optional); Type: String

If set, overrides the defaults.constraint.canary.metrics-account property. This may be required in environments with multiple TSD instances.

storageAccount: (Optional); Type: String

If set, overrides the defaults.constraint.canary.storage-account property. Not typically required.

Example

  constraints:
  - type: canary
    canaryConfigId: fn0rd5d1-6217-4fcf-aa3d-ce7f59af9cac
    beginAnalysisAfter: PT3M
    canaryAnalysisInterval: PT10M
    lifetime: PT90M
    marginalScore: 75
    passScore: 90
    capacity: 2
    source:
      account: appaccount
      cloudProvider: aws
      cluster: app-prod
    regions:
      - us-east-1
      - us-west-2
      - eu-west-1
    minSuccessfulRegions: 2

Interacting with and Overriding Constraints

Apart from the depends-on constraint, all constraint types either expect interaction from a user (in the case of manual-judgement ) or can have their automated judgements overruled by a user.

Approving or rejecting Manual Judgement constraints from Slack

If your operator has configured Slack app integration with Spinnaker, then a very convenient way to approve managed deployments gated by a Manual Judgement constraint is to use interactive Slack notifications.

All you need to do is set up Slack notifications in your delivery config, which will enable interactive notifications to a Slack channel of your choice, where you can simply click a button to approve or reject the deployment of the artifact into the environment.

Consider the following simplified delivery config:

name: myspinapp
application: myspinapp
serviceAccount: myteam@mycompany.com
artifacts:
- name: myspinapp
  type: deb
environments:
- name: test
  constraints:
  - type: manual-judgement
  notifications:
  - type: slack
    address: "#myteam"
    frequency: verbose
  resources:
  - apiVersion: bakery.spinnaker.netflix.com/v1
    kind: image
    spec:
      artifactName: myspinapp
      baseLabel: RELEASE
      baseOs: xenial
      regions:
      - us-east-1
      storeType: EBS
      application: myspinapp  
  - apiVersion: "ec2.spinnaker.netflix.com/v1"
    kind: "cluster"
    spec:
      moniker:
        app: "myspinapp"
      imageProvider:
        deliveryArtifact:
          name: "myspinapp"
          type: "deb"
      locations:
        account: "test"
        regions:
        - name: "us-east-1"

Upon evaluating whether a new Debian artifact for myspinapp should be deployed into the test environment, Spinnaker will gate the deployment on the manual-judgement constraint and send a notification that looks like the following to the #myteam Slack channel.

Once a user in the channel clicks on the Approve button, the notification changes to reflect the approval:

:warning: Note that the frequency setting for notifications does not affect this behavior – interactive notifications will always be sent unless your operator has chosen to disable this feature entirely on your company’s instance of Spinnaker.

APIs to interact with constraints

In addition to Slack integration, the following API endpoints are currently available to interact with constraints.

Note: we limit to one delivery config per app, so these APIs currently use app name.

Reading status of pending and recent constraints

GET https://gate/managed/application/{application}/environment/{environment-name}/constraints?limit=20

Returns: List<ConstraintState> consisting of:

  String deliveryConfigName;
  String environmentName;
  String artifactVersion;
  String type;
  String status;
  Instant createdAt;
  String judgedBy;
  Instant judgedAt;
  String comment;
  Map<String, Object> attributes;

Setting or Overriding Constraint State

POST https://gate/managed/application/{application}/environment/{environment-name}/constraint

POST Body:

{
  "type": "canary",
  "artifactVersion":  "fnord-1.2.3",
  "status": "OVERRIDE_PASS",
  "comment": "overriding false positive kayenta failure"
}

Valid status values: PENDING, PASS, FAIL, OVERRIDE_PASS, OVERRIDE_FAIL

Docs: Git-Based Workflows

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker supports git-based workflows to create and update Managed Delivery configurations. This allows you to keep your Managed Delivery configuration in code and follow all the usual best practices like code review and approvals before a change is introduced to your managed resources, environments or deployments.

Git support is currently achieved via a pipeline configured with a trigger that can provide source repository information (such as a Git trigger) and containing a single Import Delivery Config stage. This stage will retrieve a Delivery Config manifest from the repository associated with your pipeline’s trigger, then save (or update) it in Spinnaker so it will automatically monitor and manage the resources, environments and deployments described in the manifest.

Here’s what it looks like in the UI:

Configuration is very straightforward: your Spinnaker operator will have configured a default “base path” under which to look for manifest files in your source code repos (.spinnaker, in this example) and you only need to specify the name of the manifest file if it’s not the default shown in the placeholder text (spinnaker.yml, in the example).

Eventually, we might add support for direct monitoring of git commit events to Managed Delivery so that you don’t even need a pipeline, but for now, this is the way to do it.

Docs: Mark an Artifact as Bad

Mon, 01 Jan 0001 00:00:00 +0000

You can mark a version of an artifact in an environment as bad to ensure that that version never gets deployed to the environment. If you mark an artifact as bad in the first in a series of environments (linked together by depends-on constraints) that version of the artifact will never be promoted into the later environments.

Marking via the API

To mark the artifact via the API you’ll need the application name, the name of the environment, the reference for your artifact (defined in your delivery config, defaulted to the artifact name), and the version that is bad. The request needs to have a body that contains this information . Here’s an example:

POST /managed/application/{application}/veto

with body:


{
  "targetEnvironment": "test",
  "reference" : "my-artifact",
  "version" : "master-h10.62bbbd6"
}

Docs: Pin an Environment

Mon, 01 Jan 0001 00:00:00 +0000

You can pin an environment to a specific artifact version to ensure that all resources in the environment stay on that version. Until the pin is removed, that environment won’t change to a new version of the artifact.

Pinning via the API

To create a pin via the API you’ll need the application name, the name of the environment you want to pin, the reference for your artifact (defined in your delivery config, defaulted to the artifact name), and the version you’d like to pin to. The request needs to have a body that contains this information . Here’s an example:

POST /managed/application/{application}/pin

with body:


{
  "targetEnvironment": "test",
  "reference" : "my-artifact",
  "version" : "master-h10.62bbbd6"
}

Removing an environment pin via the API

To remove all pins from an environment, use:

DELETE /managed/application/{application}/pin/{targetEnvironment}

Removing a specific pin via the API

You may want to remove a specific pin from an environment. To do that, you’ll hit make a request like:

DELETE /managed/application/{application}/pin

with body:


{
  "targetEnvironment": "test",
  "reference" : "my-artifact",
  "version" : "master-h10.62bbbd6"
}

Docs: Resource Status

Mon, 01 Jan 0001 00:00:00 +0000

If you’re not sure what a managed resource is, check out our Overview of Managed Delivery first.

Overview

Managed resources always have a status, which describes the current state of the resource from Spinnaker’s perspective. A resource’s status can help you quickly answer questions like:

Is Spinnaker taking automatic action on this resource right now?
Does the actual state of this resource match the declarative configuration I gave to Spinnaker?
Is something wrong with this resource?

How to find a resource’s status

UI

On the Infrastructure views (Clusters, Load Balancers, Firewalls), you’ll see a color-coded “M” logo attached to any resources that are managed by Spinnaker. Hovering over the logo reveals additional context about the current status and what it means.

Example on a cluster:

API

Find the ID of a resource and call the /managed/resources/<resourceId>/status endpoint.

Example request:

curl -X GET --header "X-SPINNAKER-USER: ${SPINNAKER_USER}" "${SPIN_URL}/<resourceId>/status"

Example response:

{ "status": "HAPPY" }

Status Reference

Created

When a resource has this status, it means Spinnaker has just received the resource’s declarative configuration and hasn’t taken any action yet.

If the actual state of the resource matches the declarative configuration, the resource will transition to the Happy status the next time Spinnaker checks it.

If Spinnaker detects a drift from the declarative configuration in the future, the resource will change status to Diff . Once Spinnaker starts taking automatic action to correct the drift, the status will change to Actuating .

While Spinnaker is managing a resource, manual changes made via the UI or the API will be automatically reversed (“stomped”) in favor of the declarative configuration.

Diff

When a resource has this status, it means Spinnaker detected a drift from the declarative configuration but hasn’t taken any action yet.

Once Spinnaker starts taking automatic action to correct the drift, the status changes to Actuating .

While Spinnaker is managing a resource, manual changes made via the UI or the API are automatically reversed (“stomped”) in favor of the declarative configuration.

Actuating

When a resource has this status, it means Spinnaker detected a drift from the declarative configuration and automatic action is in progress to resolve it. You can click on “History” to see details about the detected drift and the specific tasks Spinnaker launched, along with whether they succeeded or failed.

If automatic actions successfully resolve the drift, the status will change to Happy . If automatic actions don’t help to resolve the drift, the status will change to Unhappy .

While Spinnaker is managing a resource, manual changes made via the UI or the API will be automatically reversed (“stomped”) in favor of the declarative configuration.

Happy

When a resource has this status, all is well and the actual state of the resource matches its declarative configuration.

If Spinnaker detects a drift from the declarative configuration in the future, the resource status will change to Diff . Once Spinnaker starts taking automatic action to correct the drift, the status will change to Actuating .

While Spinnaker is managing a resource, manual changes made via the UI or the API will be automatically reversed (“stomped”) in favor of the declarative configuration.

Unhappy

When a resource has this status, it means Spinnaker detected a drift from the declarative configuration but hasn’t been able to resolve it through automatic actions. This might mean that the actions Spinnaker has taken are failing, or something else is preventing the actions from having their desired effect.

While a resource is Unhappy, Spinnaker will continue trying to resolve the drift and may eventually succeed. If it does, the resource will transition to the Happy status.

You can click on “History” to see the detected drift along with actions Spinnaker is taking. If you decide manual intervention is needed to remedy any issues you might discover, you can temporarily pause management by clicking “Pause management of this resource” or by pausing the entire application on the Config view for your app.

Paused

When a resource has this status, it means Spinnaker is configured to continuously manage the resource but you’ve chosen to temporarily pause management. While management is paused Spinnaker won’t check for drift from the declarative configuration, and it won’t take any automatic actions. Depending on whether you’ve paused the specific resource or the entire application, you can either click “Resume management of this resource” or go to the Config view in your app to resume management.

When you resume management the status will change to Resumed .

Resumed

When a resource has this status, it means management was just resumed after being temporarily paused and Spinnaker hasn’t checked for drift from the declarative configuration yet.

If a drift is detected, the status will change to Diff . If no drift is detected, the status will change to Happy .

Currently unresolvable

When a resource has this status, it means something Spinnaker requires to properly manage the resource is either not ready yet or temporarily experiencing disruption. Automatic action won’t be taken while in this status, but it doesn’t necessarily mean something is broken. Some examples of things that might lead to a resource entering this status:

A downstream system (like a container registry or cloud provider) temporarily rate limiting or experiencing availability issues
A required part of the resource’s desired state (like a VM image) not being available yet

If you see this status and want to investigate the cause, you can click “History” to view additional details and understand what else was happening when the issue began. While you may decide to take some action as a result of this status, it’s unlikely the issue requires manual intervention. Once the issue is resolved the resource will transition to whatever status is applicable at the time based on its desired state.

Error

When a resource has this status, it means something went wrong while trying to check the resource for drift from its declarative configuration. Because something went wrong, Spinnaker can’t take automatic actions and drift from the declarative configuration won’t be resolved (until something changes). Some examples of problems that might lead to an error status:

A problem retrieving or processing information about the current state of the resource
A problem with the declarative configuration (invalid settings, etc.)
A bug or issue with Spinnaker itself

After an error, Spinnaker will continue trying to check the resource and may eventually succeed. If it does, the resource will transition to whatever status is applicable at that time.

If you see this status, it’s probably best to investigate the current state of the resource manually and double check that nothing is broken. You can start by clicking “History” to see the details of the error and what else was happening when the error occurred. If you decide manual intervention is needed to remedy any issues you might discover, you can temporarily pause management by clicking “Pause management of this resource”.

Unknown

When a resource has this status, it means Spinnaker was unable to determine the current status for some reason. Because the status is unknown, it’s tough to tell by the status alone whether something is wrong, or just unexpected.

After encountering this status, Spinnaker will continue trying to check the resource for its status and may eventually succeed. If it does, the resource will transition to whatever status is applicable at that time.

If you see this status, it’s probably best to investigate the current state of the resource manually and double check that nothing is broken. You can start by clicking “History” to see details about what’s been happening with the resource recently. If you decide manual intervention is needed to remedy any issues you might discover, you can temporarily pause management by clicking “Pause management of this resource”.

Docs: Managed Delivery FAQ

Mon, 01 Jan 0001 00:00:00 +0000

What is happening?

Spinnaker is managing this resource. That means that the state of this resource is controlled using a declarative config submitted to Spinnaker.

Can I make changes in the UI to this resource?

Nope. Well, you can, but they will be almost immediately stomped to return to the declared state.

Who is the source of truth for that declared state?

Spinnaker is.

Then why do I have code checked into a repo?

To help you keep your resources in a known good state. You must publish these resource configs to Spinnaker every time they change in order for that change to be reflected in Spinnaker.

What if I want to stop managing this resource declaratively?

Send us a DELETE for the config of this resource. We will stop managing the resource, and you can make changes like normal. The infrastructure will not be deleted.

Where are the getting started docs?

They’re at /docs/reference/managed-delivery/getting-started .

Docs: CI Features

Mon, 01 Jan 0001 00:00:00 +0000

One of the goals of Managed Delivery is to show you the journey of your code, from commit to deployment, in an easy, consistent way. In order to enable this experience, we created an integration between Managed Delivery and the Continuous Integration (CI) provider, which is described below.

Note: in order to take advantage of this feature, your Spinnaker operator/administrator will have to configure a CI provider which matches your company’s setup. They can look for available open-source or commercial Spinnaker CI providers, or even develop their own. See the developer guide on CI integration .

Artifact metadata

Detailed metadata (like commit message, author, timestamp) is visible in the Environments view in the UI, by clicking on an artifact version.

Here’s what it looks like in the UI:

See code changes between deployments

Now that we have git metadata for each artifact, we can easily figure out the code differences between each version. We added the See changes button in the UI, for each environment, which looks like this:

In Slack notifications:

Surface build information in the UI

A new section called “Pre-deployment” is now available in the UI. This section will surface pre-deployment steps like baking (for Debian packages only) or building. By clicking on “See details”, you’ll be taken to the viewing CI details (see below), or a CI job log which can be provided as a part of the CI integration.

Viewing CI details in Spinnaker

We recently added the option to see CI details in Spinnaker, in a new “Builds” tab. It looks like this:

This is a new tab that can be accessed from the main Spinnaker UI.