Spinnaker – Spinnaker Pipeline Reference

Docs: Pipeline Expression Reference

Mon, 01 Jan 0001 00:00:00 +0000

For more information about how pipeline expressions work, see the overview .

Syntax elements

Each pipeline expression is made of $ followed by opening/closing brackets: ${ }. Within these brackets, you can add arbitrary expressions to access and modify the details of your pipeline during pipeline execution.

Note that expressions can’t be nested: ${ expression1 ${expression2} } won’t be evaluated.

Code

Spinnaker allows you to execute Java code within a pipeline expression. This can be useful for string manipulation or more advanced custom logic than would otherwise be possible. For security reasons, you can only call methods of specifically allowed Java classes. You can find the full list of allowed classes here .

You can use methods directly on existing map values based on their Java class. For example, you can process a list of values entered as a parameter using Java’s String split() function, provided users enter them using a standard split character. You can process the list us-east-1,us-west-1,eu-west-1 with the expression ${parameters.regions.split(",")}.

You can instantiate new classes inside of an expression using the fully qualified package name. For example, you might want to use the SimpleDateFormat class to get the current date in MM-dd-yyyy format. You can do this using the expression ${new java.text.SimpleDateFormat("MM-dd-yyyy").format(new java.util.Date())}.

Similarly, you can call static methods using the syntax T(fully.qualified.class.name).methodName(). For example, to calculate the date 5 days from now, you can call: ${T(java.time.LocalDate).now().plusDays(5).toString()}.

Comparisons

You can use relational operators to compare values in an expression, such as ${instance["size"] > 400} or ${parameters["runCanary"] == "true"}.

Note that you may need to transform some values for the comparisons to work properly. In the example above, parameters["runCanary"] returns a string rather than a boolean. To use it in a comparision, you need to either compare it to a string or convert it to a boolean: ${#toBoolean(parameters["runCanary"]) == true}.

Another example is that the status attribute of a stage is actually an enum internally, not a string. To compare the status to a string, you need to call .toString() on the result. For example: ${#stage("Deploy")["status"].toString() == "SUCCEEDED"}.

Functions

The expression language has several built in helper functions that simplify common use cases. The syntax for calling built-in functions is #functionName(params). For example, #fromUrl("http://www.netflix.com"). See the full list of functions here .

Lists

You can also use lists in your expressions. Spinnaker always provides a list called stages, which you can use to access each stage by index: ${execution["stages"][0]} returns the value of the first stage in your pipeline.

Note that stages is mostly referenced here for illustration purposes. The value of stages[i] depends on the order in which your stages execute, which makes it fragile. The recommended way to access a specific stage is to use the #stage("Stage Name") helper function .

Maps

You can use maps to access data from the JSON representation of your pipeline. For example, ${trigger["properties"]["example"]} evaluates to the value of the example trigger property. To see the available values for a given pipeline, view the execution JSON:

Go to an execution of your pipeline.
Click on Details to expand the pipeline.
Click on the source link under the pipeline execution, which opens a new tab containing the JSON details of your pipeline execution. [ screenshot ]

Note that while Spinnaker supports both dot (map.value) and square bracket (map["value"]) notation, we recommend using square brackets. That is, prefer trigger["properties"]["value"] instead of trigger.properties.value. There are a few places where using dot notation produces unexpected results, such as after a filter operation or when getting nested JSON values from an URL. Bracket notation also allows you to access non-alphanumeric properties.

You can filter maps using .?. For example, to filter a list of stages by type, you would use ${execution["stages"].?[type == "bake"]}. The result is a list, and can be accessed as such: ${execution["stages"].?[type == "bake"][0]} returns the first bake stage in your pipeline.

Math

You can use arithmetic operations in your expressions, such as ${trigger["buildInfo"]["number"] * 2}. You can use helper functions such as #toInt or #toFloat if type conversion is necessary.

Strings

Strings evaluate to themselves: ${"This is a String."} becomes “This is a String.”

Helper properties

Helper properties are attribute shortcuts that you can use in Spinnaker. The specific properties are:

execution: refers to the current pipeline execution.
parameters: refers to pipeline parameters. This is a shortcut for accessing the value of trigger["parameters"].
trigger: refers to the pipeline trigger.
scmInfo: refers to the git details of either the trigger or the most recently executed Jenkins stage.
- scmInfo.sha1 returns the git commit hash of the last build
- scmInfo.branch returns the git branch name of the last build
deployedServerGroups: refers to the Server Group that was created by the last deploy stage. It should look something like: {"account":"my-gce-account", "capacity": { "desired":1.0,"max":1.0,"min":1.0}, "region":"us-central1", "serverGroup":"myapp-dev-v005"}. Note that you can use deployedServerGroups as a function to return information about an arbitrary deploy stage.

Helper functions

#alphanumerical(String)

Returns the alphanumerical value of the passed-in string. That is, the input string with all characters aside from A-Z and 0-9 stripped out.

#cfServiceKey(String stageName)

A shortcut to refer to a service key which has been created in a previous stage. Remember that the stage’s name is case-sensitive. Note also that the values for the service key are contained in a map, so one may access a property via ${#cfServiceKey("stageName")["desiredProperty"]}. For example, ${#cfServiceKey("Create MySQL Service Key")["username"]} will retrieve the username field of a service key which has been created for a MySQL service in a Create Service Key stage named “Create MySQL Service Key”.

#currentStage()

Returns the current stage.

#deployedServerGroups(String)

Takes the name of a deploy stage as an argument and returns the Server Group that was created by the specified stage.

#fromBase64(String)

Decodes a base64 string.

#fromUrl(String)

Returns the contents of the specified URL as a String. You can use this to fetch information from unauthenticated URL endpoints.

#jsonFromUrl(String)

Retrieves the contents of the given URL and converts it into either a map or a list. It uses the #fromUrl(String) helper function underneath.

#judgment(String)

Returns the selected judgment value from the Manual Judgment stage whose name matches the input string. Note that #judgment is case sensitive: ${#judgment("my manual judgment stage")} returns an error if your stage is named “My Manual Judgment Stage”. Note that this function is aliased to the spelling #judgement.

#manifestLabelValue(String stageName, String manifestKind, String labelKey)

Returns the value of a label with key labelKey from a Kubernetes Deployment or ReplicaSet manifest of kind manifestKind, deployed by a stage of type deployManifest and name stageName.

#pipelineId(String)

This function looks up the pipeline id given a pipeline name (within the same Spinnaker application). This is useful if you generate pipelines programmatically and don’t want to modify pipelines to reference a new id when a dependent pipeline is automatically regenerated. For example, ${#pipelineId("Deploy to prod")} might return 9b2395dc-7a2b-4845-b623-838bd74d059b.

#pipelineIdInApplication(String pipelineName, String applicationName)

This function looks up the pipeline id for the given pipeline name and Spinnaker application. This is helpful to retrieve the pipeline id for an existing pipeline within any application in Spinnaker.

#propertiesFromUrl(String)

Retrieves the contents of a Java properties file at the given URL and converts it into a map. You can use this to fetch information from Jenkins properties files or other similar endpoints.

#readAllYaml(String)

Converts a multi-document YAML String into a list of Maps that can then be processed further.

#readJson(String)

Converts a JSON String into a Map that can then be processed further.

#readYaml(String)

Converts a YAML String into a Map that can then be processed further.

#stage(String)

A shortcut to get the stage by name. For example, ${#stage("Bake")} allows you to access your Bake stage. Note that #stage is case sensitive: if your stage is actually named “bake”, ${#stage("Bake")} will not find it. Remember that the values for the stage are still under the context map, so you can access a property via ${#stage("Bake")["context"]["desiredProperty"]}.

#stageByRefId(String)

A shortcut to get the stage by its refId. For example, ${#stageByRefId("3")} allows you to access the stage with refId = 3.

#stageExists(String)

Checks if a given stage exists. You can search by name or id. Returns true if at least one stage is found with the name or id given. Since the id is generated at runtime, most of the time it will make sense to search by name instead. Note that stage names are set by default so if you create a Webhook stage it will be called Webhook; giving the stage a unique name when you create it makes it easier to find when using this helper function.

#toBase64(String)

Encodes the input string to base64.

#toBoolean(String)

Converts the input string to a boolean.

#toFloat(String)

Converts a value to a floating point number.

#toInt(String)

Converts a value to an integer.

#toJson(Object)

Converts an arbitrary JSON object into a JSON string.

#triggerResolvedArtifact(String name)

A shortcut to look up the resolved artifact in execution trigger by its name. If multiple artifacts are found, only 1 will be returned. For example, ${#triggerResolvedArtifact("my-image")["reference"]} might return gcr.io/spinnaker-marketplace/orca@sha256:b48dbe7d7cb580db8512e4687d31f3710185b08afcf3cb53c0203025f93f9091.

#triggerResolvedArtifactByType(String type)

A shortcut to look up the resolved artifact in execution trigger by its type. If multiple artifacts are found, only 1 will be returned. For example, ${#triggerResolvedArtifactByType("docker/image")["reference"]} might return gcr.io/spinnaker-marketplace/orca@sha256:b48dbe7d7cb580db8512e4687d31f3710185b08afcf3cb53c0203025f93f9091.

#yamlFromUrl(String)

Retrieves the contents of the given URL and converts it into a map. It uses the #fromUrl(String) helper function underneath.

Allowed Java classes

You can find the code which restricts instantiable Java classes here . The allowed classes are:

Source code

Source code for the expression language is in the spinnaker/orca repository , mostly in the following classes:

ContextParameterProcessor class
ExpressionsSupport class
Subclasses of ExpressionFunctionProvider

Pipeline expression implementation

The Pipeline Expression syntax is implemented using the Spring Expression Language (SpEL) .

Docs: Pipeline Stages

Mon, 01 Jan 0001 00:00:00 +0000

Note that when you’re creating a pipeline, you probably won’t see every stage that’s listed here. You’ll only see the stages that Spinnaker supports on your provider.

General

AWS CodeBuild

Run an AWS CodeBuild build by specifying the name of the project. The sources and the image used by the build can be from upstream stages. Artifacts produced by the build can be injected into the pipeline and used by downstream stages.

See this user guide for more information about configuring this stage.

Bake

Bake an image from the specified packages. Baking here refers to the process of creating a machine image. Spinnaker’s bakery is backed by Hashicorp’s Packer . Spinnaker provides default Packer templates and base machine images in order to get you started, but see the bakery configuration guide if you want to customize your bake process.

Note that Spinnaker skips the bake process if it detects that a new bake is unnecessary. Spinnaker generates a unique key for each bake based on the bake stage parameters: base OS, versioned packages, etc. If either the packages or the bake stage parameters have changed, Spinnaker triggers a new bake. To change the default behavior and re-bake your image each time the pipeline runs, select Rebake in the Bake Configuration section.

Canary Analysis

Use Kayenta to run automated canary analysis against the deployment before fully deploying. Note that this stage only handles analysis; you’re responsible for provisioning and cleaning up your canary instances. This is typically done within the same pipeline via stages surrounding the Canary Analysis stage.

For a step-by-step explanation of how to set up a Canary Analysis stage see the how-to guide .

Check Preconditions

Check for preconditions before continuing. For example, you can check that your cluster is a particular size, or add a pipeline expression. See the pipeline expressions guide for more information about creating and using pipeline expressions.

Clone Server Group

Copies all attributes of the original Server Group into a new Server Group (the image that was deployed to it, its capacity, etc). You can choose to override any of the properties of the original Server Group when creating the new one.

Deploy

Deploy the previously baked or found image using the specified deployment strategy. Spinnaker provides built-in support for both blue/green (formerly known as red/black) and Highlander deployment strategies. You can also choose to deploy with no impact on existing Server Groups, or build your own custom deployment strategy.

Destroy Server Group

Delete a Server Group and its resources from the specified Cluster. You must specify whether you want to delete the newest, oldest, or previous (second-most-recently deployed) Server Group when this stage starts.

Disable Cluster

Disable the specified Cluster, which means that the cluster remains up but stops handling traffic. If desired, you can leave a specific number of Server Groups running while the rest of the cluster is disabled.

Disable Server Group

The specified Server Group remains up but stops handling any traffic. Any auto-scaling policies related to this Server Group will also be disabled. Disabling Server Groups makes it easy to both route traffic to a new Server Group and roll back those changes if necessary. You must choose whether to disable the Server Group which is newest, oldest, or previous (second-most-recently deployed) when this stage starts.

Enable Server Group

Tell Spinnaker to resume sending traffic to the Server Group. The configuration of your Load Balancer determines how traffic is routed among newly-enabled Server Groups and any existing Server Groups. Enabling a Server Group also re-enables auto-scaling policies, if applicable.

Find Artifact From Execution

Find and bind an artifact from a different pipeline execution to the current one, including artifacts from different Spinnaker applications. To do this, specify the type of artifact (GCS, GitLab, Docker, etc) and its name. You can specify that Spinnaker should only consider executions that meet certain criteria, such as ones that have completed successfully or that are currently running. Spinnaker will always return the artifact from the most recent execution that matches your criteria.

Find Image From Cluster

Find an image to deploy from an existing Cluster. Make sure that you specify the Cluster and Server Group such that there is exactly one match, or this may behave in unexpected ways.

Find Image From Tags

Find the newest image that matches all specified tags. The image tag can come from any source. Regardless of whether you tagged your image through the Spinnaker Tag Image stage or through a process outside of Spinnaker, the Find Image From Tags stage can find it as long as Spinnaker has previously interacted with the image.

In Spinnaker, tags can only contain lowercase letters, numeric characters, underscores and dashes. Depending on your provider, you may need to specify what region to search for the image.

Google Cloud Build

Run a Google Cloud Build build by specifing a build config as either an artifact or as inline YAML. Artifacts produced by the build can be injected into the pipeline and used by downstream stages. You must configure Google Cloud Build in order to use this stage.

Jenkins

Run the specified job in Jenkins. You must set up Jenkins in order to use this stage. Once Jenkins is configured, your Jenkins master and available jobs are automatically populated in the respective drop-down menus.

Manual Judgment

Wait for the user to click Continue before continuing. You can specify instructions for how to decide whether to continue, or add input options that users can choose from. These input options can be used to determine pipeline behavior in downstream stages. For example, you can use the <strong>Check Preconditions</strong> stage to ensure that a given stage only runs if a particular input is specified.

Note: The Manual Judgement stage requires that Spinnaker’s Echo service is enabled in order to work.

Pipeline

Select any pipeline and run it as a sub-pipeline. You can run pipelines from both the current application and any other Spinnaker applications that you have access to. You can choose whether to wait for the results of the sub-pipeline before this stage completes. If you wait for results, the end state of this stage reflects the end state of the sub-pipeline. Otherwise, the this stage is marked successful as soon as the sub-pipeline starts.

Resize Server Group

Resize the oldest, newest, or second newest Server Group. You can resize the Server Group by either a percentage of its current size or a specific amount. The available resizing strategies are:

Scale Up, which increases the size of the target Server Group.
Scale Down, which decreases the size of the target Server Group.
Scale to Cluster Size, which increases the size of the target Server Group to match the largest Server Group in the Cluster. Optionally, you can specify additional capacity to add as well.
Scale to Exact Size, which adjusts the size of the target Server Group to match the specified capacity.

Rollback Cluster

Roll back one or more regions in a Cluster.

Run Job

Run a container. You need to set up a docker registry so that Spinnaker can access the images to run. Once you’re set up correctly, placing your cursor in the the Image field displays a drop-down menu of available images.

Scale Down Cluster

Scale down a cluster. You can prevent this stage from scaling down active Server Groups, or choose to keep a certain number of Server Groups at their current size while the rest are scaled down.

Script

Execute an arbitrary script as part of your pipeline. Spinnaker uses Jenkins to sandbox your scripts, so you need to set up Jenkins in order to use it. If you already have Jenkins set up, make sure that you have configured it to run scripts .

The only required field in this stage is Command, where you must specify the command to run the script. Otherwise, you can use any of the fields that are relevant to your use case in order to describe the script location and specifics about its output, environment, and so on.

Shrink Cluster

Shrink a given cluster to contain nothing except a specified number of either the newest or the largest Server Groups. You can choose whether to delete active Server Groups if they don’t fit the specified criteria.

Tag Image

Tag the current image in your pipeline with all specified tags. Tags can only contain lowercase letters, numeric characters, underscores and dashes. Spinnaker converts the tags to your provider’s equivalent: for example, it creates labels for GCE images, whereas it creates tags for AWS images, and so on.

Wait

Wait a specified period of time before proceeding. You can choose to manually skip some or all of the wait period during execution.

Webhook

Make an API call to an external system as part of your pipeline.

Supply the URL to send the request to and the desired HTTP method, as well as optionally any desired custom headers and a JSON payload to add to the request. At this point, this stage is marked as successful if it receives a 2XX or 3XX response, fails on a 4XX, or retries on 5XX. The webhook URL, payload, status endpoint, and final status are all shown under the pipeline execution details in the Spinnaker UI.

Note that you can use pipeline expressions in both the URL field and the payload. When the stage completes, the webhook field of the stage context contains the payload, which allows you to use it in future pipeline expressions. For example, you can reference the final status code with the expression ${#stage("My Webhook Stage")["context"]["webhook"]["statusCode"]}.

If you need more details to determine the success of your request, check the Wait for completion checkbox. Spinnaker then polls a status URL to determine the progress of the stage. You can supply the status URL to Spinnaker in one of the following ways:

GET method against webhook URL: Spinnaker polls the webhook URL using GET to determine the status.
From the Location header: Spinnaker parses the Location header of the webhook’s response call to find the status URL, and polls that URL.
From webhook’s response: Spinnaker parses the response from the original request, and polls the returned URL.

In all cases, you then need to provide details about how to access and interpret the status:

Status JsonPath is a required field which you use to specify the path to the status information in the response JSON of the status URL, such as buildInfo.status.
SUCCESS, CANCELED, and TERMINAL status mappings are comma-separated lists of strings that represent successful, canceled, or terminal (failed) statuses, respectively. These are required fields: Spinnaker continues polling until it sees one of the specified statuses in the status URL’s response.
Progress location is optional, and lets you specify the path to detailed progress information in the status URL’s response JSON, such as buildInfo.progress. It shows up in the Info field of the pipeline execution details.

If you find yourself recreating the same webhook stage repeatedly, you can create a custom webhook stage. A custom webhook stage is a webhook stage specifically named and configured for your application’s needs, which shows up in the standard pipeline stages dropdown menu.

You can add more certification authorities to trust when making webhook calls over HTTPS.

App Engine

Start a Server Group

Spin up instances for the Server Group according to its scaling settings. This is distinct from Enable Server Group because it entails starting new instances, rather than sending traffic to existing instances. You can choose whether to start the newest, oldest, or previous (second-most-recently deployed) Server Group.

Note that you can only use this stage if you’re using an App Engine flexible environment or are using manual scaling. Both options can be configured in your app.yaml.

Stop a Server Group

Scale the specified Server Group down to zero instances. This is distinct from Disable Server Group , where the specified server group remains up but stops handling traffic. You can choose whether to stop the newest, oldest, or previous (second-most-recently deployed) Server Group.

Note that you can only use this stage if you’re using an App Engine flexible environment or are using manual scaling. Both options can be configured in your app.yaml.

Edit Load Balancer

Set how much traffic a given version of your app can receive. You can specify versions by name — for example, sample-cluster-v000 — or choose to send traffic to the newest, oldest, or previous (second-most-recently deployed) Server Group. For an example of how to do this, see the App Engine Source to Prod codelab .

Note that a Spinnaker Load Balancer maps to an App Engine service, as specified in a version’s app.yaml. Your version will be deployed to the default service if one was not specified.

AWS

Modify AWS Scaling Process

Suspend/resume scaling processes.

Cloud Foundry

Bake CF Manifest

Bake a manifest with 1 or more variables files. Similar to cf push --vars-file vars.yml. See the Cloud Foundry documentation about Variable Substitution .

Create Service Key

Generate credentials for a service instance. Similar to cf create-service-key. See the Cloud Foundry documentation about how to Create a Service Key .

Delete Service Key

Delete an existing service key. Similar to cf delete-service-key. See the Cloud Foundry documentation about how to Delete a Service Key .

Deploy Service

Create a service instance. Similar to cf create-service. See the Cloud Foundry documentation about Creating Service Instances .

Destroy Service

Delete a service instance. Similar to cf delete-service. See the Cloud Foundry documentation about how to Delete a Service Instance .

Map Load Balancer

Map a Load Balancer (a Cloud Foundry route) to a server group (Cloud Foundry app). The domain must already exist in the Cloud Foundry org. If the route does not already exist, it will be created. Similar to cf map-route. See the Cloud Foundry documentation about how to Map a Route to Your App .

Share a service instance with a specific org / spaces. Similar to cf share-service. See the Cloud Foundry documentation about Sharing a Service Instance .

Unmap Load Balancer

Unmap a Load Balancer (a Cloud Foundry route) from a server group (Cloud Foundry app). Similar to cf unmap-route. See the Cloud Foundry documentation about how to Unmap a Route .

Unshare Service

Unshare a service instance with a specific org / space. Similar to cf unshare-service. See the Cloud Foundry documentation about Unsharing a Service Instance .

Kubernetes

Bake (Manifest)

Bake a manifest (or multi-doc manifest set) using a template renderer such as Helm.

Delete (Manifest)

Destroy a Kubernetes object created from a manifest. If multiple label selectors are specified, they are combined with the logical AND operator. See the Kubernetes reference page for more details.

Deploy (Manifest)

Deploy a Kubernetes manifest YAML/JSON file.

Find Artifacts From Resource

Find artifacts from a Kubernetes resource.

Patch (Manifest)

Update an already existing Kubernetes resource in place using the Kubernetes patch operation . Spinnaker can update the resource without knowing the entire resource (that is, you have to specify only the portion of the manifest you want to update).

The patch stage can be used to add a label or update a sidecar container image for a set of resources. It can also be used to implement a rainbow deployment strategy for Kubernetes by first deploying a new ReplicaSet and then patching the fronting service’s selectors to point to the new ReplicaSet.

When patching with a strategic or merge strategy, Spinnaker also supports artifact substitution for the patch content just like the resource manifest in the deploy stage.

Scale (Manifest)

Scale a Kubernetes object created from a manifest.

Undo Rollout (Manifest)

Rollback a manifest a target number of revisions.

Custom Stages

Custom Webhook

Extend Spinnaker with predefined Webhook stages .

Custom Job

Extend Spinnaker with predefined Run Job stages .

Docs: Pipeline Templates

Mon, 01 Jan 0001 00:00:00 +0000

Pipeline template JSON

A pipeline template uses the following config JSON:

{
  "schema": "v2",
  "variables": [
    {
      "type": "<type>",
      "defaultValue": <value>,
      "description": "<description>",
      "name": "<varName>"
    }
  ],
  "id": "<templateName>",           # The pipeline instance references the template using this
  "protect": <true | false>,
  "metadata": {
    "name": "displayName",          # The display name shown in Deck
    "description": "<description>",
    "owner": "example@example.com",
    "scopes": ["global"]            # Not used
  },
  "pipeline": {                     # Contains the templatized pipeline itself
    "lastModifiedBy": "anonymous",  # Not used
    "updateTs": "0",                # Not used
    "parameterConfig": [],          # Same as in a regular pipeline
    "limitConcurrent": true,        # Same as in a regular pipeline
    "keepWaitingPipelines": false,  # Same as in a regular pipeline
    "description": "",              # Same as in a regular pipeline
    "triggers": [],                 # Same as in a regular pipeline
    "notifications": [],            # Same as in a regular pipeline
    "stages": [                     # Contains the templated stages
      {
        # This one is an example stage:
        "waitTime": "${ templateVariables.waitTime }",  # Templated field.
        "name": "My Wait Stage",
        "type": "wait",
        "refId": "wait1",
        "requisiteStageRefIds": []
      }
    ]
  }
}

Pipeline JSON

A pipeline instance that implements a pipeline template uses the following config:

{
  "schema": "v2",
  "application": "<appName>",     # Set this to the app you want to create the pipeline in.
  "name": "New Pipeline Name",    # The name of your pipeline.
  "template": {
    "type": "front50/pipelineTemplate",
    "artifactAccount": "front50ArtifactCredentials",
    "reference": "spinnaker://<templateName>"
  },
  "variables": {
    "<varName>": <value>,         # Value for the template variable.
    "someOtherVar": <value>
  },
  "inherit": [],
  "triggers": [],
  "parameters": [],
  "notifications": [],
  "description": "",
  "stages": []
}

A more thorough explanation of some fields such as protect can be found in the v1 schema here .