Spinnaker – Additional Features

Docs: Configuring the Script Stage

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

The Script stage lets you run an arbitrary shell, Python, or Groovy script on a Jenkins instance as a first class stage in Spinnaker. For example, you can use it to launch a test suite from a pipeline instead of doing it manually. Using the Script stage requires some additional setup after you add Jenkins to your Spinnaker deployment.

Prerequisites

To configure a Script stage, you need:

A running Jenkins instance at $JENKINS_HOST, with a Jenkins user profile set up with access to create jobs
A running Spinnaker instance, with access to configuration files, and in which you have set up Jenkins on Spinnaker

Configure Jenkins

Download the raw job xml config file with the command:

curl -X GET \
    -o "scriptJobConfig.xml" \
    "https://storage.googleapis.com/jenkins-script-stage-config/scriptJobConfig.xml"

Create the Jenkins job where your script will run. To create the job, you need the following information:
- $JENKINS_HOST: your running Jenkins instance with protocol (http/https - e.g. https://myjenkins.com or http://somejenkins:8080).
- $JOB_NAME: the name of the Jenkins job where your script runs.
- $USER: your Jenkins username.
- $USER_API_TOKEN: the API token for your user. You can find this in Jenkins in the Configure page for your user (e.g. $JENKINS_HOST/user/$USER/configure).
Then, run the command:
```
curl -s -XPOST "$JENKINS_HOST/createItem?name=$JOB_NAME" \
    -u $USER:$USER_API_TOKEN \
    --data-binary @scriptJobConfig.xml \
    -H "Content-Type:text/xml"
```
Navigate to Jenkins » the job you just created » Configure and do the following:
1. Add the GitHub repository containing your scripts.
2. Either create a Spinnaker node in which Jenkins will run all Script jobs, or de-select the Restrict where this project can be run checkbox.
At this point, you can manually run the script job in Jenkins (including manually adding necessary parameters) and see it succeed.
If your Jenkins master is named anything other than master in your Spinnaker configuration, you’ll need to add the following to orca-local.yml in order for Spinnaker to find it:
```
script:
  master: your-jenkins-master
  job: $JOB_NAME  # from step #3
```

You can now use the Script stage in your pipelines.

Docs: Enable Slack Support Channels

Mon, 01 Jan 0001 00:00:00 +0000

You can create an application and configure it with relevant data—application attributes.

Including a Slack support channel for an application is a configurable application attribute. This feature communicates with Slack’s conversations API to expose a list of available Slack channels in your workspace. The Slack support channel feature is supported by all providers.

Configuration

Gate

You need an API token associated with your Slack workspace for gate to receive a list of channels from Slack’s API. See the Slack documentation if you need to create a Slack token. Add this token in the config for gate’s encrypted secrets.

slack:
  token: YOUR_TOKEN_HERE

The base url for the Slack API is also needed in gate.yml. This does not need to be encrypted.

slack:
  baseUrl: https://slack.com/api

Deck

The UI components in deck are protected with a feature flag in the application settings, settings.netflix.ts. To activate this feature you need to enable this flag to true.

feature: {
  slack: true,
  ...otherFeatures
}

If your workspace’s base URL is anything other than https://slack.com , then you need to configure this as well. This base URL is used to construct the link for accessing a Slack support channel.

  slack: {
    baseUrl: 'https://my-cool-workspace.slack.com',
  }

Docs: Notifications and Events Guide

Mon, 01 Jan 0001 00:00:00 +0000

Out of the box, Spinnaker allows you to configure the following types of notifications:

This is discussed in the Configuring Notifications section below.

Additionally, Spinnaker allows you to set webhooks for git triggers. See the Setting up Git Triggers in Spinnaker section.

You can also set Spinnaker to stream all its events to a downstream listener. See the Add a Webhook to Spinnaker section.

Spinnaker is also capable of handling cron-based triggers and detect changes in Jenkins builds and Docker images by enabling continuous integration configuration in spinnaker

Last, pub/sub triggering with both custom and default types is available via aws pub/sub configurations or Google pub/sub integration

Configuring notifications

Notification configurations are in echo-local.yml and settings-local.js

Spinnaker baseURL

You need to set the spinnaker.baseUrl configuration value which is used by spinnaker templates. This should point back to the url for your spinnaker’s UI ( deck ) instance. This url is used in notifications to link back to your spinnaker instance.

Email

Email in spinnaker is provided by Spring Boot Mail starter .

The settings for some popular email providers are listed below:

Email Provider	SMTP username	SMTP password	SMTP server address	SMTP port (TLS)	SMTP port (SSL)	SMTP TLS/SSL required
Gsuite/Gmail	Your email address	Your email password	smtp.gmail.com	587	465	yes
Yahoo	Your email address	Your email password	smtp.mail.yahoo.com	587	465	yes
Hotmail/live	Your email address	Your email password	smtp.live.com	587	-	yes
Outlook	Your email address	Your email password	smtp-mail.outlook.com	587	25	yes

The following is an example of using hotmail to send notifications.

echo-local.yml:

mail:
  enabled: true
  from: <myemail>@hotmail.com
spring:
  mail:
    host: smtp.live.com
    username: <myemail>@hotmail.com
    password: hunter2
    port: 587
    properties:
      mail:
        smtp:
          auth: true
          starttls:
            enable: true
        transport:
          protocol: smtp
#	debug: true <- this is useful if you are mucking around with smtp properties

in settings-local.js (deck)

window.spinnakerSettings = window.spinnakerSettings || {};
window.spinnakerSettings.notifications = window.spinnakerSettings.notifications || {};
window.spinnakerSettings.notifications.email = window.spinnakerSettings.notifications.email || {};
window.spinnakerSettings.notifications.email.enabled = true;

Microsoft Teams

Note, teams has changed their APIs. We are unable at this time to send notifications to teams, but would welcome PRs to fix teams integration.

To enable Microsoft Teams support, add the following statement to the echo-local.yml file:

microsoftteams:
  enabled: true

Spinnaker supports sending notifications to Microsoft Teams using Incoming Webhooks.

To create a custom incoming webhook in Teams, follow the instructions provided here: Add an incoming webhook to a Teams channel

When configuring a Microsoft Teams notification, enter the full incoming webhook URL in the Teams Webhook URL text box.

Slack

For slack, you need to create a custom bot user , then get the access token associated with that new bot user. Then…

Add to echo-local.yml

slack:
  botName: SpinnakerNotifications
  enabled: true
  token: ${ECHO_NOTIFICATION_SECRETS_SLACK_BOT_TOKEN}

Note: your users will need to invite the slack bot to private rooms that want to be notified.

Twilio

For Twilio, you need to add your account credentials . Then… add this to your echo-local.yml

twilio:
  enabled: true
  baseUrl: https://api.twilio.com/
  token: token
  account: TWILIO_ACCOUNT_SID
  from: twilioPhoneNumber

CDEvents

To enable CDEvents notification support, add the following statement to the echo-local.yml file:

cdevents:
  enabled: true

Spinnaker supports sending CDEvents to a configured Message Broker . The Message Broker is defined within the notification itself.

For example, Knative Eventing can be configured as a Message Broker to receive CDEvents.

When configuring a CDEvents notification, In the Edit Notification window, specify CDEvents in the Notify via field and enter the full Message Broker URL in the Events Broker URL text box.

To send a specific type of CDEvents, enter the CDEvents Type in the CDEvents Type text box. Different types of CDEvents can be found in the CDEvents Specification .

Note: Currently Spinnaker supports sending Continuous Delivery Core Events

Using notifications

Once notifications have been configured, you can use them to send changes in pipelines and in the manual judgment stage in Spinnaker.

To set up an application-wide notification, go to Application -> Config -> Notifications,

Click on ‘Add Notification’

Enter your notification details.

You can also set Notifications at the Pipeline level ( under configuration ) and at the stage level ( by clicking on the [ ] Send Notifications for this stage checkbox.

Customizing notifications

Spinnaker will send a standard message based on the event that triggered the notification, indicating the stage (if applicable), the pipeline, the application, and the status of the event. It will also include a link to the pipeline.

For stage-level notifications, you can override the message by editing the JSON of the stage, adding a customBody field (for email) or a customMessage field (for Slack). If you are customizing an email notification, you can use Markdown or HTML to customize the format of the email. If you are using Slack, you can use a more limited range of formatting .

For email messages, you can customize the subject by adding a customSubject field.

All standard SpEL expressions will be evaluated prior to sending the notifications, and can be used in the custom notification fields.

There are two special variables available when adding a custom message or body: executionId (the ID of the execution) and link (a fully-formed URL for the pipeline or stage that triggered the notification). To use either of these fields, just wrap them in two curly brackets, e.g. {% raw %}{{link}}{% endraw %}.

An example of a custom email notification: {% raw %}

{
  "customSubject": "Beginning deployment to production (started by: ${trigger.user})",
  "customBody": "*Pipeline parameters:* ${parameters.toString()}\n\n [View the stage]({{link}}) here.",
  "notifications": [
    {
      "address": "spinnakerteam@spinnaker.io",
      "level": "stage",
      "type": "email",
      "when": [
        "stage.starting"
      ],
    }
  ],
  //...
}

{% endraw %}

A custom Slack message: {% raw %}

  {
    "customMessage": "${trigger.user} started a <{{link}}|deploy to production>",
    "notifications": [
      {
        "address": "spinnaker-prod-deploys",
        "level": "stage",
        "type": "slack",
        "when": [
          "stage.starting"
        ],
      }
    ],
    //...
  }

{% endraw %}

Note: the custom fields will apply to all notifications within a stage. If you have different notifications for different events (e.g. a Slack message when the stage starts and when it completes), the custom fields will be applied to both messages.

Setting up Git Triggers in Spinnaker

Instead of listening for events from Github like in the Docker and Jenkins case, direct triggering from git requires a webhook endpoint to be set up in the corresponding VCS.

Github

Using Github as a Trigger

Go to your project’s Webhooks page, for example https://github.com/spinnaker/igor/settings/hooks

Click “Add webhook”

enter http://[spinnakerapi]/webhooks/git/github in the Payload URL, where spinnakerapi is the location of your gate installation.

select “Just the push event”

make it active

Click “Add Webhook” button

Stash

Go to settings / hooks for your repository

Add a Post Receive webhook

Add the following URL: http://[gate url]/webhooks/git/stash, where gate url is the url to your spinnaker api.

You are now ready to receive events from this repository.

BitBucket Cloud

Go to settings / webhooks for your repository

Click ‘Add webhook’

Add a ‘Repository Push’ or ‘Pull Request Merged’ webhook

Add the following URL: http://[gate url]/webhooks/git/bitbucket, where gate url is the url to your spinnaker api.

You are now ready to receive events from this repository.

Add Git Trigger in Pipeline:

Go to Pipeline > Configuration

Select Trigger Type to be Git

Set Repository Type to be github, stash, or bitbucket

Enter your Repository organization / user

Enter your Repository name

Save your pipeline

Notes

You can access the hash of the build via expression ${trigger.hash}

Add a listening webhook to spinnaker

The echo-rest module in spinnaker allows you to set downstream listeners keeping track of Spinnaker events.

Whenever Spinnaker receives an event from Orca, igor, or echo, it will forward these events to the webhooks registered.

A useful pattern is to have an intermediate service that will filter and convert events you care about into the format you expect.

Configuration in echo.yml

rest:
  enabled: true
  endpoints:
    -
      wrap: false
      url: http://listener.com
    -
      wrap: false
      url: http://listener2.com

This will post every event to listener and listener2

Wrapping

If your endpoint expects events of the form

	{ "eventName" : "abc", "payload" : "[event json]"}

you can wrap the event via this configuration:

  -
      wrap: true
      flatten: true
      url: http://listener3
      eventName: spinnaker_events
      fieldName: payload

The flatten setting will simply make the json in content and details ( see below ) into a json String.

Event types

Here is an example event:

{
    "details": {
      "source": "orca",
      "type": "orca:task:complete",
      "created": "1422487582294",
      "organization": null,
      "project": null,
      "application": "asgard",
      "_content_id": null
    },
    "content": {
      "standalone": true,
      "context": {
        "asgName": "asgard-staging-v048",
        "credentials": "test",
        "deploy.account.name": "test",
        "deploy.server.groups": {},
        "kato.last.task.id": {
          "id": "19351"
        },
        "kato.task.id": {
          "id": "19351"
        },
        "kato.tasks": [
          {
            "history": [
            ],
            "id": "19351",
            "resultObjects": []
          }
        ],
        "notification.type": "enableasg",
        "regions": ["us-west-1"],
        "targetop.asg.enableAsg.name": "asgard-staging-v048",
        "targetop.asg.enableAsg.regions": ["us-west-1"],
        "user": "clin@netflix.com",
        "zones": ["us-west-1a", "us-west-1c"]
      },
      "execution": ...
      "executionId": "62ca5574-0629-419a-b9ac-fb873aa165b2",
      "taskName": "f92239a7-b57a-408d-9d72-3a77484e050b.enableAsg.monitorAsg.9568e7e5-3c37-4699-9e93-f62118adc7c6"
    }
  }

Events have details, which will always be the same.

details.type

The type of the event will outline where the event is coming from:

orca:[task type]:[status] - where task type is either ‘pipeline’, ‘stage’ or ’task’ and status is ‘starting’, ‘complete’, ‘failed’
build - from igor Jenkins events
docker - from igor Docker events
git - from git web triggers

details.application

The application involved in the execution, this will only exist for orca: type events.

content.execution

For orca type events, this is the current execution that has the entire pipeline execution.

Docs: User Data Files and Metadata

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker refers to data injected into instances started by Spinnaker as user data. The implementation and naming of this varies from provider to provider, but the resulting functionality is similar. In AWS, it is known as User Data . In GCP, it is known as Instance Metadata . The template file used to define the user data is called the user data file which is located on the Clouddriver server. Tokens are replaced in the user data file to provide some specifics about the deployment. Every instance started has environment variables are set according to the template and the deployment. This feature is currently supported with the AWS and GCP providers.

Configuration

The filename and location of the file is configured differently for each provider. The general format for the template file a simple property file though there are some differences depending on the provider.

Example of setting a Consul server address:

CONSUL_SERVER_ADDRESS="consul.example.com"

The environment variable CONSUL_SERVER_ADDRESS will be set on every instance with the value of consul.example.com.

Typical usage would be defining environment variables with the values including the environment specific details to customize the behavior based on account, environment, region, and other things.

Here is the list of replacement tokens:

Token	Description
`%%account%%`	Name of the account
`%%accounttype%%`	Type of the account
`%%env%%`	Environment of the account
`%%app%%`	Application being deployed
`%%region%%`	The deployment region
`%%group%%`	Name of the server group
`%%cluster%%`	Name of the cluster
`%%stack%%`	Stack component of the cluster name
`%%detail%%`	Detail component of the cluster name
`%%launchconfig%%`	Name of the launch configuration

Example template file using tokens in the values:

CLOUD_ACCOUNT="%%account%%"
CLOUD_ACCOUNT_TYPE="%%accounttype%%"
CLOUD_ENVIRONMENT="%%env%%"
CLOUD_SERVER_GROUP="%%group%%"
CLOUD_CLUSTER="%%cluster%%"
CLOUD_STACK="%%stack%%"
CLOUD_DETAIL="%%detail%%"
CLOUD_REGION="%%region%%"

If the server group udf-example-cluster-v001 was deployed using this template in the account main, accountType streaming, environment prod, in the east region, the resulting user data would look like:

CLOUD_ACCOUNT="main"
CLOUD_ACCOUNT_TYPE="streaming"
CLOUD_ENVIRONMENT="prod"
CLOUD_SERVER_GROUP="udf-example-cluster-v001"
CLOUD_CLUSTER="udf-example-cluster"
CLOUD_STACK="example"
CLOUD_DETAIL="cluster"
CLOUD_REGION="east"

AWS

The location of the template file for AWS is controlled by the udf.udfRoot property and the behavior is controlled by the udf.defaultLegacyUdf property. The defaults are:

udf:
  udfRoot: /apps/nflx-udf
  defaultLegacyUdf: true

You probably want to change the location on the filesystem where the template file lives to suit your deployment. You almost certainly want to change udf.defaultLegacyUdf=false as this disables deprecated behavior that is specific to Netflix. In the udf.udfRoot directory, create a file called udf0. The contents of this file are base64 encoded and set as the user-data for the LaunchConfiguration when a new server group is created.

Note: The udf property is root level property instead of being nested under the AWS section.

Google

The path to the template file is controlled by the userDataFile property in clouddriver-local.yml. You can find more properties or information in the google account properties

With Google, the user data file is set per account. It is best practice to use the same file for different accounts to ensure consistency, but different user data files can be used for different accounts if needed. The contents of the file is parsed and set as the Instance Metadata on launched instances. Any metadata defined in the server group configuration within Spinnaker is appended to the metadata defined by the user data file. The metadata defined in the server group configuration within Spinnaker takes preferences over the metadata defined in the user data file if the metadata keys match. For example, if the environment variable CLOUD_CLUSTER=%%app%%-%%stack%% is set in the user data file and in the server group configuration, the metadata section includes CLOUD_CLUSTER=overridden-stack then the final metadata injected into the instance will be CLOUD_CLUSTER=overridden-stack.