Spinnaker – Set Up Triggers

Docs: Configuring GitHub Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

These steps show you how to configure webhook push events to send to Spinnaker from a single GitHub repository.

Prerequisites

You need Spinnaker’s API running on an endpoint that is publicly reachable. Ideally this means that you’ve configured authentication . This is required to allow GitHub’s webhooks to reach Spinnaker.

If you’re unsure of what your Spinnaker API endpoint is, check the value of services.gate.baseUrl or your ingress or services.
You need a GitHub repository to send Webhooks from.

Configuring your GitHub webhook

Under your GitHub repository, navigate to Settings > Webhooks > Add Webhook. Here, provide the following values to the form shown below:

Field	Value
Payload URL	Given that the above prerequisite API endpoint is `$ENDPOINT`, enter `$ENDPOINT/webhooks/git/github`. While all GitHub webhooks share an endpoint in Spinnaker, there is no practical limit to the number of repositories you can configure to send notifications.
Content type	`application/json`
Secret	The value is up to you, and must be provided to any GitHub webhooks triggers that you configure within Spinnaker. It’s used to ensure that only GitHub can trigger your pipelines, not an imposter.

Spinnaker configuration

Aall that’s needed is pipeline trigger configuration. An example of that can be found in this guide .

Docs: Amazon AWS Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker pipelines can be triggered from a queue in Amazon Simple Queue Service (SQS) that receives new notifications from Amazon Simple Notification Service (SNS) .

Prerequisites

You need an AWS Account and the AWS Command Line Interface (cli) installed locally and configured with credentials that allow you to manage your AWS cloud resources.

Use the AWS Command Line tool or the AWS Console website to create a SNS Topic that will receive the messages destined for the queue and Spinnaker. You can do this by running the following commands:

 bash4.4$ export AWS_TOPIC_NAME=aws-pubsub-test-topic
 bash4.4$ aws sns create-topic --name ${AWS_TOPIC_NAME}
 {
    "TopicArn": "arn:aws:sns:us-east-1:1234567890:aws-pubsub-test-topic"
 }

Make a note of the TopicArn. You will need this to configure Echo via the echo-local.yml file to tell Spinnaker to subscribe to this topic.

Creating an Simple Queue Service (SQS) Queue (Optional)

At this point, you can also create an SQS Queue that will receive the notification messages.

Spinnaker will do this part for you, provided that you specify what would be a valid name for the QueueARN in the echo-local.yml.

If you’d like to create the queue yourself manually, follow these instructions. Otherwise skip to the section below about Configuring your S3 Bucket .

Use the AWS Command Line tool or the AWS Console website to create an SQS Queue that Spinnaker will listen to. You can do this by running the following commands:

 # Create the queue
 bash4.4$ export AWS_QUEUE_NAME=aws-pubsub-test-queue
 bash4.4$ aws sqs create-queue --queue-name ${AWS_QUEUE_NAME}
 {
    "QueueUrl": "https://us-east-1.queue.amazonaws.com/1234567890/aws-pubsub-test-queue"
 }
 # Add the permission for the topic to send messages to the queue
 bash4.4$ aws sqs add-permission --queue-url <QUEUE_URL> --aws-account-ids 1234567890 \
 --label spinnaker-pubsub --actions SendMessage
 # Now get the Queue ARN, you will need this for Spinnaker.
 bash4.4$ aws sqs get-queue-attributes --queue-url <QUEUE_URL> --attribute-names QueueArn
 {
    "Attributes": {
        "QueueArn": "arn:aws:sqs:us-east-1:1234567890:aws-pubsub-test-queue"
    }
 }
 # Now subscribe the queue to the topic you created above.
 bash4.4$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:12345467890:aws-pubsub-test-topic \
 --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:1234567890:aws-pubsub-test-queue
 {
    "SubscriptionArn": "arn:aws:sns:us-east-1:1234567890:aws-pubsub-test-topic:e8d02657-e92c-43aa-a785-8d93cb8738fd"
 }

Make a note of this QueueArn, as you will need to provide it to Spinnaker in the echo-local.yml file.

You have now set up an SNS topic and created a SQS queue which is subscribed to the topic. When messages are delivered to the topic, they will be delivered to the queue.

Configuring an S3 bucket to send notifications

The next step is to create an S3 bucket, or use an existing bucket, and automatically deliver notifications when a file is uploaded. The following commands can be used to create a bucket and configure the notifications.

Now you have to tell AWS that your SNS topic can receive messages from your S3 bucket. To do this, you’ll need to change the Access Policy for the SNS topic. You can do this by issuing the following command, swapping out the values with the resources created above:

# CREATE THE BUCKET
bash4.4$ export AWS_TOPIC_ARN=<INSERT TOPIC ARN HERE>
bash4.4$ export AWS_PUBSUB_BUCKET=spin-pubsub-test-bucket
bash4.4$ aws s3 mb s3://${AWS_PUBSUB_BUCKET}/
make_bucket: spin-pubsub-test-bucket

bash4.4$ aws sns set-topic-attributes --topic-arn ${AWS_TOPIC_ARN} --attribute-name Policy --attribute-value \
"{\"Version\":\"2008-10-17\",\"Id\":\"__default_policy_ID\",\"Statement\":[{\"Sid\":\"s3Publish\",\
\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":[\"SNS:GetTopicAttributes\",\
\"SNS:Publish\"],\"Resource\":\"${AWS_TOPIC_ARN}\",\"Condition\":{\"ArnLike\":\
{\"aws:SourceArn\":\"arn:aws:s3:::${AWS_PUBSUB_BUCKET}\"}}}]}"
bash4.4$

Now that you’ve given your topic permission to have messages published to it, you can update your bucket’s put-bucket-notification-configuration with the following spin-pubsub-notification.json. You’ll need to swap out any values for the values for your configuration.

 # You need to create a file with the notification information in json format
 bash4.4$ cat << EOF > /tmp/spin-pubsub-notification.json
 {
    "TopicConfigurations": [
        {
            "TopicArn": "${AWS_TOPIC_ARN}",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
 }
EOF
 bash4.4$ aws s3api put-bucket-notification-configuration --bucket ${AWS_PUBSUB_BUCKET} \
 --notification-configuration file:///tmp/spin-pubsub-notification.json    
 bash4.4$

Congratulations! You have now done all of the configuration in AWS to send pubsub messages. All we have to do now is configure Spinnaker to subscribe to and retrieve these messages.

Writing your Echo configuration

Define the details regarding your subscription in your echo-local.yml file. If you are interested in contributing to Spinnaker in code but haven’t found something simple to start on, this might be just the thing for you!

pubsub:
  enabled: true
  amazon:
    enabled: true
    subscriptions:
    - name: aws-pubsub-test-subscription
      topicARN: arn:aws:sns:us-east-1:1234567890:aws-pubsub-test-topic
      queueARN: arn:aws:sqs:us-east-1:1234567890:aws-pubsub-test-queue
      messageFormat: S3

Turning on Amazon as an option in the UI (Deck)

As of release 1.14.6, Amazon pubsub support is not supported by default in the UI, even if it’s configured in echo-local.yaml as laid out above. In order to get amazon to show up in the list of pubsub, you must make a slight change to the settings-local.js file that is deployed with Deck.

Add a line like so:

window.spinnakerSettings.pubsubProviders = ['google', 'amazon'];

Once you have done that, redeploy Spinnaker. After the new versions of Echo, Gate and Deck have come up and reported as healthy, you should be able to see this in the UI.

Troubleshooting

If you don’t see your subscriptions after going through these steps, take a look at your Echo logs as the service starts up.

You should see messages from the SQS Provider telling you that the Subscriptions have turned on. Messaages like this should appear near the top of the logs.

INFO 1 --- [           main] c.n.s.e.p.aws.SQSSubscriberProvider      : Bootstrapping SQS for SNS topic: arn:aws:sns:us-east-1:1234567890:aws-pubsub-test
INFO 1 --- [           main] ubsubProperties$AmazonPubsubSubscription : Using message format: S3 to process artifacts for subscription: aws-pubsub-test
INFO 1 --- [pool-2-thread-1] c.n.s.echo.pubsub.aws.SQSSubscriber      : Starting arn:aws:sqs:us-east-1:1234567890:aws-pubsub-queue/SQSSubscriber

If things aren’t working and you want additional info, you can add the following lines to echo-local.yml to get debugging information specific to pubsub.

logging:
  level:
    com.netflix.spinnaker.echo.pubsub.aws: DEBUG

This will print out a message each time that Spinnaker goes out and attempts to pull messages from the queue.

Docs: Google Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker pipelines can be triggered using a Google Pub/Sub subscription .

Prerequisites

You need a Google Cloud Platform (GCP) project to with the Pub/Sub API enabled , and <code>gcloud</code> installed locally. You can check that gcloud is installed and authenticated by running:

gcloud info

A Pub/Sub subscription

The Pub/Sub integration can be used to either:

Start pipelines by sending Pub/Sub messages from a system you own.
Start pipelines using Pub/Sub messages originating from a Google Cloud system, such as Google Cloud Storage , or Google Container Registry .

Configuration and setup for all cases are described below.

Sending your own Pub/Sub messages

First, record the fact that your $MESSAGE_FORMAT is CUSTOM, this will be needed later.

MESSAGE_FORMAT=CUSTOM

You need a topic with name $TOPIC to publish messages to:

gcloud beta pubsub topics create $TOPIC

This topic needs a a pull subscription named $SUBSCRIPTION to let Spinnaker read messages from. It is important that Spinnaker is the only system reading from this single subscription. You can always create more subscriptions for this topic if you want multiple systems to recieve the same messages.

gcloud beta pubsub subscriptions create $SUBSCRIPTION --topic $TOPIC

At this point, you can use the publisher guide to learn how to publish messages programmatically to the topic you have created.

Note that your topic messages need to be valid JSON, otherwise an exception will be raised in the echo service and your pipeline will not be triggered.

Receiving messages from Google Cloud Storage (GCS)

First, record the fact that your $MESSAGE_FORMAT is GCS, this will be needed later.

MESSAGE_FORMAT=GCS

Given that you’ll be listening to changes in a GCS bucket ($BUCKET), the following command will create (or use an existing) topic with name $TOPIC to publish messages to:

gsutil notification create -t $TOPIC -f json gs://${BUCKET}

Finally, create a pull subscription named $SUBSCRIPTION to listen to changes to this topic:

gcloud beta pubsub subscriptions create $SUBSCRIPTION --topic $TOPIC

To understand the format that GCS will publish messages in, please read the reference material .

Receiving messages from Google Container Registry (GCR)

Warning

The GCR message type extracts the image digest by default, but Google App Engine doesn’t support deploying from a digest. So if you’re using artifacts from GCR messages to trigger a GAE deployment, you need to use a custom message template. The template must copy the image’s tag into the artifact’s reference and version fields.

First, record the fact that your $MESSAGE_FORMAT is GCR, this will be needed later.

MESSAGE_FORMAT=GCR

Given a project name $PROJECT, GCR will always try to publish messages to a topic named projects/${PROJECT}/topics/gcr for any repositories in $PROJECT. To ensure that GCR has a valid topic to publish to, try to create the following topic:

gcloud beta pubsub topics create projects/${PROJECT}/topics/gcr

If the command fails with “Resource already exists in the project”, then the topic already exists (which is OK).

Finally, create a pull subscription named $SUBSCRIPTION to listen to changes to this topic:

gcloud beta pubsub subscriptions create $SUBSCRIPTION \
  --topic projects/${PROJECT}/topics/gcr

To understand the format that GCR will publish messages in, please read the reference material .

Credentials

Spinnaker needs a service account to authenticate as against GCP, with the roles/pubsub.subscriber role enabled. If you don’t already have such a service account with the corresponding JSON key downloaded, you can run the following commands to do so:

SERVICE_ACCOUNT_NAME=spinnaker-pubsub-account
SERVICE_ACCOUNT_DEST=~/.gcp/pubsub-account.json

gcloud iam service-accounts create \
    $SERVICE_ACCOUNT_NAME \
    --display-name $SERVICE_ACCOUNT_NAME

SA_EMAIL=$(gcloud iam service-accounts list \
    --filter="displayName:$SERVICE_ACCOUNT_NAME" \
    --format='value(email)')

PROJECT=$(gcloud config get-value project)

gcloud projects add-iam-policy-binding $PROJECT \
    --role roles/pubsub.subscriber --member serviceAccount:$SA_EMAIL

mkdir -p $(dirname $SERVICE_ACCOUNT_DEST)

gcloud iam service-accounts keys create $SERVICE_ACCOUNT_DEST \
    --iam-account $SA_EMAIL

Warning

It’s possible to restrict access to a subscription by service account in GCP. If this is how your subscription is configured, you may need to grant $SA_EMAIL additional permissions following the instructions on the

Pub/Sub IAM page

Once you have run these commands, your GCS JSON key is sitting in a file called $SERVICE_ACCOUNT_DEST.

Editing your Pub/Sub settings

All that’s required are the following values:

# See 'Credentials' section above
SERVICE_ACCOUNT_DEST=

# See 'A Pub/Sub Subscription' section above
MESSAGE_FORMAT=
PROJECT=
SUBSCRIPTION=

# You can pick this name, it's meant to be human-readable
PUBSUB_NAME=my-google-pubsub

First, make sure that Google Pub/Sub support is enabled. Add the following to echo-local.yml

pubsub:
  enabled: true
  google:
    enabled: true
    pubsubType: GOOGLE
    subscriptions:
    - name: my-gcs-subscription
      project: my-project
      subscriptionName: my-gcs-subscription
      jsonPath: /path/to/my-gce-account.json
      ackDeadlineSeconds: 10
      messageFormat: GCS
    publishers: []

Spinnaker – Set Up Triggers

Docs: Configuring GitHub Webhooks

Prerequisites

Configuring your GitHub webhook

Spinnaker configuration

Docs: Amazon AWS Pub/Sub

Prerequisites

Creating an Simple Notification Service (SNS) Topic

Creating an Simple Queue Service (SQS) Queue (Optional)

Configuring an S3 bucket to send notifications

Writing your Echo configuration

Turning on Amazon as an option in the UI (Deck)

Troubleshooting

Docs: Google Cloud Pub/Sub

Prerequisites

A Pub/Sub subscription

Sending your own Pub/Sub messages

Receiving messages from Google Cloud Storage (GCS)

Receiving messages from Google Container Registry (GCR)

Warning

Credentials

Warning

Editing your Pub/Sub settings