Spinnaker – Extending Spinnaker

Docs: Kubernetes CRD Extensions

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker Extension Points for Custom Resource Definitions

At Google, we’ve built extension points for deep CRD integrations within Spinnaker.

This work has allowed us to support the following features within Spinnaker:

Custom models for representing CRDs as spinnakerKinds
Deploying CRDs with custom Spinnaker artifact types
Custom Kubernetes API versions
Custom Spinnaker naming strategies
Per-account, custom Spinnaker UIs that can run alongside the existing Kubernetes UIs

This guide is for developers who want to duplicate this functionality for their CRDs. It also exists as an explanation of certain code paths within Spinnaker which include hooks with no current corresponding open-source implementations.

Developers who want to implement these features will have to build their own layered version of Clouddriver (found in the monorepo) - see Adam Jorden’s blog post - and should be familiar with the Kubernetes provider and writing code for Clouddriver.

Custom Handlers

The central extension point is the KubernetesHandler - defines the relationship between Spinnaker and your Kubernetes kind.

For example, if you wanted to build a Spinnaker integration for your CRD of kind MyCRDKind, you would start with the following handler:

@Component
public class MyCRDHandler extends KubernetesHandler {

  public MyCRDHandler() {
    // Hook point for registering a custom `ArtifactReplacer`
    // for your CRD. During a deploy operation,
    // if an artifact of the type specified in the replacer is present,
    // the artifact will be inserted into the manifest using the
    // strategy described in the replacer.
  }

  @Override
  public KubernetesKind kind() {
    return KubernetesKind.from("MyCRDKind");
  }

  @Override
  public boolean versioned() {
    // If the CRD resource should be versioned - i.e., assigned a sequence
    // v001, v002, etc.
    return false;
  }

  @Override
  public SpinnakerKind spinnakerKind() {
    // The Spinnaker kind that the resource will be represented as in Spinnaker's API and UI.
    return SpinnakerKind.SERVER_GROUPS;
  }

  @Override
  public Status status(KubernetesManifest manifest) {
    // Includes logic for determining whether your CRD manifest is stable.
    // A deploy manifest operation, for example, will block until this
    // method returns a stable status.
  }

  @Override
  public KubernetesV2CachingAgentFactory cachingAgentFactory() {
    return KubernetesCoreCachingAgent::new;
  }
}

Custom Spinnaker Resource Models

You may want to change how their CRD is represented in Spinnaker’s API. By default, a CRD of spinnakerKind serverGroups will be represented with the model class KubernetesServerGroup

You may want to override this representation, for example, if you want to define how your server group’s region is resolved.

To override the default model class, MyCRDHandler should implement ServerGroupHandler (or ServerGroupManagerHandler if your resource is of spinnakerKind serverGroupManagers). You will be responsible for translating raw Spinnaker cache data into a subclass of KubernetesServerGroup.

Custom Kubernetes API Versions

If you want Spinnaker to support custom Kubernetes API versions, subclass KubernetesApiVersion.

For example,

public class MyApiVersion extends KubernetesApiVersion {
  public static MY_BETA_API_VERSION = new MyApiVersion("myApiVersion/v1beta");

  public MyApiVersion(String name) {
    // Base class maintains state.
    super(name);
  }
}

Custom Spinnaker Naming Strategies

To use a custom naming strategy for your CRD, implement NamingStrategy . For example,

@Component
public class MyManifestNamer implements NamingStrategy<KubernetesManifest> {
  @Override
  public String getName() {
    return "myManifestNamingStrategy";
  }

  @Override
  public void applyMoniker(KubernetesManifest manifest, Moniker moniker) {
    // Strategy for applying a Spinnaker `Moniker` to a Kubernetes
    // manifest prior to deployment.
  }

  @Override
  public Moniker deriveMoniker(KubernetesManifest manifest) {
    // Strategy for deriving a Spinnaker `Moniker` from a Kubernetes
    // manifest.
  }
}

This naming strategy can be referenced in a Kubernetes account config. For example:

kubernetes:
  accounts:
    - name: my-kubernetes-account
      namingStrategy: myManifestNamingStrategy

Be careful - this naming strategy will be applied to all manifests manipulated by this account.

Docs: Writing a New Stage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

To create a new stage, you need to make changes in the monorepo to orca to implement the logic of the stage, and the front-end changes in deck to implement the UI. Depending on what the stage does, you may need to implement new cloud provider-specific logic in clouddriver and/or expose new APIs in gate as well.

This doc currently only covers the backend changes made to orca.

Backend (orca)

For the backend, you need to define:

A stage class
One or more task classes associated with the stage

Stage class

A stage class must implement the com.netflix.spinnaker.orca.api.pipeline.graph.StageDefinitionBuilder interface. Note there are several abstract classes that have extended this with things like expression aware functionality that are usable.

For providing additional functionality, it can also implement other interfaces:

A CancellableStage can be cancelled.
A RestartableStage can be restarted.
A CloudProviderAware stage exposes information about the cloud provider.
An AuthenticatedStage can perform custom authentication.

Here’s an example based on a stage that is used internally at Netflix to integrate with the Chaos Automation Platform (ChAP).

The stage is composed of two tasks:

beginChap starts a new ChAP run
monitorChap waits for the ChAP run to finish

package com.netflix.spinnaker.orca.pipeline;

import com.google.common.collect.ImmutableMap;
import com.netflix.spinnaker.orca.CancellableStage;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.chap.tasks.BeginChapTask;
import com.netflix.spinnaker.orca.chap.tasks.MonitorChapTask;
import com.netflix.spinnaker.orca.pipeline.model.Execution;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class ChapStage implements StageDefinitionBuilder, CancellableStage {

  @Autowired
  public ChapService chapService;

  @Override
  public <T extends Execution<T>> void taskGraph(Stage<T> stage, TaskNode.Builder builder {
    builder
      .withTask("beginChap", BeginChapTask.class)
      .withTask("monitorChap", MonitorChapTask.class);
  }

  @Override
  public CancellableStage.Result cancel(Stage stage) {
    Run run = (Run) stage.getContext().get("run");
    if (run != null) {
      Run latestDetails = chapService.cancelChap(run.id.toString(), "");
      return new CancellableStage.Result(stage, ImmutableMap.of("run", latestDetails));
    }

    return null;
  }
}

Task classes

A task class must implement a com.netflix.spinnaker.orca.api.pipeline.Task , or an interface that extends it, such as:

A RetryableTask can be retried if it fails.
A PreconditionTask defines preconditions that the task will enforce.

To communicate that a task failed, throw a RuntimeException.

In our example, the ChapStage consists of two tasks:

BeginChapTask
MonitorChapTask

BeginChapTask

The BeginChapTask:

retrieves the testCaseId specified by the user during the task configuration stage
calls the ChAP service via REST API call (see ChapService.startChap)
returns a DefaultTaskResult, passing it the response from the ChAP REST API call

package com.netflix.spinnaker.orca.chap.tasks;

import com.netflix.spinnaker.orca.*;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class BeginChapTask implements RetryableTask {

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();
    Object testCaseId = ctx.get("testCaseId");

    if(testCaseId == null || !(testCaseId instanceof String)) {
      throw new RuntimeException("Cannot begin ChAP experiment without a testCaseId.");
    }

    Map<String, Object> params = new HashMap<>();
    params.put("testCaseId", testCaseId);
    Run chapRun = chapService.startChap(params);

    Map<String, Object> map = new HashMap<>();
    map.put("run", chapRun);
    return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Autowired
  private ChapService chapService;

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.SECONDS.toMillis(5);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.MINUTES.toMillis(1);
  }
}

MonitorChapTask

The MonitorChapTask polls the ChAP service for the status of the ChAP run.

package com.netflix.spinnaker.orca.chap.tasks;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.DefaultTaskResult;
import com.netflix.spinnaker.orca.ExecutionStatus;
import com.netflix.spinnaker.orca.RetryableTask;
import com.netflix.spinnaker.orca.TaskResult;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class MonitorChapTask implements RetryableTask {

  @Autowired
  private ObjectMapper objectMapper;

  @Autowired
  public ChapService chapService;

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();

    Run run = objectMapper.convertValue(ctx.get("run"), Run.class);

    if (run == null) {
      throw new RuntimeException("Cannot monitor Chap task without a valid Run in the context.");
    }

    Run latestDetails = chapService.getChap(run.id.toString());

    Map<String, Object> map = new HashMap<>();
    map.put("run", latestDetails);

    if(latestDetails.outcome == Run.Outcome.PASSED){
      return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
    }

    ExecutionStatus status;

    switch (latestDetails.state) {
      case COMPLETED:
        //workflow is complete, but the outcome didnt pass, consider this a failure.
      case FAILED:
        throw new RuntimeException("ChAP experiment failed.");
      case CANCELLED:
        status = ExecutionStatus.CANCELED;
        break;
      default:
        status = ExecutionStatus.RUNNING;
        break;
    }


    return new DefaultTaskResult(status, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.MINUTES.toMillis(1);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.DAYS.toMillis(1);
  }

  public ObjectMapper getObjectMapper() {
    return objectMapper;
  }

  public void setObjectMapper(ObjectMapper objectMapper) {
    this.objectMapper = objectMapper;
  }
}

Other classes used

The details of the com.netflix.spinnaker.orca.chap.Run class and com.netflix.spinnaker.orca.chap.ChapService interface aren’t directly relevant to learning how to write a Spinnaker stage, but for completeness, here’s what those look like for this case:

Run

The Run class is a Java object that is deserialized from JSON.

package com.netflix.spinnaker.orca.chap;

import com.fasterxml.jackson.annotation.JsonAnyGetter;
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

@JsonIgnoreProperties(ignoreUnknown = true)
public class Run {

  public UUID id;
  // Other properties not shown
  // ...

  // Support arbitrary properties without needing to define them explicitly
  public Map<String, Object> properties = new HashMap<>();

  @JsonAnySetter
  public void set(String fieldName, Object value) {
    this.properties.put(fieldName, value);
  }

  @JsonAnyGetter
  public Object get(String fieldName) {
    return this.properties.get(fieldName);
  }
}

ChapService

The ChapService defines a REST client API for talking to the ChAP service. It uses the Retrofit library.

package com.netflix.spinnaker.orca.chap;

import retrofit.http.Body;
import retrofit.http.GET;
import retrofit.http.POST;
import retrofit.http.Path;

import java.util.Map;

public interface ChapService {
  @POST("/v1/runs")
  Run startChap(@Body Map params);

  @GET("/v1/runs/{id}")
  Run getChap(@Path("id") String id);

  @POST("/v1/runs/{id}/stop")
  Run cancelChap(@Path("id") String id, @Body String body);
}

ChapConfig

To implement the ChapService interface, we do not define a class that extends the interface. Instead, we define a class named ChapConfig with a Spring @Configuration annotation. Note that this implementation uses the chap.baseUrl configuration value that is defined in a separate Spinnaker configuration file.

package com.netflix.spinnaker.config;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.retrofit.Slf4jRetrofitLogger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import retrofit.Endpoint;
import retrofit.RestAdapter;
import retrofit.client.Client;
import retrofit.converter.JacksonConverter;

import static retrofit.Endpoints.newFixedEndpoint;

@Configuration
@ComponentScan({
  "com.netflix.spinnaker.orca.chap.pipeline",
  "com.netflix.spinnaker.orca.chap.tasks"
})
@ConditionalOnProperty(value = "chap.baseUrl")
public class ChapConfig {

  @Bean
  Endpoint chapEndpoint(@Value("${chap.baseUrl}") String chapBaseUrl) {
    return newFixedEndpoint(chapBaseUrl);
  }

  @Bean
  ChapService chapService(Endpoint chapEndpoint,
                          Client retrofitClient,
                          RestAdapter.LogLevel retrofitLogLevel,
                          ObjectMapper objectMapper) {
    return new RestAdapter.Builder()
      .setEndpoint(chapEndpoint)
      .setClient(retrofitClient)
      .setLogLevel(retrofitLogLevel)
      .setLog(new Slf4jRetrofitLogger(ChapService.class))
      .setConverter(new JacksonConverter(objectMapper))
      .build()
      .create(ChapService.class);
  }
}