Spinnaker – Persistence

Docs: Set up Clouddriver to use SQL

Mon, 01 Jan 0001 00:00:00 +0000

These are:

Caching the state of all supported cloud resources and graphing their relationships
The cloud operations task repository
Distributed locking for the caching agent scheduler

You can also use Redis for one of the above, and SQL for another. A DualTaskRepository class is provided to enable online migrations between Redis and SQL in either direction without impact to running pipelines.

This guide covers database and Clouddriver configuration details, as well as tips on how Clouddriver is operated at Netflix.

Why SQL

Growth in Netflix’s infrastructure footprint over time exposed a number of scaling challenges with Clouddriver’s original Redis-backed caching implementation. For example, the functional use of unordered Redis sets as secondary indexes incurs linear cost while also making it challenging to shard the keyspace without hotspots. Additionally, Clouddriver’s data is highly relational, making it a good candidate for modeling to either relational or graph databases.

Small Spinnaker installations (managing up-to a few thousand instances across all applications) are unlikely to see significant performance or cost gains by running Clouddriver on an RDBMS as opposed to Redis. In the Netflix production environment, we have seen improvements of up to 400% in the time it takes to load the clusters view for large applications (hundreds of server groups with thousands of instances) since migrating from Redis to Aurora. We also see greater consistency in 99th percentile times under peak load and have reduced the number of instances required to operate Clouddriver. Besides improving performance at scale, we believe the SQL implementation is better suited to future growth. We’ve load tested Clouddriver backed by Aurora with the Netflix Production data set at more than 10x traffic rates, which previously resulted in Redis related outages.

At some point in the future, Spinnaker may drop support for Redis, but today it remains a good choice for evalauating Spinnaker as well as local development.

Note that cache provider classes within Clouddriver may need development to take full advantage of the SqlCache provider. The inital release adds secondary indexing by application specifically to accelerate /applications/{application}/serverGroups calls made by the UI, but only AWS caching agents and providers initially take advantage of this. Prior to adding application based indexing of AWS resources, Netflix still saw performance and consistency gains using Aurora over Redis. Performance of the SQL cache provider should increase over time as Clouddriver’s data access patterns evolve to better utilize features of the underlying storage.

Configuration Considerations

At Netflix, Orca and Clouddriver run with their own dedicated Aurora cluster but this isn’t required. When sizing a database cluster for Clouddriver, make sure the full dataset fits within the InnoDB buffer cache. If migrating an existing Clouddriver installation from Redis to SQL, provisioning database instances with 4x the RAM of the current Redis working set should ensure a fit with room for growth. Much of the additional memory use is accounted for by secondary indexes.

Throughput and write latency are important considerations, especially for Spinnaker installations that manage many accounts and regions. Each distinct (cloudProvider, account, region, resourceType) tuple generally gets an independently scheduled caching-agent instance within Clouddriver. The number of these instances per Clouddriver configuration can be considered the upper bound for concurrent write-heavy sessions to the database, if not limited by the sql.agent.maxConcurrentAgents property. Caching-agent write throughput can directly impact deployment times, especially in large environments with many accounts or multiple cloud providers. Clouddriver only tries to write new or modified resources to the database however, moderating write requirements after initial population.

Database Setup

Clouddriver ships with mysql-connector-java by default. You can provide additional JDBC connectors on the classpath such as mariadb-connector-j if desired. Use of a different RDBMS family will likely require some development effort at this time. Clouddriver was developed targeting Amazon Aurora’s MySQL 5.7 compatible engine.

Before you deploy Clouddriver, you need to manually create a logical database and user grants.

CREATE DATABASE `clouddriver` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
ON `clouddriver`.*
TO 'clouddriver_service'@'%'; -- IDENTIFIED BY "password" if using password based auth

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
ON `clouddriver`.*
TO 'clouddriver_migrate'@'%'; -- IDENTIFIED BY "password" if using password based auth

The following MySQL configuration parameter is required:

tx_isolation Set to READ-COMMITTED

The following MySQL configuration parameter may improve performance for large data sets:

tmp_table_size Increase if the Created_tmp_disk_tables MySQL metric regularly grows

Configure Clouddriver to use MySQL

The following parameters provide a Clouddriver configuration that entirely uses MySQL in place of Redis.
SQL configuration. Set these in your clouddriver-local.yml file.

sql:
  enabled: true
  # read-only boolean toggles `SELECT` or `DELETE` health checks for all pools.
  # Especially relevant for clouddriver-ro and clouddriver-ro-deck which can
  # target a SQL read replica in their default pools.
  read-only: false
  taskRepository:
    enabled: true
  cache:
    enabled: true
    # These parameters were determined to be optimal via benchmark comparisons
    # in the Netflix production environment with Aurora. Setting these too low
    # or high may negatively impact performance. These values may be sub-optimal
    # in some environments.
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: true
  # Enable clouddriver-caching's clean up agent to periodically purge old
  # clusters and accounts. Set to true when using the Kubernetes provider.
  unknown-agent-cleanup-agent:
    enabled: false

  connectionPools:
    default:
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      default: true
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
      user: clouddriver_service
      # password: depending on db auth and how spinnaker secrets are managed
    # The following tasks connection pool is optional. At Netflix, clouddriver
    # instances pointed to Aurora read replicas have a tasks pool pointed at the
    # master. Instances where the default pool is pointed to the master omit a
    # separate tasks pool.
    tasks:
      user: clouddriver_service
      jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
  enabled: false
  cache:
    enabled: false
  scheduler:
    enabled: false
  taskRepository:
    enabled: false

CAUTION: It’s recommended to keep Clouddrivers cache data in SQL, but continue to use the redis SCHEDULER.

As such, the above is not a recommended configuration but shows all available options.

Agent Scheduling

The above yaml configures Clouddriver to use a MySQL table as part of a locking service for the caching agent scheduler. This works well for Netflix in production against Aurora, but could result in poor overall performance for all of Clouddriver, depending on how your database is configured. If you observe high database CPU utilization, lock contention, or if caching agents aren’t consistently running at the expected frequency, you may want to try using the Redis-based scheduler to determine if this is a contributing factor.

The following modification to the above example configures Clouddriver to use SQL for the cache and task repository, and Redis for agent scheduling.

IMPORTANT NOTE FOR SQL USERS NOT USING AWS AURORADB: Problems have been reported when trying to move the Clouddriver Agent Scheduler to SQL in both Google Managed CloudSQL and Amazon Managed RDS SQL. The issue can cause unpredictable behavior concerning Clouddriver’s ability to schedule caching agents. It is recommended to keep your Agent Scheduler in redis if you’re using this type of backend for Clouddriver caching. A configuration example that does this is below. Please see this issue for additional information.

sql:
  scheduler:
    enabled: false

redis:
  enabled: true
  connection: redis://your.redis
  cache:
    enabled: false
  scheduler:
    enabled: true
  taskRepository:
    enabled: false

Maintaining Task Repository Availability While Migrating from Redis to SQL

If you’re migrating from Redis to SQL in a production environment, where you need to avoid pipeline failures and downtime, you can modify the configuration, as shown below, to configure Clouddriver use SQL for caching and new tasks, but with fallback reads to Redis for tasks not found in the SQL database.

redis:
  enabled: true
  connection: redis://your.redis
  cache:
    enabled: false
  scheduler:
    enabled: false
  taskRepository:
    enabled: true

dualTaskRepository:
  enabled: true
  primaryClass: com.netflix.spinnaker.clouddriver.sql.SqlTaskRepository
  previousClass: com.netflix.spinnaker.clouddriver.data.task.jedis.RedisTaskRepository

How Netflix Migrated Clouddriver from Redis to SQL

The following steps were taken to live migrate Clouddriver in the Netflix production Spinnaker stack from Redis to SQL.

Provision the database. In this case, a multi-AZ Aurora cluster was provisioned with 3 reader instances.

Deploy Clouddriver with SQL enabled only for the dualTaskRepository. During the migration, traffic is split between Redis-backed Clouddriver instances and SQL-backed instances. It is important that tasks can be read regardless of request routing to avoid pipeline failures.

# clouddriver.yml; there were no modifications to the redis: properties at this point. The following properties were added:
sql:
  enabled: true
  taskRepository:
    enabled: true
  cache:
    enabled: false
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: false
  connectionPools:
    default:
      default: true
      jdbcUrl: jdbc:mysql://clouddriver-aurora-cluster-endpoint:3306/clouddriver
      user: clouddriver_service
      password: hi! # actually injected from encrypted secrets
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://clouddriver-aurora-cluster-endpoint:3306/clouddriver

dualTaskRepository:
  enabled: true
  primaryClass: com.netflix.spinnaker.clouddriver.data.task.jedis.RedisTaskRepository
  previousClass: com.netflix.spinnaker.clouddriver.sql.SqlTaskRepository

Add a temporary clouddriver-sql-migration clusters with the following configuration. The dualTaskRepository class ordering is flipped.

sql:
  enabled: true
  taskRepository:
    enabled: true
  cache:
    enabled: true
    readBatchSize: 500
    writeBatchSize: 300
  scheduler:
    enabled: true
  connectionPools:
    default:
      default: true
      jdbcUrl: jdbc:mysql://clouddriver-aurora-cluster-endpoint:3306/clouddriver
      user: clouddriver_service
      password: hi! # actually injected from encrypted secrets
  migration:
    user: clouddriver_migrate
    jdbcUrl: jdbc:mysql://clouddriver-aurora-cluster-endpoint:3306/clouddriver

redis:
  enabled: true
  connection: redis://your.redis
  cache:
    enabled: false
  scheduler:
    enabled: false
  taskRepository:
    enabled: true

dualTaskRepository:
  enabled: true
  primaryClass: com.netflix.spinnaker.clouddriver.sql.SqlTaskRepository
  previousClass: com.netflix.spinnaker.clouddriver.data.task.jedis.RedisTaskRepository

A clouddriver-sql-migration-caching server group was deployed with the above configuration, followed by a 5 minute WaitStage, allowing time for cache population. Clouddriver API server groups (with caching agent execution disabled) were then deployed behind the same load balancers routing traffic to the Redis-backed server groups.
After another 5 minute wait, the Redis-backed server groups were disabled. At this time, any tasks running on the disabled Redis instance continue until finished. All new requests are routed to clouddriver-sql-migration-api server groups. If a requested taskId is not present in the SQL database, Clouddriver attempts to read it from Redis.
The clouddriver-sql-migration configuration is merged into main, with the following changes:
```
redis:
  enabled: false

dualTaskRepository:
  enabled: false
```
The disabled Redis-backed Clouddriver instances were verified as idle and the new configuration deployed via a blue/green.
The temporary migration clusters were disabled and then terminated 5 minutes later.

Docs: Set up Front50 to use SQL

Mon, 01 Jan 0001 00:00:00 +0000

You can configure Front50 to use a MySQL compatible database in place of any cloud provider-specific storage service (S3, GCS, etc.).

A migration can be performed without downtime by taking advantage of the dual-read/write capabilities in the provided CompositeStorageService.

Why SQL

Netflix has been plagued by eventual consistency-related issues around our usage of versioned S3 buckets and Front50’s expectations around read-after-write consistency.

Attempts to work around S3’s consistency model have resulted in an operationally complicated production deployment.

The SQL implementation provides a strongly consistent storage service that is performant, operationally simple and feature rich. It supports object versioning out-of-the-box.

Configuration Considerations

At Netflix, Orca, Clouddriver and Front50 run with their own dedicated Aurora cluster but this isn’t required.

Database Setup

Front50 ships with mysql-connector-java by default. You can provide additional JDBC connectors on the classpath if desired. Use of a different RDBMS family will likely require some development effort at this time. Front50 was developed targeting Amazon Aurora’s MySQL 5.7 compatible engine.

Before you deploy Front50, you need to manually create a logical database and user grants.

CREATE DATABASE `front50` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
ON `front50`.*
TO 'front50_service'@'%'; -- IDENTIFIED BY "password" if using password based auth

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
ON `front50`.*
TO 'front50_migrate'@'%'; -- IDENTIFIED BY "password" if using password based auth

Configure Front50 to use MySQL

The following yaml-based parameters provide a Front50 configuration that entirely uses MySQL. Add these to your front50-local.yml file. You will need to disable any other previously enabled storage services (s3, gcs, etc.).

sql:
  enabled: true
  connectionPools:
    default:
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      default: true
      jdbcUrl: jdbc:mysql://your.database:3306/front50
      user: front50_service
      # password: depending on db auth and how spinnaker secrets are managed    
  migration:
    user: front50_migrate
    jdbcUrl: jdbc:mysql://your.database:3306/front50

Migration

The following steps support a no downtime migration to SQL, substitute S3StorageService for the implementation specific to your environment.

It will enable primary and previous writes while leaving reads pointed at previous.

spinnaker:
  migration: 
    primaryClass: com.netflix.spinnaker.front50.model.SqlStorageService
    previousClass: com.netflix.spinnaker.front50.model.S3StorageService  
    compositeStorageService:
      enabled: true
      reads:
        primary: false
        previous: true

Enable the migrator.

spinnaker:
  migration:
    enabled: true

Lastly, you can enable primary reads and optionally fallback to previous if an object is not found.

spinnaker:
  migration:
    compositeStorageService:
      reads:
        primary: true
        previous: true  # set to `false` to disable fallback reads

Once you are satisfied with the migration, remove the spinnaker.migration block and disable the previous storage service.

How Netflix Migrated Front50 from S3 to SQL

The following steps were taken to live migrate Front50 at Netflix from S3 to SQL.

Provision the database. In this case, a multi-AZ Aurora cluster was provisioned with 2 reader instances.
A configuration file was created and scoped to a temporary front50-sql-migration cluster. This profile had SQL, S3 and the migrator enabled (see above for configuration). The resulting cluster was not added to our load balancer and thus took no traffic.
Verify the migration by looking at logs (search for StorageServiceMigrator) and the contents of the SQL database.
Another custom Spring profile was created and scoped to a temporary front50-sql cluster. This profile had SQL, S3 and the composite storage service enabled with dual writes (see above for configuration).
Verify that writes are happening against SQL and S3 (search for CompositeStorageService).
Enable reads against SQL with spinnaker.migration.compositeStorageService.reads.primary: true.
Verify reads are happening against SQL (search for CompositeStorageService).
Disable reads against S3 with spinnaker.migration.compositeStorageService.reads.previous: false.
Diable the migrator with spinnaker.migration.enabled: false.

To ensure safety, we migrated over the course of a week.

Our deployment pipelines were updated to include both SQL and non-SQL deploy stages, this made it easy to quickly revert/rollback along the way.

As of 2019-07, we are still running with the CompositeStorageService enabled (reads against SQL only, writes to SQL and S3) but the migrator has been disabled for a few months.

Docs: Set up Orca to use SQL

Mon, 01 Jan 0001 00:00:00 +0000

Orca’s execution state is stored in Redis by default, but can be configured for SQL. In this topology, Redis is still required for the work queue. Using SQL for execution state will make your Spinnaker installation more durable.

This guide will go over MySQL setup, how to configure Orca, as well as how to perform a zero-downtime migration from Redis to SQL.

If you already have an Orca deployment, you should also refer to the Redis to SQL Migration Guide .

MySQL 5.7 Setup

Orca ships with MySQL drivers by default, but you can include your own JDBC drivers on the classpath if you need to connect to a different database.

Orca has been developed and tested targeting MySQL 5.7. As part of this, setting MySQL’s tx_isolation value to READ-COMMITTED is essential to successfully running Orca in SQL. While Orca will attempt to set this on connection sessions, it is better to have it set on the database itself.

The SQL integration is configured to support a migration user and a service user. The migration user will only be used to perform schema changes on the database, whereas the service user will be used for runtime traffic.

Before deploying Orca, the schema and database uses must first be manually setup:

Set MySQL Server variable tx_isolation setting to READ-COMMITTED. Refer to MySQL Server System Variables .

From the MySQL Server command line run

set tx_isolation = 'READ-COMMITTED';

Setup the schema and database users

CREATE SCHEMA `orca` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
ON `orca`.*
TO 'orca_service'@'%'; -- IDENTIFIED BY "password" if using password based auth

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
ON `orca`.*
TO 'orca_migrate'@'%'; -- IDENTIFIED BY "password" if using password based auth

When Orca starts up, it will perform database migrations to ensure its running the correct schema. It is safe to start multiple Orca services at the same time, even if migrations need to be run.

Configuring Orca for SQL

This configuration is your baseline for Orca to talk to SQL, in orca-local.yml.

sql:
  enabled: true
  connectionPool:
    jdbcUrl: jdbc:mysql://localhost:3306/orca
    user: orca_service
    password: hunter2
    connectionTimeoutMs: 5000
    maxLifetimeMs: 30000
    # MariaDB-specific:
    maxPoolSize: 50
  migration:
    jdbcUrl: jdbc:mysql://localhost:3306/orca
    user: orca_migrate
    password: hunter2

# Ensure we're only using SQL for accessing execution state
executionRepository:
  sql:
    enabled: true
  redis:
    enabled: false

# Reporting on active execution metrics will be handled by SQL
monitor:
  activeExecutions:
    redis: false

# Use SQL for Orca's work queue
# Settings from Netflix and may require adjustment for your environment
# Only validated with AWS Aurora MySQL 5.7
# Please PR if you have success with other databases
keiko:
  queue:
    sql:
      enabled: true
    redis:
      enabled: false

queue:
  zombieCheck:
    enabled: true
  pendingExecutionService:
    sql:
      enabled: true
    redis:
      enabled: false

MariaDB

The default MySQL Connector for Aurora MySQL 5.7 should be fine, but you may also setup Orca to use the MariaDB JDBC driver over MySQL Connector.

The MariaDB driver is Aurora clustering aware, which takes care of automatic master failover operations. Due to licensing issues, Orca cannot ship with the MariaDB driver.

An example of wiring up MariaDB into Orca can be found here: robzienert/orca-mariadb-extension .

Netflix’s Amazon Aurora Example

While vanilla MySQL provides more durability and performance over Redis, Netflix additionally uses Amazon Aurora MySQL 5.7. If you’d like to configure Orca to use Aurora as well, here is how Netflix has it set up.

IMPORTANT: This configuration is for multi-region Aurora replication. If you are only deploying Aurora into a single region, don’t enable any binlog settings.

Aurora Parameter Groups

binlog_cache_size: 32768
default_tmp_storage_engine: InnoDB
general_log: 0
innodb_adaptive_hash_index: 0
innodb_buffer_pool_size: {DBInstanceClassMemory*3/4}
key_buffer_size: 16777216
log_queries_not_using_indexes: 0
log_throttle_queries_not_using_indexes: 60
long_query_time: 0.5
max_allowed_packet: 25165824
max_binlog_size: 134217728
query_cache_size: {DBInstanceClassMemory/24}
query_cache_type: 1
read_buffer_size: 262144
slow_query_log: 1
sync_binlog: 1
tx_isolation: READ-COMMITTED

Aurora DB Cluster Parameter Group

binlog_checksum: NONE
binlog_error_action: IGNORE_ERROR
binlog_format: MIXED
character_set_client: utf8mb4
character_set_connection: utf8mb4
character_set_database: utf8mb4
character_set_filesystem: utf8mb4
character_set_results: utf8mb4
character_set_server: utf8mb4
collation_connection: utf8mb4_unicode_ci
collation_server: utf8mb4_unicode_ci
innodb_checksums: 0