In this article, we will look into the basics of the CycloneDX format by generating an SBOM file and going through various entries in the document.

The CycloneDX Format

The CycloneDX SBOM specification has a comprehensive object model that can capture software components, services, interdependencies, and relationships across all inventory types including software, hardware, and other digital assets. The object model can support detailed metadata of all components and services, life cycle stages, etc. The framework is also very extensible and modular and can represent a wide range of supply chain entities and metadata.

Let us look at some of the high-level components of the CycloneDX specification:

• Top Level BOM Metadata - Contains basic information that describes the BOM itself, the tooling used, etc

• Components - Components could be software, hardware, ML models, source code, configuration, etc. Includes the first-party and third-party components involved along with the metadata of each component

• Services - External APIs that the software may call

• Dependencies - Represent the graphs of the dependency of components on other components. It can represent direct and transitive dependencies

There are several other top-level entities in the specification - an overview is available at ttps://cyclonedx.org/specification/overview/

The following visual representation will help you get a better idea of the object model:
https://cyclonedx.org/images/CycloneDX-Object-Model-Swimlane.svg

Anchore Syft for SBOM generation

Syft is a comprehensive open-source CLI and library for generating Software Bill of Materials (SBOMs) for container images and filesystems. Syft development is sponsored by the security company Anchore

Some of the notable features of syft are:

• Generates SBOMs for container images, filesystems, archives, and more to discover packages and libraries

• Supports OCI, Docker, and Singularity image formats

• Linux distribution identification

• SBOM signing/attestation with in-toto specification

• Support multiple SBOM formats - CycloneDX, SPDX, and Syft's proprietary format.

Generating SBOMs with Syft

We will demonstrate syft by generating SBOM for a container image. To begin with, install the soft binary by following the instructions at https://github.com/anchore/syft/. This will install the CLI “syft” on your OS.

The basis usage syntax is as follows:

syft [SOURCE] [FALGS]...

Syft can create SBOMs from a variety of sources including different formats for container images, different container daemons, filesystems and directories, etc. It supports a number of flags, one of the most important ones is the “-o” which can specify an output format and file.

Let us start by creating a CylconeDX formatted SBOM from the alpine docker image from the docker registry.

$syft alpine -o cyclonedx-json=alpine.cyclone.json

This will generate the SBOM file from the official docker repo at “docker.io/library/alpine” in the CycloneDX format and save it into the file alpine.cyclone.json. We will use the “jq” CLI to inspect the file and look at some of the important elements in the SBOM file.

Let us start by looking at the top-level elements of the file

$cat alpine.cyclone.json|jq '.|keys'

[
  "$schema",
  "bomFormat",
  "components",
  "dependencies",
  "metadata",
  "serialNumber",
  "specVersion",
  "version"
]

Looking at the important BOM-specific metadata

$cat alpine.cyclone.json |jq '{"$schema",bomFormat,serialNumber,specVersion,version}'
{
  "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json",
  "bomFormat": "CycloneDX",
  "serialNumber": "urn:uuid:83201860-d58d-4a8e-8acb-59431eb61ce9",
  "specVersion": "1.6",
  "version": 1
}

The SBOM file is generated with the latest CycloneDX schema - version 1.6. Every SBOM will also have a serial number and a version. When you update the SBOM for the same source, the version should be incremented and the serialNumber should remain the same.

The SBOM metadata:

$cat alpine.cyclone.json|jq '.metadata' 
{
  "timestamp": "2024-12-01T01:10:11+05:30",
  "tools": {
    "components": [
      {
        "type": "application",
        "author": "anchore",
        "name": "syft",
        "version": "1.17.0"
      }
    ]
  },
  "component": {
    "bom-ref": "c48d0ee842be961f",
    "type": "container",
    "name": "alpine",
    "version": "sha256:37224ec0ba64192fa71cf0cd764a375e6204b58af5274f9d3b2984f9d5516cbb"
  }
}

The metadata explains what kind of tools are used to create the SBOM as well as the source object (the component in the metadata ). As you can see, the type of component is “container” with the name “alpine” and the version, in this case, is the sha256 hash of the image.

Let us look at the ( software ) components in the BOM now.

$cat alpine.cyclone.json |jq -r '.components|keys[] as $k|[$k+1, .[$k].name, .[$k].type]|@tsv'|column -t -o " | "
1  | alpine-baselayout      | library
2  | alpine-baselayout-data | library
3  | alpine-keys            | library
4  | apk-tools              | library
5  | busybox                | library
6  | busybox-binsh          | library
7  | ca-certificates-bundle | library
8  | libcrypto3             | library
9  | libssl3                | library
10 | musl                   | library
11 | musl-utils             | library
12 | scanelf                | library
13 | ssl_client             | library
14 | zlib                   | library
15 | alpine                 | operating-system

Since Alpine is a lightweight container, it only has 15 components, out of which one is the operating system Alpine itself and the rest are the Alpine packages.

Now let us take a closer look at a single package, we will choose the ssl_client package.

$cat alpine.cyclone.json |jq '.components[12]|del(.properties)'
{
  "bom-ref": "pkg:apk/alpine/ssl_client@1.36.1-r29?arch=x86_64&distro=alpine-3.20.3&package-id=c5128491237ee638&upstream=busybox",
  "type": "library",
  "publisher": "Sören Tempel <soeren+alpine@soeren-tempel.net>",
  "name": "ssl_client",
  "version": "1.36.1-r29",
  "description": "EXternal ssl_client for busybox wget",
  "licenses": [
    {
      "license": {
        "id": "GPL-2.0-only"
      }
    }
  ],
  "cpe": "cpe:2.3:a:ssl-client:ssl-client:1.36.1-r29:*:*:*:*:*:*:*",
  "purl": "pkg:apk/alpine/ssl_client@1.36.1-r29?arch=x86_64&distro=alpine-3.20.3&upstream=busybox",
  "externalReferences": [
    {
      "url": "https://busybox.net/",
      "type": "distribution"
    }
  ]
}

One thing to note particularly is the “bom-ref” key which is a unique key for an element within the BOM document. This ID can be used to refer to this component anywhere within the SBOM document. Most of the other keys are self-explanatory. The CPE and PURL keys are common standards used to specify the location of a package.

Let us also look at a few properties of the package.

$cat alpine.cyclone.json |jq '.components[12].properties'
[
  {
    "name": "syft:package:foundBy",
    "value": "apk-db-cataloger"
  },
  {
    "name": "syft:package:type",
    "value": "apk"
  },
  {
    "name": "syft:package:metadataType",
    "value": "apk-db-entry"
  },
  {
    "name": "syft:location:0:layerID",
    "value": "sha256:75654b8eeebd3beae97271a102f57cdeb794cc91e442648544963a7e951e9558"
  },
  {
    "name": "syft:location:0:path",
    "value": "/lib/apk/db/installed"
  },
  {
    "name": "syft:metadata:installedSize",
    "value": "28672"
  },
  {
    "name": "syft:metadata:originPackage",
    "value": "busybox"
  },
  {
    "name": "syft:metadata:provides:0",
    "value": "cmd:ssl_client=1.36.1-r29"
  },
  {
    "name": "syft:metadata:pullChecksum",
    "value": "Q1fihnCSoO3udDb3DkQwtrfd42MJQ="
  },
  {
    "name": "syft:metadata:pullDependencies:0",
    "value": "so:libc.musl-x86_64.so.1"
  },
  {
    "name": "syft:metadata:pullDependencies:1",
    "value": "so:libcrypto.so.3"
  },
  {
    "name": "syft:metadata:pullDependencies:2",
    "value": "so:libssl.so.3"
  },
  {
    "name": "syft:metadata:size",
    "value": "4693"
  }
]

Please note that I have removed a few entries to shorten the list, the keys are mostly self-explanatory.

One of the key elements in SBOMs is the license of the component. It is common for one component to have multiple licenses as part of the software might be built from different software dependencies - all of which might have different licenses that specify how they are used in another product. Most of the Alpine packages in the SBOM had simpler licenses, so to demonstrate this point, I generated an SBOM from the Ubuntu image. Let us look at the bash package, which has multiple licenses.

$cat ubuntu.cyclone.json|jq '.components[]|select(.name=="bash")|.licenses'
[
  {
    "license": {
      "id": "BSD-4-Clause-UC"
    }
  },
  {
    "license": {
      "id": "GFDL-1.3-only"
    }
  },
  {
    "license": {
      "id": "GPL-2.0-only"
    }
  },
  {
    "license": {
      "id": "GPL-2.0-or-later"
    }
  },
  {
    "license": {
      "id": "GPL-3.0-only"
    }
  },
  {
    "license": {
      "id": "GPL-3.0-or-later"
    }
  },
  {
    "license": {
      "id": "Latex2e"
    }
  },
  {
    "license": {
      "name": "GFDL-NIV-1.3"
    }
  },
  {
    "license": {
      "name": "MIT-like"
    }
  },
  {
    "license": {
      "name": "permissive"
    }
  }
]

As you can see, there are multiple licenses associated with bash.

Next, we will look at one of the most important parts of the SBM - software dependencies. We will examine the dependencies of the ssl_client package

$cat alpine.cyclone.json |jq '.dependencies[]|select(.ref=="pkg:apk/alpine/ssl_client@1.36.1-r29?arch=x86_64&distro=alpine-3.20.3&package-id=c5128491237ee638&upstream=busybox")'
{
  "ref": "pkg:apk/alpine/ssl_client@1.36.1-r29?arch=x86_64&distro=alpine-3.20.3&package-id=c5128491237ee638&upstream=busybox",
  "dependsOn": [
    "pkg:apk/alpine/libcrypto3@3.3.2-r0?arch=x86_64&distro=alpine-3.20.3&package-id=0bd67c24de5c4187&upstream=openssl",
    "pkg:apk/alpine/libssl3@3.3.2-r0?arch=x86_64&distro=alpine-3.20.3&package-id=409f5b93e7b861be&upstream=openssl",
    "pkg:apk/alpine/musl@1.2.5-r0?arch=x86_64&distro=alpine-3.20.3&package-id=3ea0974d202d0c73"
  ]
}

If you see here, the dependency entry for the ssl_client package, all dependencies are referenced using their “bom-ref” ID. This helps in building a dependency graph of all packages within the SBOM.

As you can see, an SBOM provides a lot of valuable information about your application/containers that helps you with security audits, supply chain security, license compliances, etc.

Syft is just one tool that can help you generate an SBOM. There are more such tools in the SBOM ecosystem. A selected list of tools can be found at the CycloneDX tools center - https://cyclonedx.org/tool-center/

What is an AWS Transit Gateway?

AWS Transit Gateway is a service that is aimed at simplifying AWS network topologies. It functions as a managed transit hub that uses a hub-and-spoke model: VPCs, on-premises networks, data centers, or SD-WAN solutions can connect to the Transit Gateway. Instead of potentially managing dozens or hundreds of peering connections, tunnels, etc., engineers can now attach the relevant network entities and manage routing in a centralized place. AWS Transit Gateway provides a scalable, secure, and streamlined networking solution by eliminating the complexity of many point-to-point connections.

Key Components and Architecture

The key component is the Transit Gateway itself, which acts as the central hub and a virtual router for the VPCs and on-premise networks. In addition, the following components are part of the Transit Gateway Architecture.

• Attachments - Transit Gateway Attachments are connections that facilitate attaching different network entities like VPCs or VPNs to the transit gateway. Commonly supported attachments are:

  • VPCs

  • VPN Connections

  • Direct Connect Gateway

  • SD-WAN/Third-party Network Appliances

  • Another Transit Gateway

• Route Tables - A transit gateway comes with a default route table, but can optionally have additional route tables. Like regular route tables, these route tables determine the next hop based on the destination IP address. Route tables can have static ( manual configuration ) or dynamic ( routing protocol-based) routes and the next hope in each route will be a Transit Gateway Attachment.

• Routing Table Association - An attachment will be associated with exactly one route table. A route table within Transit Gateway might or might not be associated with an attachment

• Route Propagation - Attachments can “advertise” routes to a Transit Gateway route table. If you enable route propagation for a VPC attachment, the subnet CIDRs of that VPC are automatically advertised to the associated Transit Gateway route table. Similarly, for on-premises connections, the CIDRs of on-premises networks can be automatically propagated to the Transit Gateway.

Transit Gateway is a regional resource, but it can have associations across regions or accounts. It can also peer with other Transit Gateways to segment or simplify network topologies. The following diagram provides a view of how Transit Gateway fits into AWS network architecture.

Advantages of Using AWS Transit Gateway

• Centralized and Simplified Network Management - The biggest USPO of Transit Gateway is that it simplifies network management. Rather than handling a large number of point-to-point connections, TGW helps you centralize these connections with a hub and spoke architecture

• Scalability and High Availability - TGW is a fully managed service and scales to handle your traffic automatically. There is no need to manually provision any network components or resize the SKUs as your network traffic needs evolve. The data plane traffic is distributed across multiple availability zones for high availability.

• Consistent Performance - Transit Gateway traffic flows through the AWS backbone and that provides consistent performance for inter-VPC and hybrid connectivity. When you peer multiple Transit Gateways across Regions, traffic stays on AWS’s global private network, avoiding the unpredictable nature of the public internet.

• Centralized Security and Monitoring - Since the transit gateway consolidates network traffic across your cloud infrastructure, it is easy to apply managed policies and network ACLs on it. As of September 2024, security group referencing is also enabled on Transit Gateways. The centralized nature also makes it easy to have network analysis, packet inspection, firewall capabilities, etc to be implemented in a centrally dedicated infrastructure. The VPC Flow Logs for TGW can be used for packet analysis across the AWS network infrastructure connected to it.

• Flexible and Granular Routing Policies - The Transit gateway enables you to enforce granular control over routing. The ability to associate specific routing tables on individual attachments makes it easy to control routing. Transitive routing which was not possible between three or more VPCs is now possible with TGW. It is also possible to segment the network by controlling which parts of the network should and should not communicate with each other.

• Cost-Effectiveness Over Large-Scale Deployments - While there is a cost associated with AWS Transit Gateway attachments and data processing, the service can be more cost-effective compared to managing a large mesh of VPC peering connections or multiple VPNs. The fewer network endpoints you have to configure, monitor, and secure, the lower your total administrative overhead. Many organizations also see cost savings by consolidating connectivity through Transit Gateway, particularly when egress or data transfer charges in multi-VPC environments are taken into account.

Common use cases of Transit Gateways

TGWs are used to centralize networking and ease management burden. But let us look at some of the use cases of how customers can make use of TGW in their infrastructure.

• Interconnecting Multiple VPCs - A fundamental use case for AWS Transit Gateway is interconnecting multiple VPCs. In a complex cloud infrastructure, organizations will maintain 100s of VPCs. The regular VPC peering is one-to-one in nature and handling a large number of VPCs would require a large mesh of peer connections. TGW simplifies this mesh by providing a Hub-and-Spoke architecture for managing network infrastructure.

• Hybrid Connectivity with On-Premises Data Centers - AWS Transit Gateway offers a cohesive way to connect on-premises data centers to AWS. Enterprises can use Site-to-Site VPN for smaller or less latency-sensitive workloads or opt for Direct Connect for more demanding, data-intensive applications. Once the on-premises networks are attached to the Transit Gateway, each VPC that is also attached can communicate as needed, all managed from a single routing domain

• Multi-Region High-Availability and Disaster Recovery - For High Availability and business continuity plans, businesses invest in multi-region infrastructure. Reliable cross-connectivity between infrastructure in different regions is critical for such plans. Transit gateways can act as the central hub that facilitates the traffic routing between these regions.

• Shared Services with central control and security - Create a hub of shared services/offerings that can be accessed from customer networks by integrating with the transit gateway. This will allow businesses to offer private/dedicated services to customers while maintaining control and ensuring security. A SaaS provider with tenant VPCs for each customer would be a good example of such a case.

• SD-WAN Integration - A lot of organizations use traditional SD-WAN services from telecom/network vendors. Utilizing a transit gateway, this kind of infrastructure can be migrated to integrate with AWS, allowing for better management and security.

Additional Networking Features of Transit Gateways

Some of the additional features of the transit gateways that are worth mentioning are:

• Larger MTU - The maximum transmission unit is the size of the largest packet that can go through a network. A larger MTU means better bandwidth and faster communication. While a lot of traditional networks have an MTU of 1500 bytes, TGW offers 8500 bytes MTU for many of its attachments.

• ECMP - Equal cost multipathing is a network-level load balancing method that allows sending traffic to the same destination through multiple routes. The transit gateway supports ECMP for different types of attachments.

• Multicast support - Multicast is a network protocol that allows sending the same message to a selected set of network destinations. The transit gateway supports multicast and provides options to configure multicast groups and destinations.

Transit gateways are a game changer in simplifying cloud network architecture, eliminating the need for complex mesh topologies and legacy hub and spoke networks. It is the key to maintaining a secure and efficient network topology with simplified management. This is very critical as your cloud footprint and complexity grow.

For a very long time crontab has been the undisputed choice for scheduling jobs on Linux systems. But with the wide adoption of systemd as the system and service manager, it has given us a new option - systemd timers.

Most casual Linux users consider systemd as a modern replacement for the SysV init system. While it is partially true, systemd is much more than that. To quote the systemd website:

systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, maintains mount and automount points, and implements an elaborate transactional dependency-based service control logic.

While a discussion on the systemd capabilities is outside the scope of this article, the following points will serve as a refresher to better understand this article:

• Systemd owns PID1 and is started by the Linux kernel

• All other processes in the system are descendants of systemd

• It is responsible for filesystem initialization

• The fundamental building block of the systemd ecosystem is the concept of units

• A unit is a system resource that systemd knows how to manage

• There are different types of units, including services, devices, mountpoints, sockets, etc

• The definition and configuration of a unit is stored in unit files under different directories managed by systemd

• There are system level units and user level units

With that background set, let's get back to running crons with systemd timers.

Systemd Timers

Systemd provides an alternative to crons in the form of systemd timers. Timers are a type of unit files that define how a job or a service can be run on calendar events or monotonic events. Calendar events are similar to cron time and date fields, set based on a calendar time, and can be recurring as well. Monotonic events are configured as a time delta from the occurrence of an event, say boot time.

• A systemd timer unit file will have the extension .timer

• For every systemd timer unit file, there will be another unit file with the same name and a .service extension

• The timer unit defines the “when to run”, whereas the service unit defines the “what to run”

Let's use an example to demonstrate how timers work.

Consider the scenario of taking a MySQL backup every three hours. A crontab entry would look like this:

0 */3 * * *  /usr/local/bin/mysql-backup.sh

And the script would be

cat /usr/local/bin/mysql-backup.sh

#!/bin/bash
/usr/bin/mysqldump inventory  > /opt/db-backup/inventory-db-backup_`date +%H-%d-%m-%Y`.sql

The authentication will be taken care of with /root/.my.cnf

[client]
host=localhost
user=MYSQL_BACKUP_USER
password=MYSQL_BACKUP_PASSWORD

Now let us see how to move this cron to a systemd timer.

First we need to translate our backup script into a systemd service.

cat /etc/systemd/system/mysql-backup.service

[Unit]
Description="MySQL Backup Service"
[Service]
ExecStart=/usr/local/bin/mysql-backup.sh

Now we need to create the timer unit

cat /etc/systemd/system/mysql-backup.timer
[Unit]
Description="Run mysql-backup.service every 3 hours"

[Timer]
OnCalendar=*-*-* 00/3:00:00
Unit=mysql-backup.service

[Install]
WantedBy=multi-user.target

Most parts of these unit files are self-explanatory and are not different from normal unit files. The timer section describes what service to run ( mysql-backup.service ) and one or more time options. In this particular case, we are giving a specific calendar timing that executes the service once every three hours. We will look at a few more timer options later.

Now, verify the files are syntactically correct.

sudo systemd-analyze verify /etc/systemd/system/mysql-backup.*

If this command doesn't return any output, then we are good to proceed.

Reload systemd to update the system about new unit files.

sudo systemctl daemon-reload

Enable and start the timer unit.

sudo systemctl enable --now mysql-backup.timer

The timer is active now.

sudo systemctl status mysql-backup.timer
● mysql-backup.timer - "Run mysql-backup.service every 3 hours
     Loaded: loaded (/etc/systemd/system/mysql-backup.timer; disabled; vendor preset: enabled)
     Active: active (waiting) since Sat 2023-03-08 21:07:23 IST; 3s ago
    Trigger: Sun 2023-03-09 00:00:00 IST; 2h 52min left
   Triggers: ● mysql-backup.service

As you can see, the timer is enabled and the next run time is also listed.

Now let us look at the few ways the timers can be configured. As mentioned in the beginning, there are two categories of timers - real time ( calendar based ) and monotonic ( based on events ).

Monotonic timers

Monotonic timers are triggered after a specific time elapsed from an event, like boot time. There are different options to configure the monotonic timers, some of them are given below.

• OnBootSec: time after the machine boots up

• OnActiveSec: time after the timer unit is activated

• OnUnitActiveSec: time after the service unit was last activated

• OnUnitInactiveSec: time after the service unit was last deactivated

• OnStartupSec: time after the service manager is started

There are various formats in which you can provide values to the monotonic timer options. Some of them are

• 5hours

• 34minutes

• 5hours 34minutes

• 1y 3month

Real-time timer

Triggered by calendar events, the real-time timers have only one option: “OnCalendar”. This is the option that closely resembles crontab timers, so let's have a quick comparison.

Crontab Format:   minute hours day-of-the-month month day-of-the-week

This is a five-part format that can use *, absolute values, ranges, and lists for each part.

Systemd Timer Format:  Day Of the week      Year-Month-Date      Hour:Minute:Second

This three-part format works as follows:

• First part is the day of the week with 3 character values from Mon to Sun

• Second part is Year-Month-Date in number format with 4 digit years, two digit month and day

• Third part is time - 2 digits per hour, minute, and second

• A continuous range of values can be indicated with two dots. Eg: Mon..Wed mean Mon, Tue, Wed

• A list of values can be indicated with a comma. Eg: Sat,Sun

• Asterisk can be used as a wild card to match all valid values. Eg: “*” in the first part means all weekdays - Mon..Sun

• “/” can be used as a repetition option

• Default values can be skipped and the syntax can be shortened in specific ways

• Shorthands like minutely, hourly, yearly, etc can be used as a timer

The systemd time manpage covers the time formats in detail. Refer - https://man.archlinux.org/man/systemd.time.7

A few values for OnCalendar option are:

• Sat,Sun --* 23:00 - Every Saturday and Sunday at 11

• 2023-02-14 00:00:01 - 1 second into 14 Feb 2023

• --* 00/3:00:00 - Every three hours

• 23:40/5 - Every day every 5 minutes starting 11:40 PM

The variety of values in the OnCalendar option can be confusing. To ensure we provide the right values, systemd provides a command-line option. Let us validate the last value from this list as follows:

sudo systemd-analyze calendar --iterations=2 "23:40/5"
  Original form: 23:40/5
Normalized form: *-*-* 23:40/5:00
    Next elapse: Sat 2023-03-08 23:40:00 IST
       (in UTC): Sat 2023-03-08 18:10:00 UTC
       From now: 17min left
       Iter. #2: Sat 2023-03-08 23:45:00 IST
       (in UTC): Sat 2023-03-08 18:15:00 UTC
       From now: 22min left

As you can see, the command interprets the shorthand and expands into the normalized form, it also provides when the next few occurrences of the schedule will be ( controlled with –iterations ). It also tells you how much time from your current time will the event occur.

Conclusion

As you can see, the systemd timers are very versatile. Crontab provides the simple and efficient form of scheduling, and is available ubiquitously on all Linux operating systems, unlike systemd, which is not guaranteed to be available on all variants of Linux. Having said that, systemd is available on most popular operating systems and these are some of the advantages of timers.

• With service units, the jobs can be independently tested and run anytime without timers

• A job can be configured to depend on another systemd unit. For example, run the MySQL backup unit only if the mysqld service is running.

• Can be resource controlled with cgroups and slices

• Easy debugging with journalctl

• Time formats allow more control. Years and seconds are supported.

Do experiment with systemd timers. Most of the systemd options that can be used to customize the unit files can be used to better configure your timers and services. If you are running scheduled jobs for production systems, using the systemd timers makes more sense, and is easier to automate with configuration management tools.

In the past decade, we experienced exponential growth and transformation of software development, cloud technologies, and adoption of DevOps culture which supported the advancement of the Continuous Delivery (CD) ecosystem. With the growth, we also witnessed focused advancement of the Site Reliability Engineering (SRE) perspective.. In this blog, we present a broader outlook on the evolution of CD with SRE through the insights presented by our ambassadors.

To fully realize the potential of CD at scale, the integration of SRE principles is essential. Balancing investment in tools and upskilling for reliability vis-a-vis rapid innovation in CD would be an optimal operating model for the digital economy. However, it is important to note that as SRE comes of age, it faces scalability, growth, and complexity challenges.

Advanced resiliency needs, next-generation security threats, and exponential data integration into software products indicate the need for evolution in the SRE approach hand-in-hand with CD. The evolution of SRE with the following key features can be seen as a critical success factor for unleashing the potential of the CD Ecosystem.

Better Reliability Posture and Overall Stability

How does SRE tie to CD? Some of the core tenets of SRE are change management, incident management, and observability. The central theme around which incident and change management revolves is “change”. The CD pipeline is the vehicle of change in your application infrastructure. Having a comprehensive CI/CD solution that covers all changes to production will provide much-needed control for SREs to understand and resolve incidents faster. It also enables them to implement change management processes as well as programmatically verify that the process is adhered to. This in turn translates to better reliability posture and overall stability.

Incident Management

One of the key tenets of SRE is effective incident management. And more often than not, incidents are associated with changes in production. Now, these could be code, configuration, or infrastructure changes. Change awareness—the knowledge of the recent changes that were pushed to production applications infrastructure is vital in resolving an incident within the shortest possible time. And these resolutions involve rolling back (or forward) the problematic changes to a known good state. The Last Known Good state aka LKG is one of the primary strategies used in determining the stable state to move to. This process is closely tied to one of the most important metrics SREs track, the Mean Time to Resolve (MTTR) of an incident.

Change Management

Another aspect of SRE is the discipline of change management itself. SRE engagement models greatly vary based on the cultural and organizational aspects of companies, and this applies to change management as well. A model that works for one organization won’t necessarily fit well in another organization. But there are always some underlying principles that can be applied across teams and organizations. The older model of change approval boards and central control has given way to the peer review and approval model. This is often supplemented with automated software and security testing as part of the continuous integration pipeline so that issues are caught and resolved early on.

Observability

SREs are also responsible for monitoring, or rather observability of the application infrastructure. While this predominantly covers observability of the production infrastructure, as the reliability practices have evolved observability has also extended to the measurement of engineering excellence. This is often achieved with reliability scorecards that are tied to various aspects of engineering and application delivery. Similar to how golden signals provide observability into production infrastructure, DORA metrics provide observability into engineering excellence. To put it in the CI/CD parlance, even observability is shifting—or rather expanding—to the left

Data-Driven CD: Metrics and SLO

A lot has been done in the past with SRE principles getting mainstream, however, most of the SRE practices are still falling short of “shift left” quickly, as highlighted by Dynatrace’s State of SRE Report: 2022 Edition. More often, early integration of SRE principles and practices into CI/CD with a data-centric metric-based approach could be the next step in the evolution of CI/CD with SRE. A unified view on SLOs right from the inception stages of Continuous Delivery, some guidance in this direction could be taken from DevOps Research and Assessment (DORA).

Advanced Reliability Engineering and CD: Platform, Tools & Application

As the adoption of SRE principles scales, it is evident that the reliability engineering space remains fragmented and heavily focused on monitoring, visualization, and communication. To tailor the SRE principles to the organizational needs, SREs often have to take a hybrid approach with automation, monitoring & AIOps tools co-existing in the ecosystem. In the future with the evolution of CD, SRE tools & applications would not only need consolidation but a more standardized approach to scale. CD integration with SRE will go beyond the current hybrid, fragmented tool-based integration to a more platform-oriented approach, paving the way for a more proactive, insightful, and action-oriented integration of CD & SRE.

Emerging Technology & its Integration with CD

As CD tends to integrate emerging technologies, for example, AI-based features, more & more SRE-based tools & applications will have to be moving in the same direction with Data Observability (ML Observability as an example). Chaos Engineering is another important practice, and when integrated based on standardized interfaces and core components, can be maturing as a framework for not only experimenting but also evaluating and prioritizing resilience at every stage of continuous delivery.

Net-Zero Commitment – SRE Can Lead the Way

As CD makes its way to more and more industry segments, it is evident we need to get more serious about the Carbon Footprint and steps to mitigate and reduce Carbon impact. SREs have been toying with the ideas of cost reduction and right-sizing to reduce the TCO and bring down the carbon footprint as a side effect. By observing and managing workloads, and on-demand features for more resource-centric digital applications, products, and features, SREs can take a more conscious approach toward carbon footprint reduction. With this, SRE principles and practices will lead the way toward a carbon-aware and carbon-optimized CD Ecosystem.

Some of these thoughts and practices are traditionally considered to be part of the DevOps domain. But in any reasonably sized organization, SRE and DevOps practices are often intertwined, driving towards the common goal of achieving production reliability and stability through engineering excellence. These cross-functional practices are centrally pivoted on continuous integration and delivery.

Monitoring is the cornerstone of operating any software system or application effectively. The more visibility you have into the software and hardware systems, the better you are at serving your customers. It tells you whether you are on the right track and, if not, by how much you are missing the mark.

So what should we expect from a monitoring system? Most of the monitoring concepts that apply to information systems apply to other projects and systems as well. Any monitoring system should be able to collect information about the system under monitoring, analyze and/or process it, and then share the derived data in a way that makes sense for the operators and consumers of the system.

The meaningful information that we are trying to gather from the system is called signals. The focus should always be to gather signals relevant to the system. But just like any radio communication technology that we are drawing this terminology from, noise will interfere with signals. Noise being the unwanted and often irrelevant information that is gathered as a side effect of monitoring.

Traditional monitoring has been built around active and passive checks and the use of near-real-time metrics. The good old Nagios and RRDTools worked this way. Monitoring gradually matured to favor metrics-based monitoring, and that gave rise to popular platforms like Prometheus and Grafana.

Centralized Log analysis and deriving metrics from logs became mainstream - the ELK stack was at the forefront of this change. But the focus is now shifting to traces and the term monitoring is being replaced by observability. Beyond this, we also have all the APM (Application Performance Monitoring) and Synthetic monitoring vendors offering various levels of observability and control.

All these platforms provide you with the tools to monitor anything, but they don’t tell you what to monitor. So how do we choose the relevant metrics from all this clutter and confusion? The crowded landscape of monitoring and observability makes the job harder, not to mention the efforts needed to identify the right metrics and separate noise from the signal. When things get complicated, one way to find a solution is to reason from first principles. We need to deconstruct the problem and identify the fundamentals and build on that. In this specific context, that would be to identify what is the absolute minimum that we need to monitor and then build a strategy on that. So on that note, let’s understand the popular strategy used to choose the right metrics.

SRE Golden Signals

SRE Golden signals were first introduced in the Google SRE book - defining it as the basic minimum metrics required to monitor any service. This model was about thinking of metrics from first principles and serves as a foundation for building monitoring around applications. The strategy is simple - for any system, monitor at least these four metrics - Latency, Traffic, Errors, and Saturation.

Latency

Latency is the time taken to serve a request. While the definition seems simple enough, latency has to be measured from the perspective of a client or server application. For an application that serves a web request, the latency it can measure is - the time delta between the moment the application receives the first byte of request, to the moment the last byte of the response to this request leaves the application. This would include the time the application took to process the request and build the response and everything in between - which could include disk seek latencies, downstream database queries, time spent in the CPU queue, etc. Things get a little more complicated when measuring latency from the client's perspective because now the network between the client and server also influences the latency. The client could be of two types - the first is another upstream service within your infrastructure, and the second - and more complex - are real users sitting somewhere on the internet and there is no way of ensuring an always stable network between it and the server. For the first kind, you are in control and measure the latencies from the upstream application. For internet users, employ synthetic monitoring or Real User Monitoring (RUM) to get an approximation of latencies. These measurements get overly complicated when there is an array of firewalls, load balancers, and reverse proxies between the client and the server.

There are certain things to keep in mind when measuring latencies. The first is to identify and segregate the good latency and the bad latency, ‌the latencies endured by a successful request versus failed request. Quoting from the SRE Book, an HTTP 500 error latency should be measured as bad latency, and should not be allowed to pollute the HTTP 200 latencies - which could cause an error in judgment when planning to improve your request latencies.

Another important matter is the choice of the type of metrics for latency. Average or rate are not good choices for latency metrics as a large latency outlier can get averaged out and would blindside you. This outlier - otherwise called “tail” can be caught if the latency is measured in buckets of requests. Pick a reasonable number of latency buckets and count the number of requests per bucket. This would allow for plotting the buckets as histograms and flush out the outliers as percentiles or quartiles.

Traffic

Traffic refers to the demand placed on your system by its clients. The exact metric would vary based on what the system is serving - there could also be more than one traffic metric for a system. For most web applications this could be the number of requests served - in a specific time frame. For a streaming service like youtube, it can be the amount of video content served. For a database, it would be the number of queries served and for a cache, it could be the number of cache misses and cache hits.

A traffic metric could be further broken down based on the nature of requests. For a web request, this could be based on the HTTP code, HTTP method or even the type of content served. For video streaming, service content downloads for various resolutions could be categorized. For YouTube, the amount and size of video uploads are traffic metrics as well. Traffic can also be categorized based on geographies or other common characteristics. One way to measure the traffic metrics is to calculate traffic as a monotonically increasing value - usually of the metric type “counter” and then calculate the rate of this metric over a defined interval - say 5 minutes.

Errors

This is measured by counting the number of errors from the application and then calculating the rate of errors in a time interval. Error rate per second is a common metric used by most web applications. For eg: errors could be 5xx server-side errors, 4xx client-side errors, 2xx responses with an application-level error - wrong content, no data found, etc. It would also use a counter-type metric and then a rate calculated over a defined interval.

An important decision to make here would be what we can consider as errors. It might look like the errors are always obvious - like 5xx errors or database access errors, etc. But there is another kind of error that is defined by our business logic or system design. For example, serving wrong content for a perfectly valid customer request would still be an HTTP 200 response, but as per your business logic and the contract with the customer, this is an error. Consider the case of a downstream request that ultimately returns the response to an upstream server, but not before the latency threshold defined by the upstream times out. While the upstream would consider this an error - as it should be, the downstream may not be aware that it breached an SLO (which is subject to change and may not be part of the downstream application design) with its upstream and would consider this a successful request - unless the necessary contract is added to the code itself.

Saturation

Saturation is a sign of how used or “full” the system is. 100% utilization of a resource might sound ideal in theory, but a system that’s nearing full utilization of its resources could lead to performance degradation. The tail latencies we discussed earlier could be the side effect of a resource constraint at the application or system level. The saturation could happen to any sort of resources that are needed by the application. It could be system resources like memory or CPU or IO. Open file counts hitting the max limit set by the operating system and disk or network queues filling up are also common examples of saturation. At the application level, there could be request queues that are filling up, the number of database connections hitting the maximum, or thread contention for a shared resource in memory.

Saturation is usually measured as a “gauge” metrics type, which can go up or down, but usually within a defined upper and lower bound. While not a saturation metric, the 99th percentile request latency (or metrics on outliers) of your service could act as an early warning signal. Saturation can have a ripple effect in a multi-tiered system where your upstream would wait on the downstream service response indefinitely or eventually timeout - but also causing additional requests to queue up, resulting in resource starvation.

While the Golden signals covered in this blog are metrics-driven and ideally a good starting point to measure if something is going wrong, they are not the only things to consider. There are various other metrics not necessary to track on a daily basis, but certainly an important place to investigate when an incident takes place. We will be covering this in Part 2 of this blog series.

Irrespective of your strategy, understanding why a system exists, what are its services, and the business use cases it serves, are vital. This ‌will lead you to identify the critical paths in your business logic and help you model the metrics collection system based on that.

The traditional approach to network security

The traditional approach to securing an infrastructure is to use network perimeter-based protection — otherwise known as the castle and moat approach. The corporate firewall becomes the moat that encircles and protects the network castle and anyone inside is trusted while the rest of the world is untrusted. These parties could get access to the inside either by being located inside, in a connected corporate office/ datacentre network, or connected via VPN. All devices and users behind the firewall perimeter are trusted while the rest of the world is considered untrusted. There could be different zones of trust within the trusted environment with more restrictions ( for example DMZ). Some of the tools employed to secure the trusted perimeters are firewalls, VPNs, ACLs, IDS, IPS, etc.

The inherent weakness in this approach is the de facto classification of inside devices and users as trusted. Devices and users can be compromised and once they are compromised can be used to launch attacks using the implicit trust conferred upon them. This problem is further aggravated by the growing adoption of SaaS/IaaS cloud services, more remote users, and bring-your-own-device (BYOD) policies. The perimeters of the trusted network get extended further in such a hybrid environment. It has become hard to bring all these elements under a few trusted perimeters and classify the traffic as trusted or untrusted.

Zero trust approach to network security

The cornerstone of zero-trust security is the elimination of the primary weakness in traditional security, ie implicit trust. Zero trust doesn’t use the network location — whether a user or device is inside the perimeter — to decide on authorization or access. It rather treats all requests as hostile and enforces security for every request made.

The approach follows three principles:

1. Verify explicitly: Authenticate and authorize devices and users based on multiple criteria set by the organization’s policy. This could include but is not limited to the health of the device, the identity of the user, the classification of target service or data to be accessed, suspicious activities originating from the device or by the user, etc.

2. Use least privilege access — Restricting a user or device to the minimum permission required to perform a certain action minimizes the attack surface. It prevents the chances for lateral movements ( getting access to one location and then moving on to adjacent services and systems ).

3. Assume Breach — Assume every access/operation is hostile and then implement processes to identify and approve legitimate access.

The zero trust model is not a single technology or software implementation. It is rather a framework that can be implemented by incorporating several security technologies that exist and being used already in silos or some combinations.

Fundamentally zero trust is the system that should allow access from a subject to a resource while ensuring authentication, authorization, and transport security for every request between them.

The subject could be

• User/Identity

• Device/Endpoints

• Applications

• Systems

The resource/object could be one or a combination of the following

• Data

• Applications/APIs

• Infrastructure/Systems

The workhorse of zero-trust is the policy decision and enforcement infrastructure. This infrastructure is spread across the control and data planes of the zero-trust ecosystem.

The control plane hosts two components of the policy infrastructure:

1. Trust evaluation/policy engine — Responsible for granting access to a request from a subject to a resource. This component depends on input from multiple systems to make its decision. This includes enterprise access policy and input from various information and security systems like SIEM ( Security information and event management ), compliance services, threat intelligence services, etc.

2. Policy administrator/Access control engine — This component is responsible for authentication, authorization, and access control for every request. It is also responsible for the continued evaluation of trust while a trusted subject is accessing a resource ( and revoking the access if necessary ). This service will make use of identity management services, the policy engine, real-time threat analysis services, etc.

A data plane usually has only one component.

1. (Trusted) Proxy — This proxy is also known as the policy enforcement point. It is responsible for enabling, monitoring, and terminating connections between subject and resource. It intercepts the traffic from the subject, executes the decision from the policy engine and administrator, and ensures the adaptive access control is enforced. It is responsible for encrypting all traffic.

The following diagram represents high-level elements of the zero-trust ecosystem.

Image adapted from NIST SP 800–207

Zero trust is a reference model that can be implemented in different ways as long as the underlying principles are maintained. Hence you would find different cloud and open-source products that fall under the zero-trust security model. Irrespective of the product or service you chose, zero-trust is an iterative process and requires a lot of work beyond using some services. Some of them include

1. Identifying the data stores

2. Classify the data according to sensitivity

3. Identify all the roles and the level of access to this data required for various use cases

4. Ensure all users are brought under one or more of these roles

5. Map out all transaction flows/communications to these systems and build policies that map those flows to the role and level of access required

6. Use the above information along with company policies, compliance requirements, and security practices to create enterprise access policies.

7. Once these are sorted out work out what solutions you want to adopt and redesign your security infrastructure accordingly.

Chaos engineering is the discipline of experimenting on a software system in production to build confidence in the system’s capability to withstand turbulent and unexpected conditions.

If you are new to Chaos Engineering, go through this introduction first:

Engineering Chaos: A Gentle Introduction

In production outages, a lot of blame is attributed to the network — sometimes with reason and evidence but countless other times because there is no other visible culprit to blame.

To increase the resilience against network failures and degradation, we need to run our chaos experiments on the network. But this is not always easy — if your application is in a data center, the chances of getting your hands on the network infrastructure to introduce chaos are close to zero, and with good reason. If the application is hosted in the cloud, the network layer is mostly abstracted out from you.

What in this situation would be the right way to introduce some network chaos? Given we can't manipulate the networking infrastructure itself, the next best thing we can do is redirect the traffic to a system that we can control and then forward the traffic to the original destination. This can be achieved in different ways — manipulating routing, modifying DNS records, using forward proxies, and transparently intercepting network packets using tools like iptables or EBPF.

In this article, we are going to examine one such tool — Toxiproxy. It is a framework and TCP proxy that can simulate poor network conditions. It was developed by Shopify to test the resilience of its webstack. Toxiproxy is a network proxy that can intercept and forward TCP communication. It is highly performant and easy to configure. For any traffic flow that needs to be intercepted and tested for network degradation, that traffic can be sent through Toxiproxy and subjected to various experiments before being sent to its intended destination.

Toxiproxy has two components:

1. The control plane — the API used to manage the proxy configuration. The control plane can be managed by directly hitting the API/the toxiproxy-cli / various client libraries

2. The data plane — the proxies that are created on demand to proxy different services

The Toxiproxy ecosystem is as given below

OK, so we have installed and started toxiproxy, but how exactly does it simulate poor network conditions, and what are those conditions?

To proxy the traffic to any given downstream service, a corresponding proxy has to be created within toxiproxy with a source port of our choosing ( through which we will proxy to the destination ) and the port of the specific downstream/destination service.

For example, when you want to proxy traffic to a remote MySQL server running on default port 3306, you create a proxy with a source port of your choice ( say 4306 ) and destination port and host as :3306. Now your application will configure its MySQL client to talk to :4306.

Once the proxy is ready, it's time to introduce the fault (anomaly/poor condition). In toxiproxy, these conditions are called toxics ( hence the name toxiproxy ). These toxics have their own parameters/attributes.

While the toxics and the attributes are mostly self-explanatory, more details about toxics and attributes can be found here

Setting up and testing a proxy

Toxiproxy installation is quite easy ( it is a single binary each for the server and the CLI ). Instructions can be found here. Once the proxy is installed, run it on the default port — 8474 ( or an alternate port of your choosing — in which case you should use “— host” option with the cli ). This is the port on which the control plane API would be available.

Once the installation is done, we can start setting up proxies.

First start toxiproxy by running the server binary: toxiproxy-server . Binary can be run without any arguments to run it on interface 127.0.0.1 port 8474. Toxiproxy keeps all modifications in memory, but a config file in JSON format with predefined proxies and toxics can be provided as a command-line argument. Once the proxy is started it can be manipulated using the toxiproxy-cli binary or the client libraries in different languages. The process is outlined below.

Now let us try a practical example

• toxiproxy is running on the default port on my laptop

• the downstream we will proxy to is the Geo-location API of ipify.org. Specifically, the endpoint https://geo.ipify.org/api/v1 which returns the public IP and Geolocation of the caller

• Let's configure the proxy with

• — Unique proxy name: ipify

• — Downstream server: geo.ipify.org

• — Downstream port: 443 ( SSL )

• — Proxy port: 8443

• Toxic to inject

• — type: latency

• — attribute: latency

• — value: 1500 ( milliseconds )

• — name: latency_1500

Let us create the proxy using the toxiproxy-cli

toxiproxy-cli create ipify --listen localhost:8443 --upstream geo.ipify.org:443

Add toxic

toxiproxy-cli toxic add --toxicName latency_1500 -type latency --attribute latency=1500 ipify

Let's list the proxies and then inspect the ipify proxy

toxiproxy-cli list

Name Listen Upstream Enabled Toxics
\====================================================================
ipify 127.0.0.1:8443 geo.ipify.org:443 enabled 1

toxiproxy-cli inspect ipify

Name: ipify Listen: 127.0.0.1:8443 Upstream: geo.ipify.org:443
\====================================================================
Upstream toxics:
Proxy has no Upstream toxics enabled.

Downstream toxics:
latency_1500: type=latency stream=downstream toxicity=1.00 attributes=[ jitter=0 latency=1500 ]

Let’s hit the geo.ipify.org API directly and get my public ip and Geo location using curl. We will also print the total time taken for the request. We will filter the JSON output using jq to only pick up the country of the public IP. Please note that I have saved my API key to the shell variable IPIFY_APIKEY already.

curl -s -w "%{stderr}Total Time: %{time_total}\nCountry from public IP: " "https://geo.ipify.org/api/v1?apiKey=${IPIFY_APIKEY}" |jq .location.country

Total Time: 4.108721
Country from public IP: "IN"

The vanilla request without any proxy took approx 4100 milliseconds / 4 seconds

Now let us send the traffic via the proxy we created earlier. Not that we need to pass the host header and disable SSL checking.

curl -k -s -w "%{stderr}Total Time: %{time_total}\nCountry from public IP: " -H "Host: geo.ipify.org" "https://localhost:8443/api/v1?apiKey=${IPIFY_APIKEY}" |jq .location.country

Total Time: 5.727683
Country from public IP: "IN"

As you can see, the request now took 5700 milliseconds with the addition of roughly 1500 milliseconds. You can experiment with various toxics like this to chaos test your app against network conditions.

Note: The ipify API response time greatly varies when it is under load ( and am using a free version of their API ). Try to experiment withe either a performant public API or an internally hosted service for consistent results.

So how do we better prepare for such issues? Making fixes based on the speculation and waiting for an outage to occur to confirm our hypothesis is neither scalable nor practical. We need to attack this problem head-on. That is where chaos engineering comes into the picture.

Chaos engineering is the discipline of experimenting on a software system in production to build confidence in the system’s capability to withstand turbulent and unexpected conditions

In other words, chaos engineering looks for evidence of weakness in a production system.

The Evolution of Chaos Engineering

The term chaos engineering rose into popularity shortly after Netflix published a blog in 2010 about their journey in migrating their infrastructure into the AWS cloud. In the article, they talked about the tool Chaos Monkey, which they used to randomly kill EC2 instances. This was to simulate production outage-like scenarios and observe how their infra copped with such failures. This trend of experimenting by introducing failures into the production infrastructure quickly caught on and companies started adopting the principles and soon chaos engineering evolved into its own discipline.

While Netflix might have popularized chaos engineering, breaking infrastructure to test the resiliency has its roots in Amazon, where Jesse Robins, popularly known in Amazon as the Master of Disaster introduced Gamedays. Gameday is a program where failures are introduced into production to observe how systems and people respond to them and based on the observation the systems are fixed/rebuilt/re-architected and processes improved. This greatly helped Amazon in exposing weaknesses in its infrastructure and fixing them.

Chaos Experiments

Practical chaos engineering at its heart is the process of defining, conducting, and learning from chaos experiments.

Before we start with experiments, we need to identify a target system for which we will test the resilience. This system could be an infrastructure component or an application. Once you identify such a target, map out its dependencies/downstream — databases, external APIs, hardware, cloud/data center infrastructure, etc. The aim is to identify the impact on the target — if any — when an anomaly/fault is injected into one of its dependencies.

Once we have decided on the target and the dependency to which fault should be injected — define a steady-state for this system. A steady state of a system is the desirable state in which the target can serve its purpose optimally.

Following this, form a hypothesis about the resilience of the system. The hypothesis defines the impact that the target would suffer when the fault is injected into the dependency, causing the target to violate its steady state. As part of this, the nature and severity of fault to be injected — the real-world failure scenario that needs to be simulated — should be defined as well.

After we decide on these factors, the fault injection can begin. In an ideal world, the fault should be injected into the production system, though it doesn’t hurt to first try this out in a pre-production environment. In a reasonably well-built system, the target will withstand the fault for a while, then start failing when it crosses some threshold or as the impact of the fault aggravates. If the impact of the fault is not severe, or the target is designed to withstand the fault, the experiment and the severity of the fault need to be redefined. In either case, the experiment should define a stop condition — this is the point when an experiment is stopped — either after encountering an error/breakage in the target system or after a defined period without encountering any errors/violation to the steady state.

During the experiment, record the behavior of the target system — what was the pattern of the standard metrics, what events and logs it generated, etc. It is also important to watch upstream services that make use of the target service. If the experiment was stopped after encountering an error this would provide valuable insight into the resiliency and should be used to improve the resilience of the target system as well as upstream services.

If the target didn’t encounter any errors and the experiment finished without incidents after the defined time, the fault and its impact need to be redefined. This would entail defining more aggressive faults which would result in increased impact on the target system. This process is known as increasing the blast radius.

Rinse and repeat the process until all weaknesses are eliminated.

An example to tie it all together

Consider your company is into fleet management and last-mile delivery. There are several micro-services in your application infrastructure. There would be front-end and back-end services that handle inventory, vehicles, drivers, shifts, route planning, etc.

Note: this is a very trimmed down and hypothetical design and may not cover all possibilities

Let’s follow the process outlined in the previous section.

• Selection of target service — route planning microservice.

• The route planner uses Google Maps API, an in-memory cache like Redis, and a MySQL back-end among other things. Route planner exposes a REST API that is used by the upstream front-end services that display the optimal route the drivers should take.

• Downstream dependency service — we pick Google Maps API as the service to which fault should be injected.

• The steady-state: when operating in the optimal condition, the route planner will serve 99% of the requests under 100ms latency ( P99 latency ). Service can tolerate P99 latency upto 200ms

• Fault and severity — A latency of 50ms when accessing the Google Maps API — for each API call

• Hypothesis: Upstream front-end servers have a timeout of 200ms for getting a response from the route planner API. Even after accounting for a 50ms latency to Google Maps API, the route planner will still return a result within 150ms which is well within the expectation of the upstreams. Expect the P99 latency to be at 150ms and no significant increase in 4xx or 5xx errors for the route planner. Don’t expect the upstream services using this API to have any issues.

• Stop condition:

• — 10 minutes run without any impacts

• — P99 latency crosses 200ms

• — Increase in 5xx errors — 2% of total requests

• — Failure reports from upstream services or customers

• Experiment start and fault injection: In practice, this could be achieved in many ways, one popular option being the use of an intermediary proxy to control the latency of the outgoing traffic. For eg: Toxiproxy can be configured to sent outbound traffic with a latency of 50ms ( and proxy the traffic to google maps API ). For this to work, the route planner application should be redeployed with the Toxiproxy endpoint in the place of the Google Maps API URL. If you are on Kubernetes you can also use Chaosmesh for introducing chaos.

• Record fault injection and its impact

• Scenario 1: No issues, P99 latency was close to but less than 150ms. No visible change in the number of 5xx/4xx errors

• — Stop the experiment. Increase the blast radius by raising the latency from 50ms to 75ms. Repeat the experiment

• Scenario 2: P99 latency is between 150ms and 200ms, 5xx errors for the front end spike to 3%. Users report seeing blank pages instead of route plans.

• — Stop the experiment, and investigate the issue.

• — You find that for certain routes 3 Google Maps API calls were needed instead of 1. The route planner returned the right result within 250ms ( 100 + 50 * 3 ) but the API request from the front-end server requesting the route data has already timed out at 200ms. This causes front-end servers to show a blank page instead of a meaningful message. Since only a small percentage of requests had this issue, their 200+ ms latency was averaged out with a large number of 150 ms requests.

• — Fix the code, and see if the 3 calls can be run in parallel instead of sequential. Modify the front-end code to gracefully handle timeout and show a customer-friendly message.

Once the issues are fixed, repeat the experiments.

Best practices and checklists

• If there are single-point-of-failures / other resiliency issues in the infrastructure/application that are already known fix them before attempting to run chaos experiments in the production

• Make sure all the vital metrics for the target and dependent systems are being monitored. A robust monitoring system is essential for running chaos experiments

• While it is natural to focus primarily on the target service and the downstream dependency, it is vital to monitor the upstream services and the customers of the target service

• Use learning from previous outages and postmortems to create a better hypothesis as well as fix known problems proactively

• Use the learning from chaos experiments not only to fix software systems but also to fix any gaps in the process — oncall response, runbooks, etc.

• Once you are confident about the experiments, automate them fully and let them run periodically in production.

• To build confidence to test in production, chaos experiments can be incorporated into the testing process for pre-production build pipelines