Node Templates Overview #

In the Stackbooster platform, node templates dictate the launch parameters for Kubernetes nodes within your clusters, facilitating optimal resource allocation and cost-efficiency. They allow users to define constraints, properties and launch specifications for nodes added during cluster scaling. Key functionalities include:

• Specifying node launch configuration params, such as subnets, disk size, computer architecture, etc.
• Define the lifecycle of the nodes (spot, on-demand)
• Configuration of the instance types and families that are eligible in the node template.
• Define taints and labels for limiting the pods that can run on the node template

Node Template Variants #

1. Universal Node Template

   • Purpose: Automatically created when a user provisions a Kubernetes cluster from the Stackbooster console.
   • Features:
     • Availability Zones: Includes all zones of the EKS cluster.
     • Security Groups: Uses the security group created for worker nodes.
     • Lifecycle: Defaults to spot with an automatic fallback to on-demand.
     • Instance Families: Excludes certain AWS families to optimise cost and compatibility.
     • Architecture: Supports both amd64 and arm64.
     • Root Device: Uses gp3, 32 GiB.

2. Personalised Node Template

   • Purpose: Lets users supply custom parameters via a StackBoosterNodeTemplate YAML.
   • Configuration: Editable in the Stackbooster console or by applying YAML:

     kubectl apply -f your-custom-node-template.yaml
     

Managing Node Configurations #

Existing nodes cannot be upgraded in place. Delete the node and Stackbooster will recreate it using the new template.

Multiple Node Template Options #

You may create several templates. Design them so that a pod matches at most one template. If a pod matches more than one, Stackbooster picks the cheapest eligible instance.

Attributes of Node Templates #

IAM Instance Profile #

Required. Specifies the Amazon Resource Name (ARN) of the IAM instance profile to associate with the nodes.

instanceProfileArn: arn:aws:iam::000000000000:instance-profile/private-NodeInstanceProfile-1.

SSH Key #

Optional.. Specifies the SSH key that can be used to access nodes.

keyName: "sandbox"

Subnet Selector #

Required. Selects the subnets for node placement using subnet IDs or tag key-value pairs.

1. By Subnet IDs. Selects subnets directly by specifying their IDs. This method uses a comma-separated list of subnet IDs.

subnetSelector:
  ids: "subnet-xxxxxxxxxxxxxxx1,subnet-xxxxxxxxxxxxxxx2,subnet-xxxxxxxxxxxxxxx3"

2. By Tag Key-Value Pair. Selects subnets based on a specific tag key-value pair. This method allows selection of subnets that have tags matching exactly the specified key and value.

subnetSelector:
  Environment: "production"
  Project: "Stackbooster"

Security Group Selector #

Required. Chooses the security groups to be applied to the nodes, selectable by IDs or tag key-value pairs.

1. By Security Group IDs. Selects security groups directly by specifying their IDs. This method uses a comma-separated list of security groups IDs.

securityGroupSelector:
  ids: "sg-xxxxxxxxxxxxxxx1,sg-xxxxxxxxxxxxxxx2"

2. By Tag Key-Value Pair. Selects security groups based on a specific tag key-value pair. This method allows selection of security groups that have tags matching exactly the specified key and value.

securityGroupSelector:
  Environment: "production"

Block Device Mappings #

Optional. Configures the block devices attached to nodes, including settings for volume size and type. Default is 32 GiB and gp3 type (AWS)

blockDeviceMappings:
  - deviceName: "/dev/xvda"
    ebs:
      volumeSize: 32
      volumeType: "gp3"

Lifecycle #

Optional. Defines the lifecycle of the instance, with options like spot, od (on demand), or spot-od (spot with on-demand fallback). Default is spot-od.

lifecycle: "spot-od"

Labels #

Optional. Assigns labels to the nodes for identification and management within the cluster.

labels:
  template: "dev"
  Project: "Stackbooster"

Annotations #

Optional. Assigns annotations to the nodes.

annotations:
  function.kubernetes.io/service-type: "database"

Machine Types #

Optional. Specifies a list of machine types (instance types) that are allowed for node creation. If this list is provided, only the specified machine types will be used for nodes.

machineTypes:
   - "m5.large"
   - "t3.medium"

Node Restrictions #

Optional. Node restrictions allow you to apply constraints on where nodes can be placed based on specified labels, which helps in enforcing policies for workload segregation and resource utilization. StackBooster integrates well-known Kubernetes labels and supports cloud-specific labels. You can view a comprehensive list of supported labels below. Also, user

Labels can be incorporated into either the Node Template or the workload configuration (such as a nodeSelector in a pod’s specification). This includes well-known cloud labels, which are detailed below. The process of node selection relies on matching the criteria set forth in both the Node Template and the pod’s specifications. If there’s no overlap in requirements, StackBooster will refrain from provisioning nodes, thus ensuring that pods are placed only on appropriate nodes. In scenarios where a specific label, important for pod placement and included among the well-known cloud labels, is not set in the Node Template but is included in the pod’s specification, the system might select any available value that fits the cluster’s placement options.

Users have the option to create custom labels. Should a nodeSelector or a necessary nodeAffinity include a label that isn’t specified in the list below, Stackbooster will not deploy nodes with these labels, resulting in pods that are not scheduled. To enable Stackbooster to recognize and schedule nodes based on these custom labels, they must be included in the NodeTemplate requirements.

Properties #

• Key: Label key used for matching the restriction against node labels.
• Operator: Defines the relationship between the node label and the specified values. Possible Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt.
• Values: An array of values relevant to the restriction applied based on the operator.

nodeRestrictions:
  - key: "topology.kubernetes.io/zone"
    operator: "In"
    values: ["us-east-1a", "us-east-1b"]

Supported labels #

Tip – You can combine any of the labels above in Node Template nodeRestrictions or directly in a Pod’s nodeSelector / nodeAffinity. Stackbooster will provision the cheapest instance whose labels satisfy the union of both sets of constraints.

Taints #

Optional. Defines a list of taints to be applied to each node, influencing scheduling decisions.
Array of objects, each representing a single taint.

• Properties:
  • Effect: Specifies the effect of the taint, which determines how the taint influences pod scheduling on the node.
    • Possible Values: NoSchedule, NoExecute.
  • Key: A unique identifier for the taint.
  • Value: Provides additional information about the taint, used to match the taint against pod tolerations.
• Sample Configuration:

  taints:
    - key: "example-key"
      value: "example-value"
      effect: "NoSchedule"

Custom Kubelet Parameters #

Node Templates can include an optional kubeletConfiguration block that changes how the kubelet runs on every node provisioned from the template. Stackbooster converts the fields you specify into --kubelet-extra-args (and, when needed, --dns-cluster-ip) in the bootstrap script.

If kubeletConfiguration is omitted, nodes start with the regular EKS defaults. When the block is present, only the parameters you set are overridden; everything else keeps its default.

YAML schema #

spec:
  kubeletConfiguration:
    evictionHard:                 # [map]     -> --eviction-hard
    evictionSoft:                 # [map]     -> --eviction-soft
    evictionSoftGracePeriod:      # [map]     -> --eviction-soft-grace-period

Lowering maxPods or setting aggressive eviction thresholds can lead to unschedulable system pods. Verify capacity after any change.

StackBooster Node Template Example #

Here’s a sample configuration for a StackBooster Node Template:

apiVersion: stackbooster.io/v1alpha1
kind: StackBoosterNodeTemplate
metadata:
  name: qa
  namespace: stackbooster
spec:
  # SSH Key (Optional)
  # An SSH key to be added to new nodes.
  keyName: "sandbox"

  # IAM Instance Profile (Required)
  # The ARN of the IAM Instance Profile to assign as the node's identity.
  instanceProfileArn: arn:aws:iam::852008484434:instance-profile/private-NodeInstanceProfile-aZlgQ4Zce4ug

  # Subnet Selector (Required)
  subnetSelector:
    ids: subnet-xxxxxxxxxxxxxxx1,subnet-xxxxxxxxxxxxxxx2,subnet-xxxxxxxxxxxxxxx3,subnet-xxxxxxxxxxxxxxx4

  # Security Group Selector (Required)
  securityGroupSelector:
    ids: sg-0dc8f3cd4db83eb2b

  # Block Device Mappings (Optional)
  # Configuration for block devices attached to nodes.
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 20
        volumeType: gp2

  # Node Restrictions (Optional)
  # Restricts nodes based on labels such as instance-type, architecture, OS, zone.
  nodeRestrictions: 
   - key: "topology.kubernetes.io/zone"
     operator: "In"
     values: ["us-east-1a", "us-east-1b"]
  
  taints:
    - key: "example-key"
      value: "example-value"
      effect: "NoSchedule"

  # Lifecycle (Optional)
  # Defines the lifecycle of the instance. Default is 'spot-od'.
  lifecycle: "spot-od"

  # Labels (Optional)
  # Labels to be added to the nodes.
  labels:
    team: qa

  kubeletConfiguration:
    evictionHard:
      memory.available: "100Mi"
    evictionSoft:
      memory.available: "200Mi"
    evictionSoftGracePeriod:
      memory.available: "1m"