Production-Grade Kubernetes on CoreOS - Multi-Node Deployment Guide#

Deploying Kubernetes in production environments requires careful consideration of security, stability, and scalability. Fedora CoreOS provides an excellent foundation for Kubernetes deployments due to its minimal, immutable nature and built-in security features. This guide walks you through creating a production-ready Kubernetes cluster on CoreOS with enhanced security.

Security Overview#

This deployment includes several security enhancements:

SELinux enabled by default for mandatory access control
CRI-O container runtime with strong SELinux integration
Secure node communication through kubeadm’s built-in PKI infrastructure
Network security with kube-router for network policy enforcement
Immutable infrastructure principles through CoreOS design

Prerequisites#

Before you begin, ensure you have:

Fedora CoreOS qcow2 image (download here)
A host with podman, libvirt, and virt-install tools installed
SSH key pair generated (ssh-keygen -t ed25519)

Hardware requirements per node:

2+ vCPUs
4GB+ RAM
10GB+ storage (20GB+ recommended for production)

Initial Configuration#

1. Create Butane Configuration#

Butane is CoreOS’s configuration transpiler. Create a file named fcos.bu:

1
variant: fcos
2
version: 1.4.0
3
storage:
4
  files:
5
    # CRI-O Configuration
6
    - path: /etc/dnf/modules.d/cri-o.module
7
      mode: 0644
8
      overwrite: true
9
      contents:
10
        inline: |
11
          [cri-o]
12
          name=cri-o
13
          stream=1.17
14
          profiles=
15
          state=enabled
16

17
    # Kubernetes Repository
18
    - path: /etc/yum.repos.d/kubernetes.repo
19
      mode: 0644
20
      overwrite: true
21
      contents:
22
        inline: |
23
          [kubernetes]
24
          name=Kubernetes
25
          baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
26
          enabled=1
27
          gpgcheck=1
28
          gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
29

30
    # Network Configuration
31
    - path: /etc/modules-load.d/br_netfilter.conf
32
      mode: 0644
33
      overwrite: true
34
      contents:
35
        inline: br_netfilter
36

37
    # Kubernetes Network Parameters
38
    - path: /etc/sysctl.d/kubernetes.conf
39
      mode: 0644
40
      overwrite: true
41
      contents:
42
        inline: |
43
          net.bridge.bridge-nf-call-iptables=1
44
          net.ipv4.ip_forward=1
45
passwd:
46
  users:
47
    - name: core
48
      ssh_authorized_keys:
49
        - YOUR_SSH_PUBLIC_KEY_HERE

Replace YOUR_SSH_PUBLIC_KEY_HERE with the content of your public SSH key.

2. Generate Ignition Configuration#

Convert the Butane configuration to an Ignition file:

1
podman run --interactive --rm \
2
  quay.io/coreos/butane:release \
3
  --pretty --strict < fcos.bu > fcos.ign

3. Create VM Deployment Script#

Create a script named start_fcos.sh to launch CoreOS VMs:

1
#!/bin/sh
2

3
IGN_CONFIG=/path/to/fcos.ign
4
IMAGE=/path/to/fedora-coreos.qcow2
5
VM_NAME=node$1
6
VCPUS=2
7
RAM_MB=4096
8
DISK_GB=20
9
STREAM=stable
10

11
chcon --verbose --type svirt_home_t ${IGN_CONFIG}
12
virt-install --connect="qemu:///system" --name="${VM_NAME}" \
13
    --vcpus="${VCPUS}" --memory="${RAM_MB}" \
14
    --os-variant="fedora-coreos-$STREAM" --import --graphics=none \
15
    --disk="size=${DISK_GB},backing_store=${IMAGE}" \
16
    --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${IGN_CONFIG}"

Make the script executable:

1
chmod +x start_fcos.sh

Node Setup#

1. Launch VMs#

Start your cluster nodes. For a basic production setup, you’ll need at least three nodes (one control plane and two workers):

1
# Launch in separate terminals or tmux panes
2
./start_fcos.sh 1  # Control plane
3
./start_fcos.sh 2  # Worker
4
./start_fcos.sh 3  # Worker

Note the IP addresses assigned to each VM during boot.

2. Configure Host Resolution#

Add the node IP addresses to your host’s /etc/hosts file for easier access: