Deploying CoreDNS and StepCA with Kubernetes Manifests Using Quadlet on CoreOS#

Fedora CoreOS provides an excellent foundation for running containerized workloads, especially when combined with Podman and systemd. While Kubernetes is a popular choice for container orchestration, it can be complex to set up and manage. Quadlet offers a compelling alternative by allowing you to deploy Kubernetes-style manifests directly through systemd on CoreOS, combining the declarative power of Kubernetes manifests with the simplicity of systemd service management.

This guide demonstrates how to deploy two critical infrastructure components—CoreDNS for DNS management and StepCA for certificate authority services—using Kubernetes manifests orchestrated through Quadlet on Fedora CoreOS.

Understanding the Architecture#

Before diving into implementation, let’s understand the overall architecture:

1
graph TD
2
    A[CoreOS Host] --> B[systemd]
3
    B --> C[Quadlet]
4
    C --> D[Podman]
5
    D --> E[CoreDNS Kubernetes Deployment]
6
    D --> F[StepCA Kubernetes Deployment]
7

8
    E --> G[DNS Service]
9
    F --> H[Certificate Authority]
10

11
    I[Client Systems] --> G
12
    I --> H
13

14
    G --> J[(Zone Files)]
15
    H --> K[(Certificate Store)]

This architecture enables:

Declarative Configuration: Using Kubernetes manifests for service definition
SystemD Integration: Managing container deployments through systemd
No Kubernetes Required: Running Kubernetes-style resources without a full cluster
Simplified Management: Standard systemd commands for controlling services
Domain Name Resolution: Internal DNS with custom zones
Certificate Infrastructure: Self-hosted PKI for secure communications

Prerequisites#

Before beginning, ensure you have:

Fedora CoreOS 35 or newer
Root access to the system
Basic understanding of systemd, Kubernetes manifests, and Podman
The kubectl command-line tool installed

1
# Check CoreOS version
2
rpm-ostree status
3

4
# Verify kubectl availability or install it
5
which kubectl || rpm-ostree install kubectl
6

7
# Check Podman version (should be 4.0+)
8
podman --version

Implementation Steps#

1. Creating the Configuration Directory#

First, let’s create the directory where our Quadlet files will reside:

1
# Create the systemd directory for Quadlet files
2
sudo mkdir -p /etc/containers/systemd

2. Preparing StepCA Password#

For StepCA, we need to generate a secure password and encode it for use in the Kubernetes Secret:

1
# Generate a secure random password
2
STEPCA_PASSWORD=$(openssl rand -base64 32)
3

4
# Base64 encode it for use in Kubernetes Secret
5
STEPCA_PASSWORD_BASE64=$(echo -n "$STEPCA_PASSWORD" | base64)
6

7
# Store the plain password securely for reference
8
echo "$STEPCA_PASSWORD" | sudo tee /etc/containers/stepca.password
9
sudo chmod 600 /etc/containers/stepca.password
10

11
# Output the encoded password for use in our manifest
12
echo "Encoded password for use in manifest: $STEPCA_PASSWORD_BASE64"

3. Creating the CoreDNS Deployment Quadlet#

Now, let’s create the Quadlet file for CoreDNS:

1
# Create the CoreDNS Quadlet file
2
sudo tee /etc/containers/systemd/dns-deployment.kube > /dev/null << 'EOF'
3
#
4
# CoreDNS Kubernetes Deployment via Quadlet
5
#
6
[Unit]
7
Description=CoreDNS Kubernetes Deployment
8
After=network-online.target
9
Wants=network-online.target
10

11
[Kube]
12
# Namespace for DNS services
13
Namespace=dns
14

15
# The manifest for CoreDNS
16
Yaml='''
17
apiVersion: v1
18
kind: Namespace
19
metadata:
20
  name: dns
21
---
22
apiVersion: v1
23
kind: ConfigMap
24
metadata:
25
  name: coredns-config
26
  namespace: dns
27
data:
28
  Corefile: |
29
    .:53 {
30
        forward . 8.8.8.8
31
        log
32
        errors
33
    }
34
    invinsense:53 {
35
        file /etc/coredns/invinsense.db
36
        log
37
        errors
38
    }
39
  invinsense.db: |
40
    $ORIGIN invinsense.
41
    @       3600 IN SOA  coredns.invinsense. admin.invinsense. (
42
            2023062001 ; serial
43
            7200       ; refresh
44
            3600       ; retry
45
            1209600    ; expire
46
            3600       ; minimum
47
    )
48
    @       3600 IN NS  coredns.invinsense.
49
    coredns   IN A     192.168.122.16
50
    ca        IN A     192.168.122.76
51
---
52
apiVersion: apps/v1
53
kind: Deployment
54
metadata:
55
  name: coredns
56
  namespace: dns
57
spec:
58
  replicas: 1
59
  selector:
60
    matchLabels:
61
      app: coredns
62
  template:
63
    metadata:
64
      labels:
65
        app: coredns
66
    spec:
67
      containers:
68
      - name: coredns
69
        image: docker.io/coredns/coredns:1.11.1
70
        args: ["-conf", "/etc/coredns/Corefile"]
71
        ports:
72
        - containerPort: 53
73
          name: dns
74
          protocol: UDP
75
        - containerPort: 53
76
          name: dns-tcp
77
          protocol: TCP
78
        volumeMounts:
79
        - name: config-volume
80
          mountPath: /etc/coredns
81
      volumes:
82
      - name: config-volume
83
        configMap:
84
          name: coredns-config
85
---
86
apiVersion: v1
87
kind: Service
88
metadata:
89
  name: coredns
90
  namespace: dns
91
spec:
92
  selector:
93
    app: coredns
94
  ports:
95
  - name: dns
96
    port: 53
97
    protocol: UDP
98
  - name: dns-tcp
99
    port: 53
100
    protocol: TCP
101
  type: LoadBalancer
102
'''
103

104
[Install]
105
WantedBy=default.target
106
EOF

4. Creating the StepCA Deployment Quadlet#

Next, we’ll create the Quadlet file for StepCA. Note that we need to substitute the actual base64-encoded password:

1
# Create the StepCA Quadlet file with password substitution
2
sudo tee /etc/containers/systemd/pki-deployment.kube > /dev/null << EOF
3
#
4
# StepCA Kubernetes Deployment via Quadlet
5
#
6
[Unit]
7
Description=StepCA Kubernetes Deployment
8
After=network-online.target
9
Wants=network-online.target
10

11
[Kube]
12
# Namespace for PKI services
13
Namespace=pki
14

15
# The manifest for StepCA
16
Yaml='''
17
apiVersion: v1
18
kind: Namespace
19
metadata:
20
  name: pki
21
---
22
apiVersion: v1
23
kind: Secret
24
metadata:
25
  name: stepca-secret
26
  namespace: pki
27
type: Opaque
28
data:
29
  password: ${STEPCA_PASSWORD_BASE64}
30
---
31
apiVersion: v1
32
kind: PersistentVolumeClaim
33
metadata:
34
  name: stepca-data
35
  namespace: pki
36
spec:
37
  accessModes:
38
    - ReadWriteOnce
39
  resources:
40
    requests:
41
      storage: 1Gi
42
---
43
apiVersion: apps/v1
44
kind: Deployment
45
metadata:
46
  name: stepca
47
  namespace: pki
48
spec:
49
  replicas: 1
50
  selector:
51
    matchLabels:
52
      app: stepca
53
  template:
54
    metadata:
55
      labels:
56
        app: stepca
57
    spec:
58
      containers:
59
      - name: stepca
60
        image: smallstep/step-ca:0.24.0
61
        ports:
62
        - containerPort: 443
63
        env:
64
        - name: STEPCA_INIT_NAME
65
          value: "Invinsense CA"
66
        - name: STEPCA_INIT_DNS_NAMES
67
          value: "ca.invinsense"
68
        - name: STEPCA_INIT_ADDRESS
69
          value: ":443"
70
        - name: STEPCA_PASSWORD
71
          valueFrom:
72
            secretKeyRef:
73
              name: stepca-secret
74
              key: password
75
        volumeMounts:
76
        - name: stepca-data
77
          mountPath: /home/step
78
      volumes:
79
      - name: stepca-data
80
        persistentVolumeClaim:
81
          claimName: stepca-data
82
---
83
apiVersion: v1
84
kind: Service
85
metadata:
86
  name: stepca
87
  namespace: pki
88
spec:
89
  selector:
90
    app: stepca
91
  ports:
92
  - port: 443
93
    targetPort: 443
94
  type: LoadBalancer
95
'''
96

97
[Install]
98
WantedBy=default.target
99
EOF

5. Setting Correct Permissions#

Ensure the Quadlet files have appropriate permissions:

1
# Set correct permissions for Quadlet files
2
sudo chmod 644 /etc/containers/systemd/*.kube

6. Enabling and Starting Services#

Now, let’s enable and start the services through systemd:

1
# Reload systemd to detect new Quadlet files
2
sudo systemctl daemon-reload
3

4
# Enable and start CoreDNS
5
sudo systemctl enable --now kube-dns-deployment
6

7
# Enable and start StepCA
8
sudo systemctl enable --now kube-pki-deployment

7. Verifying Service Deployment#

Let’s verify that our services are running correctly:

1
# Check systemd services status
2
sudo systemctl status kube-dns-deployment
3
sudo systemctl status kube-pki-deployment
4

5
# Check Kubernetes resources using kubectl
6
kubectl get all -n dns
7
kubectl get all -n pki
8

9
# Check the pods
10
kubectl get pods -n dns
11
kubectl get pods -n pki

8. Configuring DNS Resolution#

To use our CoreDNS server for name resolution:

1
# Get CoreDNS service IP
2
COREDNS_IP=$(kubectl get svc -n dns coredns -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
3
echo "CoreDNS service IP: $COREDNS_IP"
4

5
# Configure systemd-resolved
6
sudo mkdir -p /etc/systemd/resolved.conf.d/
7
sudo tee /etc/systemd/resolved.conf.d/dns.conf > /dev/null << EOF
8
[Resolve]
9
DNS=$COREDNS_IP
10
Domains=~invinsense
11
EOF
12

13
# Restart systemd-resolved
14
sudo systemctl restart systemd-resolved

9. Initializing StepCA Client#

To use certificates from our StepCA installation:

1
# Install step CLI if needed
2
sudo rpm-ostree install step-cli
3

4
# Get StepCA service IP
5
STEPCA_IP=$(kubectl get svc -n pki stepca -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
6
echo "StepCA service IP: $STEPCA_IP"
7

8
# Get CA fingerprint
9
CA_POD=$(kubectl get pods -n pki -l app=stepca -o jsonpath='{.items[0].metadata.name}')
10
CA_FINGERPRINT=$(kubectl exec -n pki $CA_POD -- step certificate fingerprint /home/step/certs/root_ca.crt)
11
echo "CA Fingerprint: $CA_FINGERPRINT"
12

13
# Bootstrap with CA
14
step ca bootstrap --ca-url https://$STEPCA_IP --fingerprint $CA_FINGERPRINT

Testing the Infrastructure#

Now that we have deployed both CoreDNS and StepCA, let’s verify they’re working properly:

Testing DNS Resolution#

1
# Test DNS resolution through CoreDNS
2
dig @$COREDNS_IP ca.invinsense
3
dig @$COREDNS_IP coredns.invinsense
4

5
# Test external domain resolution
6
dig @$COREDNS_IP google.com

Testing Certificate Issuance#

1
# Request a certificate for a test domain
2
step ca certificate test.invinsense test.crt test.key
3

4
# Verify the certificate
5
step certificate verify test.crt --roots ~/.step/certs/root_ca.crt

Advanced Configuration#

Modifying CoreDNS Zones#

To add new DNS records:

Edit the Quadlet file:

1
sudo vim /etc/containers/systemd/dns-deployment.kube

Add your new entries to the zone file in the ConfigMap:

1
invinsense.db: |
2
  $ORIGIN invinsense.
3
  # ...existing entries...
4
  newservice IN A 192.168.122.100

Apply the changes:

1
sudo systemctl daemon-reload
2
sudo systemctl restart kube-dns-deployment

Adding StepCA Provisioners#

To create additional certificate provisioners:

Connect to the StepCA pod:

1
CA_POD=$(kubectl get pods -n pki -l app=stepca -o jsonpath='{.items[0].metadata.name}')
2
kubectl exec -it -n pki $CA_POD -- /bin/sh

Add a new provisioner:

1
step ca provisioner add deploy --type JWK

Exit the pod:
Terminal window
```
1
exit
```

Maintenance Procedures#

Viewing Service Logs#

1
# CoreDNS logs
2
kubectl logs -n dns -l app=coredns
3

4
# StepCA logs
5
kubectl logs -n pki -l app=stepca

Updating Services#

To update the container images:

1
# Update CoreDNS
2
kubectl set image deployment/coredns -n dns coredns=docker.io/coredns/coredns:1.11.2
3

4
# Update StepCA
5
kubectl set image deployment/stepca -n pki stepca=smallstep/step-ca:0.25.0

Backing Up Critical Data#

1
# Create a backup directory
2
sudo mkdir -p /var/backups/quadlet
3

4
# Backup Kubernetes resources
5
kubectl get all -n dns -o yaml | sudo tee /var/backups/quadlet/dns-backup.yaml
6
kubectl get all -n pki -o yaml | sudo tee /var/backups/quadlet/pki-backup.yaml
7

8
# Backup StepCA data
9
CA_POD=$(kubectl get pods -n pki -l app=stepca -o jsonpath='{.items[0].metadata.name}')
10
kubectl cp -n pki $CA_POD:/home/step /var/backups/quadlet/stepca-data
11

12
# Backup configurations
13
sudo tar -czf /var/backups/quadlet/k8s-quadlet-backup.tar.gz \
14
    /etc/containers/systemd/*.kube \
15
    /etc/containers/stepca.password

Troubleshooting Common Issues#

DNS Resolution Issues#

If DNS resolution isn’t working:

Check if CoreDNS is running:
Terminal window
```
1
kubectl get pods -n dns
```

Verify CoreDNS configuration:

1
kubectl describe configmap -n dns coredns-config

Check DNS connectivity:

1
podman run --rm alpine nslookup -type=a coredns.invinsense $COREDNS_IP

Certificate Issuance Problems#

If certificate issuance fails:

Check StepCA pod status:
Terminal window
```
1
kubectl get pods -n pki
```
Examine StepCA logs:
Terminal window
```
1
kubectl logs -n pki -l app=stepca
```
Verify connectivity to StepCA:
Terminal window
```
1
curl -k https://$STEPCA_IP/health
```

Quadlet Service Failures#

If the Quadlet services aren’t starting:

Check the systemd service status:
Terminal window
```
1
sudo systemctl status kube-dns-deployment
```
Examine journal logs:
Terminal window
```
1
sudo journalctl -u kube-dns-deployment
```

Validate the Quadlet file syntax:

1
podman kube --dry-run play /etc/containers/systemd/dns-deployment.kube

Security Considerations#

When implementing this infrastructure, consider the following security aspects:

Restrict DNS Access: Configure firewall rules to limit DNS access to your network
Secure CA Keys: StepCA private keys are critical security assets
Regular Backups: Back up the StepCA data regularly
Certificate Rotation: Set up automatic certificate rotation
Access Controls: Implement RBAC for StepCA access
Network Segmentation: Place your infrastructure on a separate network
TLS for CoreDNS: Consider enabling DNS over TLS for sensitive queries

Conclusion#

Deploying CoreDNS and StepCA using Kubernetes manifests with Quadlet provides a powerful way to bring Kubernetes-like declarative infrastructure to Fedora CoreOS without the overhead of a full Kubernetes cluster. This approach gives you:

Simplified management through standard systemd commands
Declarative configuration using familiar Kubernetes manifests
Infrastructure as code practices without complex orchestration
Critical infrastructure services for DNS and certificate management
Persistent configuration that survives system reboots

By integrating DNS and certificate management, you create a foundation for secure, service-oriented architecture within your environment. The Quadlet approach elegantly bridges the gap between traditional system administration and modern Kubernetes-based configuration, offering the best of both worlds.