Deploying CoreDNS and StepCA on CoreOS using Podman Quadlet#

Modern infrastructure requires both reliable DNS resolution and secure certificate management. Fedora CoreOS provides an excellent foundation for running containerized infrastructure services with its focus on security, atomic updates, and minimal footprint. This guide demonstrates how to deploy both CoreDNS (for DNS services) and StepCA (for certificate authority services) on CoreOS using Podman Quadlet, which integrates containers directly with systemd.

Understanding the Components#

Before diving into the implementation, let’s understand the key components:

1
graph TD
2
    A[CoreOS] --> B[Podman]
3
    B --> C[Quadlet]
4
    C --> D[systemd Integration]
5
    D --> E[CoreDNS Container]
6
    D --> F[StepCA Container]
7

8
    E --> G[DNS Resolution]
9
    F --> H[Certificate Authority]
10

11
    G --> I[Internal Domain Names]
12
    G --> J[External Domain Resolution]
13

14
    H --> K[Certificate Issuance]
15
    H --> L[Certificate Validation]
16

17
    M[Client System] --> G
18
    M --> H

CoreOS: A minimal container-focused operating system
Podman: A daemonless container engine
Quadlet: A tool that integrates containers with systemd
CoreDNS: A flexible DNS server written in Go
StepCA: A certificate authority for secure automated certificate management

Prerequisites#

To follow this guide, you need:

A Fedora CoreOS system (version 38 or newer recommended)
Administrative access (sudo privileges)
Basic knowledge of containers, DNS, and PKI concepts

Implementation Steps#

1. Verifying the Environment#

First, ensure you’re running CoreOS with Podman installed:

1
# Check CoreOS version
2
rpm-ostree status
3

4
# Verify Podman installation
5
podman --version

Make sure Podman version is 4.0 or higher, which includes Quadlet support.

2. Creating Directory Structure#

Set up the necessary directories for our configuration files:

1
# Create main directories
2
sudo mkdir -p /etc/containers/systemd
3
sudo mkdir -p /etc/containers/coredns
4
sudo mkdir -p /etc/containers/stepca

3. Creating CoreDNS Configuration Files#

Create the configuration files needed for CoreDNS:

1
# Create the Corefile for CoreDNS
2
sudo tee /etc/containers/coredns/Corefile > /dev/null << 'EOF'
3
.:53 {
4
    forward . 8.8.8.8
5
    log
6
    errors
7
}
8

9
invinsense:53 {
10
    file /etc/coredns/invinsense.db
11
    log
12
    errors
13
}
14
EOF
15

16
# Create the zone file for your internal domain
17
sudo tee /etc/containers/coredns/invinsense.db > /dev/null << 'EOF'
18
$ORIGIN invinsense.
19
@       3600 IN SOA  coredns.invinsense. admin.invinsense. (
20
        2024062201 ; serial
21
        7200       ; refresh
22
        3600       ; retry
23
        1209600    ; expire
24
        3600       ; minimum
25
)
26

27
@       3600 IN NS  coredns.invinsense.
28

29
coredns  IN A     127.0.0.1
30
ca       IN A     127.0.0.1
31
EOF

4. Setting Up StepCA Environment#

Generate a secure password for StepCA and save it in an environment file:

1
# Generate a secure random password
2
PASSWORD=$(openssl rand -base64 32)
3

4
# Create environment file
5
sudo bash -c "cat > /etc/containers/stepca.env << EOF
6
STEPCA_PASSWORD=$PASSWORD
7
EOF"
8

9
# Secure the file
10
sudo chmod 600 /etc/containers/stepca.env
11

12
# Save the password somewhere secure for reference
13
echo "Your StepCA password is: $PASSWORD"
14
echo "Make sure to save this password in a secure location!"

5. Creating Quadlet Configuration Files#

Now, let’s create the Quadlet files that define our containers as systemd services:

1
# Create CoreDNS container configuration
2
sudo tee /etc/containers/systemd/coredns.container > /dev/null << 'EOF'
3
[Unit]
4
Description=CoreDNS DNS Server
5
Documentation=https://coredns.io
6
After=network-online.target
7
Wants=network-online.target
8

9
[Container]
10
Image=docker.io/coredns/coredns:1.11.1
11
Network=host
12
Volume=/etc/containers/coredns:/etc/coredns:ro,z
13
UserNS=keep-id
14

15
[Service]
16
Restart=always
17
TimeoutStartSec=30
18
TimeoutStopSec=30
19

20
[Install]
21
WantedBy=multi-user.target
22
EOF
23

24
# Create StepCA container configuration
25
sudo tee /etc/containers/systemd/stepca.container > /dev/null << 'EOF'
26
[Unit]
27
Description=StepCA Certificate Authority
28
Documentation=https://smallstep.com/docs/step-ca
29
After=network-online.target
30
Wants=network-online.target
31

32
[Container]
33
Image=docker.io/smallstep/step-ca:0.24.0
34
Network=host
35
EnvironmentFile=/etc/containers/stepca.env
36
Environment=STEPCA_INIT_NAME=Invinsense CA
37
Environment=STEPCA_INIT_DNS_NAMES=ca.invinsense
38
Environment=STEPCA_INIT_ADDRESS=:443
39
Volume=stepca-data:/home/step
40

41
[Service]
42
Restart=always
43
TimeoutStartSec=60
44
TimeoutStopSec=30
45

46
[Install]
47
WantedBy=multi-user.target
48
EOF
49

50
# Create volume for StepCA data
51
sudo tee /etc/containers/systemd/stepca.volume > /dev/null << 'EOF'
52
[Volume]
53
Driver=local
54
Device=tmpfs
55
Options=type=tmpfs
56
UID=0
57
GID=0
58
EOF

6. Activating the Services#

Now we’ll enable and start our services:

1
# Reload systemd to detect new quadlet files
2
sudo systemctl daemon-reload
3

4
# Start CoreDNS
5
sudo systemctl enable --now container-coredns
6

7
# Start StepCA
8
sudo systemctl enable --now container-stepca

7. Configuring Local DNS Resolution#

Configure your system to use your local CoreDNS server for DNS resolution:

1
# Create resolved configuration
2
sudo mkdir -p /etc/systemd/resolved.conf.d/
3
sudo tee /etc/systemd/resolved.conf.d/dns.conf > /dev/null << 'EOF'
4
[Resolve]
5
DNS=127.0.0.1
6
Domains=~invinsense
7
EOF
8

9
# Restart resolved
10
sudo systemctl restart systemd-resolved

8. Verifying Service Operation#

Let’s verify that both services are running properly:

1
# Check CoreDNS service status
2
sudo systemctl status container-coredns
3

4
# Check CoreDNS logs
5
podman logs coredns
6

7
# Check StepCA service status
8
sudo systemctl status container-stepca
9

10
# Check StepCA logs
11
podman logs stepca

9. Testing DNS Resolution#

Test your CoreDNS configuration by resolving both internal and external domains:

1
# Test internal domain resolution
2
dig @localhost ca.invinsense
3

4
# Test external domain resolution
5
dig @localhost google.com

10. Setting Up StepCA Client#

Install and configure the Step CLI tool to interact with your certificate authority:

1
# Install step CLI if not already installed
2
sudo rpm-ostree install step-cli
3

4
# Get CA fingerprint
5
CA_FINGERPRINT=$(podman exec stepca step certificate fingerprint /home/step/certs/root_ca.crt)
6

7
# Bootstrap with CA
8
step ca bootstrap --ca-url https://ca.invinsense --fingerprint $CA_FINGERPRINT
9

10
# Request a test certificate
11
step ca certificate test.invinsense test.crt test.key

Maintenance and Troubleshooting#

Viewing Service Logs#

Monitor your services through systemd’s journaling system:

1
# View CoreDNS logs
2
journalctl -u container-coredns -f
3

4
# View StepCA logs
5
journalctl -u container-stepca -f

Restarting Services#

If you need to restart the services:

1
# Restart CoreDNS
2
sudo systemctl restart container-coredns
3

4
# Restart StepCA
5
sudo systemctl restart container-stepca

Updating Container Images#

Keep your services updated with the latest container images:

1
# Pull new CoreDNS image
2
podman pull docker.io/coredns/coredns:latest
3

4
# Pull new StepCA image
5
podman pull docker.io/smallstep/step-ca:latest
6

7
# Restart services to use new images
8
sudo systemctl restart container-coredns
9
sudo systemctl restart container-stepca

Backing Up Configuration#

Regularly back up your configuration files:

1
# Create a comprehensive backup
2
sudo tar -czf coreos-dns-ca-backup-$(date +%Y%m%d).tar.gz \
3
    /etc/containers/systemd \
4
    /etc/containers/coredns \
5
    /etc/containers/stepca.env
6

7
# For a more complete backup that includes certificates
8
podman exec stepca tar -cf - -C /home/step . | sudo tee stepca-data-backup-$(date +%Y%m%d).tar > /dev/null

Common Issues and Solutions#

DNS Resolution Not Working#

If DNS resolution isn’t working:

1
# Check if CoreDNS is running
2
sudo systemctl status container-coredns
3

4
# Verify DNS configuration
5
cat /etc/systemd/resolved.conf.d/dns.conf
6

7
# Test direct DNS queries
8
dig @127.0.0.1 ca.invinsense
9

10
# Check CoreDNS logs for errors
11
journalctl -u container-coredns | grep -i error

Certificate Issuance Problems#

If you’re having trouble issuing certificates:

1
# Check if StepCA is running
2
sudo systemctl status container-stepca
3

4
# Verify CA connectivity
5
curl -k https://ca.invinsense/health
6

7
# Check StepCA logs for errors
8
journalctl -u container-stepca | grep -i error
9

10
# Re-bootstrap the CA client
11
step ca bootstrap --ca-url https://ca.invinsense --fingerprint $CA_FINGERPRINT

For issues with the containers themselves:

1
# Check container status
2
podman ps -a
3

4
# Inspect container details
5
podman inspect coredns
6
podman inspect stepca
7

8
# Check for network conflicts
9
sudo ss -tulpn | grep '53\|443'

Advanced Configuration#

Persistent Storage for StepCA#

The default configuration uses tmpfs for StepCA data, which means certificates are lost on reboot. For a production environment, you’ll want persistent storage:

1
# Create a persistent directory for StepCA data
2
sudo mkdir -p /var/lib/stepca
3
sudo chown 1000:1000 /var/lib/stepca
4

5
# Update the StepCA container configuration
6
sudo tee /etc/containers/systemd/stepca.container > /dev/null << 'EOF'
7
[Unit]
8
Description=StepCA Certificate Authority
9
Documentation=https://smallstep.com/docs/step-ca
10
After=network-online.target
11
Wants=network-online.target
12

13
[Container]
14
Image=docker.io/smallstep/step-ca:0.24.0
15
Network=host
16
EnvironmentFile=/etc/containers/stepca.env
17
Environment=STEPCA_INIT_NAME=Invinsense CA
18
Environment=STEPCA_INIT_DNS_NAMES=ca.invinsense
19
Environment=STEPCA_INIT_ADDRESS=:443
20
Volume=/var/lib/stepca:/home/step:z
21

22
[Service]
23
Restart=always
24
TimeoutStartSec=60
25
TimeoutStopSec=30
26

27
[Install]
28
WantedBy=multi-user.target
29
EOF
30

31
# Reload systemd and restart StepCA
32
sudo systemctl daemon-reload
33
sudo systemctl restart container-stepca

Adding More DNS Zones#

You can add more DNS zones to CoreDNS by updating the Corefile and adding zone files:

1
# Add a new zone to the Corefile
2
sudo tee -a /etc/containers/coredns/Corefile > /dev/null << 'EOF'
3

4
example.com:53 {
5
    file /etc/coredns/example.com.db
6
    log
7
    errors
8
}
9
EOF
10

11
# Create the new zone file
12
sudo tee /etc/containers/coredns/example.com.db > /dev/null << 'EOF'
13
$ORIGIN example.com.
14
@       3600 IN SOA  ns1.example.com. admin.example.com. (
15
        2024062201 ; serial
16
        7200       ; refresh
17
        3600       ; retry
18
        1209600    ; expire
19
        3600       ; minimum
20
)
21

22
@       3600 IN NS  ns1.example.com.
23

24
ns1     IN A     127.0.0.1
25
www     IN A     192.168.1.100
26
app     IN A     192.168.1.101
27
EOF
28

29
# Restart CoreDNS to apply changes
30
sudo systemctl restart container-coredns

Configuring Additional StepCA Provisioners#

By default, StepCA creates an admin provisioner for certificate issuance. You can add more provisioners:

1
# Add a new ACME provisioner for Let's Encrypt-compatible clients
2
podman exec -it stepca step ca provisioner add acme --type ACME
3

4
# Add a new JWK provisioner for service authentication
5
podman exec -it stepca step ca provisioner add service --type JWK
6

7
# List all provisioners
8
podman exec -it stepca step ca provisioner list

Setting Up Automatic Certificate Renewal#

Configure automatic renewal for certificates:

1
# Create a renewal script
2
cat << 'EOF' > renew-cert.sh
3
#!/bin/bash
4
step ca renew --force certificate.crt certificate.key
5
chmod 600 certificate.key
6
EOF
7
chmod +x renew-cert.sh
8

9
# Add to user's crontab (runs daily at 2am)
10
(crontab -l 2>/dev/null; echo "0 2 * * * $PWD/renew-cert.sh") | crontab -

Security Considerations#

Securing DNS Traffic#

For production environments, consider securing DNS traffic:

1
# Update CoreDNS configuration to enable DNS-over-TLS
2
sudo tee /etc/containers/coredns/Corefile > /dev/null << 'EOF'
3
.:53 {
4
    forward . 8.8.8.8
5
    log
6
    errors
7
}
8

9
.:853 {
10
    tls /etc/coredns/cert.pem /etc/coredns/key.pem
11
    forward . 8.8.8.8
12
    log
13
    errors
14
}
15

16
invinsense:53 {
17
    file /etc/coredns/invinsense.db
18
    log
19
    errors
20
}
21

22
invinsense:853 {
23
    tls /etc/coredns/cert.pem /etc/coredns/key.pem
24
    file /etc/coredns/invinsense.db
25
    log
26
    errors
27
}
28
EOF
29

30
# Generate certificates for CoreDNS
31
step ca certificate coredns.invinsense /etc/containers/coredns/cert.pem /etc/containers/coredns/key.pem

Limiting Access to Services#

Restrict access to your DNS and CA services:

1
# Configure firewall rules
2
sudo firewall-cmd --permanent --add-service=dns
3
sudo firewall-cmd --permanent --add-port=443/tcp
4
sudo firewall-cmd --permanent --add-port=853/tcp
5
sudo firewall-cmd --reload

Regular Backup Schedule#

Implement a regular backup schedule:

1
# Create a backup script
2
cat << 'EOF' > /usr/local/bin/backup-infra.sh
3
#!/bin/bash
4
BACKUP_DIR="/var/backups/infra"
5
DATE=$(date +%Y%m%d)
6

7
# Create backup directory
8
mkdir -p $BACKUP_DIR
9

10
# Backup configurations
11
tar -czf $BACKUP_DIR/configs-$DATE.tar.gz \
12
    /etc/containers/systemd \
13
    /etc/containers/coredns \
14
    /etc/containers/stepca.env
15

16
# Backup StepCA data
17
podman exec stepca tar -cf - -C /home/step . | \
18
    tee $BACKUP_DIR/stepca-data-$DATE.tar > /dev/null
19

20
# Cleanup old backups (keep last 30 days)
21
find $BACKUP_DIR -type f -mtime +30 -delete
22
EOF
23

24
# Make executable
25
chmod +x /usr/local/bin/backup-infra.sh
26

27
# Add to crontab (weekly backup on Sunday at 1am)
28
(sudo crontab -l 2>/dev/null; echo "0 1 * * 0 /usr/local/bin/backup-infra.sh") | sudo crontab -

Conclusion#

In this guide, we’ve successfully deployed CoreDNS and StepCA on Fedora CoreOS using Podman Quadlet for systemd integration. This setup provides a robust foundation for DNS resolution and certificate management in your infrastructure, all while leveraging the benefits of containerization and the security-focused design of CoreOS.

The combination of these technologies offers several advantages:

Security-focused design with minimal attack surface
Atomic updates through CoreOS’s immutable design
Container isolation for services
Systemd integration for reliable service management
Lightweight deployment compared to full Kubernetes
Self-hosted DNS and PKI for complete control over infrastructure

By following the maintenance procedures and security considerations outlined in this guide, you can ensure that your DNS and certificate authority services remain reliable, secure, and properly managed over time.