Hirte: Deterministic Multi-Node Service Controller for Safety-Critical Systems

Table of Contents#

Introduction#

Hirte is a deterministic multi-node service controller designed specifically for highly-regulated industries with functional safety requirements. Unlike traditional orchestrators like Kubernetes, Hirte provides predictable behavior critical for safety systems while maintaining a lightweight footprint. It manages service states across multiple nodes by integrating with systemd via its D-Bus API and relaying D-Bus messages over TCP for multi-node support.

This guide provides a comprehensive overview of Hirte’s architecture, implementation, and practical deployment on Rocky Linux 9.

What is Hirte?#

Hirte (German for “shepherd”) is a minimalist service orchestrator that prioritizes:

Deterministic behavior: Critical for safety-certified systems
Lightweight architecture: Minimal resource overhead
Direct systemd integration: Leverages existing Linux infrastructure
Multi-node coordination: TCP-based D-Bus message relay
Container support: Native integration with Podman and Quadlet

Key Features#

Predictable State Management: Every action has a deterministic outcome
Minimal Dependencies: Built on standard Linux components
Safety-First Design: Suitable for FuSa (Functional Safety) environments
Simple API: Easy integration with existing tools
Container-Native: First-class support for containerized workloads

Architecture Overview#

Hirte’s architecture is elegantly simple yet powerful:

1
graph TB
2
    subgraph "Control Node"
3
        CLI[hirtectl CLI]
4
        HM[Hirte Manager<br/>Port 2020]
5
        DBUS1[D-Bus API]
6
        SD1[systemd]
7
        AGENT1[Hirte Agent]
8
    end
9

10
    subgraph "Worker Node 1"
11
        AGENT2[Hirte Agent]
12
        DBUS2[D-Bus API]
13
        SD2[systemd]
14
        SVC1[Services/<br/>Containers]
15
    end
16

17
    subgraph "Worker Node 2"
18
        AGENT3[Hirte Agent]
19
        DBUS3[D-Bus API]
20
        SD3[systemd]
21
        SVC2[Services/<br/>Containers]
22
    end
23

24
    CLI --> HM
25
    HM --> AGENT1
26
    AGENT1 --> DBUS1
27
    DBUS1 --> SD1
28

29
    HM -.TCP 2020.-> AGENT2
30
    AGENT2 --> DBUS2
31
    DBUS2 --> SD2
32
    SD2 --> SVC1
33

34
    HM -.TCP 2020.-> AGENT3
35
    AGENT3 --> DBUS3
36
    DBUS3 --> SD3
37
    SD3 --> SVC2
38

39
    style HM fill:#4a90e2,color:#fff
40
    style SD1 fill:#50e3c2,color:#000
41
    style SD2 fill:#50e3c2,color:#000
42
    style SD3 fill:#50e3c2,color:#000

Implementation Workflow#

The implementation process follows a structured approach:

1
sequenceDiagram
2
    participant Admin
3
    participant hirtectl
4
    participant Manager
5
    participant Agent
6
    participant DBus
7
    participant systemd
8
    participant Service
9

10
    Admin->>hirtectl: start node1 nginx.service
11
    hirtectl->>Manager: Service start request
12
    Manager->>Manager: Validate request
13
    Manager->>Agent: Relay D-Bus command
14
    Agent->>DBus: D-Bus method call
15
    DBus->>systemd: Start unit
16
    systemd->>Service: Execute service
17
    Service-->>systemd: Running status
18
    systemd-->>DBus: Success response
19
    DBus-->>Agent: Method response
20
    Agent-->>Manager: Operation result
21
    Manager-->>hirtectl: Success confirmation
22
    hirtectl-->>Admin: Service started

Container Integration with Podman#

Hirte seamlessly integrates with Podman through Quadlet:

1
graph TB
2
    subgraph "Container Management Flow"
3
        Q[Quadlet Files<br/>.container/.pod]
4
        SG[systemd-generator]
5
        SU[systemd Units]
6
        H[Hirte Control]
7
        P[Podman Runtime]
8
        C[Containers]
9
    end
10

11
    Q --> SG
12
    SG --> SU
13
    H --> SU
14
    SU --> P
15
    P --> C
16

17
    style Q fill:#f9a825,color:#000
18
    style H fill:#4a90e2,color:#fff
19
    style P fill:#7b1fa2,color:#fff

Hirte vs Kubernetes Comparison#

Understanding when to use Hirte versus Kubernetes:

1
graph LR
2
    subgraph "Hirte Characteristics"
3
        H1[Deterministic]
4
        H2[Lightweight<br/>~10MB]
5
        H3[Direct systemd]
6
        H4[Simple State]
7
        H5[FuSa Ready]
8
    end
9

10
    subgraph "Kubernetes Characteristics"
11
        K1[Declarative]
12
        K2[Heavy<br/>~1GB+]
13
        K3[Abstract Layers]
14
        K4[Complex State]
15
        K5[Cloud Native]
16
    end
17

18
    subgraph "Use Cases"
19
        U1[Safety Critical ✓]
20
        U2[Edge Computing ✓]
21
        U3[Regulated Industry ✓]
22
        U4[Large Scale ✗]
23
        U5[Multi-Cloud ✗]
24
    end
25

26
    H1 --> U1
27
    H2 --> U2
28
    H5 --> U3
29
    K4 --> U4
30
    K5 --> U5
31

32
    style H1 fill:#4caf50,color:#fff
33
    style H5 fill:#4caf50,color:#fff
34
    style K4 fill:#2196f3,color:#fff
35
    style K5 fill:#2196f3,color:#fff

Setup Instructions#

Prerequisites#

For this implementation, you’ll need:

Two Rocky Linux 9 servers (one primary, one agent node)
Network connectivity between nodes
Root or sudo access
Basic understanding of systemd

Step 1: Install Hirte#

Since Hirte isn’t in default repositories, we’ll use the COPR repository.

On the primary node:

1
# Enable COPR repository
2
sudo dnf install -y dnf-plugins-core
3
sudo dnf copr enable mperina/hirte-snapshot el9
4

5
# Install Hirte components
6
sudo dnf install -y hirte hirte-agent hirtectl
7

8
# Verify installation
9
hirtectl --version

On the agent node:

1
# Enable COPR repository
2
sudo dnf install -y dnf-plugins-core
3
sudo dnf copr enable mperina/hirte-snapshot el9
4

5
# Install only the agent
6
sudo dnf install -y hirte-agent

Step 2: Configure Hirte#

On the primary node (replace 192.168.1.10 with your primary node’s IP):

1
# Create Hirte manager configuration
2
sudo tee /etc/hirte/hirte.conf << EOF
3
[hirte]
4
ManagerPort=2020
5
AllowedNodeNames=primary,agent
6
LogLevel=info
7
LogTarget=journald
8
EOF
9

10
# Create agent configuration for local agent
11
sudo tee /etc/hirte/agent.conf << EOF
12
[hirte-agent]
13
NodeName=primary
14
ManagerHost=127.0.0.1
15
ManagerPort=2020
16
LogLevel=info
17
LogTarget=journald
18
HeartbeatInterval=5
19
EOF

On the agent node (replace 192.168.1.10 with your primary node’s IP):

1
# Create agent configuration
2
sudo tee /etc/hirte/agent.conf << EOF
3
[hirte-agent]
4
NodeName=agent
5
ManagerHost=192.168.1.10
6
ManagerPort=2020
7
LogLevel=info
8
LogTarget=journald
9
HeartbeatInterval=5
10
ReconnectInterval=10
11
EOF

Step 3: Configure Firewall#

On the primary node:

1
# Open Hirte manager port
2
sudo firewall-cmd --permanent --add-port=2020/tcp
3
sudo firewall-cmd --reload
4

5
# Verify firewall rules
6
sudo firewall-cmd --list-ports

Step 4: Start Services#

On the primary node:

1
# Start and enable services
2
sudo systemctl start hirte hirte-agent
3
sudo systemctl enable hirte hirte-agent
4

5
# Verify services are running
6
sudo systemctl status hirte hirte-agent

On the agent node:

1
# Start and enable agent
2
sudo systemctl start hirte-agent
3
sudo systemctl enable hirte-agent
4

5
# Verify agent is running
6
sudo systemctl status hirte-agent

Step 5: Verify Connectivity#

On the primary node:

1
# Check logs for agent connection
2
sudo journalctl -lfu hirte
3

4
# Expected output should show:
5
# "Agent 'agent' connected from 192.168.1.20"
6

7
# List connected nodes
8
sudo hirtectl list-nodes
9

10
# Expected output:
11
# NODE      STATE     LAST SEEN
12
# primary   online    now
13
# agent     online    2s ago

Testing Functionality#

Basic Service Control#

Let’s test controlling services across nodes:

1
# List all units on all nodes
2
sudo hirtectl list-units
3

4
# List units on specific node
5
sudo hirtectl list-units agent
6

7
# Install a test service on agent node
8
ssh agent-node "sudo dnf install -y httpd"
9

10
# Control the service from primary node
11
sudo hirtectl start agent httpd.service
12
sudo hirtectl status agent httpd.service
13
sudo hirtectl stop agent httpd.service
14
sudo hirtectl restart agent httpd.service
15

16
# Enable service at boot
17
sudo hirtectl enable agent httpd.service

Container Management with Podman#

First, install Podman on the agent node:

1
# On agent node
2
sudo dnf install -y podman
3

4
# Create a Quadlet container file
5
sudo mkdir -p /etc/containers/systemd
6
sudo tee /etc/containers/systemd/webserver.container << EOF
7
[Unit]
8
Description=Nginx Web Server
9
After=network-online.target
10

11
[Container]
12
Image=docker.io/nginx:latest
13
PublishPort=8080:80
14
Volume=/var/www:/usr/share/nginx/html:ro
15
Environment=NGINX_HOST=example.com
16

17
[Service]
18
Restart=always
19
TimeoutStartSec=900
20

21
[Install]
22
WantedBy=default.target
23
EOF
24

25
# Reload systemd to generate service
26
sudo systemctl daemon-reload

Now control the container from the primary node:

1
# Start container
2
sudo hirtectl start agent webserver.service
3

4
# Check container status
5
sudo hirtectl status agent webserver.service
6

7
# View container logs
8
ssh agent-node "sudo journalctl -u webserver.service"
9

10
# Stop container
11
sudo hirtectl stop agent webserver.service

Multi-Container Pod Example#

Create a pod with multiple containers:

1
# On agent node - Create pod file
2
sudo tee /etc/containers/systemd/webapp.pod << EOF
3
[Unit]
4
Description=Web Application Pod
5

6
[Pod]
7
PodName=webapp
8
Network=bridge
9
PublishPort=8080:80
10
PublishPort=9090:9090
11
EOF
12

13
# Create application container
14
sudo tee /etc/containers/systemd/webapp-app.container << EOF
15
[Unit]
16
Description=Web Application
17
After=webapp.pod
18

19
[Container]
20
Image=docker.io/myapp:latest
21
Pod=webapp
22
Environment=DB_HOST=localhost
23
Environment=DB_PORT=5432
24

25
[Service]
26
Restart=always
27

28
[Install]
29
WantedBy=default.target
30
EOF
31

32
# Create database container
33
sudo tee /etc/containers/systemd/webapp-db.container << EOF
34
[Unit]
35
Description=PostgreSQL Database
36
After=webapp.pod
37

38
[Container]
39
Image=docker.io/postgres:14
40
Pod=webapp
41
Environment=POSTGRES_PASSWORD=secret
42
Environment=POSTGRES_DB=webapp
43
Volume=webapp-db-data:/var/lib/postgresql/data
44

45
[Service]
46
Restart=always
47

48
[Install]
49
WantedBy=default.target
50
EOF
51

52
# Reload systemd
53
sudo systemctl daemon-reload

Control the pod from primary node:

1
# Start the entire pod
2
sudo hirtectl start agent webapp.service
3
sudo hirtectl start agent webapp-app.service
4
sudo hirtectl start agent webapp-db.service
5

6
# Check pod status
7
sudo hirtectl list-units agent | grep webapp

Advanced Features#

Service Dependencies#

Leverage systemd’s dependency model:

1
# Create dependent services on agent node
2
sudo tee /etc/systemd/system/app-backend.service << EOF
3
[Unit]
4
Description=Application Backend
5
After=network.target postgresql.service
6
Requires=postgresql.service
7

8
[Service]
9
Type=simple
10
ExecStart=/usr/bin/app-backend
11
Restart=on-failure
12
User=appuser
13
Group=appgroup
14

15
[Install]
16
WantedBy=multi-user.target
17
EOF
18

19
sudo tee /etc/systemd/system/app-frontend.service << EOF
20
[Unit]
21
Description=Application Frontend
22
After=app-backend.service
23
Requires=app-backend.service
24

25
[Service]
26
Type=simple
27
ExecStart=/usr/bin/app-frontend
28
Restart=on-failure
29

30
[Install]
31
WantedBy=multi-user.target
32
EOF
33

34
# Reload systemd
35
sudo systemctl daemon-reload
36

37
# Start frontend (will automatically start backend)
38
sudo hirtectl start agent app-frontend.service

Monitoring Integration#

Create a monitoring setup:

1
# Create monitoring script
2
sudo tee /usr/local/bin/hirte-monitor.sh << 'EOF'
3
#!/bin/bash
4

5
# Monitor all nodes
6
for node in $(hirtectl list-nodes | tail -n +2 | awk '{print $1}'); do
7
    echo "=== Node: $node ==="
8

9
    # Get failed units
10
    failed=$(hirtectl list-units $node | grep -c failed) || failed=0
11
    if [ $failed -gt 0 ]; then
12
        echo "WARNING: $failed failed units on $node"
13
        hirtectl list-units $node | grep failed
14
    fi
15

16
    # Check node connectivity
17
    last_seen=$(hirtectl list-nodes | grep $node | awk '{print $3, $4}')
18
    echo "Last seen: $last_seen"
19

20
    echo ""
21
done
22
EOF
23

24
sudo chmod +x /usr/local/bin/hirte-monitor.sh
25

26
# Create systemd timer for monitoring
27
sudo tee /etc/systemd/system/hirte-monitor.service << EOF
28
[Unit]
29
Description=Hirte Monitoring
30

31
[Service]
32
Type=oneshot
33
ExecStart=/usr/local/bin/hirte-monitor.sh
34
StandardOutput=journal
35
EOF
36

37
sudo tee /etc/systemd/system/hirte-monitor.timer << EOF
38
[Unit]
39
Description=Run Hirte Monitoring every 5 minutes
40

41
[Timer]
42
OnBootSec=5min
43
OnUnitActiveSec=5min
44

45
[Install]
46
WantedBy=timers.target
47
EOF
48

49
# Enable monitoring
50
sudo systemctl daemon-reload
51
sudo systemctl enable --now hirte-monitor.timer

Multi-Node Orchestration#

Create orchestration scripts for complex deployments:

1
# Orchestration script
2
sudo tee /usr/local/bin/deploy-stack.sh << 'EOF'
3
#!/bin/bash
4
set -e
5

6
echo "Deploying application stack..."
7

8
# Start database on primary node
9
echo "Starting database..."
10
hirtectl start primary postgresql.service
11
sleep 5
12

13
# Wait for database to be ready
14
until hirtectl exec primary "pg_isready -U postgres"; do
15
    echo "Waiting for database..."
16
    sleep 2
17
done
18

19
# Start backend services on agent nodes
20
echo "Starting backend services..."
21
hirtectl start agent app-backend.service
22
hirtectl start agent cache.service
23

24
# Wait for backends
25
sleep 10
26

27
# Start frontend services
28
echo "Starting frontend services..."
29
hirtectl start agent nginx.service
30
hirtectl start agent app-frontend.service
31

32
echo "Stack deployment complete!"
33

34
# Show status
35
hirtectl list-units primary | grep -E "postgresql|running"
36
hirtectl list-units agent | grep -E "app-|nginx|cache|running"
37
EOF
38

39
sudo chmod +x /usr/local/bin/deploy-stack.sh

Security Considerations#

TLS Encryption#

While Hirte doesn’t natively support TLS, you can add it using stunnel:

1
# Install stunnel
2
sudo dnf install -y stunnel
3

4
# On primary node - Create stunnel server config
5
sudo tee /etc/stunnel/hirte-server.conf << EOF
6
[hirte-manager]
7
accept = 0.0.0.0:2021
8
connect = 127.0.0.1:2020
9
cert = /etc/stunnel/hirte.pem
10
EOF
11

12
# On agent node - Create stunnel client config
13
sudo tee /etc/stunnel/hirte-client.conf << EOF
14
client = yes
15

16
[hirte-agent]
17
accept = 127.0.0.1:2020
18
connect = 192.168.1.10:2021
19
EOF
20

21
# Generate certificate (on primary)
22
sudo openssl req -new -x509 -days 365 -nodes \
23
    -out /etc/stunnel/hirte.pem \
24
    -keyout /etc/stunnel/hirte.pem \
25
    -subj "/C=US/ST=State/L=City/O=Organization/CN=hirte.local"
26

27
# Start stunnel services
28
sudo systemctl enable --now stunnel@hirte-server
29
sudo systemctl enable --now stunnel@hirte-client
30

31
# Update agent config to use local stunnel
32
sudo sed -i 's/ManagerHost=.*/ManagerHost=127.0.0.1/' /etc/hirte/agent.conf
33
sudo systemctl restart hirte-agent

Access Control#

Implement basic access control:

1
# Create hirte group
2
sudo groupadd hirte-operators
3

4
# Add users to group
5
sudo usermod -a -G hirte-operators operator1
6

7
# Restrict hirtectl access
8
sudo chown root:hirte-operators /usr/bin/hirtectl
9
sudo chmod 750 /usr/bin/hirtectl
10

11
# Create sudo rules for specific operations
12
sudo tee /etc/sudoers.d/hirte << EOF
13
# Allow hirte-operators to use hirtectl
14
%hirte-operators ALL=(root) NOPASSWD: /usr/bin/hirtectl list-units *
15
%hirte-operators ALL=(root) NOPASSWD: /usr/bin/hirtectl status * *
16
%hirte-operators ALL=(root) NOPASSWD: /usr/bin/hirtectl start * *.service
17
%hirte-operators ALL=(root) NOPASSWD: /usr/bin/hirtectl stop * *.service
18
%hirte-operators ALL=(root) NOPASSWD: /usr/bin/hirtectl restart * *.service
19
EOF

Audit Logging#

Enable comprehensive audit logging:

1
# Configure auditd rules
2
sudo tee -a /etc/audit/rules.d/hirte.rules << EOF
3
# Audit Hirte operations
4
-w /usr/bin/hirtectl -p x -k hirte_commands
5
-w /etc/hirte/ -p wa -k hirte_config
6
-w /var/log/hirte/ -p wa -k hirte_logs
7

8
# Audit systemd operations via Hirte
9
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/systemctl -k systemd_hirte
10
EOF
11

12
# Reload audit rules
13
sudo augenrules --load
14
sudo systemctl restart auditd
15

16
# Create log aggregation
17
sudo tee /usr/local/bin/hirte-audit-report.sh << 'EOF'
18
#!/bin/bash
19
echo "Hirte Audit Report - $(date)"
20
echo "=========================="
21
echo ""
22
echo "Recent Hirte Commands:"
23
ausearch -k hirte_commands -ts recent | aureport -x
24
echo ""
25
echo "Configuration Changes:"
26
ausearch -k hirte_config -ts recent | aureport -f
27
echo ""
28
echo "Service Operations:"
29
journalctl -u hirte -u hirte-agent --since "1 hour ago" | grep -E "start|stop|restart"
30
EOF
31

32
sudo chmod +x /usr/local/bin/hirte-audit-report.sh

Troubleshooting#

Common Issues and Solutions#

Connection Failures#

1
# Check if manager is listening
2
sudo ss -tlnp | grep 2020
3

4
# Test connectivity from agent
5
telnet 192.168.1.10 2020
6

7
# Check firewall on both nodes
8
sudo firewall-cmd --list-all
9

10
# Verify DNS resolution
11
ping -c 1 primary-node
12

13
# Check agent logs
14
sudo journalctl -u hirte-agent -f

Service Control Issues#

1
# Verify local systemd control works
2
sudo systemctl status httpd
3

4
# Check D-Bus connectivity
5
sudo busctl status
6

7
# Test D-Bus method calls
8
sudo busctl call org.freedesktop.systemd1 \
9
    /org/freedesktop/systemd1 \
10
    org.freedesktop.systemd1.Manager \
11
    GetUnit s "httpd.service"
12

13
# Check Hirte agent D-Bus permissions
14
sudo -u hirte-agent busctl status

Performance Issues#

1
# Monitor Hirte resource usage
2
sudo systemctl status hirte hirte-agent
3

4
# Check message queue
5
sudo ss -tnp | grep 2020
6

7
# Monitor D-Bus traffic
8
sudo busctl monitor org.freedesktop.systemd1
9

10
# Check system load
11
top -b -n 1 | head -20

Debug Mode#

Enable verbose logging for troubleshooting:

1
# Enable debug logging
2
sudo sed -i 's/LogLevel=.*/LogLevel=debug/' /etc/hirte/hirte.conf
3
sudo sed -i 's/LogLevel=.*/LogLevel=debug/' /etc/hirte/agent.conf
4

5
# Restart services
6
sudo systemctl restart hirte hirte-agent
7

8
# Watch debug logs
9
sudo journalctl -f -u hirte -u hirte-agent
10

11
# Disable debug when done
12
sudo sed -i 's/LogLevel=.*/LogLevel=info/' /etc/hirte/hirte.conf
13
sudo sed -i 's/LogLevel=.*/LogLevel=info/' /etc/hirte/agent.conf
14
sudo systemctl restart hirte hirte-agent

Best Practices#

1. Service Organization#

1
# Use consistent naming conventions
2
webapp-frontend.service
3
webapp-backend.service
4
webapp-database.service
5

6
# Group related services
7
infrastructure-*.service  # Infrastructure services
8
app-*.service           # Application services
9
monitor-*.service       # Monitoring services

2. Health Checks#

1
# Create health check script
2
sudo tee /usr/local/bin/check-hirte-health.sh << 'EOF'
3
#!/bin/bash
4

5
ERRORS=0
6

7
# Check Hirte services
8
for service in hirte hirte-agent; do
9
    if ! systemctl is-active --quiet $service; then
10
        echo "ERROR: $service is not running"
11
        ((ERRORS++))
12
    fi
13
done
14

15
# Check node connectivity
16
NODES=$(hirtectl list-nodes | tail -n +2 | wc -l)
17
if [ $NODES -lt 2 ]; then
18
    echo "ERROR: Expected at least 2 nodes, found $NODES"
19
    ((ERRORS++))
20
fi
21

22
# Check for failed units
23
FAILED=$(hirtectl list-units | grep -c failed || true)
24
if [ $FAILED -gt 0 ]; then
25
    echo "ERROR: $FAILED failed units detected"
26
    ((ERRORS++))
27
fi
28

29
exit $ERRORS
30
EOF
31

32
sudo chmod +x /usr/local/bin/check-hirte-health.sh
33

34
# Add to monitoring
35
echo "*/5 * * * * root /usr/local/bin/check-hirte-health.sh || logger -t hirte-health 'Health check failed'" | sudo tee -a /etc/crontab

3. Backup and Recovery#

1
# Backup Hirte configuration
2
sudo tee /usr/local/bin/backup-hirte.sh << 'EOF'
3
#!/bin/bash
4

5
BACKUP_DIR="/var/backups/hirte"
6
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
7

8
mkdir -p $BACKUP_DIR
9

10
# Backup configurations
11
tar -czf $BACKUP_DIR/hirte-config-$TIMESTAMP.tar.gz \
12
    /etc/hirte/ \
13
    /etc/systemd/system/*.service \
14
    /etc/containers/systemd/
15

16
# Backup node state
17
hirtectl list-nodes > $BACKUP_DIR/nodes-$TIMESTAMP.txt
18
hirtectl list-units > $BACKUP_DIR/units-$TIMESTAMP.txt
19

20
# Keep only last 7 days
21
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete
22
find $BACKUP_DIR -name "*.txt" -mtime +7 -delete
23

24
echo "Backup completed: $BACKUP_DIR/*-$TIMESTAMP.*"
25
EOF
26

27
sudo chmod +x /usr/local/bin/backup-hirte.sh
28

29
# Schedule daily backups
30
echo "0 2 * * * root /usr/local/bin/backup-hirte.sh" | sudo tee -a /etc/crontab

Production Deployment#

High Availability Setup#

For production environments, implement HA for the Hirte manager:

1
# Install keepalived on both manager nodes
2
sudo dnf install -y keepalived
3

4
# On primary manager
5
sudo tee /etc/keepalived/keepalived.conf << EOF
6
vrrp_instance HIRTE_VIP {
7
    state MASTER
8
    interface eth0
9
    virtual_router_id 51
10
    priority 100
11
    advert_int 1
12
    authentication {
13
        auth_type PASS
14
        auth_pass secret123
15
    }
16
    virtual_ipaddress {
17
        192.168.1.100/24
18
    }
19
    notify_master "/usr/local/bin/hirte-failover.sh master"
20
    notify_backup "/usr/local/bin/hirte-failover.sh backup"
21
}
22
EOF
23

24
# On backup manager
25
sudo tee /etc/keepalived/keepalived.conf << EOF
26
vrrp_instance HIRTE_VIP {
27
    state BACKUP
28
    interface eth0
29
    virtual_router_id 51
30
    priority 90
31
    advert_int 1
32
    authentication {
33
        auth_type PASS
34
        auth_pass secret123
35
    }
36
    virtual_ipaddress {
37
        192.168.1.100/24
38
    }
39
    notify_master "/usr/local/bin/hirte-failover.sh master"
40
    notify_backup "/usr/local/bin/hirte-failover.sh backup"
41
}
42
EOF
43

44
# Create failover script
45
sudo tee /usr/local/bin/hirte-failover.sh << 'EOF'
46
#!/bin/bash
47

48
STATE=$1
49

50
case $STATE in
51
    master)
52
        systemctl start hirte
53
        logger -t hirte-ha "Became master, starting Hirte manager"
54
        ;;
55
    backup)
56
        systemctl stop hirte
57
        logger -t hirte-ha "Became backup, stopping Hirte manager"
58
        ;;
59
esac
60
EOF
61

62
sudo chmod +x /usr/local/bin/hirte-failover.sh
63

64
# Start keepalived
65
sudo systemctl enable --now keepalived
66

67
# Configure agents to use VIP
68
sudo sed -i 's/ManagerHost=.*/ManagerHost=192.168.1.100/' /etc/hirte/agent.conf
69
sudo systemctl restart hirte-agent

Monitoring Dashboard#

Create a simple monitoring dashboard:

1
# Install dependencies
2
sudo dnf install -y python3 python3-flask python3-requests
3

4
# Create dashboard application
5
sudo tee /opt/hirte-dashboard.py << 'EOF'
6
#!/usr/bin/env python3
7

8
from flask import Flask, render_template_string
9
import subprocess
10
import json
11

12
app = Flask(__name__)
13

14
DASHBOARD_TEMPLATE = '''
15
<!DOCTYPE html>
16
<html>
17
<head>
18
    <title>Hirte Dashboard</title>
19
    <meta http-equiv="refresh" content="30">
20
    <style>
21
        body { font-family: Arial, sans-serif; margin: 20px; }
22
        table { border-collapse: collapse; width: 100%; margin: 20px 0; }
23
        th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
24
        th { background-color: #4CAF50; color: white; }
25
        .online { color: green; }
26
        .offline { color: red; }
27
        .running { background-color: #e8f5e9; }
28
        .failed { background-color: #ffebee; }
29
    </style>
30
</head>
31
<body>
32
    <h1>Hirte Service Controller Dashboard</h1>
33

34
    <h2>Nodes</h2>
35
    <table>
36
        <tr><th>Node</th><th>State</th><th>Last Seen</th></tr>
37
        {% for node in nodes %}
38
        <tr>
39
            <td>{{ node.name }}</td>
40
            <td class="{{ node.state }}">{{ node.state }}</td>
41
            <td>{{ node.last_seen }}</td>
42
        </tr>
43
        {% endfor %}
44
    </table>
45

46
    <h2>Services</h2>
47
    <table>
48
        <tr><th>Node</th><th>Service</th><th>State</th><th>Description</th></tr>
49
        {% for service in services %}
50
        <tr class="{{ service.state }}">
51
            <td>{{ service.node }}</td>
52
            <td>{{ service.name }}</td>
53
            <td>{{ service.state }}</td>
54
            <td>{{ service.description }}</td>
55
        </tr>
56
        {% endfor %}
57
    </table>
58

59
    <p>Last updated: {{ timestamp }}</p>
60
</body>
61
</html>
62
'''
63

64
@app.route('/')
65
def dashboard():
66
    from datetime import datetime
67

68
    # Get nodes
69
    nodes = []
70
    result = subprocess.run(['sudo', 'hirtectl', 'list-nodes'],
71
                          capture_output=True, text=True)
72
    if result.returncode == 0:
73
        lines = result.stdout.strip().split('\n')[1:]  # Skip header
74
        for line in lines:
75
            parts = line.split()
76
            if len(parts) >= 3:
77
                nodes.append({
78
                    'name': parts[0],
79
                    'state': parts[1],
80
                    'last_seen': ' '.join(parts[2:])
81
                })
82

83
    # Get services
84
    services = []
85
    for node in nodes:
86
        result = subprocess.run(['sudo', 'hirtectl', 'list-units', node['name']],
87
                              capture_output=True, text=True)
88
        if result.returncode == 0:
89
            lines = result.stdout.strip().split('\n')[1:]  # Skip header
90
            for line in lines:
91
                parts = line.split(None, 3)
92
                if len(parts) >= 3:
93
                    services.append({
94
                        'node': node['name'],
95
                        'name': parts[0],
96
                        'state': parts[1],
97
                        'description': parts[3] if len(parts) > 3 else ''
98
                    })
99

100
    return render_template_string(DASHBOARD_TEMPLATE,
101
                                nodes=nodes,
102
                                services=services,
103
                                timestamp=datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
104

105
if __name__ == '__main__':
106
    app.run(host='0.0.0.0', port=8080)
107
EOF
108

109
sudo chmod +x /opt/hirte-dashboard.py
110

111
# Create systemd service
112
sudo tee /etc/systemd/system/hirte-dashboard.service << EOF
113
[Unit]
114
Description=Hirte Dashboard
115
After=network.target hirte.service
116

117
[Service]
118
Type=simple
119
ExecStart=/opt/hirte-dashboard.py
120
Restart=always
121
User=nobody
122
Group=nobody
123

124
[Install]
125
WantedBy=multi-user.target
126
EOF
127

128
# Add sudo permissions for dashboard
129
echo "nobody ALL=(root) NOPASSWD: /usr/bin/hirtectl list-nodes, /usr/bin/hirtectl list-units *" | sudo tee -a /etc/sudoers.d/hirte-dashboard
130

131
# Start dashboard
132
sudo systemctl daemon-reload
133
sudo systemctl enable --now hirte-dashboard
134

135
# Open firewall port
136
sudo firewall-cmd --permanent --add-port=8080/tcp
137
sudo firewall-cmd --reload

Conclusion#

Hirte provides a unique solution for deterministic service orchestration across multiple nodes, making it ideal for:

Safety-critical systems requiring predictable behavior
Regulated environments where Kubernetes complexity is prohibitive
Edge computing scenarios with resource constraints
Legacy system integration leveraging existing systemd infrastructure
Container orchestration without the overhead of k8s

Key Advantages#

Deterministic Behavior: Every operation has predictable outcomes
Minimal Footprint: ~10MB memory usage vs gigabytes for k8s
Native Integration: Direct systemd control without abstraction layers
Simple Architecture: Easy to understand, audit, and validate
Container Support: First-class Podman/Quadlet integration

When to Use Hirte#

Choose Hirte when you need:

Functional safety compliance (ISO 26262, IEC 61508)
Predictable real-time behavior
Minimal resource overhead
Simple multi-node coordination
Direct systemd integration

When NOT to Use Hirte#

Consider alternatives when you need:

Large-scale deployments (1000+ nodes)
Complex scheduling algorithms
Multi-cloud portability
Extensive ecosystem of operators
Dynamic auto-scaling

Hirte fills a critical gap in the orchestration landscape, providing a safety-focused alternative to cloud-native solutions while maintaining the simplicity and reliability that regulated industries require.