UTMStack Correlation Engine - Complete Technical Documentation

Table of contents#

Executive Summary#

UTMStack’s proprietary correlation engine was built from scratch to analyze data before ingestion and maximize real-time correlation, resulting in extremely fast threat detection and response times. The engine processes over 128,000 correlation rules and operates as a unified SIEM/XDR platform with real-time threat intelligence integration.

Key Security Advantages:#

Pre-ingestion Analysis: Correlates threats before data storage, reducing attack dwell time
Real-time Processing: Immediate threat detection without indexing delays
Threat Intelligence Integration: Automated IOC correlation from multiple feeds
Memory-based Cache: High-speed rule processing for time-sensitive alerts
Defensive Architecture: Containerized microservices with strong authentication

Architecture Overview#

Core Components Deep Dive#

1. Rule Processing Engine#

The correlation engine supports two primary processing modes:

Cache-Based Processing

Purpose: High-speed analysis for time-sensitive rules (≤1 hour timeframe)
Storage: In-memory cache for rapid access
Iteration: Multi-stage rule evaluation with state persistence
Performance: Optimized for real-time threat detection

Search-Based Processing

Purpose: Complex analysis requiring historical data (>1 hour timeframe)
Storage: Direct OpenSearch/Elasticsearch queries
Flexibility: Complex JSON queries with advanced filtering
Use Case: Long-term pattern analysis and forensic investigation

1
graph LR
2
    A[Incoming Logs] --> B{Rule Processor}
3
    B --> C[Cache-Based Rules<br/>≤1 hour timeframe]
4
    B --> D[Search-Based Rules<br/>>1 hour timeframe]
5
    C --> E[In-Memory Cache]
6
    D --> F[OpenSearch Query]
7
    E --> G[Real-time Alerts]
8
    F --> G

2. Rule Structure and Components#

UTMStack correlation rules are written in YAML format with three main components: threat documentation, logic/frequency blocks, and alert information blocks.

Rule Metadata Structure

1
name: "Rule Name" # Alert identifier
2
severity: "High|Medium|Low" # Risk classification
3
description: "Attack description" # Detailed explanation
4
solution: "Remediation steps" # Response guidance
5
category: "Attack category" # MITRE ATT&CK alignment
6
tactic: "Attack tactic" # Threat actor methodology
7
reference: ["URL1", "URL2"] # Additional resources
8
dataTypes: ["log_type"] # Applicable log sources
9
frequency: 30 # Check interval (seconds)

Processing Logic Blocks

Cache-Based Rules:

1
cache:
2
  - allOf: # AND logic - all conditions must match
3
      - field: "log.field.path"
4
        operator: "=="
5
        value: "expected_value"
6
    minCount: 5 # Minimum event threshold
7
    timeLapse: 300 # Time window (seconds)
8
    save: # Field preservation for next iteration
9
      - field: "source.ip"
10
        alias: "SourceIP"

Search-Based Rules:

1
search:
2
  - query: '{
3
      "size": 500,
4
      "query": {
5
        "bool": {
6
          "must": [{"match": {"field": "value"}}],
7
          "filter": [{"range": {"@timestamp": {"gte": "now-5m"}}}]
8
        }
9
      }
10
    }'
11
    minCount: 3
12
    save:
13
      - field: "destination.ip"
14
        alias: "DestinationIP"

3. Operator Types and Security Implications#

The correlation engine supports multiple comparison operators with specific security use cases:

Operator	Type	Security Use Case
==	Exact match	Specific value detection
!=	Not equal	Anomaly detection
< > <= >=	Numeric comparison	Threshold analysis
contain	Substring search	Pattern matching
not contain	Exclusion	Whitelist filtering
regexp	Regular expression	Complex pattern detection
in	List membership	Multiple value matching
not in	List exclusion	Blacklist implementation
exist	Field presence	Schema validation
not exist	Field absence	Missing data detection
in cidr	IP range matching	Network analysis

Security-Critical Operators:

in cidr: Network-based threat detection (lateral movement, reconnaissance)
regexp: Advanced pattern matching for obfuscated attacks
contains: Payload analysis and signature detection
exist/not exist: Field presence anomaly detection

4. Multi-Stage Correlation Processing#

1
sequenceDiagram
2
    participant Log as Log Source
3
    participant Engine as Correlation Engine
4
    participant Cache as Memory Cache
5
    participant Search as OpenSearch
6
    participant Alert as Alert System
7

8
    Log->>Engine: Raw Event
9
    Engine->>Engine: DataType Filter
10
    Engine->>Cache: Check Previous State
11
    Cache-->>Engine: Previous Iteration Data
12
    Engine->>Engine: Evaluate Conditions
13

14
    alt Cache-Based Rule
15
        Engine->>Cache: Store Current State
16
    else Search-Based Rule
17
        Engine->>Search: Historical Query
18
        Search-->>Engine: Query Results
19
    end
20

21
    Engine->>Engine: Check Thresholds
22
    Engine->>Alert: Generate Alert

Data Flow and Processing Pipeline#

1. Log Ingestion and Normalization#

UTMStack employs Logstash to parse logs from various sources like firewalls, AWS, Office 365, etc. These logs are then processed through input, filter, and output plugins and sent to the UTMStack correlation engine.

1
graph TD
2
    A[Log Sources] --> B[Logstash]
3
    B --> C[Input Plugins]
4
    C --> D[Filter Plugins]
5
    D --> E[Output Plugins]
6
    E --> F[Correlation Engine]
7
    F --> G[Alert Generation]

2. Real-time Correlation Process#

Phase 1: Event Classification

DataType filtering based on rule applicability
Field extraction and normalization
Timestamp validation and parsing

Phase 2: Rule Evaluation

Cache lookup for existing correlation state
Multi-condition evaluation (allOf/oneOf logic)
Threshold counting and time window validation

Phase 3: Alert Generation

Alias resolution for alert fields
GeoIP enrichment for network indicators
Severity assignment and categorization

3. Cache Management and Performance#

1
graph LR
2
    A[Event Stream] --> B{Cache Manager}
3
    B --> C[Active Rules Cache<br/>TTL: 1 hour]
4
    B --> D[Field State Cache<br/>TTL: Rule-specific]
5
    B --> E[Counter Cache<br/>TTL: Time window]
6
    C --> F[Rule Processor]
7
    D --> F
8
    E --> F
9
    F --> G[Alert Decision]

Threat Intelligence Integration#

1. IOC Correlation Engine#

All logs the system receives are aggregated and correlated for indicators of compromise (IOCs) using several open threat intelligence feeds. This feature is enabled by default, and there is no need for custom correlation rules or configurations.

1
graph TB
2
    A[Incoming Logs] --> B[IOC Extraction]
3
    B --> C{IOC Types}
4
    C --> D[IP Addresses]
5
    C --> E[Domains]
6
    C --> F[File Hashes]
7
    C --> G[URLs]
8

9
    D --> H[Threat Intel Feeds]
10
    E --> H
11
    F --> H
12
    G --> H
13

14
    H --> I{Match Found?}
15
    I -->|Yes| J[Generate Alert]
16
    I -->|No| K[Continue Processing]

2. Automated Threat Detection Rules#

The system includes built-in rules for:

IP-based Threats: Blacklisted IP correlation
Domain Intelligence: Malicious domain detection
Hash-based Detection: Known malware signatures
Behavioral Analysis: Anomalous activity patterns

Security Architecture and Threat Modeling#

1. Attack Surface Analysis#

1
graph TD
2
    A[Attack Surface] --> B[External Interfaces]
3
    A --> C[Internal Components]
4

5
    B --> D[Log Collection APIs]
6
    B --> E[Management Console]
7
    B --> F[Alert Delivery]
8

9
    C --> G[Correlation Engine]
10
    C --> H[Data Storage]
11
    C --> I[Processing Pipeline]
12

13
    D --> J[Input Validation]
14
    E --> K[Authentication]
15
    F --> L[Encryption]
16

17
    G --> M[Rule Injection Prevention]
18
    H --> N[Access Control]
19
    I --> O[Resource Limits]

2. Defensive Security Measures#

All data in transit between agents and UTMStack servers is encrypted using TLS. UTMStack services are isolated by containers and microservices with strong authentication. Connections to the UTMStack server are authenticated with a +24 characters unique key.

Key Security Features:

Transport Security: End-to-end TLS encryption
Authentication: 24+ character unique connection keys
Isolation: Containerized microservice architecture
Input Validation: Comprehensive log sanitization
Access Control: Role-based permission system

3. Threat Detection Capabilities#

Advanced Persistent Threat (APT) Detection:

Multi-stage attack correlation
Long-term behavioral analysis
Attribution through TTPs mapping

Insider Threat Detection:

Privilege escalation monitoring
Abnormal access pattern recognition
Data exfiltration indicators

Infrastructure Security:

Network reconnaissance detection
Lateral movement identification
Command and control communication

Performance and Scalability#

1. Processing Architecture#

1
graph TB
2
    A[Load Balancer] --> B[Correlation Engine Cluster]
3
    B --> C[Engine Node 1]
4
    B --> D[Engine Node 2]
5
    B --> E[Engine Node N]
6

7
    C --> F[Shared Cache Layer]
8
    D --> F
9
    E --> F
10

11
    F --> G[Redis Cluster]
12

13
    C --> H[OpenSearch Cluster]
14
    D --> H
15
    E --> H

2. Scalability Considerations#

Horizontal Scaling:

Distributed cache architecture
Multi-node rule processing
Load-balanced log ingestion

Performance Optimization:

In-memory correlation cache
Optimized rule evaluation order
Efficient field indexing

Rule Development and Management#

1. Custom Rule Creation Process#

1
graph LR
2
    A[Threat Research] --> B[Rule Design]
3
    B --> C[YAML Definition]
4
    C --> D[Testing]
5
    D --> E[Validation]
6
    E --> F[Deployment]
7
    F --> G[Monitoring]
8
    G --> H[Tuning]
9
    H --> B

2. Rule Categories and Use Cases#

Authentication & Access Control:

Failed login attempts correlation
Privilege escalation detection
Account compromise indicators

Network Security:

Port scanning identification
DDoS attack detection
Malicious traffic analysis

Endpoint Security:

Malware execution correlation
Process injection detection
Registry modification tracking

Data Protection:

Data exfiltration patterns
Unauthorized access attempts
Sensitive file monitoring

Integration and API Architecture#

1. External System Integration#

1
graph LR
2
    A[UTMStack Core] --> B[REST API]
3
    B --> C[SIEM Integration]
4
    B --> D[SOAR Platform]
5
    B --> E[Ticketing System]
6
    B --> F[Threat Intel Platform]
7

8
    C --> G[Splunk/QRadar]
9
    D --> H[Phantom/XSOAR]
10
    E --> I[ServiceNow/Jira]
11
    F --> J[MISP/ThreatConnect]

2. API Security and Authentication#

Authentication Methods:

API key-based authentication
JWT token validation
Role-based access control

Data Protection:

Request/response encryption
Rate limiting and throttling
Input sanitization and validation

Compliance and Regulatory Framework#

1. Supported Compliance Standards#

UTMStack automates compliance controls and evidence tracking for GDPR, GLBA, HIPAA, SOC 2 and CMMC, with specific reporting capabilities for:

HIPAA: Healthcare data protection monitoring
PCI-DSS: Payment card industry security
SOC 2: Service organization controls
GDPR: Data privacy regulation compliance
CMMC: Cybersecurity maturity model certification

2. Audit Trail and Evidence Collection#

1
graph TD
2
    A[Security Events] --> B[Correlation Engine]
3
    B --> C[Compliance Mapping]
4
    C --> D{Compliance Framework}
5

6
    D --> E[HIPAA Controls]
7
    D --> F[PCI-DSS Requirements]
8
    D --> G[GDPR Articles]
9
    D --> H[SOC 2 Criteria]
10

11
    E --> I[Evidence Repository]
12
    F --> I
13
    G --> I
14
    H --> I
15

16
    I --> J[Compliance Reports]
17
    I --> K[Audit Documentation]

Advanced Features and AI Integration#

1. Machine Learning Components#

The threat detection engine is composed of rule-based correlation systems, scanners, and AI-powered machine learning algorithms that enable the system to learn from its environment.

AI-Powered Capabilities:

Behavioral Analytics: Anomaly detection using baseline learning
Threat Hunting: Automated IOC discovery and correlation
False Positive Reduction: ML-based alert filtering
Predictive Analysis: Threat trend identification

2. Advanced Correlation Techniques#

1
# Example: Multi-Stage Attack Detection
2
name: "APT Campaign Detection"
3
severity: "Critical"
4
description: "Detects multi-stage APT campaign activity"
5

6
cache:
7
  # Stage 1: Initial Compromise
8
  - allOf:
9
      - field: "event.action"
10
        operator: "contain"
11
        value: "exploit"
12
    minCount: 1
13
    timeLapse: 3600
14
    save:
15
      - field: "source.ip"
16
        alias: "AttackerIP"
17

18
  # Stage 2: Lateral Movement
19
  - allOf:
20
      - field: "event.category"
21
        operator: "=="
22
        value: "network"
23
      - field: "source.ip"
24
        operator: "=="
25
        value: "$AttackerIP"
26
    minCount: 5
27
    timeLapse: 7200
28
    save:
29
      - field: "destination.ip"
30
        alias: "TargetSystems"
31

32
  # Stage 3: Data Exfiltration
33
  - allOf:
34
      - field: "network.bytes"
35
        operator: ">"
36
        value: 1000000
37
      - field: "source.ip"
38
        operator: "in"
39
        value: "$TargetSystems"
40
    minCount: 1
41
    timeLapse: 3600

Operational Procedures and Best Practices#

1. Deployment Strategy#

Security-by-Design Principles:

Minimal attack surface configuration
Defense-in-depth architecture
Continuous security validation
Threat model-driven development

2. Monitoring and Maintenance#

Performance Monitoring:

Rule processing metrics
Cache hit ratios
Alert generation rates
System resource utilization

Security Monitoring:

Failed authentication attempts
Unusual system behavior
Configuration changes
Access pattern anomalies

3. Incident Response Integration#

1
sequenceDiagram
2
    participant Alert as Alert System
3
    participant IR as IR Platform
4
    participant Analyst as Security Analyst
5
    participant Tools as Security Tools
6

7
    Alert->>IR: High Severity Alert
8
    IR->>Analyst: Notification
9
    Analyst->>IR: Acknowledge
10
    IR->>Tools: Automated Enrichment
11
    Tools-->>IR: Context Data
12
    IR->>Analyst: Enhanced Alert
13
    Analyst->>IR: Investigation Actions
14
    IR->>Tools: Response Execution
15
    Tools-->>IR: Action Results
16
    IR->>Alert: Update Status

Conclusion and Security Recommendations#

The UTMStack correlation engine represents a comprehensive approach to real-time threat detection and response. Its pre-ingestion analysis capability, combined with extensive rule coverage and threat intelligence integration, provides significant security advantages for XDR/OXDR platform development.

Key Recommendations for Security Architects:#

Implement Cache-Based Rules for high-priority, time-sensitive threats
Leverage Threat Intelligence integration for automated IOC correlation
Design Multi-Stage Rules for complex attack pattern detection
Utilize Field Aliasing for consistent alert formatting and response automation
Monitor Performance Metrics to ensure optimal correlation engine performance
Implement Custom Rules aligned with organization-specific threat models

Advanced Security Considerations:#

Regular rule validation and false positive analysis
Continuous threat intelligence feed updates
Performance tuning for high-volume environments
Integration with automated response systems
Compliance reporting and audit trail maintenance

This documentation provides the foundation for understanding and implementing UTMStack’s correlation engine in enterprise security architectures, with particular emphasis on defensive programming practices and security-by-design principles essential for robust XDR/OXDR platform development.

UTMStack OpenSearch Connector - Architecture & Functionality#

Overview#

The UTMStack OpenSearch Connector acts as a crucial bridge that facilitates seamless communication between UTMStack’s SIEM/XDR platform and OpenSearch clusters, abstracting complex HTTP/JSON operations into intuitive Java methods and data structures.

Core Architecture Diagram#

1
graph TB
2
    A[UTMStack Core] --> B[OpenSearch Connector]
3
    B --> C{Connection Pool}
4
    C --> D[REST High Level Client]
5
    C --> E[REST Low Level Client]
6

7
    D --> F[Index Operations]
8
    D --> G[Search Operations]
9
    D --> H[Bulk Operations]
10

11
    E --> I[Direct HTTP Calls]
12
    E --> J[Custom Endpoints]
13

14
    F --> K[OpenSearch Cluster]
15
    G --> K
16
    H --> K
17
    I --> K
18
    J --> K

Detailed Component Interaction Flow#

1
sequenceDiagram
2
    participant App as UTMStack Application
3
    participant Conn as OpenSearch Connector
4
    participant Pool as Connection Pool
5
    participant Client as REST Client
6
    participant OS as OpenSearch Cluster
7

8
    App->>Conn: Initialize Connection
9
    Conn->>Pool: Create Connection Pool
10
    Pool->>Client: Initialize REST Clients
11
    Client->>OS: Establish Connection
12
    OS-->>Client: Connection Confirmed
13

14
    App->>Conn: Index Security Event
15
    Conn->>Pool: Get Available Connection
16
    Pool->>Client: Execute Index Request
17
    Client->>OS: HTTP PUT Request
18
    OS-->>Client: Index Response
19
    Client-->>Conn: Process Response
20
    Conn-->>App: Return Result

Key Functionality Breakdown#

1
graph LR
2
    A[OpenSearch Connector] --> B[Connection Management]
3
    A --> C[Index Operations]
4
    A --> D[Search Operations]
5
    A --> E[Bulk Operations]
6
    A --> F[Cluster Operations]
7

8
    B --> B1[Connection Pooling]
9
    B --> B2[Load Balancing]
10
    B --> B3[Failover Handling]
11

12
    C --> C1[Create Index]
13
    C --> C2[Update Mappings]
14
    C --> C3[Delete Index]
15

16
    D --> D1[Query DSL]
17
    D --> D2[Aggregations]
18
    D --> D3[Scroll API]
19

20
    E --> E1[Bulk Index]
21
    E --> E2[Bulk Update]
22
    E --> E3[Bulk Delete]
23

24
    F --> F1[Health Checks]
25
    F --> F2[Node Stats]
26
    F --> F3[Cluster Settings]

Security Operations Integration#

1
graph TD
2
    A[Security Events] --> B[OpenSearch Connector]
3
    B --> C{Event Type}
4

5
    C --> D[Alerts]
6
    C --> E[Raw Logs]
7
    C --> F[Correlation Results]
8
    C --> G[Threat Intel]
9

10
    D --> H[alerts-* indices]
11
    E --> I[logs-* indices]
12
    F --> J[correlation-* indices]
13
    G --> K[threat-* indices]
14

15
    H --> L[Time-based Rotation]
16
    I --> L
17
    J --> L
18
    K --> L

Data Flow & Processing Pipeline#

1
flowchart LR
2
    A[Log Sources] --> B[UTMStack Ingestion]
3
    B --> C[Normalization]
4
    C --> D[Enrichment]
5
    D --> E[OpenSearch Connector]
6

7
    E --> F{Indexing Strategy}
8
    F --> G[Real-time Index]
9
    F --> H[Batch Index]
10
    F --> I[Archive Index]
11

12
    G --> J[Hot Storage]
13
    H --> K[Warm Storage]
14
    I --> L[Cold Storage]
15

16
    J --> M[Search & Analytics]
17
    K --> M
18
    L --> M

Technical Implementation Details#

1
// Connector initialization example
2
public class UTMStackOpenSearchConnector {
3
    private final RestHighLevelClient client;
4
    private final ConnectionPool pool;
5
    private final RetryPolicy retryPolicy;
6

7
    public UTMStackOpenSearchConnector(OpenSearchConfig config) {
8
        this.pool = new ConnectionPool(config.getMaxConnections());
9
        this.client = buildClient(config);
10
        this.retryPolicy = new ExponentialBackoffRetry(3, 1000);
11
    }
12

13
    // Index security event with automatic retry
14
    public IndexResponse indexSecurityEvent(SecurityEvent event) {
15
        return retryPolicy.execute(() -> {
16
            IndexRequest request = new IndexRequest("security-events")
17
                .id(event.getId())
18
                .source(event.toJson(), XContentType.JSON);
19

20
            return client.index(request, RequestOptions.DEFAULT);
21
        });
22
    }
23

24
    // Bulk index for high-throughput scenarios
25
    public BulkResponse bulkIndexEvents(List<SecurityEvent> events) {
26
        BulkRequest bulkRequest = new BulkRequest();
27

28
        events.forEach(event -> {
29
            bulkRequest.add(new IndexRequest("security-events")
30
                .id(event.getId())
31
                .source(event.toJson(), XContentType.JSON));
32
        });
33

34
        return client.bulk(bulkRequest, RequestOptions.DEFAULT);
35
    }
36
}

Performance & Scalability Features#

1
graph TB
2
    A[Performance Features] --> B[Connection Pooling]
3
    A --> C[Bulk Operations]
4
    A --> D[Async Processing]
5
    A --> E[Query Optimization]
6

7
    B --> B1[Thread-safe Pools]
8
    B --> B2[Connection Reuse]
9
    B --> B3[Keep-alive Settings]
10

11
    C --> C1[Configurable Batch Size]
12
    C --> C2[Automatic Flushing]
13
    C --> C3[Error Handling]
14

15
    D --> D1[Non-blocking I/O]
16
    D --> D2[Future-based API]
17
    D --> D3[Callback Support]
18

19
    E --> E1[Query Caching]
20
    E --> E2[Field Selection]
21
    E --> E3[Aggregation Pipeline]

Security & Authentication Flow#

1
sequenceDiagram
2
    participant Connector as OpenSearch Connector
3
    participant Auth as Authentication Layer
4
    participant TLS as TLS Handler
5
    participant OS as OpenSearch
6

7
    Connector->>Auth: Initialize with Credentials
8
    Auth->>Auth: Load Certificates
9
    Auth->>TLS: Setup TLS Context
10

11
    Connector->>TLS: Secure Connection Request
12
    TLS->>OS: TLS Handshake
13
    OS-->>TLS: Certificate Validation
14
    TLS-->>Connector: Secure Channel Established
15

16
    Connector->>Auth: Generate Auth Headers
17
    Auth-->>Connector: Bearer Token / Basic Auth
18
    Connector->>OS: Authenticated Request
19
    OS-->>Connector: Authorized Response

Key Benefits & Security Advantages#

🔒 Security Benefits#

Encryption in Transit: All communications use TLS/SSL
Authentication Integration: Support for multiple auth methods
Role-based Access Control: Fine-grained permission system
Audit Trail: Complete operation logging for compliance

⚡ Performance Benefits#

Connection Pooling: Efficient resource utilization
Bulk Operations: High-throughput data processing
Async Processing: Non-blocking operations
Query Optimization: Intelligent query building

🛡️ Reliability Features#

Circuit Breaker Pattern: Prevents cascade failures
Automatic Retry Logic: Handles transient failures
Health Monitoring: Continuous cluster health checks
Failover Support: Seamless cluster failover

📊 SIEM-Specific Advantages#

Real-time Indexing: Immediate log availability for correlation
Time-based Indices: Efficient log retention and archival
Complex Query Support: Advanced threat hunting capabilities
Aggregation Processing: Statistical analysis for anomaly detection

Integration Points in UTMStack#

The OpenSearch Connector serves as the foundation for UTMStack’s core security operations:

Log Ingestion Pipeline: Real-time indexing of security events
Correlation Engine: Historical data queries for pattern matching
Alert Storage: Persistent alert and incident management
Compliance Reporting: Long-term data retention and reporting
Threat Intelligence: IOC storage and correlation
Dashboard Analytics: Real-time metrics and visualization
Forensic Analysis: Historical data investigation and analysis

This connector is essential for UTMStack’s XDR capabilities, enabling seamless integration between the platform’s security logic and the underlying search and analytics engine.