Aqua Alert Details
Alert ID: AQUA-CONTAINER-ADMIN-1609-7842
Alert Time: 2024-02-13 14:30:22 EST
Severity: HIGH (85/100)
Source: Aqua Security Cloud Native Protection
Rule: “Unauthorized kubectl exec into Production Container”
MITRE ATT&CK: T1609 – Container Administration Command
Alert Details:
Detection: kubectl exec command executed in production environment
Cluster: prod-eks-cluster-01
Namespace: payment-processing
Pod: payment-api-v2-7d8f9c4d5-abcde
Container: api
Time: 14:28 EST
Command Details:
– User: jenkins-deploy (Service Account)
– Source IP: 10.0.45.78 (Internal – Jenkins server)
– Command: kubectl exec -it payment-api-v2-7d8f9c4d5-abcde -c api — /bin/bash
– Session Duration: 2 minutes (14:28-14:30)
Activities Inside Container:
– 14:28:20 – ls -la /app/config
– 14:28:35 – cat /app/config/secrets.env
– 14:28:50 – curl http://metadata.google.internal/computeMetadata/v1/ (GCP metadata)
– 14:29:10 – apt-get update && apt-get install -y nmap
– 14:29:30 – nmap -sS 10.0.0.0/8 -p 22,80,443,3306,5432
– 14:30:00 – exit
Anomaly Detection:
– Jenkins service account normally does NOT exec into pods
– Jenkins only used for CI/CD deployments (kubectl apply)
– Exec command unusual for automation
– Reconnaissance activities inside container
– Metadata service access attempt
Threat Intelligence:
– Pattern matches container breakout attempts
– Scanning internal network from compromised pod
– Possible credential harvesting from secrets.env
SOC Investigation Process
Step
Action
Tools Used
Findings
1. Alert Validation
Verify Aqua alert
Aqua Console
Confirmed unauthorized kubectl exec
2. Immediate Containment
Terminate pod, block exec
Kubernetes RBAC, Aqua
Pod terminated; service account permissions revoked
3. Source Investigation
Check Jenkins server
CrowdStrike, Jenkins Logs
Jenkins server compromised via vulnerable plugin
4. Impact Assessment
Review accessed secrets
Vault, Secrets Manager
secrets.env contained database credentials (rotated)
5. Network Scanning Check
Check for lateral movement
VPC Flow Logs, Calico
Scans detected but blocked by network policies
6. Remediation
Patch Jenkins, rotate secrets
DevOps Team
Jenkins plugin updated; all secrets rotated
Jira Incident Report
Ticket: SOC-2024-067
Summary: T1609 – Unauthorized Container Administration Command in Production
Status: RESOLVED
Resolution: MALICIOUS – Container Compromised
Priority: P1 – CRITICAL
Labels: T1609, container-admin, kubernetes, aqua, jenkins-compromise
Components: Container-Security, Cloud-Native
INCIDENT ANALYSIS REPORT
1. Initial Context:
Detection Source: Aqua Security Cloud Native Protection.
Alert: “Unauthorized kubectl exec into Production Container”.
Cluster: prod-eks-cluster-01, namespace payment-processing.
Time: 2024-02-13 14:30 EST.
Technique: MITRE ATT&CK T1609 – Container Administration Command.
2. Technical Analysis:
Attack Chain:
1. Jenkins server (10.0.45.78) compromised via CVE-2024-1234 (RCE in Jenkins plugin)
2. Attacker used Jenkins service account credentials (jenkins-deploy)
3. kubectl exec into payment-api container
4. Inside container: recon, metadata access, network scanning
5. Attempted lateral movement (blocked by network policies)
Container Activities:
Accessed secrets.env (database credentials, API keys)
Queried GCP metadata service (attempted to get instance service account)
Installed nmap and scanned internal network
Scanned targets: database servers, other pods, internal services
Jenkins Compromise:
Vulnerability: CVE-2024-1234 (CVSS 9.8) in “Pipeline: Declarative” plugin
Attacker gained remote code execution on Jenkins master
Stole service account credentials from environment variables
Network Policies:
Calico network policies blocked scanning traffic
No successful connections to other pods/services
Metadata service access blocked by pod identity restrictions
3. Investigation Findings:
Timeline:
14:25 – Jenkins compromise via CVE-2024-1234
14:26 – Attacker enumerates kubeconfig, finds service account
14:27 – kubectl exec into payment-api pod
14:28-14:30 – Reconnaissance activities
14:30 – Aqua alert triggers
14:31 – Pod terminated
14:32 – Service account permissions revoked
Data Exposed:
Database credentials for payment-processing (rotated)
No customer data accessed
No persistent backdoor established
Indicators of Compromise (IoCs):
Network:
– Jenkins IP: 10.0.45.78 (compromised)
– Scans: to 10.0.0.0/8 ports 22,80,443,3306,5432
Container:
– Pod: payment-api-v2-7d8f9c4d5-abcde (terminated)
– Commands: kubectl exec, nmap, curl to metadata
4. Containment Actions:
Immediate Actions:
Terminated compromised pod (Kubernetes).
Revoked jenkins-deploy service account permissions.
Isolated Jenkins server from network.
Blocked scanning IPs at firewall (already blocked by policies).
Credential Rotation:
Rotated all database credentials exposed in secrets.env.
Rotated Jenkins service account token.
Rotated any other secrets in same namespace.
Jenkins Remediation:
Patched vulnerable plugin.
Removed unnecessary service account credentials.
Implemented network segmentation for Jenkins.
5. Root Cause Analysis:
Primary Cause: Jenkins vulnerability allowing RCE.
Contributing Factors:
Jenkins service account had excessive permissions (exec).
Secrets stored in environment variables (not Vault).
No network policies blocking exec from Jenkins.
6. Business Impact:
Operational Impact: Payment API pod offline for 30 minutes.
Data Exposure: Database credentials exposed (rotated).
Financial Impact: None.
7. Remediation & Prevention:
Completed Actions:
Pod terminated.
Jenkins patched.
Secrets rotated.
Service account permissions reduced.
Technical Controls Enhanced:
Removed exec permissions from CI/CD service accounts.
Implemented OPA policies to block exec in production.
Moved secrets to Vault with dynamic rotation.
Deployed network policies to restrict pod-to-pod communication.
Enabled Aqua runtime protection for exec commands.
8. Conclusion:
Attackers exploited a Jenkins vulnerability to gain access to a production container via kubectl exec. They performed reconnaissance and attempted lateral movement, but were blocked by network policies and detected by Aqua. All exposed credentials were rotated.
Closure Rationale: Container terminated; Jenkins secured; secrets rotated; controls enhanced.
Analyst: [Walter White], SOC Analyst
Date: 2024-02-13 16:00 EST