Aqua Alert Details
Alert ID: AQUA-CONTAINER-ADMIN-1609-7842
Alert Time: 2024-02-13 14:30:22 EST
Severity: HIGH (85/100)
Source: Aqua Security Cloud Native Protection
Rule: “Unauthorized kubectl exec into Production Container”
MITRE ATT&CK: T1609 – Container Administration Command
Alert Details:
Detection: kubectl exec command executed in production environment
Cluster: prod-eks-cluster-01
Namespace: payment-processing
Pod: payment-api-v2-7d8f9c4d5-abcde
Container: api
Time: 14:28 EST
Command Details:
– User: jenkins-deploy (Service Account)
– Source IP: 10.0.45.78 (Internal – Jenkins server)
– Command: kubectl exec -it payment-api-v2-7d8f9c4d5-abcde -c api — /bin/bash
– Session Duration: 2 minutes (14:28-14:30)
Activities Inside Container:
– 14:28:20 – ls -la /app/config
– 14:28:35 – cat /app/config/secrets.env
– 14:28:50 – curl http://metadata.google.internal/computeMetadata/v1/ (GCP metadata)
– 14:29:10 – apt-get update && apt-get install -y nmap
– 14:29:30 – nmap -sS 10.0.0.0/8 -p 22,80,443,3306,5432
– 14:30:00 – exit
Anomaly Detection:
– Jenkins service account normally does NOT exec into pods
– Jenkins only used for CI/CD deployments (kubectl apply)
– Exec command unusual for automation
– Reconnaissance activities inside container
– Metadata service access attempt
Threat Intelligence:
– Pattern matches container breakout attempts
– Scanning internal network from compromised pod
– Possible credential harvesting from secrets.env
SOC Investigation Process
| Step | Action | Tools Used | Findings |
|---|---|---|---|
| 1. Alert Validation | Verify Aqua alert | Aqua Console | Confirmed unauthorized kubectl exec |
| 2. Immediate Containment | Terminate pod, block exec | Kubernetes RBAC, Aqua | Pod terminated; service account permissions revoked |
| 3. Source Investigation | Check Jenkins server | CrowdStrike, Jenkins Logs | Jenkins server compromised via vulnerable plugin |
| 4. Impact Assessment | Review accessed secrets | Vault, Secrets Manager | secrets.env contained database credentials (rotated) |
| 5. Network Scanning Check | Check for lateral movement | VPC Flow Logs, Calico | Scans detected but blocked by network policies |
| 6. Remediation | Patch Jenkins, rotate secrets | DevOps Team | Jenkins plugin updated; all secrets rotated |
Jira Incident Report
Ticket: SOC-2024-067
Summary: T1609 – Unauthorized Container Administration Command in Production
Status: RESOLVED
Resolution: MALICIOUS – Container Compromised
Priority: P1 – CRITICAL
Labels: T1609, container-admin, kubernetes, aqua, jenkins-compromise
Components: Container-Security, Cloud-Native
INCIDENT ANALYSIS REPORT
1. Initial Context:
- Detection Source: Aqua Security Cloud Native Protection.
- Alert: “Unauthorized kubectl exec into Production Container”.
- Cluster: prod-eks-cluster-01, namespace payment-processing.
- Time: 2024-02-13 14:30 EST.
- Technique: MITRE ATT&CK T1609 – Container Administration Command.
2. Technical Analysis:
- Attack Chain:
1. Jenkins server (10.0.45.78) compromised via CVE-2024-1234 (RCE in Jenkins plugin)
2. Attacker used Jenkins service account credentials (jenkins-deploy)
3. kubectl exec into payment-api container
4. Inside container: recon, metadata access, network scanning
5. Attempted lateral movement (blocked by network policies)
- Container Activities:
- Accessed secrets.env (database credentials, API keys)
- Queried GCP metadata service (attempted to get instance service account)
- Installed nmap and scanned internal network
- Scanned targets: database servers, other pods, internal services
- Jenkins Compromise:
- Vulnerability: CVE-2024-1234 (CVSS 9.8) in “Pipeline: Declarative” plugin
- Attacker gained remote code execution on Jenkins master
- Stole service account credentials from environment variables
- Network Policies:
- Calico network policies blocked scanning traffic
- No successful connections to other pods/services
- Metadata service access blocked by pod identity restrictions
3. Investigation Findings:
- Timeline:
14:25 – Jenkins compromise via CVE-2024-1234
14:26 – Attacker enumerates kubeconfig, finds service account
14:27 – kubectl exec into payment-api pod
14:28-14:30 – Reconnaissance activities
14:30 – Aqua alert triggers
14:31 – Pod terminated
14:32 – Service account permissions revoked
- Data Exposed:
- Database credentials for payment-processing (rotated)
- No customer data accessed
- No persistent backdoor established
- Indicators of Compromise (IoCs):
Network:
– Jenkins IP: 10.0.45.78 (compromised)
– Scans: to 10.0.0.0/8 ports 22,80,443,3306,5432
Container:
– Pod: payment-api-v2-7d8f9c4d5-abcde (terminated)
– Commands: kubectl exec, nmap, curl to metadata
4. Containment Actions:
- Immediate Actions:
- Terminated compromised pod (Kubernetes).
- Revoked jenkins-deploy service account permissions.
- Isolated Jenkins server from network.
- Blocked scanning IPs at firewall (already blocked by policies).
- Credential Rotation:
- Rotated all database credentials exposed in secrets.env.
- Rotated Jenkins service account token.
- Rotated any other secrets in same namespace.
- Jenkins Remediation:
- Patched vulnerable plugin.
- Removed unnecessary service account credentials.
- Implemented network segmentation for Jenkins.
5. Root Cause Analysis:
- Primary Cause: Jenkins vulnerability allowing RCE.
- Contributing Factors:
- Jenkins service account had excessive permissions (exec).
- Secrets stored in environment variables (not Vault).
- No network policies blocking exec from Jenkins.
6. Business Impact:
- Operational Impact: Payment API pod offline for 30 minutes.
- Data Exposure: Database credentials exposed (rotated).
- Financial Impact: None.
7. Remediation & Prevention:
Completed Actions:
Pod terminated.
Jenkins patched.
Secrets rotated.
Service account permissions reduced.
Technical Controls Enhanced:
Removed exec permissions from CI/CD service accounts.
Implemented OPA policies to block exec in production.
Moved secrets to Vault with dynamic rotation.
Deployed network policies to restrict pod-to-pod communication.
Enabled Aqua runtime protection for exec commands.
8. Conclusion:
Attackers exploited a Jenkins vulnerability to gain access to a production container via kubectl exec. They performed reconnaissance and attempted lateral movement, but were blocked by network policies and detected by Aqua. All exposed credentials were rotated.
Closure Rationale: Container terminated; Jenkins secured; secrets rotated; controls enhanced.
Analyst: [Walter White], SOC Analyst
Date: 2024-02-13 16:00 EST