04 - Patch Management and Remediation
Finding and prioritizing vulnerabilities is useless unless you actually fix them. Patch management is the operational process of deploying updates to software and systems.
Patch Deployment Strategies
Patching must balance security with system stability. Applying a patch blindly can break production applications.
- Testing Environments: Always deploy patches to a QA or Staging environment first. Run automated regression tests to ensure the application still functions.
- Phased Rollouts (Canary): Deploy the patch to a small subset of the production fleet. Monitor for errors or crashes before rolling out to the rest.
- Immutable Infrastructure: In containerized or cloud-native environments, do not patch live servers. Instead, update the base image (e.g., Dockerfile) and redeploy the entire fleet.
Zero-Day Emergency Patching Workflows
When a critical, actively exploited zero-day vulnerability (like Log4Shell) drops, standard SLA timelines go out the window.
Emergency Workflow:
- Identify Exposure: Instantly query asset inventory/scanners for the affected software.
- Assess Threat Intel: Check EPSS and CISA KEV. Determine if active exploitation is occurring.
- Emergency CAB (Change Advisory Board): Convene an emergency meeting to bypass standard change control windows.
- Deploy Mitigations: If a patch isn't ready, deploy Virtual Patching immediately.
- Deploy Patches: Roll out vendor patches as soon as they are validated.
Virtual Patching
What happens when a vendor hasn't released a patch yet, or a legacy system cannot be taken offline for an update?
Virtual Patching is the implementation of a security policy or enforcement rule to prevent the exploitation of a known vulnerability, without modifying the underlying source code or binary.
Mechanisms:
- Web Application Firewalls (WAF): Deploying rules to drop malicious payloads targeting a specific CVE (e.g., blocking
${jndi:strings to mitigate Log4j). - API Gateways: Rate limiting, schema validation, and blocking malicious headers.
- Intrusion Prevention Systems (IPS): Dropping network packets that match exploit signatures.
- Endpoint Controls: Disabling vulnerable services or restricting file execution (e.g., AppLocker).
Note: Virtual patching is a temporary mitigation, not a permanent fix. The underlying vulnerability still exists.
Exception Handling Policies
Sometimes, you cannot remediate a vulnerability. Common reasons include:
- The vendor no longer supports the software (End of Life).
- Applying the patch breaks a critical business application.
- The system is an isolated air-gapped lab where the vulnerability cannot be exploited.
In these cases, a Risk Exception must be filed.
- Time-bound: Exceptions should not be permanent. They should expire (e.g., 6 or 12 months) and trigger a review.
- Compensating Controls: To approve an exception, the team must prove that mitigating controls (like network segmentation or virtual patching) have reduced the risk.
- Executive Sign-off: High-risk exceptions must be signed off by a risk owner (e.g., VP or CISO) to ensure the business formally accepts the risk.