Monitoring, Observability & Capacity Planning Guide

1. Introduction and Scope

1.0 Normative Language (Mandatory Keywords)

The following keywords are used to express requirement strength in this standard:

• SHALL / MUST: Mandatory requirement. Non-compliance requires an approved exception.
• SHOULD: Recommended requirement. If not implemented, rationale should be documented.
• MAY: Optional requirement.
• MUST NOT / SHALL NOT: Prohibited requirement.

1.1 Purpose

The purpose of this standard is to define how the Government of Malawi monitors systems and applications (metrics, logs, traces, security events), how alerts are configured and handled, how dashboards and SLOs/SLIs are used, how demand is forecast and resources are sized, and how quotas and scaling policies prevent service degradation. It applies to infrastructure, applications, and security monitoring for government systems and includes the discipline of capacity planning and resource management.

1.2 Scope of Application

This standard applies to all government ministries and departments; state-owned enterprises and parastatals; local government authorities; and government contractors when they operate monitoring, observability, and capacity planning for government systems. It covers on-premises, government cloud, and vendor-managed systems where the government retains responsibility for monitoring and capacity decisions. It includes infrastructure, application, and security monitoring; capacity forecasting, sizing, scaling, and quotas; and log retention and analysis.

1.3 Definitions and Key Terms

Monitoring – The collection and use of metrics, logs, and traces to detect issues, support alerting, and enable investigation; includes infrastructure, application, and security monitoring.

Observability – The capability to infer the internal state of a system from its external outputs (metrics, logs, traces); supports rapid detection and root-cause analysis.

Metrics – Quantitative measurements (e.g. CPU, memory, latency, throughput, error rate) collected from systems and applications; stored with defined retention and used for alerts and dashboards.

SIEM (Security Information and Event Management) – A system that aggregates and correlates security-relevant logs and events; supports threat detection, alerting, and incident investigation; per Government Server Security Hardening.

SLO (Service Level Objective) – A target level of service (e.g. availability 99.9%, latency p99 under 500ms) for a critical service; breach is tracked and reviewed.

SLI (Service Level Indicator) – A measured indicator (e.g. uptime, latency) that reflects service level; used to evaluate SLO compliance.

Alert – A notification triggered when a metric or event exceeds a threshold or matches a rule; critical and high alerts handled per Incident Response timelines.

Log retention – The period for which logs are kept; minimum 90 days online, 7 years archived for audit-relevant logs per Server Hardening; aligned with Logging and Evidence Standards.

2. Core Objectives and Success Metrics

2.1 Primary Objectives

Coverage – 100% of critical and high-impact systems under monitoring within 18 months; metrics, logs, and (where applicable) traces collected and retained per policy.

Detection and response – Critical alerts acknowledged and acted upon per Incident Response SLA; security events ingested into SIEM and correlated; no critical alert unhandled beyond SLA.

SLOs and SLIs – Critical services have defined SLO and SLI; reviewed quarterly with System Owner; breach tracked and drives improvement.

Integrity and retention – Logs retained per policy; no critical gap; logs protected from tampering per Logging and Evidence Standards; personal data in logs minimised and protected per Data Protection Act.

Integration – Monitoring aligns with Incident Response (alert handling), Server Hardening (logging, SIEM), Backup and Recovery (log retention), and Capacity Planning (metrics for capacity decisions).

2.2 Key Performance Indicators (KPIs)

2.3 Minimum Baseline and Prohibitions

• Minimum controls (floor): Critical and high-impact systems SHALL be under monitoring (metrics, logs); security events SHALL be ingested into SIEM or equivalent. Critical alerts SHALL be acknowledged and acted upon per Incident Response SLA. Critical services SHALL have defined SLO and SLI; reviewed quarterly. Log retention SHALL meet policy (90 days online, 7 years archived for audit-relevant per Server Hardening). All exceptions SHALL be in the exception register with compensating controls and expiry.
• Prohibitions: MUST NOT leave critical or high-impact systems without monitoring without approved exception. MUST NOT grant exception without recording in exception register.

3. Governance and Compliance Framework

3.1 Governance Structure

Document Owner – The Ministry of Information and Communication Technology (MoICT), Department of eGovernment, owns this standard and is responsible for its maintenance, interpretation, and coordination with other government standards (including Incident Response, Server Hardening, and Logging and Evidence).

System Owner – Accountable for monitoring of their systems; approves SLOs and exception requests within their remit.

Security Officer – Validates that security monitoring and SIEM coverage meet this standard; receives security alerts; governs exceptions.

Operations or SRE – Configures and operates monitoring (metrics, logs, traces); maintains dashboards and alerts; handles alerts per Incident Response; supports Capacity Planning with trend data.

SIEM or security analyst – Manages SIEM correlation rules and security alerts; supports incident investigation and post-incident review.

3.2 Metrics

Metrics are collected from infrastructure (CPU, memory, disk, network), applications (throughput, latency, errors), and business-relevant counters. Metrics are stored in a monitoring system with defined retention. Critical metrics have alerts (threshold or anomaly) that trigger per Incident Response. Dashboards provide at-a-glance view for operations and management. Metrics support Capacity Planning (trends, usage).

3.3 Logging

Application, system, and security logs are collected. Log format and fields are standardised where feasible (e.g. timestamp, source, level, message). Logs are centralised (e.g. SIEM, log aggregation) for critical and high-impact systems per Government Server Security Hardening Standards. Logs are protected from tampering and access is restricted. Retention: minimum 90 days online, 7 years archived for audit-relevant logs per Server Hardening; other logs per policy. Logs containing personal data are minimised and protected per Data Protection Act.

3.4 Security Monitoring and SIEM

Security-relevant events (auth failures, privilege escalation, config change, malware alert, etc.) are ingested into SIEM or equivalent. Correlation rules and alerts are configured for known threats and anomalies. Alerts are routed to Security Officer and/or Incident Response; handling follows Malawi Government Incident Response Standards. SIEM and log data are used for incident investigation and post-incident review. Daily or continuous review of security logs is performed per Server Hardening.

Procedure: (1) Configure collection of security-relevant logs from critical and high-impact systems into SIEM or central logging; document source and retention. (2) Define and tune correlation rules and alerts; route critical and high alerts per Incident Response SLA. (3) Perform daily or continuous review of security logs; document and escalate anomalies. (4) Retain logs per policy (minimum 90 days online, 7 years archived for audit-relevant); retain alert and response evidence for audit.

3.5 SLOs and SLIs

Critical services have defined SLOs (e.g. availability 99.9%, latency p99 under 500ms). SLIs are measured from metrics or logs. Breach of SLO is tracked and reviewed; repeated breach triggers capacity or architecture improvement. SLOs are reviewed quarterly with System Owner.

3.6 Alerting and Response

Alerts are prioritised (critical, high, medium, low). Critical and high alerts are handled per Incident Response timelines. Alert fatigue is avoided by tuning thresholds and consolidating alerts. On-call or escalation paths are defined for 24/7 coverage where required for critical systems.

3.6.1 Alert Severity Taxonomy (Baseline)

To reduce ambiguity, alerts SHOULD be classified consistently:

• Critical: Immediate service outage or active security compromise suspected; requires immediate response.
• High: Severe degradation or high-confidence security event; urgent response.
• Medium: Degradation or suspicious event; standard response.
• Low: Informational; queued response.
• Each alert definition SHOULD include: owner, routing, runbook link, and suppression/deduplication guidance.

3.8 Minimum Dashboards (Baseline for Critical Services)

Critical services SHOULD have, at minimum, the following dashboards:

• Service health (availability, error rate, latency, throughput)
• Infrastructure health (CPU, memory, disk, network saturation)
• Dependency health (database/queue/downstream calls where applicable)
• Security signals (auth failures, privilege events, policy violations where available)
• Error budget / SLO view (SLO compliance over time)

3.7 Safe Exception Process

Exceptions to mandatory monitoring requirements (e.g. SIEM coverage, retention, SLOs) SHALL be requested, documented, and reviewed as follows. Exception requests must include: system or service name; requirement(s) from which exception is sought; justification; compensating controls and timeline; proposed end date; and sign-off from System Owner and Security Officer.

Request via government form or template.

• Documentation: justification; compensating controls and timeline; risk acceptance by System Owner and Security Officer; maximum exception duration (not to exceed 12 months unless re-approved).
• Review: at least quarterly; extensions require re-approval.
• Register: all exceptions SHALL be recorded in an exception register and made available for audit.

4. Capacity Planning and Resource Management

This section integrates capacity planning and resource management into the observability framework. Proactive capacity planning prevents service degradation, reduces unplanned outages, and optimises capital and operational spending. Capacity decisions are informed by monitoring data (trends, utilization, growth rates) and aligned with business demand forecasts.

4.1 Demand Forecasting and Headroom Strategy

Demand forecasting projects the growth of CPU, memory, storage, network, and database resources over 12 months. Headroom (buffer above projected peak demand) prevents service degradation when demand spikes and accommodates forecast error. A minimum headroom of 20–30% is SHALL be maintained for critical systems. High-impact systems SHOULD maintain minimum 15% headroom. Demand is forecast at least quarterly, reviewed with System Owner and communicated to procurement for capacity investment.

Procedure: (1) Use 12 months of monitoring data (CPU, memory, disk, throughput, transactions) to fit growth trend. (2) Project forward 12 months; add 20–30% headroom for critical systems (or 15% for high-impact) to peak projection. (3) If projected utilization + headroom exceeds current capacity, initiate procurement and deployment of additional resources. (4) Document forecast, headroom calculation, and actions in Capacity Plan (Appendix D.3).

4.2 Quotas and Fair-Use Policies

On shared infrastructure (cloud, multi-tenant systems), quotas (hard limits on CPU, memory, storage, API calls, concurrent connections) prevent a single application or tenant from exhausting resources and degrading service for others. Quota breach triggers alert and runbook for temporary mitigation (e.g. terminate runaway job, rate-limit API) and permanent remedy (e.g. resource request, code optimization).

Procedure: (1) For each shared system, define quotas per tenant or application. (2) Configure system to enforce quotas (e.g. cgroup, API gateway rate limit). (3) Log quota breach and alert. (4) Implement runbook for on-call response (temporary mitigation). (5) Escalate for permanent remedy; track resolution.

4.3 Scaling and Provisioning Policies

Scaling policies define when resources are added (scale-up), removed (scale-down), or redistributed. Auto-scaling (dynamic based on demand) is preferred for variable workloads; manual scaling (planned) for others. Scaling policy includes: trigger (e.g. CPU >80% for 5 minutes), action (e.g. +2 compute nodes), cool-down period (e.g. 10 minutes before next scale action to prevent oscillation), and approval gate if applicable.

Procedure: (1) Define scaling policy per critical or high-impact system. (2) Document trigger, action, cool-down, and approval gate. (3) Implement auto-scaling or manual runbook. (4) Test scaling quarterly; document results. (5) Review and adjust policy annually based on historical demand and cost.

4.4 Capacity Plan Baseline Requirements

All critical and high-impact systems SHALL have a documented Capacity Plan (template in Appendix D.3), containing: system or application name; current utilization (CPU, memory, storage, throughput, transactions per second); 12-month demand forecast with headroom calculation; relevant quotas and scaling policies; procurement timeline if expansion required; and owner contact and review date. Capacity Plan is reviewed at least quarterly and updated when significant change is forecast or observed.

5. Verification and Evidence Baseline (Audit-Ready Monitoring and Observability)

To make this standard auditable, ministries and operations SHOULD maintain evidence that demonstrates: monitoring coverage for critical/high systems; alert handling and SIEM coverage; SLO/SLI definitions and quarterly reviews; log retention compliance; exception register.

• What internal audit checks: Sample of systems for coverage and retention; alert and SLO evidence.
• Sampling: Quarterly sample of critical systems; annual review of exception register.
• Escalation: Non-compliance → System Owner and Security Officer → remediation plan; repeated → Document Owner and senior management.

5.1 Compliance Reporting

6. Legal Compliance and Training

6.1 Legal and Related Standards

Electronic Transactions and Cybersecurity Act (2016): cybersecurity, record-keeping. Data Protection Act (2024): where logs contain personal data, minimise, protect, retain per policy. Related: Incident Response, Server Hardening, Backup and Recovery, Capacity Planning, Logging and Evidence. Training: Operations and security staff on this standard, alert handling, SIEM use, Incident Response, and capacity planning (demand forecasting, quota configuration, scaling decision-making).

7. Framework Alignment

This standard contributes to compliance with international frameworks:

• ISO/IEC 27001:2013 – control families A.12, A.9, A.16
• NIST SP 800-53 Rev. 5 – families SI, AU, IR
• Benchmarks – none specific

Refer to `ISO27001_Mapping.md` and `Multi_Framework_Mapping.md` for the complete mapping tables.

8. Appendices

Appendix A: Related Standards

Malawi Government Incident Response Standards; Government Server Security Hardening Standards; Malawi Government Backup and Recovery Standards; Malawi Government Capacity Planning Standards; Malawi Government Logging and Evidence Standards.

• Appendix B: Capacity Plan Template (Outline)
• System/Application: [Name] Owner: [Name, Contact] Review Date: [Date]

• Current State
• CPU Utilization (Average/Peak): [e.g., 45% avg, 72% peak]
• Memory Utilization (Average/Peak): [e.g., 60% avg, 78% peak]
• Storage (Current / Max): [e.g., 450 GB / 500 GB]
• Transactions Per Second (TPS) or Throughput: [e.g., 250 TPS average, 400 TPS peak]
• Key Dependencies: [Database, queues, external APIs, etc.]
• 12-Month Demand Forecast
• Growth Rate (Annual % or Compound Growth): [e.g., 15% per annum]
• Projected Peak Utilization (12 months): [e.g., CPU 92%, Memory 88%]
• Headroom Target: [20% for critical, 15% for high-impact]
• Headroom After Growth: [e.g., 8% shortfall; expansion required]
• Capacity Actions
• If Expansion Required: [Procurement lead time, budget, timeline for deployment]
• If Headroom Adequate: [Next review date]
• Quotas and Scaling
• Current Quotas: [CPU cores, memory GB, API rate limits, concurrent connections]
• Scaling Policy: [Trigger (e.g., CPU >80% for 5 min), action, cool-down, approval gate]
• Last Scaling Drill: [Date]
• Notes
• [Planned workload changes, seasonal variation, one-time events, etc.]

Appendix C: Control and Legal Mapping

• Appendix D: Evidence and Artifact Checklist
• Monitoring: Metrics and log coverage; SIEM or central logging; SLO/SLI definitions and breach tracking.
• Alerting: Alert routing and response; exception register with requirement, justification, and review/expiry dates.
• Appendix E: Templates (Minimum)

D.1 SLO Definition (Template)

• Service name and owner
• SLI definition (how it’s measured)
• Target (SLO) and measurement window
• Error budget policy (what happens on burn)
• Alerting thresholds and runbooks
• Review cadence and change linkage

D.2 Alert Definition (Template)

• Alert name, severity, owner
• Condition and data source
• Routing/escalation path
• Runbook link
• Suppression/deduplication guidance

References

NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations (AU-2, AU-3, AU-6, SI-4).

Government Server Security Hardening Standards.

Electronic Transactions and Cybersecurity Act (2016) (Malawi).

Data Protection Act No. 3 of 2024 (Malawi).