From 54d1fd58213e087d63331a0d94ea723617207e20 Mon Sep 17 00:00:00 2001
From: Brendan Golden <git@brendan.ie>
Date: Fri, 2 Feb 2024 17:04:43 +0000
Subject: [PATCH] post: Rough overview of the extended downtime from Jan 2023

---
 .../2023-01-12_Loss-of-network-access.md      | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 src/postmortem/2023-01-12_Loss-of-network-access.md

diff --git a/src/postmortem/2023-01-12_Loss-of-network-access.md b/src/postmortem/2023-01-12_Loss-of-network-access.md
new file mode 100644
index 0000000..0b6266e
--- /dev/null
+++ b/src/postmortem/2023-01-12_Loss-of-network-access.md
@@ -0,0 +1,50 @@
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+title = '2023-01-12 Loss of Network Access'
+date  = 2023-01-12
+
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+# 2023-01-12 Loss of Network Access Postmortem
+
+Key people: Brendan (silver).
+
+## What happened
+On 2023-01-12 we received an email from ITD stating that 193.1.99.123 was registered as an attack source by Heanet and external access was removed by ITD to contain it.  
+Later the same day it had been upgraded to our entire subnet, the ITD security team saw our servers as a security risk.
+
+## What was the root cause
+Unlike other incidents we knew from the outset what the root cause was.  
+One of the Wordpress instances on www.skynet.ie decided its true calling was to become a spambot, this is what caught the attention of Heanet.
+
+## Restoring network access
+In order to restore network access ITD had two requirements:
+
+1. Servers are patched on both an OS & application level.
+2. Maintained in such a manner to prevent unauthorized access or misuse
+
+
+## Rebuilding services
+Due to the age of the software on virtually all machines in place upgrades were neither feasible nor maintainable.  
+
+As a result user data and config files were backed up from the servers that would be reused in the future.  
+Other servers were archived in case of a need to get more data in the future.
+
+Using the newest server with adequate hard drive bays Proxmox was installed.  
+Various containers were then created to serve the roles needed to run the cluster.  
+The OS chosen for these containers was NixOS, a config based operating system, which allows us to easily update with consistency.  
+
+This process took until the end of April.  
+We were then able to request that ITD open specific ports for servers.  
+Over the summer more services became active.
+
+
+## Outcomes
+As a result of all this: 
+
+* We have a far better relationship with ITD
+* We have reliable systems
+* We have improved security and access controls.
+* We have embraced automation
+* We are far more open and transparent (config is open source)
+