doc: added how to find information on updating proxmox

Got started on how to init a new server

.githooks/post-checkout

@@ -1,3 +0,0 @@
-#!/bin/sh
-command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting the 'post-checkout' file in the hooks directory (set by 'core.hookspath'; usually '.git/hooks').\n"; exit 2; }
-git lfs post-checkout "$@"

.githooks/post-commit

@@ -1,3 +0,0 @@
-#!/bin/sh
-command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting the 'post-commit' file in the hooks directory (set by 'core.hookspath'; usually '.git/hooks').\n"; exit 2; }
-git lfs post-commit "$@"

.githooks/post-merge

@@ -1,3 +0,0 @@
-#!/bin/sh
-command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting the 'post-merge' file in the hooks directory (set by 'core.hookspath'; usually '.git/hooks').\n"; exit 2; }
-git lfs post-merge "$@"

.githooks/pre-commit

@@ -1,4 +0,0 @@
-#!/usr/bin/env bash
-
-tar czf _git.tar.gz --exclude .git/lfs .git
-git add _git.tar.gz

.githooks/pre-push

@@ -1,3 +0,0 @@
-#!/bin/sh
-command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting the 'pre-push' file in the hooks directory (set by 'core.hookspath'; usually '.git/hooks').\n"; exit 2; }
-git lfs pre-push "$@"

flake.nix

@@ -19,8 +19,6 @@
         packages = with pkgs.python3Packages; [
           mkdocs
           mkdocs-material
-          mkdocs-git-authors-plugin
-          mkdocs-git-revision-date-localized-plugin
         ];
       in {
         formatter = alejandra.defaultPackage.${system};
@@ -28,7 +26,7 @@
           name = "skynet-wiki";
           src = self;
           buildInputs = packages;
-          buildPhase = "tar -zxf _git.tar.gz && mkdocs build";
+          buildPhase = "mkdocs build";
           installPhase = "mkdir -p $out; cp -R site/* $out;";
         };
 

mkdocs.yml

@@ -25,10 +25,10 @@ theme:
     view: material/eye
 plugins:
   - search
-  - git-authors:
-      show_email_address: false
-  - git-revision-date-localized:
-      enable_creation_date: true
+#  - git-authors:
+#      show_email_address: false
+#  - git-revision-date-localized:
+#      enable_creation_date: true
 markdown_extensions:
   - tables
   - pymdownx.caret

src/procedures/skynet/admin_modify.md

@@ -0,0 +1,78 @@
+# Change Admins
+A Skynet Admin is a user with teh highest level of access to the cluster, having been trained up over a year.  
+This guide will cover adding and removing an Admin.
+
+The normal Committee Member process applies to them as well, so check [here](./committee_modify.md#adding) for teh specific details.
+
+A user cannot add or remove themselves from being an admin, an existing admin must do so (one who will still be admin after the procedure).
+
+## Add
+Steps to add a new Admin
+
+### NixOS
+There are several steps related to the NixOS repo on Forgejo.
+
+#### Username Added
+Like a normal committee member the users username must be added to the [``config/users.nix``][users.nix], specifically the admin section.  
+This gives teh user teh correct Skynet roles on our services.  
+This also enables them to use their personal account to login to all the servers.  
+
+#### SSH Key
+##### Root Account
+There is a ``root`` account that admins can use.  
+To be able to login as root you have to add you key to the keys for ``root``
+
+You can add it in the [``machines/_base.nix``][base.nix].
+
+##### Secrets Management
+We store our secrets encrypted in the repo using a tool called ``agenix``, a nix implementation of [age][age].  
+In order to create/edit any secret your key has to be added to [``secrets/secrets.nix``][secrets.nix].
+
+Add the key mimicking the format for the other admins.  
+Then add it to the users list/array.  
+Finally re-key the secrets
+
+
+###### Re-key Secrets
+In order to re-key the secrets your key must have been previously added (this just not work if you have just added your key, an existing admin must do this).  
+
+
+```shell
+# enter the devshell
+nix develop
+
+# Move into teh secrets folder
+cd secrets
+
+# Re-key the secrets
+agenix -R
+```
+
+Then commit the changes.
+
+### VPN
+The VPN is what allows admins to get access to the internal parts of teh cluster when not on campus.  
+It is provided by ITD.
+
+1. Add an entry to the bottom of [``ITD/VPN_Admins.csv``][VPN_Admins.csv]
+2. Add an entry to [``ITD/VPN_Admins_changes.csv``][VPN_Admins_changes.csv] with ``Pending`` for teh Action.
+3. Open up a TopDesk request with ITD to add the new Admin (TODO: TopDesk procedure)
+4. Add the ticket reference to [``ITD/VPN_Admins_changes.csv``][VPN_Admins_changes.csv]
+5. When complete mark the entry ``Added`` on [``ITD/VPN_Admins_changes.csv``][VPN_Admins_changes.csv].
+
+## Remove
+For teh most part the removal/retirement of an admin is just the opposite of the adding.  
+The main difference is contacting ITD to remove them from the VPN.
+
+### VPN
+1. Open up a TopDesk request with ITD to remove the old Admin.
+2. Add the ticket reference to [``ITD/VPN_Admins_changes.csv``][VPN_Admins_changes.csv]
+3. When complete mark the entry ``Removed`` on [``ITD/VPN_Admins_changes.csv``][VPN_Admins_changes.csv].
+
+
+[users.nix]: https://forgejo.skynet.ie/Skynet/nixos/src/commit/b46eca16b0b207d14e173d4e40286160749c5c07/config/users.nix#L76
+[base.nix]: https://forgejo.skynet.ie/Skynet/nixos/src/commit/b46eca16b0b207d14e173d4e40286160749c5c07/machines/_base.nix#L75
+[secrets.nix]: https://forgejo.skynet.ie/Skynet/nixos/src/commit/b46eca16b0b207d14e173d4e40286160749c5c07/secrets/secrets.nix#L3-L18
+[VPN_Admins.csv]: https://forgejo.skynet.ie/Skynet/nixos/src/commit/b46eca16b0b207d14e173d4e40286160749c5c07/ITD/VPN_Admins.csv
+[VPN_Admins_changes.csv]: https://forgejo.skynet.ie/Skynet/nixos/src/commit/b46eca16b0b207d14e173d4e40286160749c5c07/ITD/VPN_Admins_changes.csv
+[age]: https://github.com/FiloSottile/age

src/procedures/skynet/alumni_verify.md

@@ -0,0 +1,95 @@
+# Alumni Verification
+We are an old Society (1992-<span id="year">now</span>) with a long history of members staying members long after they leave UL.  
+In 2023 we had an outage ([see here for more information][outage_2023]) which lead to us loosing contact with many of these since their contact details were their Skynet email.    
+We are also required to ensure that all active Skynet accounts are linked to a UL Computer Society membership.  
+This put us in the catch22, where folks needed their Skynet account in order to gain access to their Skynet account.  
+
+Many of these people want to regain access to their accounts and will make a request for help.
+
+## How to process Requests
+These requests to gain access to an old Skynet account may come in through one of our Email addresses (``contact@skynet.ie``) or from our [Discord][discord]
+
+### Email
+Keep an eye on the inboxes ye have access to.  
+Ensure that you have set up the aliases correctly (TODO: Email Aliases).  
+Aside from that be polite.
+
+### Discord
+Generally users will bring up their query in ``help-and-support``.  
+It is recommended that you create a Private thread and ping them to bring them into it.  
+This ensures that no private information will leak out.
+
+#### How to create a Private Thread
+
+1. Bottom left there is a Plus icon, click on that and select the ``Thread`` option.  
+   ![plus_icon.png](alumni_verify/plus_icon.png)
+2. In teh segment that popped up tick the Private Thread checkbox.  
+   ![private_thread.png](alumni_verify/private_thread.png)
+
+## The user recovers their own account.
+The easiest way is that the user can recover their own account.  
+This works if they know their own username/password.  
+See [Account Recovery here][account_recovery] for more details.
+
+If the user's account is old enough that the password is hashed with CRYPT then tehy will be unable to reset it themselves (even if they know teh password).  
+This is due to CRYPT being specific to the *machine* it is on, and the LDAP server has moved several times since in the last two decades.  
+In which case check out the following methods.
+
+## Check if there is ``.forward``
+A ``.forward`` file was placed in a users home directory to forward all incoming mail to that address.  
+
+```shell
+USER="username"
+
+# Output a list of paths if one exists for this user
+find  /skynet_old/{mailconfig,home}/$USER-name '.forward'
+
+# use cat to output the contents to teh terminal
+cat /skynet_old/mailconfig/$USER/.forward
+```
+
+### Usernames
+One possible contents of the file is one or more Skynet usernames.
+
+These are not too useful for our use case. 
+```
+usera
+userb
+```
+
+### Email(s)
+What we are looking for is an email (identity) to tie the Skynet account to.  
+In which case you can check with the user that they still have access to teh email.  
+Get them to send you a mail from that account and verify the headers (TODO: Email headers)
+```
+username@example.com
+```
+
+## Find a link to Personal Email
+Nothing in the ``.forwards`` or they no longer have access to the email its time to see if you can tie their current identity to skynet.  
+This is mostly possible since many members used to send mail between their Skynet account and their private account.
+
+SSH into ``skynet.skynet.ie`` and run:
+```shell
+EMAIL="email@example.ie"
+USER="username"
+grep -r "$EMAIL" /skynet_old/{home,mail,mailconfig}/$USER
+```
+It may take a while to run depending on the contents of these folders.  
+in some cases they may have proof of their ID, in which case you can search for that.
+
+Honestly the output of this is vibes based.  
+If there are forwarded email headers its a strong indication.  
+If a file in their ``public_html`` shows up that is also a strong indication.  
+The easiest one is if nothing pops up.
+
+Remember you are the guardian of data on Skynet, if a user cannot prove a connection you are not obligated to grant them access.
+
+[outage_2023]: ../../support/recovery.md
+[discord]: https://discord.skynet.ie
+[account_recovery]: ../../support/recovery#account-recovery
+
+<script>
+  /* I am a lazy fucker */
+  document.getElementById('year').textContent = new Date().getFullYear().toString()
+</script>

src/procedures/skynet/committee_modify.md

@@ -0,0 +1,48 @@
+# Change Committee Members
+This page covers the adding and removing of UL Computer Society committee members.  
+
+## Adding
+Before starting the committee member in question *must* have a Skynet Account.   
+This is because we have several services that require authentication to access.  
+Details on how to create one can be found [here][skynet_account_creation].  
+
+Once they have an account add their username to [``config/users.nix``][users.nix].  
+Then commit and push.
+
+Once everything is deployed they should have access to all resources within 15 min.
+If you require it sooner than that see the [Force Update](#force-update) section below.
+
+Next checkout the page on [VaultWarden](./vaultwarden.md) to add the user to the password manager.
+
+## Removing
+Removing is essentially the same as adding.
+
+Remove their username to [``config/users.nix``][users.nix].  
+Then commit and push.
+
+## Force Update
+If you need to hasten an update you can log into teh server to give it a (virtual) kick.
+
+### SSH into Kitt  
+Kit is teh home of the user accounts.
+```shell
+ssh root@kitt.skynet.ie
+```
+### Once attached run these commands  
+```shell
+# reboot the main process
+systemctl reboot skynet_ldap_backend.service
+
+# Update the data (this ensures that folks are current members)
+systemctl reboot skynet_ldap_backend-update_data.service
+
+# Apply the new group roles
+systemctl reboot skynet_ldap_backend-update_groups.service
+```
+### Exit the terminal.  
+```shell
+quit
+```
+
+[skynet_account_creation]: ../../tutorials/skynet/create_account.md
+[users.nix]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/config/users.nix#L52

src/procedures/skynet/server_new.md

@@ -0,0 +1,178 @@
+# New Server (LXC)
+
+This is the instruction guide for setting up a new LXC server.
+
+Rough steps are as follows:
+1. Plan the servers config
+2. Login to Proxmox
+3. Create Container using the base LXC image
+4. Login to Server
+5. Push new configuration
+
+## Plan server Configuration
+To allocate he correct resources there are a few questions that need to be asked and answered.
+
+1. What will this server be **For**?
+2. What will its **Name** be?
+3. What will its **IP** be?
+4. Fill the details into the tracking sheet.
+
+### What is it **For**?
+What a server is for dictates what hardware resources need to be allocated.  
+Is there one already existing that you can copy the configuration of?  
+Do you have prior experience with what will be hosted on it?  
+Is there documentation that you can use as a foundation?
+
+When you know these, write it down and save for later.
+
+### What is its **Name**?
+In our cluster we have a very definite naming scheme for the servers we have.  
+There are two rules:
+
+1. The login server (where folks have their home dirs and websites) is called Skynet.
+    * [This can be traced back to 2007][server_name_skynet]
+    * > By popular demand, the skynet name was retained for the login server
+2. All other servers are also named after AI's
+
+
+In the current cluster we tend to use groups of AI names for particular functions.  
+For example:
+
+* Vigil/Vendetta were AI's in the Mass effect series, pointing Shepherd forward, so they are our DNS servers
+* Glados/Wheatly from Portal, that game runs on the Source Engine, so they are our Source control servers
+* Optimus/Bumblebee from Transformers, their origin is a line of toys, so fittingly our games servers.
+* Neuromancer/Wintermute, from Neuromancer, each with multiple minds, thus became our backup (redundancy) servers.
+
+If at all possible try to get the name to match its task.  
+Some past names and ideas for others can be found on [the nixos wiki][server_names]
+
+
+### What is its **IP** address?
+We have a ``/26`` allocation, so about 60 IP's we can make use of.  
+Like with the names above several servers are grouped together IP wise.  
+Check the [Server Inventory][server_inventory] to see what addresses are available.
+
+### Tracking sheet
+Now that ye have all the details about the build it is time to add them to the [Server Inventory][server_inventory].  
+Add a new one, incrementing the index, and fill in the rest of the information.  
+
+For the IP address if the last segment (ABC in this: 193.1.99.ABC) is less than 100 then add a leading ``0`` to it.  
+This is so that it can be easily sorted.
+
+## Login to Proxmox
+
+Login to Proxmox (TODO: insert link to accessing Proxmox here)
+
+## Create Container Using the Base LXC Image
+The Proxmox documentation for LXC's is available [here][proxmox_lxc].
+
+Top right there is a button [Create CT], that brings up a window.  
+Each section below is one of the tabs in the window.
+
+### General
+#### Hostname
+This is the **Name** of the server, lowercase.
+#### Unprivileged
+Ensure this is ticked.
+#### Nesting
+Ensure this is ticked.
+#### SSH Public Keys
+Enter the ``root`` pub ssh key.  
+This is used to login to teh container later.
+
+### Template
+Select the container image, most likely ``nixos-system-x86_64-linux.tar.xz``.
+
+### Disks
+#### Storage
+Most likely it is ``main_pool``, it should have a significant amount of storage available.
+**DO NOT** use ``local-zfs``, this is on Proxmox's own drive and not suited for container data.
+
+#### Disk Size
+Self-explanatory, how much space you want to give teh container.  
+A minimum of 30Gb is suggested.
+
+### CPU
+One core minimum, larger servers will require up to 6 or so.
+
+### Memory
+Nixos will happily run on 512Mb if its load is not too intensive
+
+### Network
+#### Bridge
+The main bridge we use is ``vmbr0`` which is for most servers as it connects to the normal Skynet DMZ.
+
+We also have ``vmbr1`` which is for ``skynet.skynet.ie`` and connects to Skynet-EXT DMZ.  
+This is due to our users needing ssh access.
+
+#### IPv4
+##### IPv5/CIDR
+This is the **IP** followed by ``/26``.  
+For example ``193.1.99.75/26``
+
+##### Gateway
+The main gateway we use is ``193.1.99.65``.  
+There is a secondary one for ``skynet.skynet.ie`` which is ``193.1.96.161``.
+
+### DNS
+You can either use ``use host settings`` or fill in ``193.1.99.120`` and ``193.1.99.109``.
+
+### Confirm
+Use this as a chance to review all the options.  
+There are a few gotcha's outlined above that ye do have to look over.
+
+#### Start after created
+Tick this box if you want it to boot up immediately after being installed.  
+
+## Login to Server
+Now that the server is up and running it is time to login to it.  
+Assuming you have your SSH configured like (TODO: admin ssh config).  
+You just have to use ``ssh root@IP`` (for example ``ssh root@192.99.1.111``)
+
+### Getting the server ssh key
+We are logging in because we need to get the servers own ssh key.  
+You can find it in ``/etc/ssh``.  
+You have a choice between ``ssh_host_ed25519_key.pub`` and ``ssh_host_rsa_key.pub``.  
+``ssh_host_ed25519_key.pub`` is the recommended one.
+
+### Using the server ssh key
+This key is used to decrypt secrets stored in our Nixos repo and as such needs to be added there.  
+1. Add the key with the other system keys [here][nixos_secrets].  
+2. Add it to the systems array underneath that.
+3. In the ``secrets`` folder run ``cd secrets && agenix -r`` to rekey the secrets.
+    * This is to give the new server access.
+4. Commit all the changed files
+
+
+## Push new configuration
+### Create config
+An example server config for nixos is [available here][nixos_template].  
+Copy it to a new file and name it ``name.nix``.  
+Fill in all relevant details (name/ip/name details/its purpose/...).
+
+Finally add it to the bottom of the [``flake.nix``][nixos_flake].  
+Save and commit the files.
+
+### Push Config
+There are two ways to test out the config, manual and pipeline.  
+Each has pros and cons.
+
+#### Manually
+This requires your key to be added the the keys for the [root account][nixos_root] and your ``~/.ssh/config`` to be set up properly (TODO: SSH tutorial)
+
+1. Build it with ``colmena build --on name``
+2. If it builds ye can test deployment
+3. Deploy it using ``colmena apply --on name``
+
+#### Pipeline
+Just push it to the repo and the pipeline will handle building and deployment.  
+The disadvantage of this is you cannot fix any mistakes before they are pushed.
+
+[server_name_skynet]: https://2009.skynet.ie/history.html
+[server_names]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/Possible_Server_Names.md
+[server_inventory]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/ITD/Server_Inventory.csv
+[proxmox_lxc]: https://pve.proxmox.com/wiki/Linux_Container
+[nixos_secrets]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/secrets/secrets.nix#L35
+[nixos_template]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/machines/_template.nix
+[nixos_flake]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/flake.nix#L156
+[nixos_root]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/machines/_base.nix#L75

src/procedures/skynet/server_update_nixos.md

@@ -0,0 +1,84 @@
+# Updating Servers (NixOS)
+A short guide on how to update NixOS servers.  
+This is required at least once a semester in order to keep the systems up to date.
+
+These are some of the easiest servers we have to update.  
+This is how you update it locally, and then deploy.
+
+## Update
+Invoke the devshell for the NixOS repo locally on your computer (TODO: Setup nix and into to devshell).  
+This will give you all the tools you need to update the NixOS servers.
+
+In that shell you can run these commands.
+
+```shell
+# this will update every input, sometimes that is not desired
+nix flake update
+
+# This will update a single input, nixpkgs which will update the OS's
+nix flake update nixpkgs colmena
+
+# Update multiple at once, these are the core tools and utilities
+# Recommended command
+nix flake update nixpkgs flake-utils agenix arion alejandra colmena
+```
+
+## Build
+Now that the inputs have been updated it is time to build it locally to ensure no errors pop up.  
+For any errors that pop up you can use ``--key-result`` in order to explore the system output.
+
+```shell
+# This will build all the servers locally
+colmena build
+
+# Build a single one
+colmena build --on glados
+
+# Build a logical group of them
+colmena build --on @active-dns
+
+# Build all but keep the outputs in ``.gcroots``
+# This is useful if you need to explore the config locally
+colmena build --keep-result
+```
+
+
+## Deploy
+It is now time to commit and deploy the updates.
+
+
+### Manually
+Most of the flags from the build step above apply for teh deploy step.  
+You dont need to have the files committed to do this, so it cna be useful for testing.
+
+```shell
+# This will build all the servers locally
+colmena deploy
+
+# Build a single one
+colmena deploy --on glados
+
+# Build a logical group of them
+colmena deploy --on @active-dns
+```
+
+
+### Pipeline
+To deploy via the pipeline simply commit and push to Forgejo.  
+The pipeline will build and deploy to almost all the servers.  
+Downside of this is slow iteration for testing, but it is useful if you are pretty sure of teh changes.
+
+#### Git Server/Runner
+The CI/CD pipelines we have run on Glados and Wheatly, as such the pipelines are not able to update these two servers on their own.  
+These need to be updated manually using any of these commands:
+
+```shell
+# This will deploy all the servers
+colmena deploy
+
+# Deploy a single one
+colmena deploy --on glados wheatly
+
+# Deploy a logical group of them
+colmena deploy --on @active-git
+```

src/procedures/skynet/server_update_proxmox.md

@@ -0,0 +1,16 @@
+# Updating Servers (Proxmox)
+A short guide on how to update servers running Proxmox VM Host
+
+There is documentation for Proxmox available on teh [web][docs_web] and locally on teh server [itself][docs_inbuilt].  
+The inbuilt ones are preferred since they match teh version of Proxmox we are using.  
+In both there is a chapter called [``System Software Updates``][docs_inbuilt_update] which tells you how to update.
+
+When there is a major version released there will be a new link on [this page][docs_wiki] in teh format of:  
+``Upgrade from $VersionCurrent to $VersionNext``
+
+
+
+[docs_inbuilt]: https://193.1.99.73:8006/pve-docs/pve-admin-guide.html
+[docs_inbuilt_update]: https://193.1.99.73:8006/pve-docs/pve-admin-guide.html#system_software_updates
+[docs_web]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html
+[docs_wiki]: https://pve.proxmox.com/wiki/Category:Upgrade

src/skynet/services/email.md

@@ -23,21 +23,49 @@ This work is being done with Rust.
 
 Use your Skynet ``username@skynet.ie`` and ``password`` to login.
 
+### IMAP / SMTP settings
+
+Use your full Skynet email ``username@skynet.ie`` and ``password``.
+
+#### SMTP
+Server: ``mail.skynet.ie``
+Port: ``993``
+Authentication Method: ``Normal Password``
+Connection Security: ``SSL/TLS``
+
+#### IMAP
+Server: ``mail.skynet.ie``
+Port: ``465``
+Authentication Method: ``Normal Password``
+Connection Security: ``SSL/TLS``
+
 ## Sieve Scripts
 Sieve scripts allow you to sort and manage yer email in a programmatic way.
 
 Recommended tool: <https://github.com/thsmi/sieve>  
-Small tutorial will follow later.
+
+* Download and open the tool.
+* Enter settings for skynet.
+    - Hostname: ``mail.skynet.ie``
+    - Port: ``4190``
+    - Username: ``username@skynet.ie``
+* Hit Create.
+* Hit Connect and enter your ``password``.
 
 ### ``.forwards`` replacement
 Since we no longer support ``.forwards`` this is the sieve script counterpart.  
 It copies all incoming mail to the target address.  
 
+* Continuing from above
+* Hit Create new script, enter a name and press edit.
+* Enter code below. (You may also use the gui drag and drop builder to create the script)
 ```sieve
 require "copy";
 
 redirect :copy "sending_mail_to@example.ie";
 ```
+* Save the script.
+* Go back to the home tab and press activate on your newly created script. This will enable the forwarding
 
 ### Committee/Admins
 There is an inbuilt sieve script that passes mail from committee/admin addresses into a folder in their Skynet inbox.