Merge pull request 'Teh to The' (#33) from generically/wiki:main into main

Reviewed-on: Skynet/wiki#33

.forgejo/workflows/build.yaml

@@ -10,6 +10,7 @@ on:
       - .forgejo/**/*
       - mkdocs.yml
 
+
 jobs:
   # Build it locally, this helps caching for later
   build:

README.md

@@ -10,7 +10,7 @@ Uses [mkdocs][2] to generate the site.
 
 
 ## Hooks
-Run this command to set up the hooks properly so teh git information can be stored within the repo
+Run this command to set up the hooks properly so the git information can be stored within the repo
 
 ```bash
 git config --local core.hooksPath .githooks/

mkdocs.yml

@@ -30,4 +30,7 @@ plugins:
   - git-revision-date-localized:
       enable_creation_date: true
 markdown_extensions:
-  - tables
+  - tables
+  - pymdownx.caret
+  - pymdownx.mark
+  - pymdownx.tilde

src/skynet/nix.md

@@ -1,3 +1,244 @@
 # Nix/NixOS
+The [Skynet Cluster][nixos_skynet] is (for the most part) running on a Linux variant called NixOS.  
+This article aims to introduce you to Nix and Nixos in order to get you up to speed to administer the cluster.
 
-{add warnign that git and git-lfs should also need to be in teh path}
+## What it is
+### Nix
+#### Package Manager
+The word Nix refers to two things: a language and a package manager.  
+These are deeply interlinked together with the language being how the package manager is able to do its job.   
+Nix grew out of a [PhD by Eelco Dolstra][nix_paper] wherein he proposes a slightly different way to manage dependencies on a system.
+
+For most Linux systems programs make use of other software installed on the computer, for the most part this works fine.  
+Where issue may arise is if one program needs to update one of these dependencies, specifically a minor or major patch where backwards compatibility is not guaranteed.  
+If another program is using this (system-wide) dependency then it may run into interface issues when using it.  
+In a sense updating one program can break another on the system.
+
+The route the Nix package manager takes is it treats each program as a function.  
+Using the Nix language a function for that package is created which states what inputs are required, what is needed to turn those inputs into the program as well as the name for the output.  
+The output is then saved in a read only location in the format of ``/nix/store/$hash-program-name-version``.  
+This output can either be used as the input of another program or be used as is by the system/user.  
+Using this format means that any change in the inputs or the program itself will result in a different output.  
+This means that multiple versions of the program (some even the same version but different commit) can co-exist on the one system.  
+An example using different versions of Firefox:  
+![img.png](nix/firefox_co-existing.png)
+
+##### Example
+An example of packaging an application can be found here:  
+[Sieve Editor GUI on Nixpkgs][nix_pkgs_sieve]
+
+This is packaging up a GUI node.js application.  
+The application itself allows the user to edit sieve scripts.  
+Once you have [downloaded and installed](#how-we-use-it) Nix you will be able to install and run it like so:  
+```shell
+nix-shell -p sieve-editor-gui
+sieve-editor-gui .
+```
+
+#### Language
+There are two partially difficult problems in computer science:
+
+1. Off-by-one errors
+2. Caching
+3. Naming things
+
+Nix falls into this last pitfall.  
+The programming language used by the Nix package manager is called Nix, not NixLang (as like Erlang) but rather the same name as primary tool that uses it.  
+For clarity for the remainder of this subsection we are only talking about Nix the language.  
+The Nix Package manager is sometimes known as CppNix for reasons we will get into later.
+
+Nix is a lazily evaluated functional language which also has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python.  
+As a whole it takes strong influences from OCaml and other ML derived languages.
+
+##### Types
+It has most of the normal types that you would expect of a programming language, along with a few extra to deal with the filesystem:  
+```nix
+a = 1              # int
+b = 1.001          # float
+c = /path/to/thing # path
+d = "42"           # string
+e = true           # boolean
+```
+Of these the ``path`` type will be new to most people.  
+This can take either an absolute or relative path.
+
+##### Functions
+If you look at the section below it will seem that these are another type of assignment to a variable.  
+That is half right, these are akin to function pointers that you would see in C or C++.  
+Functions in Nix do not have types for either parameters or return.  
+This is due to it being lazily evaluated, like Python or Javascript.  
+As such the ``double`` function will accept any numeric value
+```nix
+double = x: x*2
+mul = a: b: a*b
+
+double 2
+double 4.2
+mul 7 6
+```
+
+##### Attribute Sets
+In most languages the way to group data would be either an Object or a Struct.  
+Nix has a similar data structure:
+```nix
+s = { foo = "bar"; biz = "baz"; }
+s.foo # bar
+s.biz # baz
+```
+
+##### More data
+This is a rough quickstart introduction to Nix.  
+For more detailed information I recommend these resources.
+
+* [Official Guide][nix_guide_official]
+* [Nix Pills][nix_guide_pills]
+
+### Flakes
+A Flake is one of the best ways of interacting with nix.  
+Despite it having some issues and still being marked as experimental it has become a de-facto standard.  
+This is also the format that we use in Skynet.
+
+The [Official Wiki Page][nix_flake] will be more informative than what can be shoved into this article.
+
+### Nixos
+With the package manager we are able to create packages in a deterministic manner and store them in a way that does not suffer path conflicts.  
+
+Some (possibly crazy) folks saw this and decided to apply this to an entire operating system.  
+The advantages are clear, the required programs are added to the path for the current iteration of the system.  
+If any error arises the system can be rolled back to a previous config.
+
+Configuration is done via ``*.nix`` files, which are then converted into the native config for the application in question.  
+
+For example [this file][nix_dns] turns a list of attributes.  
+```nix
+{
+    record = "forgejo";
+    r_type = "CNAME";
+    value = "glados.skynet.ie";
+}
+```
+Into a config usable by the BIND DNS server.
+
+### Lix
+Nix is an old enough project now, and as such has accumulated crust over the years.  
+This is a combination of technical and societal/governance.  
+                        
+On the technical side nix is built using c++ and a max of build systems that make it hard to expand it.  
+For a good long time the nix binary used in the package manager was locked at v2.18 due to issues.  
+It took most of a year for a higher version to be used on an official basis.  
+
+Regarding governance there has been several attempts to make it better for folks to contribute and to decouple everything from requiring Eelco to have an input.  
+Those attempts did not succeed.  
+The final straw for some of the more technical core contributors was Eelco's forming a company, hiding it from the community and trying to get military sponsorship.  
+This did not vibe well with folks.
+
+Due to all of this many core maintainers forked Nix at 2.18 and started working to apply fixes for both code and organisational.  
+The result of their efforts is [Lix][nix_lix].  
+For a full explanation of its key features I would like to point you to the [Lix About page][nix_lix_about].  
+We use Lix instead of CppNix as the goals of Lix align with the viewpoints and ideologies that our members hold and what we want to represent as a (computer) society.
+
+## Why we use it
+Back in [January 2023 we got disconnected from the internet][skynet_disconnect].  
+For the purposes of this document the root cause is not of importance.
+What *is* the mad Indian Jones ~~treasure~~ config hunt that it triggered.   
+![Brendan delving for configs][skynet_disconnect_reenactment]  
+Additionally, ITD require us to update our systems regularly (as they should be).  
+Technically this config delving is an ongoing effort, the old hard drives are occasionally connected up and raided.
+
+As you would imagine this is not ideal, an exasperating factor was that many programs had changed both the location and format of their configuration over the decades that Skynet has been using them.  
+
+This is where the strengths of NixOS lie.  
+The config for the entire cluster is located in a singular location.  
+Using modules which work as a translation layer if the requirements for the programs change this don't impact us.  
+Not to mention we can fearlessly (and regularly) update our systems it is a match made is heaven.  
+
+Currently, the Skynet cluster comprises 15 servers which have NixOS on them.  
+A combination of LXC's (Linux Containers) and physical bare metal servers.   
+Since these systems share a base config (with their individual applications layered on top) we are able to efficiently build them, building a package for one will also build it for other servers.  
+Combine that with the ability to deploy them via our own self-hosted CI/CD we have a strong foundation to work off of.
+
+## How we use it
+Nix cannot run on Windows, though it cna be installed into WSL.  
+Please refer to the below downloads to get a base system up and running
+
+* [Git][git_git]
+    * As we are using a git repo this is a hard requirement.
+* [Git LFS][git_lfs]
+    * For storing non text files such as images.
+* [Nix][nix_install]
+    * Cant really use nix without thi installed.
+
+### Dev Shell
+After cloning the repo use ``nix develop`` to set up a terminal shell with the environment for working with Skynet NixOS.  
+It (currently) adds [Colmena][dev_colmena], [Agenix][dev_agenix] and [Attic][dev_attic].
+
+Another example of a dev shell can be [our discord bot][dev_discord-bot].  
+This one sets up a rust environment.  
+
+Of course, you can also look at the dev shell for this [wiki][dev_wiki].  
+For the wiki we need to ensure that the dependencies for building it are present for all users so they don't have to figure out how to manually install them.
+
+### Colmena
+[Colmena][dev_colmena] is our build and deployment tool.
+
+Building is pretty easy, just run ``colmena build``.  
+Downside of that command is it will try to build everything all at once which is a *lot*.  
+A more practical approach is to build a single server or a group of servers.  
+```shell
+# build the Skynet server, names can be found in the flake.nix
+colmena build --on skynet
+
+# build a group of servers, in this case any one with the tag of active-core
+colmena build --on @active-core
+```
+
+To be able to deploy to Skynet two things are required:
+
+* Be on the UL network
+    * This can also be accomplished by being on the VPN.
+* Have an ssh key configured that can access the servers
+    * TLDR be an admin.
+
+### Agenix
+[Agenix][dev_agenix] is our secrets manager.  
+To be able to use this tool your pub ssh key must be in ``secrets/secrets.nix``.  
+As you would expect this is an admin only tool.  
+This file also defines the names and permissions for each secret.
+
+```shell
+# have to be in the secrets folder for all these commands.
+cd secrets
+
+# edit the secret
+agenix -e path/to/secret.age
+
+# re-key all secrets, this is done when a new key is added or removed.
+agenix -r
+```
+
+### Attic
+[Attic][dev_attic] is the tool that we use for our nix cache (hosted at <vhttps://nix-cache.skynet.ie/>).  
+It is not often used by the developer/admin and its own [documentation][dev_attic_docs] covers it best.
+
+
+
+[nixos_skynet]: https://forgejo.skynet.ie/Skynet/nixos
+[nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf
+[nix_guide_official]: https://nix.dev/tutorials/first-steps/
+[nix_guide_pills]: https://nixos.org/guides/nix-pills/#
+[nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix
+[nix_flake]: https://wiki.nixos.org/wiki/Flakes
+[nix_dns]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/applications/dns/dns.nix
+[nix_lix]: https://lix.systems/
+[nix_lix_about]: https://lix.systems/about/
+[skynet_disconnect]: https://public.skynet.ie/postmortem/2023-01-12_Loss-of-network-access.html
+[skynet_disconnect_reenactment]: https://forgejo.skynet.ie/Computer_Society/presentations_compsoc/media/branch/main/src/slides/skynet/0_intro_img1.png
+[git_git]: https://git-scm.com/downloads
+[git_lfs]: https://git-lfs.com/
+[nix_install]: https://nixos.org/download/
+[dev_colmena]: https://colmena.cli.rs/unstable/
+[dev_agenix]: https://github.com/ryantm/agenix
+[dev_attic]: https://github.com/zhaofengli/attic
+[dev_attic_docs]: https://docs.attic.rs/introduction.html
+[dev_discord-bot]: https://forgejo.skynet.ie/Skynet/discord-bot/src/commit/80c9191eeec29ba20ef4084713eca7fe0cab7412/flake.nix#L65
+[dev_wiki]: https://forgejo.skynet.ie/Skynet/wiki/src/commit/ab0add44756d4992fc2b2da4eba163016ccb3d1c/flake.nix#L35

src/skynet/services/email.md

@@ -23,11 +23,49 @@ This work is being done with Rust.
 
 Use your Skynet ``username@skynet.ie`` and ``password`` to login.
 
+### IMAP / SMTP settings
+
+Use your full Skynet email ``username@skynet.ie`` and ``password``.
+
+#### SMTP
+Server: ``mail.skynet.ie``
+Port: ``993``
+Authentication Method: ``Normal Password``
+Connection Security: ``SSL/TLS``
+
+#### IMAP
+Server: ``mail.skynet.ie``
+Port: ``465``
+Authentication Method: ``Normal Password``
+Connection Security: ``SSL/TLS``
+
 ## Sieve Scripts
 Sieve scripts allow you to sort and manage yer email in a programmatic way.
 
 Recommended tool: <https://github.com/thsmi/sieve>  
-Small tutorial will follow later.
+
+* Download and open the tool.
+* Enter settings for skynet.
+    - Hostname: ``mail.skynet.ie``
+    - Port: ``4190``
+    - Username: ``username@skynet.ie``
+* Hit Create.
+* Hit Connect and enter your ``password``.
+
+### ``.forwards`` replacement
+Since we no longer support ``.forwards`` this is the sieve script counterpart.  
+It copies all incoming mail to the target address.  
+
+* Continuing from above
+* Hit Create new script, enter a name and press edit.
+* Enter code below. (You may also use the gui drag and drop builder to create the script)
+```sieve
+require "copy";
+
+redirect :copy "sending_mail_to@example.ie";
+```
+* Save the script.
+* Go back to the home tab and press activate on your newly created script. This will enable the forwarding
 
 ### Committee/Admins
 There is an inbuilt sieve script that passes mail from committee/admin addresses into a folder in their Skynet inbox.  

src/skynet/services/forgejo.md

@@ -3,7 +3,7 @@
 
 Forgejo is an online git repo host.  
 Like [Gitlab](./gitlab.md) it is self hostable meaning that Skynet has its own instance of it.  
-Structure wise it has more in common with Github with teh flatter ``owner/repo`` structure as well as being compatible with Github Actions
+Structure wise it has more in common with Github with the flatter ``owner/repo`` structure as well as being compatible with Github Actions
 
 Honestly I (@silver) just love the slogan of Forgejo:
 > Beyond coding. We Forge.

src/tutorials/skynet/create_account.md

@@ -8,7 +8,7 @@
 3. Enter the same email used for UL Wolves.
     * This is the ``Preferred Contact Email`` on your [profile page][profile].
 4. You will get an email to verify your address, follow the link in the email.
-    * You may need to check teh spam folder.
+    * You may need to check the spam folder.
     * It may also take up to 15 min to be delivered.
 5. Choose a ``username`` and ``password``.
 

src/tutorials/skynet/create_ssh.md

@@ -4,7 +4,10 @@ To be able to gain remote access to the Skynet.
 
 ``$USERNAME`` Refers to your Skynet username, for example I would replace ``$USERNAME`` with ``silver``
 
-## Create Key
+## Windows
+If you are using Windows then you should use PowerShell, not ``cmd``.
+
+## Prep
 First we set up the ssh folder and create a skynet folder within it for neatness
                 
 ```bash
@@ -12,13 +15,22 @@ mkdir -f -p ~/.ssh/skynet
 cd ~/.ssh/skynet
 ```
 
-Now we will create the ssh key itself.  
-Location: ``$USERNAME``, your skynet username.  
-Password: Press Enter twice for no password on the key.  
+## Create Key
+Now we will create the ssh key itself.
 
 ```bash
 ssh-keygen -t ed25519 -C "<comment>"
 ```
+
+* ``<comment>``: this is a comment to yerself about what the key is for
+    * I often use ``username@host``, ``silver@skynet``.
+* Location: ``./$USERNAME``, your skynet username.
+    * ``./silver`` for example.
+* **Password: Press Enter twice for no password on the key.**  
+
+If you are creating this key for a CI/CD pipeline (``user_deploy*``) then adding a password will cause it to fail.
+
+
 It will create two files: ``$USERNAME`` and ``$USERNAME.pub`` inside ``~/.ssh/skynet``
 
 ### Linux Only
@@ -67,7 +79,7 @@ nano config
 Or open up in a text editor.
 
 ### Windows/Linux
-This is what we want to have in teh file.
+This is what we want to have in the file.
 ```
 Host *.skynet.ie
    User $USERNAME
@@ -79,7 +91,7 @@ Host *.skynet.ie
 
 
 ## Add key to account
-Go to [the modify SSH page](https://account.skynet.ie/modify_ssh) and paste in teh contents of ``$USERNAME.pub``.
+Go to [the modify SSH page](https://account.skynet.ie/modify_ssh) and paste in the contents of ``$USERNAME.pub``.
 
 You will now be able to SSH into Skynet like so:
 

src/tutorials/skynet/create_website.md

@@ -1,11 +1,42 @@
 # Create your own website on Skynet
 
-## Login
-Once ye have [logged in](./create_ssh.md) you will be able to create a website on Skynet.
+One of the services that Skynet provides is access to a Linux server which also has a webserver.  
+Additionally each member gets their own domain:
+
+* ``$USERNAME.users.skynet.ie``
+    * This is the core web address
+    * <https://silver.users.skynet.ie>
+* ``skynet.ie/~$USERNAME``
+    * This is backwards compatible to help avoid link rot
+    * Redirects to the core address above
+    * <https://skynet.ie/~silver>
 
 ## Website
-### Manually
+There are two ways to create a website on Skynet, Modern and Manual.  
+Modern is using Source Control (git), pipelines (CI/CD) and pinch of magic.  
+Manual gets you hands on with the process.
 
+
+### Skynet Account requirement
+No matter if you choose Modern or Manual you will need a Skynet account.  
+Instructions to do so can be found [on the Create Skynet Account page](./create_account.md).
+
+### Modern
+Modern involves using source control to version your site.  
+This is then leveraged to automatically deploy to your Skynet account/website.
+
+The core repo for this is [here][deploy_user], it contains all the required info to get it working.  
+If this does not suit your needs you can take a look at the [forks which cover different tools][deploy_user_forks].
+
+[deploy_user]: https://forgejo.skynet.ie/Skynet/deploy_user
+[deploy_user_forks]: https://forgejo.skynet.ie/Skynet/deploy_user/forks
+
+### Manual
+#### Login
+Once ye have [logged in](./create_ssh.md) you will be able to create a website on Skynet.
+
+
+#### Create Files
 In your home folder follow these commands to create the folder that can be used to host a website
 
 ```bash
@@ -15,12 +46,32 @@ chmod -R 755 ~/public_html
 cd ~/public_html 
 ```
 
-To transfer files graphically you can use these programs:
-* WinSCP
-* FileZilla
+In this directory you can create all the relevant files using ``nano`` or ``vim``.  
 
-Anything put there will be accessible under ``https://$USERNAME.users.skynet.ie``.  
-For backwards compatability ``https://skynet.ie/~$USERNAME`` will redirect to ``https://$USERNAME.users.skynet.ie``.
+```bash
+# create/edit index.html usign nano
+# this opens up a termial based text editor.
+nano index.html
+```
+
+To exit ``nano`` you need to use ``[ctrl]+[x]`` keys.
+
+#### Transfer files
+If you already have the files locally you can copy them over using ``scp``, although it is recommended to do it graphically.
+
+To transfer files graphically you can use these programs:
+
+* [WinSCP][download_winscp]
+    * Windows  
+* [FileZilla][download_filezilla]
+    * Windows
+    * Mac
+    * Linux
+
+[download_winscp]: https://winscp.net
+[download_filezilla]: https://filezilla-project.org/download.php?type=client
+
+Anything put there will be accessible under ``https://$USERNAME.users.skynet.ie``.
 
 ## More info
 There is a slideshow that might be of use to you: <https://public.skynet.ie/slides/skynet/1_setup.html>