From e74e31752a622370a7f32919424cc75f1d75c217 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Mon, 7 Oct 2024 09:21:11 +0100 Subject: [PATCH 01/11] prep: shell of the documentation to be filled out --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 29 ++++++++++++++++++++++++++++- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index db39c25..b1bf13f 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:adbe687c5f5f0512ea64680ad16a63879e8be0827a40e9b23d9a573853cb9a9c -size 331908 +oid sha256:3df8f69e441454833dabe16025b924632137959cb60aad501a1900e27ce6b555 +size 337666 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 71fb71a..17fa865 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -1,3 +1,30 @@ # Nix/NixOS -{add warnign that git and git-lfs should also need to be in teh path} \ No newline at end of file +## What it is +### Nix + +### Flakes + +### Nixos + +### Lix + +## Why we use it +{Details of how the config was ascattered and hard to find} +{Also embracing devops and reduced manpower} + +## How we use it +### Requirements +{add warnign that git and git-lfs should also need to be in teh path} + +### Download + +### Colmena +#### Local +##### Building + +##### Repl + +#### Deployment + + -- 2.46.1 From d98c090fb7ba9a5ddc1da4df77acd49d46f5fdbe Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Mon, 14 Oct 2024 00:21:51 +0100 Subject: [PATCH 02/11] assignment: saving current progress for this evening --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 27 ++++++++++++++++++++++++++ src/skynet/nix/firefox_co-existing.png | 3 +++ 3 files changed, 32 insertions(+), 2 deletions(-) create mode 100644 src/skynet/nix/firefox_co-existing.png diff --git a/_git.tar.gz b/_git.tar.gz index b1bf13f..bfb994e 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3df8f69e441454833dabe16025b924632137959cb60aad501a1900e27ce6b555 -size 337666 +oid sha256:26358636ab2431f4688f5a3a8e2c47fd59ad842d840488b4d9577791d192ba64 +size 341053 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 17fa865..f8d2305 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -1,7 +1,30 @@ # Nix/NixOS +The [Skynet Cluster][nixos_skynet] is (for the most part) running on a Linux variant called NixOS. +This article aims to introduce you to Nix and Nixos in order to get you up to speed to administer the cluster. ## What it is ### Nix +#### Package Manager +The word Nix refers to two things: a language and a package manager. +These are deeply interlinked together with the language being how the package manager is able to do its job. +Nix grew out of a [PhD by Eelco Dolstra][nix_paper] wherein he proposes a slightly different way to manage dependencies on a system. + +For most Linux systems programs make use of other software installed on the computer, for the most part this works fine. +Where issue may arise is if one program needs to update one of these dependencies, specially a minor or major patch where backwards compatibility is not guaranteed. +If another program is using this (system wide) dependency then it may run into interface issues when using it. +In a sense updating one program can break another on the system. + +The route the Nix package manager takes is it treats each program as a function. +Using teh Nix language a function for that package is created which states what inputs are required, what is needed to turn those inputs into teh program as well as the name for the output. +The output is then saved in a read only location in the format of ``/nix/store/$hash-program-name-version``. +This output can eitehr be used as the input of another program or be used as is by the system/user. +Using this format means that any change in the inputs or the program itself will result in a different output. +This means that multiple versions of the program (some even the same version but different commit) can co-exist on the one system. +An example using different versions of Firefox: +![img.png](nix/firefox_co-existing.png) + +#### Language + ### Flakes @@ -28,3 +51,7 @@ #### Deployment + + +[nixos_skynet]: https://forgejo.skynet.ie/Skynet/nixos +[nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf \ No newline at end of file diff --git a/src/skynet/nix/firefox_co-existing.png b/src/skynet/nix/firefox_co-existing.png new file mode 100644 index 0000000..f2699d9 --- /dev/null +++ b/src/skynet/nix/firefox_co-existing.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3223a97f158347125132e42050ef77f293bb650d45bb49d8956107368b2a8b14 +size 77103 -- 2.46.1 From 550748da7717ca82d33fc62eabda818139babcd2 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Wed, 16 Oct 2024 01:53:45 +0100 Subject: [PATCH 03/11] assignment: added section on the nix programming language --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 59 +++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index bfb994e..27b759b 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26358636ab2431f4688f5a3a8e2c47fd59ad842d840488b4d9577791d192ba64 -size 341053 +oid sha256:d68d0b80a4c4150a73887f196b992c2ce1a75dcddae7d6b80705d241cd5e69d0 +size 344824 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index f8d2305..094cdd9 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -17,14 +17,67 @@ In a sense updating one program can break another on the system. The route the Nix package manager takes is it treats each program as a function. Using teh Nix language a function for that package is created which states what inputs are required, what is needed to turn those inputs into teh program as well as the name for the output. The output is then saved in a read only location in the format of ``/nix/store/$hash-program-name-version``. -This output can eitehr be used as the input of another program or be used as is by the system/user. +This output can either be used as the input of another program or be used as is by the system/user. Using this format means that any change in the inputs or the program itself will result in a different output. This means that multiple versions of the program (some even the same version but different commit) can co-exist on the one system. An example using different versions of Firefox: ![img.png](nix/firefox_co-existing.png) #### Language +There are two partially difficult problems in computer science: +1. Off by one errors +2. Caching +3. Naming things +Nix falls into this last pitfall. +The programming language used by teh Nix package manager is called Nix, not Nixlang (as like Erlang) but rather the same name as primary tool that uses it. +For clarity for teh remainder of this subsection we are only talking about Nix the language. + +Nix is a lazily evaluated functional language which al has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python. +As a whole it takes strong influences from OCaml and other ML derived languages. + +##### Types +It has most of the normal types that you would expect of a programming language, along with a few extra to deal with the filesystem: +```nix +a = 1 # int +b = 1.001 # float +c = /path/to/thing # path +d = "42" # string +e = true # boolean +``` +Of these the ``path`` type will be new to most people. +This can take either an absolute or relative path. + +##### Functions +If you look at the section below it will seem that these are another type of assignment to a variable. +That is half right, these are akin to function pointers that you would see in C or C++. +Functions in Nix do not have types for either parameters or return. +This is due to it being lazily evaluated, like Python or Javascript. +As such the ``double`` function will accept any numeric value +```nix +double = x: x*2 +mul = a: b: a*b + +double 2 +double 4.2 +mul 7 6 +``` + +##### Attribute Sets +In most languages the way to group data would be either an Object or a Struct. +Nix has a similar datastructure: +```nix +s = { foo = "bar"; biz = "baz"; } +s.foo # bar +s.biz # baz +``` + +##### More data +This is a rough quickstart introduction to Nix. +For more detailed information I recommend these resources. + +* [Official Guide][nix_guide_official] +* [Nix Pills][nix_guide_pills] ### Flakes @@ -54,4 +107,6 @@ An example using different versions of Firefox: [nixos_skynet]: https://forgejo.skynet.ie/Skynet/nixos -[nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf \ No newline at end of file +[nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf +[nix_guide_official]: https://nix.dev/tutorials/first-steps/ +[nix_guide_pills]: https://nixos.org/guides/nix-pills/# \ No newline at end of file -- 2.46.1 From a0c503896aca652db2c4e3d4ac90e4b3a757d3d5 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Wed, 16 Oct 2024 11:37:26 +0100 Subject: [PATCH 04/11] assignment: added an example package --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index 27b759b..c3eabc5 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d68d0b80a4c4150a73887f196b992c2ce1a75dcddae7d6b80705d241cd5e69d0 -size 344824 +oid sha256:35f765ba7020634d0dd962b1e0cec26d12cb746cd2580928eeb12eb18fa2a46d +size 349027 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 094cdd9..2b6143c 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -23,6 +23,18 @@ This means that multiple versions of the program (some even the same version but An example using different versions of Firefox: ![img.png](nix/firefox_co-existing.png) +##### Example +An example of packaging an application can be found here: +[Sieve Editor GUI on Nixpkgs][nix_pkgs_sieve] + +This is packaging up a GUI node.js application. +The application itself allows the user to edit sieve scripts. +Once you have [downloaded and installed](#download) Nix you will be able to install and run it like so: +```shell +nix-shell -p sieve-editor-gui +sieve-editor-gui . +``` + #### Language There are two partially difficult problems in computer science: 1. Off by one errors @@ -80,6 +92,7 @@ For more detailed information I recommend these resources. * [Nix Pills][nix_guide_pills] ### Flakes + ### Nixos @@ -109,4 +122,5 @@ For more detailed information I recommend these resources. [nixos_skynet]: https://forgejo.skynet.ie/Skynet/nixos [nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf [nix_guide_official]: https://nix.dev/tutorials/first-steps/ -[nix_guide_pills]: https://nixos.org/guides/nix-pills/# \ No newline at end of file +[nix_guide_pills]: https://nixos.org/guides/nix-pills/# +[nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix \ No newline at end of file -- 2.46.1 From e749191a47478c4672e0cb1cd3be355fd34d2592 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Wed, 16 Oct 2024 11:45:41 +0100 Subject: [PATCH 05/11] assignment: added section on flakes --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 9 +++++++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index c3eabc5..c5be389 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35f765ba7020634d0dd962b1e0cec26d12cb746cd2580928eeb12eb18fa2a46d -size 349027 +oid sha256:4c3be06c72432a2bcd9d057a97c7506e8b1f4065460fee1283ec5142531749ff +size 353210 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 2b6143c..72c13ef 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -92,7 +92,11 @@ For more detailed information I recommend these resources. * [Nix Pills][nix_guide_pills] ### Flakes - +A Flake is one of teh best ways of interacting with nix. +Despite it having some issues and still being marked as experimental it has become a de-facto standard. +This is also the format that we use in Skynet. + +The [Official Wiki Page][nix_flake] will be more informative than what can be shoved into this article. ### Nixos @@ -123,4 +127,5 @@ For more detailed information I recommend these resources. [nix_paper]: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf [nix_guide_official]: https://nix.dev/tutorials/first-steps/ [nix_guide_pills]: https://nixos.org/guides/nix-pills/# -[nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix \ No newline at end of file +[nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix +[nix_flake]: https://wiki.nixos.org/wiki/Flakes \ No newline at end of file -- 2.46.1 From 741bd1d948f7465454186a0a39acf53b1050035c Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Wed, 16 Oct 2024 20:04:42 +0100 Subject: [PATCH 06/11] assignment: added section (roughly) on nixos --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 20 +++++++++++++++++++- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index c5be389..eaf503f 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c3be06c72432a2bcd9d057a97c7506e8b1f4065460fee1283ec5142531749ff -size 353210 +oid sha256:38d0593fccaba86da9f3c2cdfbdccc261e70e9194b3951772dd63932309f9bac +size 357908 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 72c13ef..67e2771 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -99,6 +99,23 @@ This is also the format that we use in Skynet. The [Official Wiki Page][nix_flake] will be more informative than what can be shoved into this article. ### Nixos +With teh package manager we are able to create packages in a deterministic manner and store them in a way that does not suffer path conflicts. + +Some (possibly crazy) folks saw this and decided to apply this to an entire operating system. +The advantages are clear, the required programs are added to teh path for teh current iteration of the system. +If any error arises the system can be rolled back to a previous config. + +Configuration is done via ``*.nix`` files, which are then converted into teh native config for the application in question. + +For example [this file][nix_dns] turns a list of attributes. +```nix +{ + record = "forgejo"; + r_type = "CNAME"; + value = "glados.skynet.ie"; +} +``` +Into a config usable by the BIND DNS server. ### Lix @@ -128,4 +145,5 @@ The [Official Wiki Page][nix_flake] will be more informative than what can be sh [nix_guide_official]: https://nix.dev/tutorials/first-steps/ [nix_guide_pills]: https://nixos.org/guides/nix-pills/# [nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix -[nix_flake]: https://wiki.nixos.org/wiki/Flakes \ No newline at end of file +[nix_flake]: https://wiki.nixos.org/wiki/Flakes +[nix_dns]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/applications/dns/dns.nix \ No newline at end of file -- 2.46.1 From 4389f2a3fa12279faf393322462a05b3d0eb72a8 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Fri, 18 Oct 2024 00:00:23 +0100 Subject: [PATCH 07/11] assignment: added section on Lix --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 23 +++++++++++++++++++++-- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index eaf503f..f1e0d7b 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38d0593fccaba86da9f3c2cdfbdccc261e70e9194b3951772dd63932309f9bac -size 357908 +oid sha256:8fb0159b7b034f4ef4096ba21efd5ac101a00b739f593fadd1c79f7911480602 +size 363383 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 67e2771..23ae6a2 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -43,7 +43,8 @@ There are two partially difficult problems in computer science: Nix falls into this last pitfall. The programming language used by teh Nix package manager is called Nix, not Nixlang (as like Erlang) but rather the same name as primary tool that uses it. -For clarity for teh remainder of this subsection we are only talking about Nix the language. +For clarity for teh remainder of this subsection we are only talking about Nix the language. +The Nix Package manager is sometimes known as CppNix for reasons we will get into later. Nix is a lazily evaluated functional language which al has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python. As a whole it takes strong influences from OCaml and other ML derived languages. @@ -118,6 +119,22 @@ For example [this file][nix_dns] turns a list of attributes. Into a config usable by the BIND DNS server. ### Lix +Nix is an old enough project now, and as such has accumulated crust over the years. +This is a combination of technical and societal/governance. + +On the technical side nix is built using c++ and a max of build systems that make it hard to expand it. +For a good long time the nix binary used in the package manager was locked at v2.18 due to issues. +It took most of a year for a higher version to be used on an official basis. + +Regarding governance there has been several attempts to make it better for folks to contribute and to decouple everything from requiring Eelco to have an input. +Those attempts did not succeed. +The final straw for some of the more technical core contributors was Eelco's forming a company, hiding it from the community and trying to get military sponsorship. +This did not vibe well with folks. + +Due to all of this many core maintainers forked Nix at 2.18 and started working to apply fixes for both code and organisational. +The result of their efforts is [Lix][nix_lix]. +For a full explanation of its key features I would like to point you to the [Lix About page][nix_lix_about]. +We use Lix instead of CppNix as the goals of Lix align with the viewpoints and ideologies that our members hold and what we want to represent as a (computer) society. ## Why we use it {Details of how the config was ascattered and hard to find} @@ -146,4 +163,6 @@ Into a config usable by the BIND DNS server. [nix_guide_pills]: https://nixos.org/guides/nix-pills/# [nix_pkgs_sieve]: https://github.com/NixOS/nixpkgs/blob/a3c0b3b21515f74fd2665903d4ce6bc4dc81c77c/pkgs/by-name/si/sieve-editor-gui/package.nix [nix_flake]: https://wiki.nixos.org/wiki/Flakes -[nix_dns]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/applications/dns/dns.nix \ No newline at end of file +[nix_dns]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/applications/dns/dns.nix +[nix_lix]: https://lix.systems/ +[nix_lix_about]: https://lix.systems/about/ \ No newline at end of file -- 2.46.1 From eef7877d25371d47fb815bdb6ea42c9a85d8bb6f Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Fri, 18 Oct 2024 01:36:52 +0100 Subject: [PATCH 08/11] assignment: added section on Why we use it. Also added strikethrough support --- _git.tar.gz | 4 ++-- mkdocs.yml | 5 ++++- src/skynet/nix.md | 30 ++++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 9 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index f1e0d7b..8a81847 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8fb0159b7b034f4ef4096ba21efd5ac101a00b739f593fadd1c79f7911480602 -size 363383 +oid sha256:6d132d72e9f11832b7b7ec8a4394df7ebaa50765c9da822b1be35eec1f4196d8 +size 370321 diff --git a/mkdocs.yml b/mkdocs.yml index a198ca1..0fd35f5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,4 +30,7 @@ plugins: - git-revision-date-localized: enable_creation_date: true markdown_extensions: - - tables \ No newline at end of file + - tables + - pymdownx.caret + - pymdownx.mark + - pymdownx.tilde \ No newline at end of file diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 23ae6a2..80613c4 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -38,8 +38,8 @@ sieve-editor-gui . #### Language There are two partially difficult problems in computer science: 1. Off by one errors -2. Caching -3. Naming things + 2. Caching + 3. Naming things Nix falls into this last pitfall. The programming language used by teh Nix package manager is called Nix, not Nixlang (as like Erlang) but rather the same name as primary tool that uses it. @@ -90,7 +90,7 @@ This is a rough quickstart introduction to Nix. For more detailed information I recommend these resources. * [Official Guide][nix_guide_official] -* [Nix Pills][nix_guide_pills] + * [Nix Pills][nix_guide_pills] ### Flakes A Flake is one of teh best ways of interacting with nix. @@ -137,8 +137,24 @@ For a full explanation of its key features I would like to point you to the [Lix We use Lix instead of CppNix as the goals of Lix align with the viewpoints and ideologies that our members hold and what we want to represent as a (computer) society. ## Why we use it -{Details of how the config was ascattered and hard to find} -{Also embracing devops and reduced manpower} +Back in [January 2023 we got disconnected from teh internet][skynet_disconnect]. +For the purposes of this document the root cause is not of importance. +What *is* the mad Indian Jones ~~treasure~~ config hunt that it triggered. +![Brendan delving for configs][skynet_disconnect_reenactment] +Additionally ITD require us to update our systems regularly (as they should be). +Technically this config delving is an ongoing effort, the old hard drives are occasionally connected up and raided. + +As you would imagine this is not ideal, a exasperating factor was that many programs had changed both the location and format of their configuration over the decades that Skynet has been using them. + +This is where teh strengths of NixOS lie. +The config for the entire cluster is located in a singular location. +Using modules which work as a translation layer if the requirements for the programs change this dont impact us. +Not to mention we can fearlessly (and regularly) update our systems it is a match made is heaven. + +Currently the Skynet cluster is comprised of 15 servers which have NixOS on them. +A combination of LXC's (Linux Containers) and physical bare metal servers. +Since these systems share a base config (with tehir individual applications layered on top) we are able to efficiently build them, building a package for one will also built it for other servers. +Combine that with teh ability to deploy them via our own selfhosted CI/CD we have a strong foundation to work off of. ## How we use it ### Requirements @@ -165,4 +181,6 @@ We use Lix instead of CppNix as the goals of Lix align with the viewpoints and i [nix_flake]: https://wiki.nixos.org/wiki/Flakes [nix_dns]: https://forgejo.skynet.ie/Skynet/nixos/src/branch/main/applications/dns/dns.nix [nix_lix]: https://lix.systems/ -[nix_lix_about]: https://lix.systems/about/ \ No newline at end of file +[nix_lix_about]: https://lix.systems/about/ +[skynet_disconnect]: https://public.skynet.ie/postmortem/2023-01-12_Loss-of-network-access.html +[skynet_disconnect_reenactment]: https://forgejo.skynet.ie/Computer_Society/presentations_compsoc/media/branch/main/src/slides/skynet/0_intro_img1.png \ No newline at end of file -- 2.46.1 From 029a181e7e1c58fd6ff3e768515948df073b6412 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Fri, 18 Oct 2024 02:49:26 +0100 Subject: [PATCH 09/11] assignment: added section on basic use cases including devshell. --- _git.tar.gz | 4 +-- src/skynet/nix.md | 72 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 66 insertions(+), 10 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index 8a81847..e77017d 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d132d72e9f11832b7b7ec8a4394df7ebaa50765c9da822b1be35eec1f4196d8 -size 370321 +oid sha256:d9b374501f6d61ccc6c223e50623dbf3735912c8ec28a1629184b0f213efb9ab +size 377633 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 80613c4..7ca0654 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -157,19 +157,66 @@ Since these systems share a base config (with tehir individual applications laye Combine that with teh ability to deploy them via our own selfhosted CI/CD we have a strong foundation to work off of. ## How we use it -### Requirements -{add warnign that git and git-lfs should also need to be in teh path} +Nix cannot run on Windows, though it cna be installed into WSL. +Please refer to the below downloads to get a base system up and running -### Download +* [Git][git_git] + * As we are using a git repo this is a hard requirement. +* [Git LFS][git_lfs] + * For storing non text files such as images. +* [Nix][nix_install] + * Cant really use nix without thi installed. + +### Dev Shell +After cloning the repo use ``nix develop`` to set up a terminal shell with the environment for working with Skynet NixOS. +It (currently) adds [Colmena][dev_colmena], [Agenix][dev_agenix] and [Attic][dev_attic]. + +Another example of a dev shell can be [our discord bot][dev_discord-bot]. +This one sets up a rust enviroment. + +Of course you can also look at the dev shell for this [wiki][dev_wiki]. +For the wiki we need to ensure that the dependencies for building it are present for all users so tehy dont have to figure out how to manually install them. ### Colmena -#### Local -##### Building +[Colmena][dev_colmena] is our build and deployment tool. -##### Repl +Building is pretty easy, just run ``colmena build``. +Downside of that command is it will try to build everything all at once which is a *lot*. +A more practical approach is to build a single server or a group of servers. +```shell +# build the Skynet server, names cna be found in teh flake.nix +colmena build --on skynet -#### Deployment +# build a group of servers, in this case any one with teh tag of active-core +colmena build --on @active-core +``` +To be able to deploy to Skynet two things are required. +* Be on teh UL network + * This can also be accomplished by being on the VPN. +* Have an ssh key configured that can access the servers + * TLDR be an admin. + +### Agenix +[Agenix][dev_agenix] is our secrets manager. +To be able to use this tool your pub ssh key must be in ``secrets/secrets.nix``. +As you would expect this is an admin only tool. +This file also defines the names and permissions for each secret. + +```shell +# have to be in teh secrets folder for all these commands. +cd secrets + +# edit the secret +agenix -e path/to/secret.age + +# re-key all secrets, this is done when a new key is added or removed. +agenix -r +``` + +### Attic +[Attic][dev_attic] is teh tool that we use for our nix cache (hosted at ). +It is not often it is used by the dev and its own [documentation][dev_attic_docs] covers it best. @@ -183,4 +230,13 @@ Combine that with teh ability to deploy them via our own selfhosted CI/CD we hav [nix_lix]: https://lix.systems/ [nix_lix_about]: https://lix.systems/about/ [skynet_disconnect]: https://public.skynet.ie/postmortem/2023-01-12_Loss-of-network-access.html -[skynet_disconnect_reenactment]: https://forgejo.skynet.ie/Computer_Society/presentations_compsoc/media/branch/main/src/slides/skynet/0_intro_img1.png \ No newline at end of file +[skynet_disconnect_reenactment]: https://forgejo.skynet.ie/Computer_Society/presentations_compsoc/media/branch/main/src/slides/skynet/0_intro_img1.png +[git_git]: https://git-scm.com/downloads +[git_lfs]: https://git-lfs.com/ +[nix_install]: https://nixos.org/download/ +[dev_colmena]: https://colmena.cli.rs/unstable/ +[dev_agenix]: https://github.com/ryantm/agenix +[dev_attic]: https://github.com/zhaofengli/attic +[dev_attic_docs]: https://docs.attic.rs/introduction.html +[dev_discord-bot]: https://forgejo.skynet.ie/Skynet/discord-bot/src/commit/80c9191eeec29ba20ef4084713eca7fe0cab7412/flake.nix#L65 +[dev_wiki]: https://forgejo.skynet.ie/Skynet/wiki/src/commit/ab0add44756d4992fc2b2da4eba163016ccb3d1c/flake.nix#L35 -- 2.46.1 From fcac8c2448da3bc3542346eb25dea299ae142673 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Fri, 18 Oct 2024 02:57:29 +0100 Subject: [PATCH 10/11] assignment: formatting and typos --- _git.tar.gz | 4 +-- src/skynet/nix.md | 74 ++++++++++++++++++++++++----------------------- 2 files changed, 40 insertions(+), 38 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index e77017d..606b096 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9b374501f6d61ccc6c223e50623dbf3735912c8ec28a1629184b0f213efb9ab -size 377633 +oid sha256:c309ee38fc47772caa79d06f06ce8789c4872af0b35fbc30e589fbd2e7112751 +size 384902 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index 7ca0654..e29f33a 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -11,11 +11,11 @@ Nix grew out of a [PhD by Eelco Dolstra][nix_paper] wherein he proposes a slight For most Linux systems programs make use of other software installed on the computer, for the most part this works fine. Where issue may arise is if one program needs to update one of these dependencies, specially a minor or major patch where backwards compatibility is not guaranteed. -If another program is using this (system wide) dependency then it may run into interface issues when using it. +If another program is using this (system-wide) dependency then it may run into interface issues when using it. In a sense updating one program can break another on the system. The route the Nix package manager takes is it treats each program as a function. -Using teh Nix language a function for that package is created which states what inputs are required, what is needed to turn those inputs into teh program as well as the name for the output. +Using the Nix language a function for that package is created which states what inputs are required, what is needed to turn those inputs into the program as well as the name for the output. The output is then saved in a read only location in the format of ``/nix/store/$hash-program-name-version``. This output can either be used as the input of another program or be used as is by the system/user. Using this format means that any change in the inputs or the program itself will result in a different output. @@ -29,7 +29,7 @@ An example of packaging an application can be found here: This is packaging up a GUI node.js application. The application itself allows the user to edit sieve scripts. -Once you have [downloaded and installed](#download) Nix you will be able to install and run it like so: +Once you have [downloaded and installed](#how-we-use-it) Nix you will be able to install and run it like so: ```shell nix-shell -p sieve-editor-gui sieve-editor-gui . @@ -37,13 +37,14 @@ sieve-editor-gui . #### Language There are two partially difficult problems in computer science: -1. Off by one errors - 2. Caching - 3. Naming things + +1. Off-by-one errors +2. Caching +3. Naming things Nix falls into this last pitfall. -The programming language used by teh Nix package manager is called Nix, not Nixlang (as like Erlang) but rather the same name as primary tool that uses it. -For clarity for teh remainder of this subsection we are only talking about Nix the language. +The programming language used by the Nix package manager is called Nix, not NixLang (as like Erlang) but rather the same name as primary tool that uses it. +For clarity for the remainder of this subsection we are only talking about Nix the language. The Nix Package manager is sometimes known as CppNix for reasons we will get into later. Nix is a lazily evaluated functional language which al has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python. @@ -78,7 +79,7 @@ mul 7 6 ##### Attribute Sets In most languages the way to group data would be either an Object or a Struct. -Nix has a similar datastructure: +Nix has a similar data structure: ```nix s = { foo = "bar"; biz = "baz"; } s.foo # bar @@ -90,23 +91,23 @@ This is a rough quickstart introduction to Nix. For more detailed information I recommend these resources. * [Official Guide][nix_guide_official] - * [Nix Pills][nix_guide_pills] +* [Nix Pills][nix_guide_pills] ### Flakes -A Flake is one of teh best ways of interacting with nix. +A Flake is one of the best ways of interacting with nix. Despite it having some issues and still being marked as experimental it has become a de-facto standard. This is also the format that we use in Skynet. The [Official Wiki Page][nix_flake] will be more informative than what can be shoved into this article. ### Nixos -With teh package manager we are able to create packages in a deterministic manner and store them in a way that does not suffer path conflicts. +With the package manager we are able to create packages in a deterministic manner and store them in a way that does not suffer path conflicts. Some (possibly crazy) folks saw this and decided to apply this to an entire operating system. -The advantages are clear, the required programs are added to teh path for teh current iteration of the system. +The advantages are clear, the required programs are added to the path for the current iteration of the system. If any error arises the system can be rolled back to a previous config. -Configuration is done via ``*.nix`` files, which are then converted into teh native config for the application in question. +Configuration is done via ``*.nix`` files, which are then converted into the native config for the application in question. For example [this file][nix_dns] turns a list of attributes. ```nix @@ -137,45 +138,45 @@ For a full explanation of its key features I would like to point you to the [Lix We use Lix instead of CppNix as the goals of Lix align with the viewpoints and ideologies that our members hold and what we want to represent as a (computer) society. ## Why we use it -Back in [January 2023 we got disconnected from teh internet][skynet_disconnect]. +Back in [January 2023 we got disconnected from the internet][skynet_disconnect]. For the purposes of this document the root cause is not of importance. What *is* the mad Indian Jones ~~treasure~~ config hunt that it triggered. ![Brendan delving for configs][skynet_disconnect_reenactment] -Additionally ITD require us to update our systems regularly (as they should be). +Additionally, ITD require us to update our systems regularly (as they should be). Technically this config delving is an ongoing effort, the old hard drives are occasionally connected up and raided. -As you would imagine this is not ideal, a exasperating factor was that many programs had changed both the location and format of their configuration over the decades that Skynet has been using them. +As you would imagine this is not ideal, an exasperating factor was that many programs had changed both the location and format of their configuration over the decades that Skynet has been using them. -This is where teh strengths of NixOS lie. +This is where the strengths of NixOS lie. The config for the entire cluster is located in a singular location. -Using modules which work as a translation layer if the requirements for the programs change this dont impact us. +Using modules which work as a translation layer if the requirements for the programs change this don't impact us. Not to mention we can fearlessly (and regularly) update our systems it is a match made is heaven. -Currently the Skynet cluster is comprised of 15 servers which have NixOS on them. +Currently, the Skynet cluster comprises 15 servers which have NixOS on them. A combination of LXC's (Linux Containers) and physical bare metal servers. -Since these systems share a base config (with tehir individual applications layered on top) we are able to efficiently build them, building a package for one will also built it for other servers. -Combine that with teh ability to deploy them via our own selfhosted CI/CD we have a strong foundation to work off of. +Since these systems share a base config (with their individual applications layered on top) we are able to efficiently build them, building a package for one will also build it for other servers. +Combine that with the ability to deploy them via our own self-hosted CI/CD we have a strong foundation to work off of. ## How we use it Nix cannot run on Windows, though it cna be installed into WSL. Please refer to the below downloads to get a base system up and running * [Git][git_git] - * As we are using a git repo this is a hard requirement. + * As we are using a git repo this is a hard requirement. * [Git LFS][git_lfs] - * For storing non text files such as images. + * For storing non text files such as images. * [Nix][nix_install] - * Cant really use nix without thi installed. + * Cant really use nix without thi installed. ### Dev Shell After cloning the repo use ``nix develop`` to set up a terminal shell with the environment for working with Skynet NixOS. It (currently) adds [Colmena][dev_colmena], [Agenix][dev_agenix] and [Attic][dev_attic]. Another example of a dev shell can be [our discord bot][dev_discord-bot]. -This one sets up a rust enviroment. +This one sets up a rust environment. -Of course you can also look at the dev shell for this [wiki][dev_wiki]. -For the wiki we need to ensure that the dependencies for building it are present for all users so tehy dont have to figure out how to manually install them. +Of course, you can also look at the dev shell for this [wiki][dev_wiki]. +For the wiki we need to ensure that the dependencies for building it are present for all users so they don't have to figure out how to manually install them. ### Colmena [Colmena][dev_colmena] is our build and deployment tool. @@ -184,18 +185,19 @@ Building is pretty easy, just run ``colmena build``. Downside of that command is it will try to build everything all at once which is a *lot*. A more practical approach is to build a single server or a group of servers. ```shell -# build the Skynet server, names cna be found in teh flake.nix +# build the Skynet server, names cna be found in the flake.nix colmena build --on skynet -# build a group of servers, in this case any one with teh tag of active-core +# build a group of servers, in this case any one with the tag of active-core colmena build --on @active-core ``` -To be able to deploy to Skynet two things are required. -* Be on teh UL network - * This can also be accomplished by being on the VPN. +To be able to deploy to Skynet two things are required: + +* Be on the UL network + * This can also be accomplished by being on the VPN. * Have an ssh key configured that can access the servers - * TLDR be an admin. + * TLDR be an admin. ### Agenix [Agenix][dev_agenix] is our secrets manager. @@ -204,7 +206,7 @@ As you would expect this is an admin only tool. This file also defines the names and permissions for each secret. ```shell -# have to be in teh secrets folder for all these commands. +# have to be in the secrets folder for all these commands. cd secrets # edit the secret @@ -215,7 +217,7 @@ agenix -r ``` ### Attic -[Attic][dev_attic] is teh tool that we use for our nix cache (hosted at ). +[Attic][dev_attic] is the tool that we use for our nix cache (hosted at ). It is not often it is used by the dev and its own [documentation][dev_attic_docs] covers it best. -- 2.46.1 From 8212fa77809f8a0bbf85bb7073f6361136d0c672 Mon Sep 17 00:00:00 2001 From: Brendan Golden Date: Fri, 18 Oct 2024 17:47:29 +0100 Subject: [PATCH 11/11] assignment: fix, improvements suggested --- _git.tar.gz | 4 ++-- src/skynet/nix.md | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/_git.tar.gz b/_git.tar.gz index 606b096..21bff93 100644 --- a/_git.tar.gz +++ b/_git.tar.gz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c309ee38fc47772caa79d06f06ce8789c4872af0b35fbc30e589fbd2e7112751 -size 384902 +oid sha256:db1b1f700ddb6190c672e09108a1e14f88f99df904b19b25afc6f649d2f02f51 +size 392511 diff --git a/src/skynet/nix.md b/src/skynet/nix.md index e29f33a..7b0f93e 100644 --- a/src/skynet/nix.md +++ b/src/skynet/nix.md @@ -10,7 +10,7 @@ These are deeply interlinked together with the language being how the package ma Nix grew out of a [PhD by Eelco Dolstra][nix_paper] wherein he proposes a slightly different way to manage dependencies on a system. For most Linux systems programs make use of other software installed on the computer, for the most part this works fine. -Where issue may arise is if one program needs to update one of these dependencies, specially a minor or major patch where backwards compatibility is not guaranteed. +Where issue may arise is if one program needs to update one of these dependencies, specifically a minor or major patch where backwards compatibility is not guaranteed. If another program is using this (system-wide) dependency then it may run into interface issues when using it. In a sense updating one program can break another on the system. @@ -47,7 +47,7 @@ The programming language used by the Nix package manager is called Nix, not NixL For clarity for the remainder of this subsection we are only talking about Nix the language. The Nix Package manager is sometimes known as CppNix for reasons we will get into later. -Nix is a lazily evaluated functional language which al has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python. +Nix is a lazily evaluated functional language which also has REPL (Read, Evaluate, Print, and Loop) capability like what you would see in Python. As a whole it takes strong influences from OCaml and other ML derived languages. ##### Types @@ -185,7 +185,7 @@ Building is pretty easy, just run ``colmena build``. Downside of that command is it will try to build everything all at once which is a *lot*. A more practical approach is to build a single server or a group of servers. ```shell -# build the Skynet server, names cna be found in the flake.nix +# build the Skynet server, names can be found in the flake.nix colmena build --on skynet # build a group of servers, in this case any one with the tag of active-core @@ -218,7 +218,7 @@ agenix -r ### Attic [Attic][dev_attic] is the tool that we use for our nix cache (hosted at ). -It is not often it is used by the dev and its own [documentation][dev_attic_docs] covers it best. +It is not often used by the developer/admin and its own [documentation][dev_attic_docs] covers it best. -- 2.46.1