In the pioneering spirit of exploration and discovery, amidst skeptics and freedom advocates, a new vision emerges. It's time to redefine our financial future.

As the era of digitization advances at a breathtaking pace, we find ourselves at the threshold of a global financial metamorphosis. The transformation of traditional systems into the digital realm will bring us Central Bank Digital Currencies (CBDCs) – digital Euros, digital Dollars, and the like. But this moment prompts an essential question: Should we accept this change passively, or should we aspire to something more significant, something truly revolutionary?

This is for all those who see the profound value in decentralized digital cash. We, who value freedom of choice and harbor skepticism towards the looming dominance of CBDCs. We, who desire to change the system, to make a difference.

We're builders, tinkerers, high achievers and sometimes procrastinators. We believe in our collective capacity to conceive and develop an improved version of digital cash: open, inclusive, and free for everyone.

The Power of Decentralized Digital Cash

Public trust in financial institutions has been steadily eroding over time. There's a growing discomfort towards the level of control that centralized entities could potentially exert over a digital currency. A decentralized version of cash stands as a viable solution. It implies that no single authority can control, manipulate, or decide who has the right to use it.

Free from central authority, digital cash embodies the essence of economic sovereignty and personal freedom. It's available to anyone with internet access, not discriminating based on location, identity, or wealth. It's an empowering tool for individuals, rather than an instrument of control for institutions.

Dreaming Up the Future of Money

The path forward is filled with challenges and complexities. We need the collective intelligence and creativity of many to tackle issues of scalability, privacy, security, and user experience. Yet, there's an enduring belief that together, we can build this system. We can create digital cash that is as free as the air we breathe, as universal as human rights, and as groundbreaking as the invention of the internet itself.

To every builder and tinkerer brave enough to reimagine our financial systems, to every high achiever who won't settle for the status quo, to every procrastinator waiting for the perfect moment to make a mark - this is your call to action. It's your chance to contribute to something transformative, something monumental.

Together, we can redefine money, not as a tool of control but as a means of empowerment. Where access to financial services is a right, not a privilege. With technology, we have the power to not merely replicate financial inequalities of the past, but to create a future of financial inclusion and freedom.

People like us, build things like this. Join in to shape our future!

Introduction:

In this blog post, we'll dive into Nano and we'll learn how to us the nano_node RPC to receive our first nano transaction and forward it to a new address.

Nano is a near-instant cryptocurrency with no fees and a small ecological footprint.
To interact with the network, the nano_node software exposes a variety of remote procedure calls (RPCs) that let you browse the ledger, create Nano transactions and more.

To help me with this task, I asked ChatGPT how to use the Nano RPC to create a wallet and receive a transaction from a faucet.
While the example it created wasn't quite perfect, it was easy to make it work with a few adjustments.

All the additions and corrections I made are highlighted like this.

If you want to follow along and try out the Nano RPC for yourself, you can

• use the following Postman collection with all the relevant commands or
• watch my video where I go trough the whole process in about 5 minutes.

I created a fully synced node with public access to the RPC so you don't need to setup your own nano_node.
If you're interested in setting up your own Nano node, you can read this blog article, or spin up a dockerized node with just one command.

Adjusted example created by ChatGPT

1) Creating a New Wallet

Follow along on your own : Postman collection

To create a new wallet using the Nano RPC, we will send a POST request to the endpoint https://rpcproxy.bnano.info/live with the following payload:

{
    "action": "wallet_create"
}

curl -X POST --data '{"action":"wallet_create"}' https://rpcproxy.bnano.info/live

The response will contain the seed for our new wallet:

{
    "wallet": "E861D8_truncated_for_brevity"
}

2) Adding an Account

To add an account to our wallet, we will send a POST request to the endpoint https://rpcproxy.bnano.info/live with the following payload:

{
    "action": "account_create",
    "wallet": "E861D8_truncated_for_brevity"
}

curl -X POST --data '{"action":"account_create","wallet":"E861D8_truncated_for_brevity"}' https://rpcproxy.bnano.info/live

The response will contain the private key and the public address for our new account:

{
    "accounts": [
        {
            "private": "45A3_truncated_for_brevity",
            "public": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4"
        }
    ]
}

3) Receiving a First Transaction from a Faucet

To receive funds, we will need to provide the public address of our account to the faucet.

You can visit https://freenanofaucet.com/faucet to receive some free nano

4) Check your account balance

Once the faucet has sent the funds, we can check the balance of our account using the following payload:

{   
	"action": "account_balance",   
	"account": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4" 
}

curl -X POST --data '{"action":"account_balance","account":"nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4"}' https://rpcproxy.bnano.info/live

The response will contain the current balance of the account:

{
    "balance": "0", //corrected version
    "pending": "12000000000000000000000000",
    "receivable": "12000000000000000000000000"
}

This is where chatGPT was wrong for the first time. Your account will show "balance": "0" . The funds are in a "receivable" state.
Before you can use and forward the funds to another account, you first need to actively receive what has been sent to your account.
I told ChatGPT it missed a step and asked to receive the funds. It got the receive command right, but listed the wrong arguments.

5) Query the receivable block

Again ChatGPT was missing a step. To receive the funds, we need to query the receivable blocks in our wallet first like this:

{  
 "action": "wallet_receivable",  
 "wallet": "E861D8_truncated_for_brevity" 
}

The response will contain all the accounts in our wallet with their receivable blocks

{
  "blocks": {
    "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4": ["AD628C1F_truncated_for_brevity"]
  }
} 

6) Create the receivable block

To receive the funds, we need to specify the source of the funds (i.e., the public address of the sender), the wallet and the account that the funds should be received into, and the amount of funds block being received.
Here is the payload for the "receive" action:

{  
 "action": "receive",  
 "wallet": "E861D8_truncated_for_brevity",  
 "account": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4",  //ChatGPT suggested "source" instead of "account"
 "block": "AD628C1F_truncated_for_brevity"  // ChatGPT suggested "amount" instead of "block"
}

And here is the curl command:

curl -X POST --data '{"action":"receive","wallet":"E861D8_truncated_for_brevity","account":"nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4","block":"AD628C1F_truncated_for_brevity"}' https://rpcproxy.bnano.info/live 

The response will contain a block hash, which can be used to track the confirmation of the transaction on the Nano network. The funds will now be available in your account and you can verify the balance using the "balance" action.

With just a few tweaks we got a working RPC example from ChatGPT.
We'll now look at the RPC commands to send our funds to a new account

7) Check our wallet

To see all the funds in our wallet, we can use the "wallet_info" command

{
    "action": "wallet_info",
    "wallet": "E861D8_truncated_for_brevity"
}

The result will list the total balance of all our accounts. In our current example, we only have a single account "accounts_count": "1"

{
    "balance": "12000000000000000000000000",
    "pending": "0",
    "receivable": "0",
    "accounts_count": "1",
    "accounts_block_count": "1",
    "accounts_cemented_block_count": "1",
    "deterministic_count": "1"
    ...  
}

8) Send to a new address

To send our funds to a new address we have to provide our wallet, the source account we want to send from, the destination account and the amount.
We can provide an optional "id" that makes sure that even if we execute the send command twice, we don't actually send our funds twice.

{
  "action": "send",
  "wallet": "E861D8_truncated_for_brevity",
  "source": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4",
  "destination": "nano_118tih7f81iiuujdezyqnbb9aonybf6y3cj7mo7hbeetqiymkn16a67w8rkp",
  "amount": "12000000000000000000000000",
  "id": "a_unique_id_62386187gd"
} 

The response is a block. We will query the content of the block in the next step

{
  "block": "6ADE56_truncated_for_brevity"
}

9) Query the content of our send block

To validate that our block has been confirmed by the network, we can use the "block_info" command, which will return all the information of our transaction.

{  
  "action": "block_info",
  "json_block": "true",
  "hash": "6ADE56_truncated_for_brevity"
}

The result has "confirmed" field that is "true" if the block has been confirmed by the network.
At this point the transaction is final and can't be reveresed.

{
  "block_account": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4",
  "amount": "12000000000000000000000000",
  "balance": "0",
  "height": "2",
  "local_timestamp": "0",
  "successor": "0000000000000000000000000000000000000000000000000000000000000000",
  "confirmed": "true",
  "contents": {
    "type": "state",
    "account": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4",
    "previous": "CE898C131AAEE25E05362F247760F8A3ACF34A9796A5AE0D9204E86B0637965E",
    "representative": "nano_3arg3asgtigae3xckabaaewkx3bzsh7nwz7jkmjos79ihyaxwphhm6qgjps4",
    "balance": "12000000000000000000000000",
    "link": "5D1AA8A45F8736519D707FCB375976A7F9AF795091021D7E9C7548D6F45DD8D5",
    "link_as_account": "nano_118tih7f81iiuujdezyqnbb9aonybf6y3cj7mo7hbeetqiymkn16a67w8rkp",
    "signature": "82D41BC16F313E4B2243D14DFFA2FB04679C540C2095FEE7EAE0F2F26880AD56DD48D87A7CC5DD760C5B2D76EE2C205506AA557BF00B60D8DEE312EC7343A501",
    "work": "8a142e07a10996d5"
  },
  "subtype": "send"
}

I hope you enjoyed. Make sure to try it for yourself!

Conclusion

In this article, we have seen how to interact with the Nano RPC to perform various actions such as creating a wallet, adding an account, receiving funds and forwarding the funds to a new address.
The Nano RPC is a powerful tool for developers looking to build applications on top of the Nano network, and the use of Postman or curl allows for quick and easy testing of the API.

If you like my work, please show your love 💛

Integration testing is an essential step in the development of any decentralised network. It verifies that the nodes in the network can communicate, and interact with each other as expected.
However, conducting these tests manually can be very time-consuming.

Nano-node-ci is a tool that streamlines the process of testing the nano node, ensuring its reliability and stability in a local network.
By automatically testing each new commit and pull request made to the nano-node repository, nano-node-ci helps identify any potential issues early on, so they can be corrected before deployment.

How nano-node-ci works

Nano-node-ci is triggered on each new commit and on each pull request made to the nano-node repository.

• The nano-node is then built using nano-node-builder (see blog post for more details)
• If a new build is detected, a test suite of 23 tests is executed.
  The results of these tests are then published to nano-node-ci, providing a comprehensive overview in a PASS/FAIL manner as shown in the screenshot below:

Each testrun generates 3 links :

• results : a simple text file with test results,
• telemetry : a dashboard with detailed node telemetry data for additional debugging information in case of any test case failure.
• graphs : a dashboard with a performance overview per test case

The following shows an example of the performance dashboard.
5 out of 23 tests are shown :

The Benefits of Automated Integration Testing

Examples of things that need to be tested during integration testing of the nano node include:

• Communication with other nodes in the network
• Synchronization of the nodes with the network
• Consensus participation (voting on transactions)
• Verification of transactions (signature validation)

The goal of nano-node-ci is to validate that new versions of the nano-node meet all the requirements above.

• It helps to identify any potential issue with new nano-node versions before they are deployed.
• This ensures that bugs can be corrected early in the process, improving the overall quality of the code.
• Additionally, by providing test results for each build, the nano-node performance can be tracked over time and regression can be identified quickly.

In conclusion, nano-node-ci provides integration testing to the nano node, ensuring its reliability and stability. Improved network reliability and increased development speed are key goals of nano-node-ci.

With reliable automated testing, potential issues can be identified and corrected early in the development process, leading to improved code quality, faster bug detection, better collaboration, and increased development speed.

The Beta Network serves as a testing ground where new nano-node versions are tested regularly.

Before a new version is live, one or more release candidates are deployed to the Beta Network and undergo a series of tests by beta node operators.
With around 10% of the node count of the live network, the Beta Network provides a real-world environment to validate new node versions.

In the past, these tests have been mainly manual.

• Users simulate high network traffic by publishing lots of blocks.
• Results are visually compared to testruns of previous versions.

Beta Tools (Explorer, Wallets, Visualizer)

Over the past six months, I have been working on improving the ease of use of the nano beta network and on introducing automated testing.
To this end, I have forked the respective GitHub repositories to have:

• Beta Block Explorer (cloned nanolooker.com)
• Beta Wallet (cloned nault.cc)
• Beta Network visualizer (cloned nanoticker.info)
• Beta Vote Visualizer (cloned nanovisual.numsu.dev

In addition, the Beta Network is used to validate node changes before a new version is released.
To aid in identifying and troubleshooting potential issues that occur during beta tests, we increased the detail of available node telemetry.
This includes dashboards with both simple and advanced telemetry provided by the nodes.

• Dashboard with simple telemetry provided by the nodes
• Dashboard with advanced telemetry pushed by node operators to a centralised database

With the help of these advanced metrics, we now can

• compare the behavior of beta nodes in detail,
• identify how well each beta node performs,
• test how changes in configurations impact the outcome of any given test,
• identify portential causes of any arising issue

Automated Beta testing

I introduced a small daily speedtest that publishes 7500 blocks on each run and measures the confirmations per second (cps) of the network.

In the graph below we can see the daily cps rate.

The speed-test itself is not 100% stable so the rate at which blocks are published to the network (bps) is not stable from day to day. It varies from about 300 to 1k bps
This explains the spikes in cps which follow the rate at which blocks are published.

If you like my work, please show your love 💛

In a traditionl application, telemetry data is usually used to give developers quick feedback of how a change impacts the production environment.
This way, by monitoring the telemetry data closely, a new change pushed to the application that causes unforseen problems in production is often caught and fixed before any customer is visibly impacted.

The nano-node provides telemetry data aswell.
I created a dashboard that monitors the telemetry data provided by the top#20 representatives. This way we have a tool that leverages the available information  so we can proactively look for and fix potential issues.

What data does the nano-node telemetry provide ?

The following is an example-response provided by the telemetry rpc request :

{
    "metrics": [
        {
            "block_count": "173834771",
            "cemented_count": "173834771",
            "unchecked_count": "0",
            "account_count": "30874506",
            "bandwidth_cap": "10485760",
            "peer_count": "208",
            "protocol_version": "18",
            "uptime": "9570208",
            "genesis_block": "991CF19009...AA4A6734DB9F19B728948",
            "major_version": "23",
            "minor_version": "3",
            "patch_version": "0",
            "pre_release_version": "0",
            "maker": "0",
            "timestamp": "1674466711662",
            "active_difficulty": "fffffff800000000",
            "node_id": "node_38d87w3o99k5np...nz36kui3re5",
            "signature": "745514A037472E2986DC755...370E1AF08",
            "address": "::ffff:159.XXX.XXX.206",
            "port": "7075"
        }
    ]
}        

The dashboard visualizes the following metrics for each node :

• checked blocks (and missing checked blocks)
• cemented blocks (and missing cemented blocks)
• unchecked blocks
• total accounts (and missing accounts)
• bandwidth limit
• peer count
• node version

Can we identify weak nodes by looking at the telemetry data ?

Identifying the best nodes by looking at the telemetry data is not obvious.
However identifying weks nodes seems possible with the metrics provided in the telemetry.
As an example, we will look at the  block_count graph .

During the daily speedtest performed by nanotps.com, 5000 blocks are published to all nodes.
What we see, is that most top#20 representatives check most of the blocks within 30 seconds, while 2 nodes take considerably longer.

An even better insight can be gained from looking at the following graph that shows the  missing block_count .

We see the same 2 outliners Trustable NN1 and Nano Charts. We can also see that the red line My Nano Ninjadisappears when the speedtest starts. This means that the connection to that node was lost and not telemetry data was reported. The node probably died down during the test and came back online about 4 minutes later.

Having the above in mind, we an look at how quickly nodes are able to cement the checked blocks.
A transaction is cemented by the node, as soon as it has gathered 67% of online vote weight from all online representives. It means that quorum has been reached and the transaction is final.
The following shows a graph of missing cemented blocks :

In the graph above, we can see that after 5 minutes, some nodes are still struggeling and have missing cemented blocks.

5 minutes after the speedtest of 5000 blocks :

• My Nano Ninja representative has 4813 missing cemented blocks.  (It's the node that stopped reporting telemetry data at the start of the speedtest)
• Trustable NN1 has 1698 missing cemented blocks.
• nanocafe.cc has 483 missing cemented blocks.
• Nano Chartscaught up well and has almost no (21) missing cemented blocks.
• RsNano.com has 293 missing cemented blocks, but it started with this many missing blocks even before the speedtest began. So overall it coped very well.

What can be done to improve the situtaion?

In a decentralized network every user has some responsibility to make sure the network operates as intended.

• Node providers can upgrade their hardware.
• Users can change their representatives if they delegate to a weak node.
• New nano-node versions can reduce hardware ressource usage.

If you like my work, please show your love 💛

• Clones the official nanocurrency/nano-node repository.
• Builds a docker image by using the official scripts.
• Pushes the docker image to dockerhub .
  The goal is to have very quick release cycles and to be able to update my node with the latest unreleased features and improvements

Is this an official release ?

No it's not.
The nano-node is officially maintained and improved by the Nano Foundation.
In the past, a new major release has dropped once or twice a year.
While this schedule is perfectly fine for exchanges, the downside of long release cycles is that bug-fixes are slow to be integrated into the live network.

New commits are pushed to the nano-node repository almost daily.

How do I run the latest node version ?

If you run a dockerized nano-node, just switch your current docker tag (nanocurrency/nano-node:V24.0) to gr0v1ty/nano-node:latest

Is this safe to run ?

There are 2 aspects to the question :

1. How can I be sure no malicious code has been injected into this version ?

• the whole process of cloning, building and pushing is done in a public github action workflow.
• it can be easily inspected and monitored for changes.

This way you can be sure that only 100% official code is used to created the docker images made available to dockerhub.

2. Should code that hasn't been tested on a beta network be run in production ?

• running the latest commit is NOT recommended for any principal representative.
• however, when facing issues with the official release or you require a specific fix that has already been committed but not released, this project helps you out.

Explain the workflow

The workflow can be found here
The following shows a flow diagram of what the workflow does

However none of these projects  are both simple and solved my needs.
I wanted to have self-hosted runners that meet the following requirements:

• If the workflow can be run with a github runner, it can be run with the self-hosted runner.
• It should be possible to define multiple runners in parallel.
• It must not share the PAT with our dedicated work-server (this requirement was not obvious at the beginning and added some complexity)

Currently the runner is limited to the scope of repositories. But you can add it o as many repositories as you want.
If you are looking to setup self-hosted runners with an organisation scope, the scripts have to be adjusted to meet that need!

Jump to the final solution

The journey begins ... Trial and error

My goal was to run a dockerized testsuite for the nano currency that spins up multiple docker-containers and writes some files to disk inside a self-hosted runner... Let's go!

Attempt 1 : Setup a simple github runner on a dedicated server

TLDR; This does not meet my needs

• I can't run multiple workflows in parallel without changing my current workflow
• I have to create additional cleanup logic before a new runner starts
• I can't use the default cleanup hooks provided by github runners, beacuse my dockerized testsuite writes files to disk as root

Full Experience
Setting up your self-hosted runner on a dedicated server is as easy as following the github example :
1) Download

# Create a folder   
$ mkdir actions-runner && cd actions-runner 
# Download the latest runner package   
$ curl -o actions-runner-osx-x64-2.300.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.300.2/actions-runner-osx-x64-2.300.2.tar.gz 
# Optional: Validate the hash   
$ echo "59814d103186d379123da8d2e7b002305a7b57f509fdd0cf34e4f86394dae9a4 actions-runner-osx-x64-2.300.2.tar.gz" | shasum -a 256 -c 
# Extract the installer   
$ tar xzf ./actions-runner-osx-x64-2.300.2.tar.gz

2) Configure

# Create the runner and start the configuration experience   
$ ./config.sh --url https://github.com/{owner}/{repo} --token AUN... 
# Last step, run it!   
$ ./run.sh

3) Using your self-hosted runner

# Use this YAML in your workflow file for each job   
runs-on: self-hosted

In my workflow all I have to do is to change from runs-on: ubuntu-22.04 to runs-on: self-hosted and the workflow is executed

jobs:
  my_job_name:
    #runs-on: ubuntu-22.04
    runs-on: self-hosted

On the first run, this works fine. However on the second run the workflow fails. The self-hosted runners don't clean up after the workflow has ended.
In my case this means :

• docker-containers are still running
• the _work folder used by default by the github runner is not emptied

So the major challenge with this approach lies in defining some cleanup logic per workflow. Even after I created all the cleanup logic, the workflow still failed to execute successfully on a second run. What happened ?

Self-hosted runners provide some hooks which allow a script to be executed before or after the workflow is run.
The easiest way is to create a file named .env within the self-hosted runner application directory with the following content

ACTIONS_RUNNER_HOOK_JOB_STARTED=/cleanup_before_workflow_starts.sh
ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/cleanup_after_workflow_ends.sh

Inside cleanup_before_workflow_starts.sh you'd write all the actions required before a workflow runs.
Inside cleanup_after_workflow_ends.sh you'd write all the actions required after a workflow has ended.
These scripts can be located anywhere. In this case they are located  within  the self-hosted runner application directory

So what I did was :

• Create a script that deletes all the files inside _work folder
• Modify my workflow to stop all docker containers at the end

My self-hosted runners use a special user with restricted rights.
My dockerized testsuite executes as root and writes files to disk as root.
So the cleanup hooks performed by the github runner did not have the required permissions to remove all files inside the _work folder created by the testsuite.

I stopped investigating further. I could have modified the dockerized testsuite to run with the same user as my self-hosted github runner. However the workflow runs correctly on github, so there must be a better way to make it work on my self-hosted runner.

Attempt 2: Dockerized self-hosted github runners

TLDR; This does not work either

• running docker containers inside a dockerized github runner doesn't work well
• the docker containers of my testsuite are still visible on the host machine
• files written to disk by the testsuite are still shared with the host

Full Experience
While on the surface it looked very easy to setup a dockerized runner it only works well when you run non-dockerized workflows .
Running my dockerized testsuite inside a dockerized github-runner comes with a range of problems that are not trivial to work around.

• One major limitation is mounting volumes. The mount path of the volume depends on the host machines’ absolute path instead of the github-runner relative path.
• Another limitation is that dockerized github runner and the dockerized testsuite don’t share the same network by default. This means that we can’t curl into our testsuite.

Attemp 3 : self-hosted github runner inside LXD - almost there

TLDR; Almost... If you are comfortable with sharing your PAT with your work-server, this solution is for you!

2 of 3 criteria are met :

• the dockerized testsuite runs correctly on each run
• we can set an arbitrary number of github runners
• However the  repo PAT is shared with our dedicated work-server

This is an overview of the architecture

Full experience
Linux containers  (LXD) is a next generation system container and virtual machine manager.
Linux containers can easily host docker containers.

I found an interesting project that creates github runner in LXD with 2 simple scripts.
The first script(prepare-instance.sh)

• creates a base container with all the configuration needed for the self-hosted github runner.
• adds a valid github registration-token is added to the base container
  The second script (spawn.sh)
• defines the numer of parallel github runners
• creates a new LXD container with a unique name by copying the base container
• starts the container (the github runner is now active and waiting to receive a workflow)

I encountered 2 issues with these scripts :

• the disk space claimed by the workflow was not freed for new runners.
• the registration-token expires after 60 minutes

Github offers an api to create a registration-token from  a PAT.
So what I did was :

• simplify the prepare_instance script and remove the registration of the self-hosted runner
• merge the registration of the self-hosted runner with spawn script into a respawnscript.

The new respawn script takes among others a PAT as input argument and does the following :

• convert the PAT into registration-token needed to register a container as self-hosted runner
• update registration-token of the base container. So this container always has a valid registration-token
• the base container is always stopped, so it never actually starts as a self-hosted github runner
• respawn any missing self-hosted github runners based on the parallelism that is defined
• add the possibility to run workers for multiple repos

So if you feel comfortable sharing the repo PAT with your dedicated work-server, this solution works very well.
Simply define a cronjob tha runs each minute to spin up new runners, if needed and renew the registration-token.

$ crontab -e
## Add the following line as cronjob. Replace the path and RUNNER_COUNT (=parallel runners)
* * * * * /path/to/lxd-runner/respawn gh-runner RUNNER_COUNT https://github.com/ORG/REPO PAT

The requirement to keep the PAT safe only came after this solution was implemented.
If you can't share the PAT with your dedicated work-server the next attempt will show you how to do it.

Final solution

Success on attempt 4 : self-hosted github runner inside LXD without compromising your PAT

The final solution that meets all the above requirements is available as github-project here

• it runs any workflow that runs on github runners
• you can define the number of active runners per repository
• your PAT never leaves github

This is an overview of the final architecture.
We will go through this picture step by step

TLDR;

1. Fork the project https://github.com/gr0vity-dev/lxd-github-runner or create your own github project to host the infrastructure of self-hosted runners
2. Install and configure your LXD environment
3. Create one LXD base-container per repo that needs self-hosted runners
4. Renew the registration-token needed to start a self-hosted runner via a scheduled github workflow

1) Fork the project

In our final solution, you'll have a github workflow that makes sure your self-hosted runners always have a valid registration token.
To run a self-hosted github runner, you need a registration token.
You can generate a valid registration-token via github api and your PAT.
By generating the registration inside a github actions workflow, you make sure that your PAT never leaves github

2) Install and configure your LXD environment

Install LXD

sudo apt-get install lxd-client

Initialise LXD and keep the defaults

lxd init

Add a new storge pool called "docker" (for docker to run properly in a Linux container it needs btrfs.) The storage name is "docker". The scripts rely on keeping that name
Allocate 50GB for all github runners (you might adjust it for your needs)

lxc storage create docker btrfs size=50GB

3) Create one LXD base-container for every repo that needs self-hosted runners

The  prepare_instance script creates the base images with all the required dependencies.
You might want to adjust the following lines of the script for your needs :

# By default,we use ubunutu 20 as the base container
./prepare-instance gh-runner-{repo-name}

! Make sure to specify a unique name per base-container. I recommend using the following : gh-runner-{repo-name}

4) Renew the registration-token needed to start a self-hosted runner via a scheduled github workflow

On github, setup a PAT (Personal access token) with access to the repos and orgs you want serviced.

Enable the scopes listed below for your PAT:

• repo
• workflow
• admin:repo_hook

The following shows an example of the workflow that

• renews our registration-token
• ssh's into our work-server to update the base-container with the valid token.

name: Renew ephemeral self-hosted runner token
on:
  schedule:
    - cron: '0/30 * * * *'  # "every 30 minutes
jobs:
  renew_gh-runner_token:
    name: renew reg token
    runs-on: ubuntu-latest
    steps:
      - name: Convert PAT into registration-token
        id: get_token
        run: |          
          GH_RUNNER_TOKEN=$(curl \
          --location --request POST 'https://api.github.com/repos/{OWNER}/{REPO}/actions/runners/registration-token' \
          --header 'Authorization: Bearer ${{ secrets.GH_PAT }}' \
          | jq -r '.token')
          echo "::add-mask::$GH_RUNNER_TOKEN"
          echo "::set-output name=GH_RUNNER_TOKEN::$GH_RUNNER_TOKEN"

      - name: Execute remote ssh commands using user and private key
        uses: appleboy/ssh-action@v0.1.7
        with:
          host: ${{ secrets.GH_RUNNER_HOST }}
          username: ${{ secrets.GH_RUNNER_USER }}
          key: ${{ secrets.GH_RUNNER_PRV_KEY }}
          script: ./git/lxd-github-actions/renew_runner_token {work-server-gh-runner-name} https://github.com/{OWNER}/{REPO} ${{ steps.get_token.outputs.GH_RUNNER_TOKEN }}

The workflow that updates my self-hosted runners can be found here

! Make sure to create the following repo secrets

- secrets.GH_PAT (your personal access token used to create a registration-token for your repo)
- secrets.GH_RUNNER_HOST (work server ip address)
- secrets.GH_RUNNER_USER (work-server user)
- secrets.GH_RUNNER_PRV_KEY (private key to ssh into the work-server)

! Additionally make sure to replace the following variables in the example above with the relevant content:

- {OWNER} # your github user
- {REPO} # your repo that requires the self-hosted runner
- {work-server-gh-runner-name} # the gh-runner name you defined when running the 'prepare-instance' script.

Explanation of why this workflow is needed :

• The registration-token expires after 60 minutes.
• The  workflow above runs every 30 minutes and updates the registration-token inside your base-container to make sure your token is always valid.
• The workflow ssh's into your work-server to execute the  renew_runner_token
  A PAT is still required but in this case it never leaves github.

4) Create and renew self-hosted runners as soon as a workflow has finished

The respawn_runners script copies the configuration of the base-container and inherits its valid registration-token.

Run this script as a cronjob to make sure you always have available runners

$ crontab -e
## Add the following line as cronjob. Replace the path and RUNNER_COUNT (=parallel runners)
* * * * * /path/to/lxd-runner/respawn_runners gh-runner-{repo-name} RUNNER_COUNT https://github.com/ORG/REPO

! Make sure to replace the following variables in the above cronjob:

gh-runner-{repo-name} # the name of your base-container
ORG (your github user)
REPO (your github repo that nees the self-hosted runner)

If you like my work, please show your love 💛