Validator

Vote Account: NeodymeDFipD7eA1ShrLJAZTBdHWcFsDB9YkoHshZNk
Voting Identity: NdMV1C3XMCRqSBwBtNmoUNnKctYh95Ug4xb6FSTcAWr

Hardware

We’re running our validator on self-owned hardware located in a highly secure tier-3 colocation rack in Berlin.

In order to provide maximum resiliency we run a hot-spare validator in the same rack, as well as a hot-spare fallback validator at Latitude which we could switch to in cases of failures of both self-owned-hardware validators or the datacentre.

Software

Our servers are configured using NixOS, resulting in fully declarative and reproducible server configurations. This requires us to build the validator packages ourselves, which we have packaged for nix.

Client

We’re running an unmodified version of the upstream Jito client.

Upgrades

Software upgrades are done in a staggered manner: We’re updating one validator to a new version and then have this one become the active voter. The other validators will still run the old software in order to facilitate quick downgrades in should issues arise. When, after some time, we’re satisfied with the new version, it will be promoted to our main version and the fallback validators will receives updates upgrading to this version as well.

Monitoring

All of our validators, as well as the general chain state, are integrated into our monitoring, resulting in alerts being fired for any anomalies. Depending on the severity of the issue a critical PagerDuty alert (including calls to operators) will also be triggered.

Security

Keys

The individual standby keys of the validators have been generated using solana-keygen grind on one of our self-owned-hardware servers. The same goes for the shared voting identity NdMV1C3XMCRqSBwBtNmoUNnKctYh95Ug4xb6FSTcAWr.

The authorised withdrawer has been generated on a Ledger device and the recovery seedphrase is stored encrypted in a secure offline location.

The vanity vote account key NeodymeDFipD7eA1ShrLJAZTBdHWcFsDB9YkoHshZNk has been brute-forced on vast.ai servers, however as we did not want to leak the private key to whomever was hosting the server at that time, as well as for bruteforcing speed reasons, we decided to use a seeded account. This means that the server only saw the derivation base (DKiDHLVkGvBkzVKrNp3HbyZ4645sZN8ajAzezv9cTCFB) pubkey and brute-forced the seed 5qDYRQwJWxnt6bzK, allowing us to create the vote-account using hardware-wallet-only keys. You can find the related transaction here.

Operation

The colocation datacentre we’re hosting our validators has rigorous access controls requiring one to pass through multiple keycard and pin protected gateways in order to get to the rack. Only authorised infrastructure personnel at Neodyme have access to the rack.

Additionally only persons involved with validator operations have SSH-access to the validator servers.

The OS disks of all validators are fully encrypted, as is configured swap space. Runtime secrets, such as the individual standby identity and shared voting identity, are additionally encrypted using SOPS, resulting in them being version controlled in a secure manner. These secrets are decrypted at boot or configuration switch and only stored in a file on an unswapable tmpfs, resulting in unencrypted secrets never touching the disk.