ToC

• Overview
• Prerequisites
• First Steps
• Address Verification
• SAFE Multisig
• Operator Joining Flow
• Create your ENR
• DKG Ceremony
• Additional required configs
• Add Operator ID to .env
• Monitoring
• Start your Distributed Node
• Validator Deposit & Activation
• Claiming Rewards
• Exiting Validators
• Relay List
• Resources

Overview

Hi everyone, thank you for joining the 3rd round of testing for Lido’s integration of Obol based DVT on Holesky. All participating members have been split into various clusters that represent your group to operate DVs (distributed validators) and will represent an entry in the Lido Node Operator registry. Each cluster will coordinate in a corresponding Discord channel thread that will be used to set up a SAFE Multisig, represent an entry in the Lido Node Operator registry, coordinate a DKG ceremony, and run distributed validators on Holesky.

Prerequisites

• A server to run your node
• One you can give a static public IP to and open ports on
• Since Holesky is a high-stress network, we recommend the following specs if you’re going to use the template docker-repo that contains the Beacon Node and Execution Client. Charon alone uses negligible disk space of not more than a few MBs.
• 4 physical cores (8 virtual) with good CPU single-core performance
• 32GB RAM
• 1-2TB NVME storage
• The template docker-compose repo for lido DVs. This will simplify the testing process as support for various CL clients is under active development.
• Agreement on who will deploy the SAFE, stage transactions, and execute the transactions once approved. This person is referred to as the cluster coordinator. The cluster coordinator should have prior experience setting up a SAFE Multisig and operating an Obol DV cluster.
• Join the relevant Discord thread for your cluster. The name of the thread is also the name of your cluster that should be referenced where needed.

First Steps

When your team has chosen the cluster coordinator, please tag @kimonsh and @perrier. A form (linked below) will be used to collect each cluster member’s Holesky addresses that they plan to use to sign messages in the SAFE, sign the distributed validator config and to receive validator’s rewards to.

Each cluster member must submit the form and verify the address(es) they would like to use. Each cluster member should submit up to two addresses:

2. Optional: Your reward address. This will be the address added to the reward splitter contract. The splitter contract will evenly distribute rewards between all members of your cluster. This rewards address should be used if you would like to use an address other than the one used for your manager address to collect rewards. If you provide a reward address you will need to sign a message verifying ownership of it in addition to the manager address’s verification.

Each individual participant is responsible for the security and storage of the private key(s) related to these addresses. Please confirm that you have your seed phrase backed up and the address is secure before submitting.

Use this form to submit your address and verification link (see verification instructions below): https://forms.gle/rCKvBhU4AzDHgqpY8

Address Verification

Use the following guide to prepare and sign a message that will be used for verification of your chosen manager and (where applicable) reward address:

In case of using externally owned account (EOA):

1. Sign the message on mainnet using the text (add your specific details within <>: with the private key you’ll use as the signing key. One of the options is using Etherscan.
1. Go to https://etherscan.io/verifiedSignatures
2. Press the “Sign Message” button and connect your wallet
3. Input the address you’re verifying (your Individual Manager Address or Individual Rewards Address)
4. Enter your message using the following template (add your specific details within <>: “<my name> is joining Lido x Obol cluster <cluster name> with address <public address>”
5. Press sign message and if the message is successfully signed, publish it.
6. Submit the verified signature URL via the form (e.g. https://etherscan.io/verifySig/27336).

SAFE Multisig

Once the form is filled out, please tag @kimonsh and @perrier and wait for confirmation to proceed further.

Once each cluster members has submitted the form and Lido NOM team has approved, the addresses will be shared to the group and the cluster coordinator will create the 5/7 threshold SAFE Multisig using each cluster member’s individual manager address on Holesky (https://holesky-safe.protofire.io/welcome). When the SAFE is created, share the SAFE Holesky URL for your cluster to review and tag @kimonsh and @perrier. This address will be your cluster’s representation in the Lido Node Operator Registry on Holesky.

Each cluster should also pin a message with basic info in their threads:

<participant name>   :  <ETH address here>   | <participant cluster identifier>
<participant2 name>  :  <ETH address here>   | <participant3 cluster identifier>
<participant3 name>  :  <ETH address here>   | <participant3 cluster identifier>

repeat above 'n' times where 'n' = cluster size

The cluster identifier will be obtained at the end of the DKG (steps below). Update the message to reflect it once done.

During this time, the Simple DVT Module Committee will add your cluster to the Lido Node Operator Registry using the SAFE multisig you provided.

Operator Joining Flow

Create your ENR

All operators should familiarize themselves with the Obol Launchpad Quickstart Guide - Operator Journey and launchpad walkthrough video.

All cluster participants should have already shared their individual manager and reward addresses in their cluster-specific form.

To prepare for the distributed key generation (DKG) ceremony, you need to create an ENR. On the machine where you plan to run your Obol DVs, run the following commands to get your ENR.

# Clone this repo
git clone https://github.com/ObolNetwork/lido-charon-distributed-validator-node.git
 
# Change directory
cd lido-charon-distributed-validator-node
 
# Create your charon ENR private key, this will create a charon-enr-private-key file in the .charon directory
docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon" obolnetwork/charon:v0.17.2 create enr

You should see an output similar to the below:

Created ENR private key: .charon/charon-enr-private-key

enr:-JG4QGQpV4qYe32QFUAbY1UyGNtNcrVMip83cvJRhw1brMslPeyELIz3q6dsZ7GblVaCjL_8FKQhF6Syg-O_kIWztimGAYHY5EvPgmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQKzMe_GFPpSqtnYl-mJr8uZAUtmkqccsAx7ojGmFy-FY4N0Y3CCDhqDdWRwgg4u

Creating the Obol Distributed Validator

Each cluster config will be created by the Obol team who will send a specific invite code to each cluster as a URL link.

Operators will follow the invite link to their cluster set up page in the DV Launchpad, they will review the cluster configuration, input their previously-generated ENR, and sign a message using their individual manager address provided earlier to confirm the cluster configuration.

Here are the steps to follow in detail once on the invite link:

• Connect your wallet using the Holesky individual manager address provided in the form.
• Review the other operator manager addresses in the cluster and click Get Started to continue
• Review and accept the advisories
• Review the configuration - confirm that the withdrawal address and fee recipient match Lido’s requirements.
• The withdrawal address for Lido withdrawal credentials:

0xF0179dEC45a37423EAD4FaD5fCb136197872EAd9

• The Fee Recipient address for Lido Execution Layer Rewards Vault :

0xE73a3602b99f1f913e72F8bdcBC235e206794Ac8

• Input the ENR that you previously generated.
• Sign the following with your wallet
• The config hash. This is a hashed representation of all of the details for this cluster.
• Your own ENR. This signature authorises the key represented by this ENR to act on your behalf in the cluster.
• Wait for all the other operators in your cluster to do the same.

DKG Ceremony

Once all operators have successfully signed the operator configuration, the next step is to perform the Distributed Key Generation ceremony.

A command will be automatically displayed in the launchpad once all operators have signed the config. Running that command will start the DKG process for your node.

Please note this process requires semi-synchronous coordination between all cluster operators. Your cluster should determine a time window of ~ 6 hours that all members can meet to conduct this one-time process.

When you start the DKG ceremony, you shouldn’t stop the process until all members have completed the ceremony.

You can’t just run the command and then turn your machine off! You must wait for everyone to run the command and the DKG to finish (<60sec). However, you can let the command run in the background as it will retry the DKG until successful.

When the DKG ceremony has been completed, operators will see the following messages:

All peers connected, starting DKG ceremony
dkg Successfully completed DKG ceremony 🎉

The artifacts of the DKG ceremony will be created in the .charon folder for each member (inside their lido-distributed-validator-node repo).

• deposit-data.json file (the same across all operators)
• cluster-lock.json file (the same across all operators)
• validator_keys/ folder >> Unique to each operator, make a backup!

💡 Please ensure that ALL the operators have the above files created successfully before moving forward.

Additional required configs

Add Operator ID to .env

While the quick-start guide linked above is adequate for general-purpose deployments, this repository needs further setup to integrate with Lido’s smart contracts:

1. grab your Operator ID from https://operators-holesky.testnet.fi/ (it will be present after Simple DVT Module Committee adds your cluster to the registry)
2. Run the following three separate commands to add the Operator ID into the .env file in the validator-ejector config section:

cp .env.sample .env

export MY_OPERATOR_ID=your operator id here

sed -i.bak "s|#VE_OPERATOR_ID=|VE_OPERATOR_ID=$MY_OPERATOR_ID|g" .env

Reference the README for more information.

Monitoring

The cluster will need to push distributed validator metrics to Obol’s central Prometheus service to monitor, analyze and improve cluster performance. This data will also be made available to the operators via a login to the Grafana Dashboards.

The below token needs to be added in prometheus/prometheus.yml replacing $PROM_REMOTE_WRITE_TOKEN.

oboln!auNAZyJs!IYneXhQviJICT0H?mcxuZjO2g=WXqJTbjs-9r2P52q!vlDNpq?eLx7gbgmJgKDCmnxtoQMgZ5KmDeXTMttmlRsF/dNzxoePjkIbKWuGY25v2fc9RO

The final prometheus/prometheus.yml will look like:

global:
  scrape_interval: 30s # Set the scrape interval to every 30 seconds.
  evaluation_interval: 30s # Evaluate rules every 30 seconds.

remote_write:
  - url: https://vm.monitoring.gcp.obol.tech/write
    authorization:
credentials: oboln!auNAZyJs!IYneXhQviJICT0H?mcxuZjO2g=WXqJTbjs-9r2P52q!vlDNpq?eLx7gbgmJgKDCmnxtoQMgZ5KmDeXTMttmlRsF/dNzxoePjkIbKWuGY25v2fc9RO
    write_relabel_configs:
      - source_labels: [job]
        regex: "charon"
        action: keep # Keeps charon metrics and drop metrics from other containers.

scrape_configs:
  - job_name: "geth"
    metrics_path: /debug/metrics/prometheus
    static_configs:
      - targets: ["geth:6060"]
  - job_name: "lighthouse"
    static_configs:
      - targets: ["lighthouse:5054"]
  - job_name: "charon"
    static_configs:
      - targets: ["charon:3620"]
  - job_name: "lodestar"
    static_configs:
      - targets: [ "lodestar:5064" ]

Please let us know if you have any questions by using your cluster thread in the simple-dvt-testnet-cluster-coordination channel in Discord (either the general channel or group-specific thread).

Start your Distributed Node

Prior to starting your node,

• ⚠️ Ensure Docker is up to date.
• ⚠️ Ensure that you have at least 8GB of RAM allocated to Docker (via Docker settings in Docker Desktop. Shouldn’t be an issue on Ubuntu).

Start the cluster and verify the EL (default to geth) and CL (default to Lighthouse) sync to the head of the chain. Also, port forward

• 30303 (Geth)
• 9000 (Lighthouse)
• 3610 (Charon)

If using a clean or new machine, the following step is all that should be required. If not, some modifications could be required to the docker-compose.yml.

# Start cluster
cd lido-charon-distributed-validator-node
docker compose up -d
docker compose logs -f

Note: Solo stakers running an existing Obol cluster on a single machine should be aware that it will require changing a number of port mappings and config files to run two clusters side-by-side. If you do plan on running this Lido trial on the same machine as your existing Obol cluster, please reach out to the Lido or Obol teams and we will be able to assist.

For more information, check this section of the Obol Quickstart Guide.

Validator Deposit & Activation

Do not go through the standard deposit flow (either in the DV Launchpad or on ethereum.org), please follow the steps below to deposit and activate the validators instead:

Once each member's DV node is synced (both the EL, CL and that Charon is happy), it is time to activate the validator. You can check the health of your cluster by opening your local Grafana dashboard.

# Open Grafana dashboard
open http://localhost:3000/d/singlenode/

• The cluster coordinator will submit the deposit data. This can be done through the Lido testnet UI, by pasting the data from the deposit-data.json file from the .charon folder. The cluster coordinator should use the WalletConnect integration to stage this transaction in the Safe UI.
• The rest of the operators need to use the Safe UI to confirm the transaction. Each Operator should CAREFULLY confirm that the transaction data matches the deposit-data.json file they have. Once enough signers approve the transaction, it can be executed by the coordinator.

Please state when the two steps above have been completed in your Discord thread and await confirmation from the Lido team. Please tag @kimonsh and @perrier.

• Following a review that the key has been correctly submitted, the Simple DVT Module Committee will increase the clusters limit to 5 validators.
• The Lido and Obol teams will continue monitoring the performance of the cluster as the validators come online. Use the Grafana client to check if your node and the whole cluster are healthy.

Claiming Rewards

The Obol Team will have provided you with the splitter contract address from which you can claim your wstETH rewards. Until the DV Launchpad and the 0xSplits UI support Holesky, we recommend using Etherscan to claim your rewards by calling the withdraw() method.

1. Go to your splitter address on Holesky
2. Click on Contract
3. Click on Write Contract
4. Connect your wallet
5. Fill out the data for that method
1. account(address) = [YOUR_REWARD_ADDRESS]
2. withdrawETH (uint256) = 0
3. tokens (address[]) = 0x8d09a4502Cc8Cf1547aD300E066060D043f6982D
6. Click Write to execute the transaction in your wallet
7. Your share of the rewards should be in your wallet as wstETH

Exiting Validators

[Instructions Pending]

Relay List

Flashbots: https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-holesky.flashbots.net Eden: https://0xb1d229d9c21298a87846c7022ebeef277dfc321fe674fa45312e20b5b6c400bfde9383f801848d7837ed5fc449083a12@relay-holesky.edennetwork.io Titan: https://0xaa58208899c6105603b74396734a6263cc7d947f444f396a90f7b7d3e65d102aec7e5e5291b27e08d02c50a050825c2f@holesky.titanrelay.xyz Bloxroute https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.holesky.blxrbdn.com

Resources

Simple DVT Splitters

Simple DVT Proposal