Intro
Table of Contents
Summary
The purpose of this document is to provide a collaborative resource to aid Node Operators participating in the Lido protocol to timely and effectively execute any updates to their infra or processes necessary for the smooth execution of the Lido V2 upgrade and resulting changes w.r.t. validator exit processing.
Validator Exits Policy
NO Responsibilities
Expectations and requirements are detailed in the Lido Validator Exits Policy.
Validator exit requests are signaled to all Node Operators via the on-chain Ejector Oracle reports multiple times a day. Node Operators have a duty to set up infrastructure to capture signaled validator exit requests and process relevant (i.e. pertaining to their validators) requests as soon as possible.
Node Operators participating in the Lido curated registry have a duty to exit validators in a timely manner when they are requested to do so. The actual mechanisms for validators to be exited are at the discretion of Node Operators. Open source tooling is available to aid Node Operators in identifying signaled validator exit requests, as well as in processing these requests.
Monitoring
The below are excerpts from the policy. Please consult the full policy for details.
If Node Operators are processing signaled validator exit requests as soon as they are available, the shortest possible time for a validator exit request to go from “signaled” to “processed” will be somewhere within the range of a few minutes to an hour. With respect to validator exit performance, each Node Operator may be considered to have one of the below three statuses.
- In good standing - validator exit requests are being processed fully, correctly, and timely.
- Delayed - validator exit requests are being processed incompletely, incorrectly, or not within the desired time frame.
- Delinquent - validator exit requests are being processed incompletely, incorrectly, or not within the maximum acceptable time frame.
- If a Node Operator has a status of Delayed, contributors from the NOM workstream will raise an issue in internal communications with the Node Operator and request remediative action.
- If a Node Operator has a status of Delayed or Delinquent, the Exit Bus Oracle will assume that the Node Operator is unresponsive and re-route new incoming validator exit requests to operators that are not considered delinquent or delayed. Due to the re-routing of validator exit requests, the DAO should consider (via an ad-hoc vote) overriding the total limit of active validators for the relevant Node Operator such that if/when they resume a status of in good standing, they are not benefiting at the expense of Node Operators who took over the processing of the re-routed exits requests.
- Once a Delinquent Node Operator has processed all signalled validator exit requests (and thus their number of Delinquent validators in next Accounting Oracle report is updated to 0), they will recommence receiving validator exit requests. Their status shall revert to “In good standing” after 5 days (i.e. provided any newly received validator exit requests are processed timely). During this 5 day “cooldown period” they will continue to not receive new stake and receive halved rewards.
Penalties
A node operator may be penalized if they have strictly more “stuck” validators than “refunded” validators (see reimbursement section of policy). While this condition is met, the Node Operator receives only half of the rewards and does not receive any new stake. Once the Node Operator manages to either withdraw the required number of validators or compensates for the lost validators and increases the “refunded” count through DAO voting, the node operator is considered under penalty for STUCK_PENALTY_DELAY seconds and then returns to the normal state. Rewards are automatically restored to normal, but in order to start to receive new stake, the Node Operator (or anyone else) have to call the permissionless method clearNodeOperatorPenalty.
NO Tooling
- Key API Service - a Node Operator-hosted API that loads the operator’s public validator keys, calculates the next keys to be exited, and requests via other tooling (e.g. eth-do, a custom script for web3 signer, etc) signed VEMs to be stored (as pre-signed messages) + made available to the Validator Ejector.
- Validator Ejector - a daemon that listens for Ejector Oracle reports, retrieves pre-signed exit messages for exiting validators, and processes the validator exit requests (i.e. submits the VEMs to the CL for broadcast).
- Lido-CLI - a CLI to interact with various aspects of the Lido on Ethereum protocol. Useful for Node Operators to grab data about protocol activity and status, and numerous details about validator exits statuses.
Other tooling / scripts / resources
- Some Node Operators are additional running Oracles and/or DSM Guardian Daemons — they’re not relevant to everyone, so if you’ve never heard of them before you can ignore them.
Resources
Name | Description | Link |
Validator Exits policy | The Lido Policy regarding exits | |
Validator exits discussion | Governance forum post with all relevant discussions leading up to the policy | |
Goerli V2 PSA | Segment in NOM Community Call #4, introducing the steps necessary for the upcoming Goerli hardfork | |
Ejector Oracle Specs | Technical specs on how the Ejector Oracle will function | |
New tooling works | Information on the works necessary to set up the suggested tooling | |
Exits Deep-Dive #1 | Recording of one of the 2 technical deep-dives into exits. Aimed at Node operators. | |
Exits Deep-Dive #2 | Recording of one of the 2 technical deep-dives into exits. Aimed at Node operators. | |
Zheijiang exit walkthrough #1 | Recording of one of the 2 technical deep-dives showing validator exit flow. | Zoom recording (online) (passcode .K0V#wj9 )
Google drive (video, chat transcript) |
Zheijiang exit walkthrough #2 | Recording of one of the 2 technical deep-dives showing validator exit flow. | Zoom Recording (online) (passcode 5A9@i#=5 )
Google drive (video, chat transcript) |
Lido keys api repo | KAPI (Key API) repository | |
Lido Validator Ejector repo | Validator Ejector (eventlistening daemon) repository | |
Lido CLI repo |