A run-time bug created the outage in the durable nonce transactions feature, which determines how the blockchain handles a specific form of the transaction intended for offline use.
06-01-22 Solana Mainnet Beta Outage Report: The durable nonce transaction feature was disabled in releases v1.9.28/v1.10.23 to prevent the network from halting if the same situation were to arise again. https://t.co/2HwkSWkLaR
— Wu Blockchain (@WuBlockchain) June 5, 2022
Validators began restarting the network four and a half hours after disabling the “durable nonce transactions.” The following day, at 9:00 p.m. UTC, block production resumed, and network operators continued to restore client services over the next several hours.
The recent outage has shown Solana’s inability to handle durable nonces. Instead of considering these niche inbounds as a single transaction, the network’s validators double-counted them as a single transaction at two separate block heights. Solana’s consensus mechanism was effectively broken by this impossible predicament.
Solana takes proactive steps
To maximise throughput, Solana uses parallel processing of nonoverlapping transactions. Networks that execute transactions serially can use an incrementing nonce; Solana uses a different mechanism to ensure transactions are not handled twice. Since durable nonce transactions are not meant to expire, they require a different approach to avoid double processing and are handled sequentially.
The processing of a durable nonce transaction revealed a runtime bug that prevented the network from progressing during the outage. A successful transaction would not have caused this problem because the durable nonce transaction would have failed.
In releases v1.9.28/v1.10.23, the durable nonce transaction functionality was deactivated to prevent the network from stalling if the same circumstance arose again. Durable nonce transactions will not be executed until the mitigation is implemented and the feature is reactivated in a later version.
