CONDUIT Releases DEGEN CHAIN Downtime Incident Report: Improved Alerting and Monitoring of ORBIT Chain

Odaily Planet Daily News Conduit released a post-mortem analysis report on the previous downtime of Degen Chain:
On Friday, May 10 (PST time), Conduit increased the batch size of Degen and Proof of Play Apex to 10MB to reduce costs, which delayed the batch release of data from these networks to their parent chains.
On Sunday, May 12, around 1 pm, this configuration was restored to fix the batch release. This led to a reorganization on both networks because the batch was released after the 24-hour mandatory inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay these transactions with a new timestamp.
After the reorganization, the node will come back with a corrupted database because its depth is not well handled by geth. This requires resynchronizing the data directory from genesis. The synchronization time for each network is more than 40 hours, and the replay rate is about 100M gas/s.
Once the nodes are resynchronized, Conduit attempts various transaction replay schemes, although not all transactions can be recovered because some transactions rely on precise timestamps. In consultation with each rollup team, Conduit discussed and tried various strategies in parallel to bring the network online and restore the state before the reorganization.
Degen Chain went online at 7:30 pm on May 14, about 54 hours after the network was paralyzed. Proof of Play's Apex chain was restored at about the same time, but only became available to the public at 4 pm on May 15 after implementing another recovery scheme.
In response, Conduit said it has improved alerts and monitoring of the Orbit chain to cover this situation and is committed to working with Offchain Labs to improve observability for all Orbit chain operators. The team will continue to invest and research mechanisms to better simulate mainnet conditions and transaction payloads in a test environment.
Degen Chain Explorer has been displaying the latest status of Degen Chain normally.

Conduit releases Degen Chain outage report: Alerting and monitoring of Orbit chain has been improved