On July 21, 2021, the OpenEthereum team noticed their node had stopped progressing on Ropsten at block
10679538. The issue, originally thought to be in OpenEthereum, was in how the go-ethereum implementation checked the balance of the transaction sender for 1559-style transactions. Because Ropsten miners were exclusively running go-ethereum, an “invalid” transaction whose sender balance was too low was included in a block and then accepted by other go-ethereum miners, despite being rejected by some other clients on the network. Specifically, OpenEthereum and Besu rejected the transaction/block, while Nethermind, go-ethereum, and Erigon (whose implementation was partially forked from the go-ethereum code) accepted them. The cause of the issue was found and fixed in the following PRs:
- Network: Ropsten
- Number: 10679538
- Hash: 0x1252a34c4f2b061adc609e909d958c02e1ac39043e2e60c0ec47e565e3f625f1
- OpenEthereum debug logs
Note: all times listed are in Pacific Time
- 6:53: OpenEthereum developer posts in the #1559-dev channel of the Ethereum R&D discord that their node has stopped on block 10679538 on Ropsten.
@smixxsays their Ropsten node is at block 10680453.
- 7:36: Besu confirms they are also rejecting block 10679538.
- 7:51: Miner of block 10679538 is confirmed to be a go-ethereum node.
- 7:55: Confirmation that go-ethereum miners are still progressing on the chain beyond block 10679538.
- 7:56: Confirmation that Nethermind also accepted block 10679538.
- 8:08: Possible root cause for the issue identified in go-ethereum.
- 8:43: Pull request opened in go-ethereum with candidate fix.
- 8:46: Pull request opened in Erigon with candidate fix.
- 9:01: Updated go-ethereum and Besu miners restarted on Ropsten (erroneous chain now up to block 10,680,803).
- 9:43: EthereumJS confirmed to have the same issue as go-ethereum, Erigon and Nethermind.
- 10:57 Pull request opened in Nethermind with candidate fix.
This commit added new assertions to the validity of EIP-1559 transactions. Specifically, on line 217 it adds the following assertion:
assert sender.balance >= gasLimit * transaction.max_fee_per_gas
A few lines above (L207), though,
sender.balance is modified to substract from it the transaction’s amount (
signer.balance -= transaction.amount). This led to confusion, as some client teams used the full
sender.balance (i.e. pre-subtraction of
transactiion.amount) when checking the assertion defined on line 217, rather than the updated value.
One suggested fix is to move this assertion closer to when the
signer.balance value is updated, similarly to the other assertion on line 208.