This blog post provides more detail about the mainnet incident that occurred during the settlement of the LINK/USDT perpetual futures market, which was triggered via governance. Date & Time 2024/02/16 11:00 (GMT) Status Post Mortem Incident Report Classification CRITICAL For full details of how incidents are classified, please see this blog. Description During the governance voted…

USDT

LINK

<p>This blog post provides more detail about the mainnet incident that occurred during the settlement of the LINK/USDT perpetual futures market, which was triggered via governance.</p><img src="https://sosovalue.s3.us-west-2.amazonaws.com/sosponge/web3image/24/02/17/v31gt0h37wxk.png" alt=""/><h3>Date & Time</h3><p>2024/02/16 11:00 (GMT)</p><h3>Status</h3><p>Post Mortem Incident Report</p><h3>Classification</h3><p>CRITICAL</p><p>For full details of how incidents are classified, please see this <a href="https://blog.vega.xyz/mainnet-incident-classification-8efa5f927b7a" target="_blank">blog</a>.</p><h3>Description</h3><p>During the governance voted termination of the LINK/USDT market, on the 31st January 2024, a bug affected the calculation of funding payments. The issue had not been caught in testing as it was specific to a particular and unusual set of conditions that had not been anticipated and covered with test cases.</p><p>This bug led to a small number of parties’ gain/loss calculations being incorrect for the final funding payment at market termination. The calculation error is described in <a href="https://github.com/vegaprotocol/vega/issues/10520" target="_blank">this issue</a> and was resolved in patch release <a href="https://github.com/vegaprotocol/vega/releases/tag/v0.73.13" target="_blank">0.73.13</a>, which has now been deployed by node operators.</p><p><strong>Note:</strong> At no time was there any risk to VEGA tokens staked or held in the vesting contract due to this incident. Only balances on the Vega network by users with a position on the affected market could have been at risk, or, should this scenario have been realised on other markets.</p><p>Several users that identified and assisted with the diagnosis and fixing of this bug were awarded discretionary bug bounties under our <a href="https://vega.xyz/bug-bounties" target="_blank">bug bounty programme</a>.</p><h3>Analysis</h3><p>The root cause of the issue was in the calculation of the Time Weighted Average Price (TWAP) for the internal data points used to calculate funding payments on perpetuals. When a funding period ends, the last data point in that period is carried forward and contributes to the next funding period’s TWAP. If a subsequent data point is then received at time t1, the contribution from this carry-over point from the previous period will be price x (t1 — funding-period-start). Vega in effect snaps the timestamp of the internal point forward to the start of the funding period. In almost all cases this was working correctly.</p><p>When the LINK/USDT market was terminated, it was necessary for the protocol to recalculate the contribution to the TWAP of the last data-point received before termination. A bug in this recalculation meant that the protocol did not correctly snap the timestamp to the start of the funding period.</p><p>In active markets this issue would not occur since new internal data points would be supplied throughout the funding period, so no snapping was required. On the LINK/USDT market this was not the case (as the market was suspended) and the only internal data point available was carried forward from many funding periods ago. The calculation for the TWAP should have been:</p><p>TWAP = Price x (funding-period-end — funding-period-start) / (funding-period-end — funding-period-start)</p><p>But without snapping it became:</p><p>TWAP = Price x (funding-period-end — internal-point-timestamp) / (funding-period-end — funding-period-start)</p><p>Where internal-point-timestamp was much further in the past than funding-period-start. Since:</p><p>(funding-period-end — internal-point-timestamp) > (funding-period-end — funding-period-start)</p><p>The output of the final TWAP calculation was bigger than it should have been resulting in a funding payment that was bigger than it should have been.</p><h3>Resolution</h3><p>The calculation was corrected in patch release<a href="https://github.com/vegaprotocol/vega/releases/tag/v0.73.13" target="_blank"> 0.73.13</a>. Erroneous previous funding payments were not adjusted or undone by this patch, but future funding payments will be calculated correctly.</p>

Post Mortem: LINK/USDT Market Termination Funding Rate Bug