In late August, many STON.fi users experienced hours-long transaction delays followed by a complete blockchain outage. During this period, STON.fi took the precautionary measure of temporarily pausing swap functionality to protect users’ assets. This post-mortem walks through the key events, from the initial delays caused by DOGS token claims to the temporary pause of STON.fi swaps, and finally to the restoration of TON Blockchain. Our goal is to help you understand what caused these issues, explain the actions we took in response, and outline our plans to prevent similar incidents in the future.
Technical Background
To understand what happened, it’s important to know some key technical aspects:
Sharding is a technique used to optimize blockchain load and data processing efficiency. It involves breaking data into smaller segments called shards, which allows for load distribution and increased operating power for each network node. Nodes accumulate and process all network data, and reducing the load on each node enables more efficient operation.
Additionally, TON Blockchain employs a protection system that prevents a single contract from consuming all computational resources within one shard. This system prioritizes transactions containing a single outgoing message, while transactions with multiple messages are queued.
Transaction processing speed decreases when the network handles cross-shard messages—when it works with data across multiple shards. Furthermore, the network may create a processing queue if a transaction contains several outgoing messages.
Timeline of Events
August 26, 08:51 UTC: DOGS users take action
DOGS is a Telegram bot that distributed virtual DOGS tokens and encouraged users to compete with each other. Rewards depended on the age of each user’s Telegram account. This simple gameplay attracted over 50 million users to the project.
On August 26, DOGS users were invited to claim their airdrop to non-custodial wallets, such as Tonkeeper. To complete this process, users had to sign the transaction in their wallet and pay both the network fee and the DOGS project fee.
DOGS user wallet addresses and the DOGS fee address were located in different blockchain shards, which increased the load on network nodes and led to an increase in transaction latency, as described earlier.
Typically, TON Blockchain processes 3-4 million transactions per day. However, on August 26, the network experienced an exceptionally high load of 9 million transactions, many involving cross-shard messages. As a result, transaction processing across the entire blockchain slowed down.
Many users, after claiming their DOGS tokens, wanted to swap them for TON or USDT. Since STON.fi is the largest decentralized exchange on TON, integrated with Tonkeeper and TON Space, most DOGS token swaps were processed by STON.fi DEX. Some STON.fi swap transactions contain multiple outgoing messages, which, according to the logic of TON Blockchain protection system, moved them lower in the processing queue.
Within the first hour of the DOGS liquidity pool launch, users made 19,000 swaps on STON.fi. Only 3,000 were processed, with the rest queued for validation. Delays reached 3 hours, and half of the transactions were eventually declined due to slippage (the difference between the expected and actual price), with users receiving refunds.
For context: during the Notcoin token claim, STON.fi DEX was processing over 16,000 transactions per hour, which was a record until the DOGS token claim.
August 27, 22:30 UTC: blockchain shutdown
The high activity of millions of DOGS users sending cross-shard messages, combined with the transaction queue and performance issues with some validators, led to a blockchain outage. TON Blockchain stopped producing blocks for several hours.
August 28, 05:30-15:00 UTC: blockchain recovery, new queues and STON.fi swaps pause
By 05:30 on August 28, the blockchain resumed normal operations and began processing all queued transactions, including those from STON.fi users. However, as users started claiming DOGS tokens again in the morning, network congestion reoccurred.
In response to this situation, we, as STON.fi DEX, took action to protect our users:
- Temporarily disabled the swap functionality on app.ston.fi to mitigate ongoing issues such as slippage and potential asset freezing.
- Importantly, kept liquidity transactions fully operational during this time to protect the assets of liquidity providers.
These measures were implemented to address the immediate challenges posed by the blockchain congestion and to prevent further complications for our users’ assets and transactions.
August 28, 17:19 UTC: second shutdown
The blockchain stopped functioning again, and TON core team released an urgent update for validators.
August 29, 01:30 UTC: TON and STON.fi fully operational
TON Blockchain came back online and began processing the queued transactions. By morning, the STON.fi team took the following steps:
- Reopened swaps on app.ston.fi, restoring full functionality to our platform.
- Confirmed that all user assets were secure, validating the effectiveness of our protective measures during the outage.
Normal operations for both TON Blockchain and STON.fi DEX resumed fully. The swap functionality had been disabled for approximately 18 hours, during which we continuously monitored the situation to ensure a safe and timely restart of services.
Summary
The unforeseen activity of millions of users, combined with a surge in cross-shard messages, drastically increased the load on blockchain nodes and created a transaction queue. Additionally, the logic of TON Blockchain protection system, which deprioritizes complex transactions like DEX swaps, led to significant delays in swap transactions and asset refunds in cases of unsuccessful transactions.
This series of events caused significant delays in swap transactions and temporarily halted TON Blockchain. In response, the STON.fi team made the decision to temporarily disable swap transactions to ensure the safety of users’ assets until the blockchain operational issues were resolved.
Future Prevention Strategies
To prevent similar issues in the future, we’re optimizing how sharding and cross-shard messages are handled at STON.fi DEX. Here’s our action plan:
1. We will add a deadline parameter, which will return tokens to the user’s wallet if a transaction takes too long to process. This will expedite the queue processing of cross-shard messages sent to the DEX router. Such a queue forms due to limitations on the number of transactions a single router can process under heavy load. We will reduce this load by decreasing the number of messages in a single swap/refund transaction from the current 8 to 5, which will improve overall performance.
2. We will deploy 16-64 additional routers across different blockchain shards. This way, even if a queue forms in one shard due to high transaction volume, it will affect only 1/16-1/64 of the pools, rather than all of them. This will maintain the high performance of the entire DEX, even under heavy load in one shard.
3. In the future, we plan to implement an architecture with no central router contracts. This will reduce the number of potential failure points and distribute transaction processing more evenly across the entire network.
These measures aim to significantly improve the resilience and efficiency of our system, ensuring a smoother experience for our users even during periods of extremely high activity.
