Why Auditing the Code is Not Enough: A Discussion on Solana Upgrade Authorities

This blog post is intended for DeFi users looking to securely invest their assets, and for developers looking for guidance on how to manage their upgrade authority.

Intro¶

Recently, there’s been a lot of buzz around a DAO vote of Solend – one of Solana’s largest lending projects. It seeks to enact restrictions on large positions, and to temporarily take control of an existing user’s position in order to liquidate it in a controlled fashion. This can be done by upgrading the smart contract’s code.

But wait! Aren’t smart-contracts supposed to be immutable?

Only in a perfect world. No code is perfect and smart contracts aren’t either, so it can be necessary to change or fix them. This is called a program upgrade.

In this post, we’ll give an overview of one of the most fundamental and yet somehow often-overlooked aspects of the security of a smart contract, namely: Who has the power to initiate program upgrades? How can users be sure that the developers don’t make undesired changes? Or even worse, just run off with their money?

Upgrade Authorities on Solana¶

On Solana, smart contracts are stored in executable accounts containing the program binary. Each such account has an upgrade authority. This authority can replace said smart contract at will. As such, it is one of the most important factors to consider when deciding whether to put your money into a Solana program.

The upgrade authority has arbitrary control over the deposited funds. They could, for example, replace a DEX with a program that does nothing but provide a function to withdraw all deposited liquidity to the upgrader’s wallet.

We provide a more technical explanation of how program updates work on Solana in the appendix of this post. Note that this way of upgrading contracts is entirely different from proxy upgrades, which is the standard way of upgrading on Ethereum.

The upgrade authority of a contract can be a DAO, a multisig, or just an ordinary keypair. In the following, we will discuss the pros and cons of these different options.

Different Ways to Manage the Upgrade Authority¶

Let’s talk about the basic solutions for how the upgrade authority of a program might be controlled. Each is on a spectrum of the trade-off between decentralization and ease of upgrade; if you have more of one, you have less of the other.

We can roughly classify program upgrades into three classes:

• Major upgrades that bring new and important functionality
• Boring upgrades that add a specialized feature, fix a minor bug or tweak some parameters
• Urgent security upgrades that fix a major bug

Each option of managing the upgrade authority of a contract must be able to cope with all types of program upgrades.

Hot Wallet and Cold Wallet¶

The simplest option for managing the upgrade authority is if it is controlled by a single entity, which might, for example, be the project’s main developer. This means the dev can patch any urgent bugs immediately after they are found and deploy any program upgrades without pestering anyone else to okay the upgrade. However, it also means that this entity has complete control over the funds the contract holds. It can patch in an instruction that allows a specific developer-controlled account to transfer all funds controlled by the contract to that account. Crucially, this means that verifying that the smart contract code has no back door is not enough to conclude that the project cannot be rugged.

One general distinction for this option that has significant security implications is whether the upgrade authority is a hot wallet or a cold wallet. Due to the authority’s power, the private key should be kept as secure as possible. Ideally offline, in hardware wallets.

Unfortunately, some projects use their main upgrade authority for lower importance tasks as well, such as regular automatic maintenance calls. This should be a separate authority to reduce attack surface on the upgrade key.

We highly advise projects not to use these hot wallet upgrade authorities. Cold wallets are always preferable. Would you hold 9-10 figures worth of crypto in a hot wallet?

Multisig¶

A more decentralized option for managing the upgrade authority is a multisig, where several known entities have to sign the upgrade instruction. These often take the form of m-of-n multisigs, where m out of the total of n parties must sign to execute the instruction, avoiding problems when some parties are unreachable or turn malicious. Of course, to evaluate the security of a multisig, one must look at the parties that are part of the multisig. For example, it might only consist of team members of the project, or it might consist of a combination of VC people, known entities from other projects, trusted public figures, or known community members.

Multisigs where the upgrade power is concentrated on too few people, are not much different from single-entity upgrade authority and hence suffer the same drawbacks. On the other hand, if too many parties are part of the multisig, problems arise for boring and urgent upgrades. Not every party of the multisig will want to verify and sign minor changes to the contract, especially since the entities in the multisig usually have a high degree of trust, which almost always coincides with them being busy people. Furthermore, multisigs with too many members present a risk during urgent security upgrades — a single member leaking the bug or exploiting it themselves would be catastrophic for the project.

Nonetheless, a stable multisig is what we usually encourage projects to adopt. When the multisig parties are chosen well, this option strikes a good balance between decentralization and ease of upgrade.

DAO¶

The most decentralized option is if a DAO controls the upgrade authority. This means that every program upgrade must be voted on and accepted by the governance DAO that controls the upgrade authority. While this is more the pinnacle of decentralization, it presents significant security risks.

The most obvious risk is that developers must publicly announce any major bug fixes before they are fixed. This means anyone can exploit the bug before the proposal containing its fix is passed, making any major bug catastrophic for the project. There is, of course, the option of migrating liquidity to a new contract. This must either be done by the users themselves — which means having to announce the bug publicly — or via exploiting the bug if it permits stealing funds. The latter can be detected on-chain and possibly front-run, which also means loss of user funds.

Furthermore, boring upgrades have the same problems as wide multisigs since it is usually infeasible for every governance token holder to verify each minor upgrade.

Of course, it is also straightforward for the developers to obfuscate who controls a DAO. They can distribute a large amount of the governance tokens to dummy accounts they control. When done correctly, this is indistinguishable from a decentralized DAO when in reality, a single entity still has complete control over the upgrade authority.

This is also a problem we face as auditors: Our audit reports usually have two target audiences, the developers themselves, to warn them of any bugs in their contract, and the general public, to warn them of any unfixed bugs or rug pull mechanisms. Since the upgrade authority is a significant part of the security of a contract, we always try our best to verify who has control over it. However, it is outside our scope to properly doxx all members of a multisig or verify the token distribution of the governance DAOs.

Non-Upgradeable Contracts¶

Finally, the option of making the program non-upgradeable is also available. However, we heavily discourage projects from doing so in almost all cases. If a bug is discovered in the contract, fixing the bug can only be done by migrating liquidity. We have already discussed the drawbacks of this above.

Combinations Of Different Upgrade Types¶

An interesting idea that has been suggested on Twitter on how to combine the advantages of multiple different types of upgrade authorities is to have, for example, two contracts, one that is not upgradeable and manages user funds, and one that uses a more practical upgrade authority like a cold wallet. This claims to have the advantage that bugs can still be fixed while allowing the user full custody of their funds.

However, this is unfortunately not practical for most DeFi protocols. The upgradeable contract still needs to be able to access the funds to do non-trivial tasks like swapping from one token to another in an AMM or executing trades in a DEX. You could move the logic on how much the upgradeable contract is allowed to take into the non-upgradeable program, but then you also moved most of the complexity into the part you can’t upgrade.

Or consider other protocols that do things yield farming or lending. In these cases, the protocol can not give the users full custody over their funds because they might be lent out.

This approach might still work for some applications with a simple component that can function independently. We usually advise against deploying multiple programs as it increases complexity and attack surface, but it might be worthwhile in some edge cases.

Authority Sleuthing¶

This is probably the most exciting question for the average DeFi degen. You are about to put your life savings into a protocol that will use them to play roulette on the on-chain options market. How do you assess the program upgrade authority’s risk for your investment?

Unfortunately, this is also not an easy question to answer. Devs often omit this crucial information from their documentation. Let’s do some sleuthing and find out with some examples:

Solend¶

First of all, we need to find Solends on-chain program address. This is very easy, fortunately, because the Solana Explorer has a database of well-known programs. Just type Solend Program into the search bar, and you will get to this page. Here we can see that Solends upgrade authority is 2Fwvr3MKhHhqakgjjEWcpWZZabbRCetHjukHi1zfKxjk. Clicking on this address, we get an overview of the transactions that this key was involved with: The very latest transaction is just a regular signed Upgrade instruction. This means that Solend uses a standard key pair as their upgrade authority. As you can also see, many transactions were executed within a brief time, all landing in the same slot. These were transactions that wrote the new binary into the buffer account. This heavily suggests that Solend is using a hot wallet, as signing and landing such a large amount of transactions in such a short time is practically impossible without durable nonces when using a proper cold wallet.

Mango Markets¶

Repeating the previous steps, we first get the upgrade authority of Mango Program v3: 3fpD42WxHsZSi5yjKcNei8H6gzndRtDLGg4otQbKNW5K.

This key has some activity, but not nearly as much as the Solend key. Looking at the latest transaction, we can see that the Mango DAO manages this key. Doing some sleuthing, we find the governance realm controlling it: Mango Developer Council v2. The tokens required to vote in this council were distributed in this proposal of the main Mango DAO. A majority established an upgrade council of 7 members. This means that while a DAO manages this key, it is equivalent to a multisig.

Coral Multisig (previously Serum Multisig)¶

Next, let’s take a look at Corals Multisig program, which provides multisig functionality to other programs. We find the program address msigmtwzgXJHj2ext4XJjCDmpbcMuufFb5cHuwg6Xdt from the projects GitHub Readme. We can see at once that this program is not upgradeable:

This is one of very few cases where this is a good idea. The code of the contract is very small and has been extensively reviewed. Furthermore, its use-case is very clearly defined and unlikely to change over time.

Indeed, if someone were able to change the binary of this program arbitrarily, they would also be able to control the upgrade authority of all these other programs. These client projects will want some assurances that nobody will be able to do that. Making the multisig contract non-upgradeable guarantees this. Note, however, that the on-chain binary is not anchor-verified and hence it is not guaranteed that the on-chain binary corresponds to the published code.

Other Types of Authorities¶

We’ve now talked at length about the security risks of different types of upgrade authorities and what you should verify about them as a protocol user. Of course, these are not the only types of authority you should be worried about. The contract may have a “manager” authority that, for example, may have control over the protocol fees. One possible rug pull is then to set protocol fees to 100% and hence funnel all contract volume into its treasury, which the manager authority usually also has control over.

Usually, we encourage projects to restrict authorities’ power as much as possible. However, it is not always possible to structure the authorities such that none of them have the power to perform a rug-pull. As a user, you should be aware of what authorities exist and how they may be able to rug the protocol. Audit reports, if they exist, are usually a good first step in researching this.

So What Should You Do?¶

Always make sure that you know the risks before putting your money into a DeFi protocol. Part of this is the risks that come from poorly managed upgrade authorities. Always consider who has access to your funds via the upgrade authority and how well they protect their keys. Is there a multisig to ensure that no single party can make off with the funds? Is whoever holds the keys doxxed? Do they keep their keys in a hot wallet or a cold wallet?

Unfortunately, there is no documentation on managing your upgrade authority properly, so we can forgive projects if they don’t follow best practices. This results in the unfortunate reality that most Solana programs have a single hot wallet upgrade authority. We hope this blog post doesn’t just serve as a guideline for users but also for devs in their considerations on managing their upgrade authority.

Appendix: How Program Upgrades Work on Solana¶

This appendix contains technical information on how Solana program upgrades actually work. Frequently also called a smart contract, a Solana program is a BPF SBF binary, compiled most often from Rust programs.

Like everything on Solana, these binaries are stored in accounts. Let’s take Solend, for example: The interesting parts for us in this context are the Executable Data, Last Deployed Slot, Upgradeable and Upgrade Authority fields.

Executable Data is a so-called buffer account that stores the actual binary. When we upgrade our program, we first upload our new binary into a buffer account, finalize the buffer (at that point, the buffer can no longer be changed), and then execute a single transaction that sets the Executable Data field to that new buffer. This last transaction is where the program upgrade actually occurs. All transactions before this point will execute the old binary, while all transactions after this point will execute the new binary.

Last Deployed Slot stores the slot in which the program was last upgraded. This can be interesting for multiple reasons, for example, to quickly fact-check claims from devs or to get a good guess as to what version from a source code repository is running on-chain.

Upgradeable and Upgrade Authority go hand in hand: when an Upgrade Authority is present, we call the program upgradeable. This key must sign on the transaction that sets a new buffer as the Executable Data. This key also has the authority to change the Upgrade Authority and can also remove it. Upon removal of the upgrade authority, the program is non-upgradeable forever.