Skip to main content

Upgrade Preparation

Purpose of this document

Technical instructions on preparing a protocol upgrade.

Instructions to create a new upgrade and apply it to both Alpha & Beta TesNet.

You MUST follow this PRIOR TO the MainNet Release Procedure.

Required Prerequisites Setup, Reading & Knowledge ​

Table of Contents ​

1. Avoid On-Chain Non-Determinism​

⚠️ The most common cause of chain halts is non-deterministic onchain behavior ⚠️

No amount of code reviews or testing can fully catch this. Here is a suggested and opinionated potential solution:

  1. Start a session of Claude code or CLI agent of choice

  2. Identify the tag of the previous MainNet release (e.g. v0.1.28)

  3. Ask Claude the following:

    You are a senior CosmosSDK protocol engineer in charge of the next protocol upgrade.

    Do a git diff v0.1.28.

    Identify any potential bugs, edge cases or issues.

    In particular, focus on any onchain behaviour that can result in non-deterministic outcomes. For example, iterating a map without sorting the keys first.

    This is critical to avoid chain halts. Take your time and provide a comprehensive analysis.

2. Prepare a New Upgrade Handler​

  1. Identify the version of the latest release from the full list of releases (e.g. v0.1.20)

  2. Prepare a new upgrade handler by copying vNEXT.go to the next release (e.g. v0.1.21) like so:

    cp app/upgrades/vNEXT.go app/upgrades/v0.1.21.go

  3. Open v0.1.21.go and replace all instances of vNEXT with v0.1.21.

  4. Remove all the general purpose template comments from v0.1.21.go.

  5. Open app/upgrades.go and:

    • Comment out the old upgrade
    • Add the new upgrade to allUpgrades

  6. Prepare a new vNEXT.go by copying vNEXT_Template.go to vNEXT.go like so:

    cp app/upgrades/vNEXT_Template.go app/upgrades/vNEXT.go

  7. Open vNEXT.go and remove all instances of Template.

  8. Create a PR with these changes and merge it. v0.1.21 PR Example:

  9. Note that the upgrade handler MAY need business logic related to onchain state changes. See other upgrades for reference.

3. Test Locally​

Follow the instructions in Testing Protocol Upgrades Locally.

Expertise required for complex upgrades

⚠️ If your upgrade handle had complex business logic, you MUST test it locally to avoid a chain halt. ⚠️

Follow Testing Protocol Upgrades BEFORE submitting any transactions.

If you find an issue, you'll need to:

  1. Delete the previous release
  2. Delete the previous tag
  3. Implement and merge in the fix
  4. Prepare a new release
  5. Regenerate the artifacts
  6. Repeat the process above

This requires jumping back and forth between some of the steps on this page.

4. Create a GitHub Release​

  1. Tag the release by running the following command:

    make release_tag_minor

  2. Publish the release by:

    • Following the onscreen instructions after running the command above
    • Drafting a new release
    • Use the tag above to auto-generate the release notes

  3. Set as a pre-release (change to latest release after upgrade completes).

  4. Trigger the workflow to build new release artifacts by running:

    gh workflow run "Release artifacts"
    😎 Keep Calm and Wait for CI 😅

    Wait for the Release Artifacts CI job to build artifacts for your release.

    It'll take ~20 minutes and will be auto-attached to the release under the Assets section once complete.

5. Prepare the Upgrade Transactions​

Generate the new upgrade transaction JSON files. For example, for v0.1.21, run:

./tools/scripts/upgrades/prepare_upgrade_tx.sh v0.1.21

This will create:

tools/scripts/upgrades/upgrade_tx_v0.1.21_alpha.json
tools/scripts/upgrades/upgrade_tx_v0.1.21_beta.json
tools/scripts/upgrades/upgrade_tx_v0.1.21_local.json
tools/scripts/upgrades/upgrade_tx_v0.1.21_main.json

Make sure to commit these to GitHub once you're done.

Note that the height is not populated in the *.json files. This will be updated in subsequent steps below.

Example JSON snippet:
{
  "body": {
    "messages": [
      {
        "@type": "/cosmos.upgrade.v1beta1.MsgSoftwareUpgrade",
        "authority": "pokt10d07y265gmmuvt4z0w9aw880jnsr700j8yv32t",
        "plan": {
          "name": "v0.0.4",
          "height": "30",
          "info": "{\"binaries\":{...}}"
        }
      }
    ]
  }
}
Optional: Validate the Upgrade Binary URLs

Install go-getter if you don't have it:

go install github.com/hashicorp/go-getter/cmd/go-getter@latest

And check all binary URLs:

RELEASE_VERSION=<VERSION> # E.g. "v0.1.11"
for file in ./tools/scripts/upgrades/upgrade_tx_${RELEASE_VERSION}*; do
  echo "Processing $file"
  jq -r '.body.messages[0].plan.info | fromjson | .binaries[]' "$file" | while IFS= read -r url; do
    go-getter "$url" .
  done
done

Expected output should look like the following:

2025/04/16 12:11:36 success!
2025/04/16 12:11:40 success!
2025/04/16 12:11:44 success!
2025/04/16 12:11:48 success!

6. Prepare Snapshots​

Generate new snapshots for each network and ensure they are available here.

Manual Process by Grove 🌿

This is currently a manual process maintained by the team at Grove.

The instructions are currently maintained in an internal Notion document.

7. Submit the Upgrade on Alpha & Beta TestNet​

If you are submitting the upgrade for v0.1.21, follow the instructions generated by these commands:

./tools/scripts/upgrades/submit_upgrade.sh alpha v0.1.21
./tools/scripts/upgrades/submit_upgrade.sh beta v0.1.21

⚠️ Make sure to ONLY move to the next network after the prior one finished successfully ⚠️

8. Troubleshooting & Canceling an Upgrade​