Skip to main content

Protocol Upgrade Release Procedure

warning

This guide is intended for core protocol developers.

Protocol Upgrade Release: Step-by-Step 🚶 ​

This is a complete, 📠-🍝-ready checklist for releasing protocol upgrades.

  • Every step is numbered and must be completed in order.
  • Do not skip steps if you are not experienced in protocol upgrades.
  • Most commands are ready to copy/paste. If you found an issue, please update this doc.
  • Keep track of all the steps and merge everything to the main branch for history and visibility.

Table of Contents ​

0. Prerequisites & Sanity Checks​

Before you start:

  • You have push/publish access to the repo and GitHub releases
  • You have the following CLI tools: git, make, jq, sed, curl, go, brew, pocketd, etc.
  • You have reviewed or are familiar with previous upgrades for reference
  • You have read the full Protocol Upgrade Introduction
  • You understand the difference between state-breaking and consensus-breaking changes
  • You know how to test your changes locally (see Testing Upgrades)

1. Ensure ConsensusVersion is updated​

  • Bump the ConsensusVersion for all modules with state-breaking changes.
  • This requires manual inspection and understanding of your changes.
  • Merge these changes to main before continuing.

🔗 See all ConsensusVersion uses

⚠️DO NOT PROCEED until these changes are merged⚠️

2. Prepare a New Upgrade Plan​

Reference

  1. Select SHAs

    • Find the SHA of the last public release

    • Find the SHA for the new release (usually main)

    • Compare them:

      https://github.com/pokt-network/poktroll/compare/v<LAST_RELEASE_SHA>..<NEW_RELEASE_SHA>

  2. Identify Breaking Changes

    • Manually inspect the diff for parameter/authorization/state changes

  3. Update Upgrade Plan

    • If any protobufs were changed, make sure to review protobuf deprecation
    • Edit app/upgrades.go and add your upgrade to allUpgrades
    • Examples:
      • v0.1.2 - State-breaking change with a new parameter; simple upgrade
      • v0.1.3 - Node software update without consensus-breaking changes; empty upgrade

⚠️DO NOT PROCEED until these changes are merged⚠️

3. Create a GitHub Release​

note

You can review all prior releases here.

  1. Tag the release using one of the following and follow on-screen prompts:

    make release_tag_bug_fix
    # or
    make release_tag_minor_release

  2. Publish the release by:

  3. Update the description in the release by:

    • Clicking Generate release notes in the GitHub UI

    • Add this table ABOVE the auto-generated notes (below)

      ## Protocol Upgrades

      | Category                     | Applicable | Notes                                |
      | ---------------------------- | ---------- | ------------------------------------ |
      | Planned Upgrade              | ✅         | New features.                        |
      | Consensus Breaking Change    | ✅         | Yes, see upgrade here: #1216         |
      | Manual Intervention Required | ❓         | Cosmosvisor managed everything well. |

      | Network       | Upgrade Height | Upgrade Transaction Hash | Notes |
      | ------------- | -------------- | ------------------------ | ----- |
      | Alpha TestNet | ⚪             | ⚪                       | ⚪    |
      | Beta TestNet  | ⚪             | ⚪                       | ⚪    |
      | MainNet       | ⚪             | ⚪                       | ⚪    |

      **Legend**:

      - ⚠️ - Warning/Caution Required
      - ✅ - Yes
      - ❌ - No
      - ⚪ - Will be filled out throughout the release process / To Be Determined
      - ❓ - Unknown / Needs Discussion

      ## What's Changed

      <!-- Auto-generated GitHub Release Notes continue here -->

  4. Set as a pre-release (change to latest release after upgrade completes).

4. Write an Upgrade Transaction (JSON file)​

tip

See v0.1.2 upgrade transactions for examples.

Generate the new upgrade transaction JSON files like so:

./tools/scripts/upgrades/prepare_upgrade_tx.sh v<YOUR_VERSION>.<YOUR_RELEASE>.<YOUR_PATCH>

For example, running:

./tools/scripts/upgrades/prepare_upgrade_tx.sh v0.1.2

Will create:

tools/scripts/upgrades/upgrade_tx_vX.Y.Z_alpha.json
tools/scripts/upgrades/upgrade_tx_vX.Y.Z_beta.json
tools/scripts/upgrades/upgrade_tx_vX.Y.Z_local.json
tools/scripts/upgrades/upgrade_tx_vX.Y.Z_main.json
height is not populate in the *.json files

You will need to update the height before submitting each one. More on this later...

Example JSON snippet:

{
  "body": {
    "messages": [
      {
        "@type": "/cosmos.upgrade.v1beta1.MsgSoftwareUpgrade",
        "authority": "pokt10d07y265gmmuvt4z0w9aw880jnsr700j8yv32t",
        "plan": {
          "name": "v0.0.4",
          "height": "30",
          "info": "{\"binaries\":{...}}"
        }
      }
    ]
  }
}

5. Validate the Upgrade Binary URLs (Live Network Only)​

⚠️The binary URLs and checksum must be correct for LIVE NETWORKS. Otherwise,cosmovisor will fail⚠️

Install go-getter if you don't have it:

go install github.com/hashicorp/go-getter/cmd/go-getter@latest

And check all binary URLs:

jq -r '.body.messages[0].plan.info | fromjson | .binaries[]' $PATH_TO_UPGRADE_TRANSACTION_JSON | while IFS= read -r url; do
  go-getter "$url" .
done

Expected output should look like the following:

2025/04/16 12:11:36 success!
2025/04/16 12:11:40 success!
2025/04/16 12:11:44 success!
2025/04/16 12:11:48 success!

⚠️DO NOT PROCEED until all URLs validate⚠️

6. Test the New Release Locally​

  • Follow Testing Protocol Upgrades before submitting any transactions.
  • If you find an issue, you'll need to repeat the steps as needed (update plan, release, transactions, etc.).

7. Submit the Upgrade on Alpha TestNet​

This step is parameterized so you can use it for any network (Alpha, Beta, or MainNet). Substitute the variables below as needed.

Variables:

  • NETWORK: one of (pocket-alpha, pocket-beta, pocket)
  • RPC_ENDPOINT: The RPC endpoint for the network (e.g., https://shannon-testnet-grove-rpc.alpha.poktroll.com)
  • UPGRADE_TX_JSON: Path to the upgrade transaction JSON (e.g., tools/scripts/upgrades/upgrade_tx_v0.1.2_alpha.json)
  • FROM_ACCOUNT: The account submitting the transaction (e.g., pnf_alpha)
  • TX_HASH: The hash of the submitted transaction (for monitoring)

Step-by-Step instructions:

  1. Get the RPC endpoint for NETWORK. Example for Alpha here.

  2. Update the height in your upgrade transaction JSON ():

    # Get the current height
    CURRENT_HEIGHT=$(pocketd status --node ${RPC_ENDPOINT} | jq '.sync_info.latest_block_height' | tr -d '"')
    # Add 5 blocks (arbitrary, adjust as needed)
    UPGRADE_HEIGHT=$((CURRENT_HEIGHT + 5))
    # Update the JSON
    sed -i.bak "s/\"height\": \"[^\"]*\"/\"height\": \"$UPGRADE_HEIGHT\"/" ${UPGRADE_TX_JSON}
    # Cat the output file
    cat ${UPGRADE_TX_JSON}
    Export UPGRADE_TX_JSON and RPC_ENDPOINT first
    export RPC_ENDPOINT=https://shannon-testnet-grove-rpc.alpha.poktroll.com
    export UPGRADE_TX_JSON="tools/scripts/upgrades/upgrade_tx_v0.1.4_alpha.json"

  3. Submit the transaction:

    pocketd \
      --keyring-backend="test" --home="~/.pocket" \
      --fees=300upokt --chain-id=${NETWORK} --node ${RPC_ENDPOINT} \
      tx authz exec ${UPGRADE_TX_JSON} --from=${FROM_ACCOUNT}
    Grove Employee Helpers 🌿

    If you're a Grove Employee, you can use the helpers here to use this wrapper:

    pkd_<NETWORK>_tx authz exec ${UPGRADE_TX_JSON} --from=${FROM_ACCOUNT}

  4. Verify the upgrade is planned onchain:

    pocketd query upgrade plan --node ${RPC_ENDPOINT}

  5. Watch the transaction (using the TX_HASH from step 3):

    watch -n 5 "pocketd query tx --type=hash ${TX_HASH} --node ${RPC_ENDPOINT}"

  6. Verify node version aligns with what's in <UPGRADE_TX_JSON>:

    curl -s ${RPC_ENDPOINT}/abci_info | jq '.result.response.version'

  7. Once the upgrade is complete, make sure to:

    • Record the upgrade height and tx_hash in the GitHub Release
    • Commit the {UPGRADE_TX_JSON} file with the final height to main
Grove Employees 🌿
  • Use logging/observability tools to monitor full nodes & validators.
  • Only proceed to Beta/MainNet after Alpha is successful.
  • Connect to our cluster to inspect logs and pod status

⚠️ DO NOT PROCEED until the changes from step (2) are merged assuming the upgrade suceeded⚠️

8. Update the homebrew-tap Formula​

Once the upgrade is validated, update the tap so users can install the new CLI.

Steps:

git clone git@github.com:pokt-network/homebrew-pocketd.git
cd homebrew-pocketd
make tap_update_version
git commit -am "Update pocket tap from v.X1.Y1.Z1 to v.X1.Y2.Z2"
git push

Reinstall the CLI:

brew reinstall pocketd

Or install for the first time:

brew tap pocket-network/homebrew-pocketd
brew install pocketd

See pocketd CLI docs for more info.

9. Submit the Upgrade on Beta & MainNet​

Repeat Step 7: Submit the Upgrade Onchain with the appropriate parameters for Beta and MainNet:

  • Use the correct <RPC_ENDPOINT> for Beta or MainNet
  • Use the correct <UPGRADE_TX_JSON> (e.g., upgrade_tx_v0.1.2_beta.json or upgrade_tx_v0.1.2_main.json)
  • Use the correct sender account for each network

This ensures a single, copy-pasta-friendly process for all networks.

⚠️ DO NOT PROCEED until the changes from step (2) are merged assuming the upgrade succeeded⚠️

Grove Employees 🌿

If you're a Grove Employee, you can use the helpers here to use this wrapper:

See the instructions in this PR for copy-pasta commands on how v0.1.3 was submitted on Alpha & Beta.

10. Troubleshooting & Canceling an Upgrade​

If you need to cancel, see:

Checklist:

  1. Follow Protocol Upgrade Procedure
  2. Update the Upgrade List
  3. Deploy a full node on TestNet and verify sync (see Full Node Quickstart Guide)

Before You Finish​

  • All steps above are checked off
  • All releases, plans, and transactions are published and tested
  • The upgrade transaction json files with the updated height are merged in
  • You have communicated upgrade details to the team/community
  • You have prepared for rollback/troubleshooting if needed
  • You have updated the published release with the final upgrade height on each network

TODO​

The following improvements will streamline this process further

  • Concrete examples of PR examples & descriptions along the way
  • Additional helpers (not automation) for some of the commands throughout