Leaked a secret? Revoke it, then clean history

If a password, token or API key reaches a Git repository, revoke or rotate it first, at the service that issued it, and only then clean up. Deleting the file in a new commit does not remove it: every earlier commit still holds it, and so does every clone, fork and pipeline that copied the repository in the meantime. Once the secret no longer works, every copy is harmless, which may be all you need. Rewriting history comes second, and it is disruptive: git filter-repo rewrites every commit from the leak onwards, everyone must replace their clones, and the host may keep copies that only it can delete. Then you prevent the next leak with .gitignore, a pre-commit scanner and push protection. This lesson shows each step on a practice repository with a key that is fake, as every key on this page is: why deleting is not enough, where the copies hide, a full rewrite with git filter-repo, the pull that undoes it, and a hook that stops the next leak.

Deleting the file is not enough

A key is committed, and the next commit deletes it:

from gitsession import session

session(r"""
git init -q shop
cd shop
mkdir config
printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > config/payments.env
git add config/payments.env
git commit -q -m "Connect to the payment provider"
# Deleting the file in a new commit
git rm -q config/payments.env
git commit -q -m "Remove the key from the repository"
ls
# The key is still in history, one command away
git log --oneline
git log --oneline -S'paydemo_live_'
git show HEAD~1:config/payments.env
""")

$ git init -q shop
$ cd shop
$ mkdir config
$ printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > config/payments.env
$ git add config/payments.env
$ git commit -q -m "Connect to the payment provider"
# Deleting the file in a new commit
$ git rm -q config/payments.env
$ git commit -q -m "Remove the key from the repository"
$ ls
# The key is still in history, one command away
$ git log --oneline
7ea18b4 Remove the key from the repository
ae03c71 Connect to the payment provider
$ git log --oneline -S'paydemo_live_'
7ea18b4 Remove the key from the repository
ae03c71 Connect to the payment provider
$ git show HEAD~1:config/payments.env
PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON

The working tree is empty, yet git log -S'paydemo_live_', the pickaxe, lists both commits: the one that added the key and the one that removed it. git show HEAD~1:config/payments.env prints the file as it was. Anyone who can read the repository can do the same, and if it was pushed to a public repository, you must assume that someone already has. GitHub’s own documentation on removing sensitive data starts with the same point: when the data is a password, token or credential, the first step is to revoke and/or rotate it.

First, revoke or rotate

Revoking makes the key stop working; rotating replaces it with a new one and updates everything that uses it. The order of the whole response:

1. Revoke or rotate. Disable the key at its provider and issue a new one, kept in a secret store, never in the code. Every copy of the old key becomes useless at once.
2. Check its use. Read the provider’s logs for calls made with the old key since the commit was pushed. A leak that was used is an incident, with more to do.
3. Find every copy. Branches, tags, stashes, forks, CI logs, chat and tickets, so that the clean-up misses nothing.
4. Decide on a rewrite. A revoked key may need none; personal data, or a key you cannot revoke, does. A rewrite is disruptive for everyone.
5. Rewrite and push. git filter-repo in a fresh clone, then git push --force --mirror, then the host’s own clean-up, so that the shared repository stops handing out the old commits.
6. Replace every clone. Collaborators re-clone and fork owners clean their forks, because one old clone can push it all back.
7. Prevent. .gitignore, a pre-commit scanner and push protection, so that it does not happen again.

The OWASP Secrets Management Cheat Sheet puts the same first steps in its incident response: exposed keys are revoked immediately, a new secret is created quickly, preferably by an automated process, and the old one is removed from the systems where it was exposed, code and logs included. GitHub’s documentation on secret scanning gives the same advice for an alert: rotate the credential immediately, and it notes that scrubbing history is time-consuming and often unnecessary once the credential is revoked.

Find every copy

A key rarely stays in one file. In this clone it went into a configuration file that was later renamed and deleted, into the commit a tag points at, into a test script on an old branch, and into a change that was stashed:

from gitsession import session

# Asha's clone after a few weeks: the key went into a file that was later renamed and deleted, into a tag, into a
# script on an old branch, and into a stash.
HISTORY = r"""
git init -q shop
cd shop
mkdir app config
printf 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\n' > app/payments.py
git add app
git commit -q -m "Start the payments client"
printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > config/payments.env
git add config
git commit -q -m "Connect to the payment provider"
git tag -a v1.0.0 -m "shop 1.0.0"
git switch -q -c try-refunds
mkdir scripts
printf 'curl -H "Authorization: Bearer paydemo_live_FAKE0KEY0FOR0THIS0LESSON" https://pay.example.com/refunds\n' > scripts/smoke_test.sh
git add scripts
git commit -q -m "Try the refunds endpoint"
git switch -q main
git mv config/payments.env config/payments.local.env
git commit -q -m "Rename the payments config"
git rm -q config/payments.local.env
git commit -q -m "Remove the key from the repository"
printf 'KEY = "paydemo_live_FAKE0KEY0FOR0THIS0LESSON"\n' >> app/payments.py
git stash -q
"""

session(r"""
cd shop
# Commits on any branch or tag that added or removed the key
git log --all --oneline -S'paydemo_live_'
# A stash is a merge commit, and log skips the changes of merges unless asked (-m)
git log --all --oneline -m -S'paydemo_live_'
# Every file of every commit that still holds it
git grep -l 'paydemo_live_' $(git rev-list --all --abbrev-commit)
# The refs that lead to those commits
git for-each-ref --format='%(refname)'
git stash list
""", setup=HISTORY)

$ cd shop
# Commits on any branch or tag that added or removed the key
$ git log --all --oneline -S'paydemo_live_'
70e36a2 Remove the key from the repository
7d01d1f Try the refunds endpoint
24683b3 Connect to the payment provider
# A stash is a merge commit, and log skips the changes of merges unless asked (-m)
$ git log --all --oneline -m -S'paydemo_live_'
70e36a2 Remove the key from the repository
7d01d1f Try the refunds endpoint
8d2d2f2 (from 70e36a2) WIP on main: 70e36a2 Remove the key from the repository
8d2d2f2 (from 3335ec1) WIP on main: 70e36a2 Remove the key from the repository
24683b3 Connect to the payment provider
# Every file of every commit that still holds it
$ git grep -l 'paydemo_live_' $(git rev-list --all --abbrev-commit)
7d01d1f:config/payments.env
7d01d1f:scripts/smoke_test.sh
8d2d2f2:app/payments.py
24683b3:config/payments.env
6a33668:config/payments.local.env
# The refs that lead to those commits
$ git for-each-ref --format='%(refname)'
refs/heads/main
refs/heads/try-refunds
refs/stash
refs/tags/v1.0.0
$ git stash list
stash@{0}: WIP on main: 70e36a2 Remove the key from the repository

git log --all -S searched every branch and tag, and its first answer is incomplete. The pickaxe lists commits that change how many times the text occurs, so the rename, which only moved the key, is missing; and it skips the changes of merge commits unless you add -m, so the stash, which Git stores as a merge commit, only appears in the second search. git grep -l over git rev-list --all is the thorough check: it searches every file of every commit that any ref leads to, and it found the key in five places, among them 6a33668:config/payments.local.env, the renamed file, and 8d2d2f2:app/payments.py, the stash. git for-each-ref lists the four refs that lead there: two branches, the stash and the tag.

Look beyond the repository as well: other clones, forks, CI logs that printed the key, build artefacts, chat messages and tickets where someone pasted it. A scanner helps with the repository itself. gitleaks git -v --log-opts="--all" runs gitleaks, a scanner with rules for hundreds of key formats, over the whole history, and the Secret Scanner below checks pasted text in your browser.

Rewrite history with git filter-repo

git filter-branch, the old built-in tool, now carries a warning in its own manual: its pitfalls cannot be fixed compatibly, and it recommends another tool such as git filter-repo instead. git filter-repo is a single Python script under the MIT licence, which the Git project’s own documentation points to; install it with brew install git-filter-repo or pipx install git-filter-repo. GitHub’s guide needs version 2.47 or newer, for the --sensitive-data-removal option. The repository below is the one Asha pushed and Ravi cloned: the key was committed in the second of four commits and deleted in the fourth, and the tag v1.0.0 points at the third.

import filterrepo
from gitsession import session
from shop_history import HISTORY

# git filter-repo prints how long it took and, while it works, a running count of commits: both change from run to
# run, so they are written as "…" and as the final count.
TIMES = [(r"\d+\.\d+ seconds", "… seconds"), (r"(?:Parsed \d+ commits\s*)*Parsed (\d+) commits\s*", r"Parsed \1 commits\n")]

session(r"""
# The key was revoked at the provider first; that happens on the provider's website, not here.
# Rewrite in a fresh clone of the shared repository, never in a clone you work in
git clone -q --no-local shop.git cleanup
cd cleanup
printf 'paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > ../replacements.txt
git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
# Check: no commit, on any branch or tag, adds or removes the key any more
git log --all --oneline -m -S'paydemo_live_'
git show v1.0.0:config/payments.env
git log --oneline
# Push every branch and tag over the old ones
git push --force --mirror origin
cd ../shop.git
# fa3d285, the first changed commit filter-repo named, is still on the server until it collects its garbage
git cat-file -t fa3d285
git gc -q --prune=now
! git cat-file -t fa3d285
""", setup=HISTORY, mask=TIMES, path=[filterrepo.unpack("tools-unpacked")])

# The key was revoked at the provider first; that happens on the provider's website, not here.
# Rewrite in a fresh clone of the shared repository, never in a clone you work in
$ git clone -q --no-local shop.git cleanup
$ cd cleanup
$ printf 'paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > ../replacements.txt
$ git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
NOTICE: Fetching all refs from origin to make sure we rewrite
        all history that may reference the sensitive data, via
      git fetch -q --prune --update-head-ok --refmap "" origin +refs/*:refs/*

Parsed 4 commits
HEAD is now at 3c7ee3b Remove the key from the repository

New history written in … seconds; now repacking/cleaning...
You rewrote 3 (of 4) commits.

NOTE: First Changed Commit(s) is/are:
  fa3d28547aaf50dde687499233ea8c1912628c10
NOTE: LFS object orphaning not checked (LFS not in use)

Repacking your repo and cleaning out old unneeded objects
Completely finished after … seconds.

NEXT STEPS FOR YOUR SENSITIVE DATA REMOVAL:
  * If you are doing your rewrite in multiple steps, ignore these next steps
    until you have completed all your invocations of git-filter-repo.
  * See the "Sensitive Data Removal" subsection of the "DISCUSSION" section
    of the manual for more details about any of the steps below.
  * Inspect this repository and verify that the sensitive data is indeed
    completely removed from all commits.
  * Force push the rewritten history to the server:
      git push --force --mirror origin
  * Contact the server admins for additional steps they need to take; the
    First Changed Commit(s) may come in handy here.
  * Have other colleagues with a clone either discard their clone and reclone
    OR follow the detailed steps in the manual to repeatedly rebase and
    purge the sensitive data from their copy.  Again, the First Changed
    Commit(s) may come in handy.
  * See the "Prevent repeats and avoid future sensitive data spills" section
    of the manual.

# Check: no commit, on any branch or tag, adds or removes the key any more
$ git log --all --oneline -m -S'paydemo_live_'
$ git show v1.0.0:config/payments.env
PAYMENTS_API_KEY=***REMOVED***
PAYMENTS_URL=https://pay.example.com
$ git log --oneline
3c7ee3b Remove the key from the repository
6c14f22 Add refunds
626734b Connect to the payment provider
dd4a789 Start the payments client
# Push every branch and tag over the old ones
$ git push --force --mirror origin
To ~/shop.git
 + 0edbae7...3c7ee3b main -> main (forced update)
 + 2a347f2...80fe27c v1.0.0 -> v1.0.0 (forced update)
$ cd ../shop.git
# fa3d285, the first changed commit filter-repo named, is still on the server until it collects its garbage
$ git cat-file -t fa3d285
commit
$ git gc -q --prune=now
$ git cat-file -t fa3d285
fatal: Not a valid object name fa3d285
[exit status 128]

Step by step:

• A fresh clone, made with --no-local because the shared repository is a folder here. filter-repo checks that it runs in a fresh clone, because a rewrite throws away anything that exists only in your copy, and the manual asks for --no-local with local paths.
• --sensitive-data-removal first fetched every ref of origin, so that refs outside branches and tags are rewritten too, and then reported the first changed commit, fa3d285, and the steps that follow. GitHub’s guide says to keep that ID for the host.
• --replace-text read replacements.txt, one secret per line, and replaced each with ***REMOVED*** in every file of every commit, keeping the rest of the file, as git show v1.0.0:config/payments.env shows. To remove a whole file instead, use --invert-paths --path config/payments.env, and add a --path for every name it had through renames. A secret in a commit message needs --replace-message, which uses the same file format. Never commit replacements.txt: it holds the secret.
• Three of four commits got new IDs: every commit from the leak onwards changes, because each ID covers its parent’s. The pickaxe now finds nothing on any branch or tag.
• git push --force --mirror origin replaced every branch and tag on the server. A protected branch that blocks force pushes has to be unprotected for this push and protected again afterwards.

The last lines are the part people miss. After the push, the server still had the old commit fa3d285: a forced push changes which commits the refs point at, but old objects stay until the server collects its garbage, and until then a host may still show them to anyone with an old commit ID, as GitHub’s cached views do. Here git gc --prune=now on the server removed it.

On GitHub you cannot run that command. GitHub’s guide says to open a support ticket with the first changed commits and the number of affected pull requests, which filter-repo also records: pull request refs (refs/pull/…) are read-only, so your push cannot rewrite them, and GitHub Support can remove the affected pull requests’ references, clear cached views and run garbage collection. GitHub states that it will help only with sensitive data, and only where rotating the credential cannot remove the risk. Forks keep the old commits; only their owners can delete them.

A rewrite has other costs that GitHub’s guide lists: open pull requests lose their diffs and review comments, so merge or close them first; and git filter-repo strips commit and tag signatures, even from commits older than the leak.

Every old clone can bring it back

After the rewrite, Ravi still has his clone from before it. He commits and pulls as on any other day:

import filterrepo
from gitsession import session
from shop_history import HISTORY

# The rewrite of the previous session, done without showing it: a fresh clone, git filter-repo, a forced push.
CLEANED = HISTORY + r"""
git clone -q --no-local shop.git cleanup
cd cleanup
printf 'paydemo_live_FAKE0KEY0FOR0THIS0LESSON\n' > ../replacements.txt
git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
git push -q --force --mirror origin
"""

session(r"""
# The server is clean
git -C shop.git log --all --oneline -m -S'paydemo_live_'
# Ravi still has his clone from before the rewrite, and works as usual
cd ravi
printf 'def receipt(order): ...\n' > app/receipts.py
git add app/receipts.py
git commit -q -m "Add receipts"
git pull -q --no-rebase
git push -q
# One pull and one push later, the old history and the key are back on the server
git -C ../shop.git log --all --oneline -m -S'paydemo_live_'
git -C ../shop.git log --oneline --graph
""", setup=CLEANED, path=[filterrepo.unpack("tools-unpacked")])

# The server is clean
$ git -C shop.git log --all --oneline -m -S'paydemo_live_'
# Ravi still has his clone from before the rewrite, and works as usual
$ cd ravi
$ printf 'def receipt(order): ...\n' > app/receipts.py
$ git add app/receipts.py
$ git commit -q -m "Add receipts"
$ git pull -q --no-rebase
$ git push -q
# One pull and one push later, the old history and the key are back on the server
$ git -C ../shop.git log --all --oneline -m -S'paydemo_live_'
0edbae7 Remove the key from the repository
fa3d285 Connect to the payment provider
$ git -C ../shop.git log --oneline --graph
*   ae9dc23 Merge branch 'main' of ../shop
|\  
| * 3c7ee3b Remove the key from the repository
| * 6c14f22 Add refunds
| * 626734b Connect to the payment provider
* | c1304ff Add receipts
* | 0edbae7 Remove the key from the repository
* | 3832569 Add refunds
* | fa3d285 Connect to the payment provider
|/  
* dd4a789 Start the payments client

git pull found two histories that share only their first commit and merged them, and git push sent the old commits back, with the key. The graph shows every rewritten commit twice. This is why both GitHub’s guide and the filter-repo manual insist on the clean-up of other clones:

• the simple way: everyone deletes their clone and clones again, after saving any work that exists only there;
• the careful way, for a clone with work worth keeping: delete all tags and fetch them again with git fetch --prune --tags, rebase every local branch onto the rewritten history (never merge it, since one merge brings the old history back), then drop the old objects with git reflog expire --expire=now --all and git gc --prune=now, which also deletes every stash entry;
• check the result with git cat-file -t fa3d285, using each first changed commit that filter-repo printed: it has to fail, as it did on the server above.

Ask everyone to stop pushing while you clean up, because anything pushed in between is either lost or has to be cleaned again.

Prevent the next leak

Three layers, from the cheapest to the strongest:

import shutil

from gitsession import session


def hook(home):
    (home / "hooks").mkdir()
    shutil.copy("hooks/pre-commit", home / "hooks" / "pre-commit")


session(r"""
git init -q shop
cd shop
mkdir app config
# Ignore the real settings before the first commit, and commit an example without values
printf 'config/*.env\n' > .gitignore
printf 'PAYMENTS_API_KEY=\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env.example
printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env
printf 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\n' > app/payments.py
git add .
git status --short
git check-ignore -v config/payments.env
git commit -q -m "Start the payments client"
# A pre-commit hook checks every commit before it is made
cp ~/hooks/pre-commit .git/hooks/pre-commit
printf 'KEY = "paydemo_live_FAKE0KEY0FOR0THIS0LESSON"  # quick test\n' > app/settings.py
git add app/settings.py
! git commit -q -m "Hard-code the key for a quick test"
printf 'import os\nKEY = os.environ.get("PAYMENTS_API_KEY", "")\n' > app/settings.py
git add app/settings.py
git commit -q -m "Read the key from the environment"
git log --oneline
""", prepare=hook)

$ git init -q shop
$ cd shop
$ mkdir app config
# Ignore the real settings before the first commit, and commit an example without values
$ printf 'config/*.env\n' > .gitignore
$ printf 'PAYMENTS_API_KEY=\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env.example
$ printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env
$ printf 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\n' > app/payments.py
$ git add .
$ git status --short
A  .gitignore
A  app/payments.py
A  config/payments.env.example
$ git check-ignore -v config/payments.env
.gitignore:1:config/*.env	config/payments.env
$ git commit -q -m "Start the payments client"
# A pre-commit hook checks every commit before it is made
$ cp ~/hooks/pre-commit .git/hooks/pre-commit
$ printf 'KEY = "paydemo_live_FAKE0KEY0FOR0THIS0LESSON"  # quick test\n' > app/settings.py
$ git add app/settings.py
$ git commit -q -m "Hard-code the key for a quick test"
pre-commit: these staged lines look like secrets, so the commit was stopped:
  app/settings.py, line 1: a payment key (paydemo_live_...)
Move the value to an environment variable or a file that .gitignore lists, then stage again.
If it was ever pushed or shared, revoke it first.
[exit status 1]
$ printf 'import os\nKEY = os.environ.get("PAYMENTS_API_KEY", "")\n' > app/settings.py
$ git add app/settings.py
$ git commit -q -m "Read the key from the environment"
$ git log --oneline
2bc006e Read the key from the environment
cf7c8bf Start the payments client

• Ignore the file before the first commit. config/*.env in .gitignore kept the real settings out of git add ., and git check-ignore -v says which rule did it. The committed config/payments.env.example shows everyone which settings exist, without their values. The application reads the key from its environment.
• Check every commit before it is made. Git runs .git/hooks/pre-commit before each commit and stops the commit when the hook exits with a non-zero status. This one reads only the staged lines and stopped the hard-coded key. It is a short example; gitleaks ships a ready-made hook for the pre-commit framework (a .pre-commit-config.yaml that names https://github.com/gitleaks/gitleaks and the hook gitleaks), with its full set of rules. Hooks live in each clone and git commit --no-verify skips them, so they catch mistakes but cannot enforce anything.
• Let the server refuse it. GitHub’s push protection blocks a push that contains a supported secret before it reaches the repository. For your own account it is on by default and covers pushes to public repositories; for a repository or an organisation an administrator turns it on, and it needs GitHub Secret Protection. GitHub also runs secret scanning on public repositories at no cost, over the whole history, and when it finds a secret in the format of one of its partners, it tells that provider, which may revoke it.

#!/usr/bin/env python3
"""A pre-commit hook that stops a commit when the staged changes contain something that looks like a secret.

Copy it to .git/hooks/pre-commit and make it executable (chmod +x). Git runs it before every commit; a non-zero exit
status stops the commit. It reads only the lines being added, so old history and unchanged files do not matter. The
patterns are examples: a real project uses a maintained scanner such as gitleaks, which knows hundreds of formats.
"""

import re
import subprocess
import sys

PATTERNS = [
    ("a payment key (paydemo_live_...)", re.compile(r"paydemo_live_[A-Za-z0-9]{16,}")),
    ("a private key", re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----")),
    # api_key = "a long literal": a quoted value of 16 or more characters, not a placeholder such as ${...} or <...>
    ("a secret assigned to a name", re.compile(r"(?i)\w*(?:key|token|secret|password)\w*\s*[:=]\s*['\"][^'\"\s$<{]{16,}['\"]")),
    # NAME_TOKEN=value, as in .env files
    ("a secret in a settings line", re.compile(r"^\+\s*[A-Z0-9_]*(?:KEY|TOKEN|SECRET|PASSWORD)[A-Z0-9_]*=[^\s$<{'\"]{16,}\s*$")),
]

diff = subprocess.run(["git", "diff", "--cached", "--no-color", "--unified=0", "--diff-filter=ACMR"],
                      capture_output=True, text=True, check=True).stdout
found, path, line = [], None, 0
for text in diff.splitlines():
    if text.startswith("+++ b/"):
        path = text[6:]
    elif text.startswith("@@"):
        line = int(re.search(r"\+(\d+)", text).group(1))
    elif text.startswith("+") and not text.startswith("+++"):
        for what, pattern in PATTERNS:
            if pattern.search(text):
                found.append(f"  {path}, line {line}: {what}")
                break
        line += 1
if found:
    print("pre-commit: these staged lines look like secrets, so the commit was stopped:")
    print("\n".join(found))
    print("Move the value to an environment variable or a file that .gitignore lists, then stage again.")
    print("If it was ever pushed or shared, revoke it first.")
    sys.exit(1)

Exercise: scrub a repository

scrub_check.py makes scrub-practice/origin.git the first time you run it: a shared repository whose history holds the fake key in four places, a file, the same file after a rename, the commit a tag points at and a script on an old branch. Remove the key from all of them with git filter-repo, push the result back to origin.git, and write scrub-practice/INCIDENT.md, a short incident note whose first step is the right one. Run the file again after each step; it reads every object that a clone of origin.git would receive.

"""Scrub practice. The first run makes scrub-practice/origin.git, a shared repository whose history holds a fake
payment key in four places: a file, the same file after a rename, a tag and a script on an old branch. Remove the key
from every commit with git filter-repo, push the result back, write scrub-practice/INCIDENT.md, and run this file
again. It reads every object that a clone of origin.git would receive, and checks that the rest of the history is
still there.
"""

import os
import subprocess
import sys
from pathlib import Path

PRACTICE = Path("scrub-practice")
ORIGIN = PRACTICE / "origin.git"
SECRET = b"paydemo_live_FAKE0KEY0FOR0THIS0LESSON"
SAFE = ["-c", "commit.gpgSign=false", "-c", "tag.gpgSign=false", "-c", "core.hooksPath=no-hooks", "-c", "init.defaultBranch=main"]
WHO = {"GIT_AUTHOR_NAME": "Asha Rao", "GIT_AUTHOR_EMAIL": "asha@example.com",
       "GIT_COMMITTER_NAME": "Asha Rao", "GIT_COMMITTER_EMAIL": "asha@example.com"}
STEPS = [  # (message, files to write, files to rename or delete)
    ("Add a README", {"README.md": "# Shop\n"}, {}),
    ("Start the payments client", {"app/payments.py": 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\n'}, {}),
    ("Connect to the payment provider", {"config/payments.env": f"PAYMENTS_API_KEY={SECRET.decode()}\n"}, {}),
    ("Add refunds", {"app/payments.py": 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\ndef refund(order): ...\n'}, {}),
    ("Rename the payments config", {}, {"config/payments.env": "config/payments.local.env"}),
    ("Remove the key from the repository", {".gitignore": "config/*.env\n"}, {"config/payments.local.env": None}),
]


def git(*args, cwd=PRACTICE, when=None, data=None):
    env = {**os.environ, **WHO}
    if when is not None:
        env["GIT_AUTHOR_DATE"] = env["GIT_COMMITTER_DATE"] = f"@{1767240000 + 3600 * when} +0530"
    done = subprocess.run(["git", *SAFE, *args], cwd=cwd, env=env, input=data, capture_output=True)
    if done.returncode:
        sys.exit(f"git {' '.join(args)} failed:\n{done.stderr.decode()}")
    return done.stdout


def make_practice():
    work = PRACTICE / "work"
    git("init", "-q", "--bare", "origin.git")
    git("clone", "-q", "origin.git", "work")
    for n, (message, writes, moves) in enumerate(STEPS):
        for name, text in writes.items():
            (work / name).parent.mkdir(parents=True, exist_ok=True)
            (work / name).write_text(text)
            git("add", name, cwd=work)
        for old, new in moves.items():
            git(*(("mv", old, new) if new else ("rm", "-q", old)), cwd=work)
        git("commit", "-q", "-m", message, cwd=work, when=n)
        if message == "Connect to the payment provider":
            git("tag", "-a", "v0.9.0", "-m", "shop 0.9.0", cwd=work, when=n)
        if message == "Add refunds":
            git("switch", "-q", "-c", "try-refunds", cwd=work)
            (work / "scripts").mkdir()
            (work / "scripts/smoke_test.sh").write_text(f'curl -H "Authorization: Bearer {SECRET.decode()}" https://pay.example.com/refunds\n')
            git("add", "scripts/smoke_test.sh", cwd=work)
            git("commit", "-q", "-m", "Try the refunds endpoint", cwd=work, when=n)
            git("switch", "-q", "main", cwd=work)
    git("push", "-q", "--follow-tags", "origin", "main", "try-refunds", cwd=work)
    print(f"Made {ORIGIN}: its history holds the key. Clean it, then run this file again.")


if not ORIGIN.exists():
    PRACTICE.mkdir(exist_ok=True)
    make_practice()

# Every object a clone would receive: files (blobs), folders (trees), commits and tags. Which ones hold the key?
ids = [line.split(" ", 1)[0] for line in git("rev-list", "--objects", "--all", cwd=ORIGIN).decode().splitlines()]
batch = git("cat-file", "--batch", cwd=ORIGIN, data="".join(f"{i}\n" for i in ids).encode())
leaky_files, where, at = set(), set(), 0
for object_id in ids:
    header_end = batch.index(b"\n", at)
    _, kind, size = batch[at:header_end].split()
    body = batch[header_end + 1 : header_end + 1 + int(size)]
    at = header_end + 1 + int(size) + 1
    if SECRET in body:
        if kind == b"blob":
            leaky_files.add(object_id)
        else:
            where.add(f"the message of {kind.decode()} {object_id[:7]}")
# The paths, in any commit, at which a file holding the key appears.
for commit in git("rev-list", "--all", cwd=ORIGIN).decode().split():
    for entry in git("ls-tree", "-r", commit, cwd=ORIGIN).decode().splitlines():
        meta, path = entry.split("\t", 1)
        if meta.split()[2] in leaky_files:
            where.add(path)
refs = git("for-each-ref", "--format=%(refname:short)", cwd=ORIGIN).decode().split()
subjects = git("log", "--format=%s", "main", cwd=ORIGIN).decode().splitlines()
payments = git("show", "main:app/payments.py", cwd=ORIGIN).decode() if "main" in refs else ""
note = PRACTICE / "INCIDENT.md"
first = next((l for l in note.read_text().splitlines() if l.strip() and not l.startswith("#")), "") if note.exists() else ""

checks = [
    (not where, "the key is in no object a clone receives" if not where else f"the key is still in: {', '.join(sorted(where))}"),
    ({"main", "try-refunds", "v0.9.0"} <= set(refs), f"branches and tags kept: {', '.join(sorted(refs))}"),
    ({"Add a README", "Start the payments client", "Add refunds"} <= set(subjects) and "def refund" in payments,
     "the commits that never held the key are still on main"),
    (any(word in first.lower() for word in ("revoke", "rotate")),
     "INCIDENT.md starts with revoking or rotating the key" if note.exists() else "INCIDENT.md is missing"),
]
for ok, text in checks:
    print(("✓ " if ok else "✗ ") + text)
passed = sum(ok for ok, _ in checks)
print(f"{passed} of {len(checks)} checks pass")
sys.exit(0 if passed == len(checks) else 1)

Made scrub-practice/origin.git: its history holds the key. Clean it, then run this file again.
✗ the key is still in: config/payments.env, config/payments.local.env, scripts/smoke_test.sh
✓ branches and tags kept: main, try-refunds, v0.9.0
✓ the commits that never held the key are still on main
✗ INCIDENT.md is missing
2 of 4 checks pass

First hint: work in a fresh clone, git clone --no-local scrub-practice/origin.git scrub-practice/cleanup. Either replace the key with --replace-text, or remove the files with --invert-paths and one --path for every name the file had, including the old branch’s script. The check also wants the commits that never held the key, so do not throw away more than the key.

Interview questions

You committed an API key and deleted the file in the next commit. Is the key gone, and what do you do first?

No. The commit that added the key still holds it, and git log -S or git show <commit>:<path> prints it for anyone who can read the repository, in every clone, fork and CI copy made since. The first step is to revoke or rotate the key at the service that issued it, because that makes every copy useless at once. Then check the provider’s logs for use since the push, find every other copy, and decide whether the history needs rewriting with git filter-repo, which forces every collaborator to replace their clone. If the key was never pushed anywhere, git reset or an interactive rebase in your own clone is enough, but rotating it is still the safe choice.

Run the sessions yourself

Three helper files sit behind these sessions. Save them with a session script, add hooks/pre-commit for prevent.py and a tools folder holding git-filter-repo-2.47.0.tar.xz from the project’s releases page, then start the script with Python 3, say python3 recontaminate.py. Nothing leaves the learner folder the helper creates, which it treats as the home directory as well, and commit times are fixed, so the IDs agree with this page. filterrepo.py checks the archive’s SHA-256 before unpacking it, and shop_history.py holds the commands that build the leaky repository.

"""Plays a terminal session with real Git: each command is printed after "$ ", then everything it printed.

The session runs in a new folder called learner/, which is also its home folder, so your own Git settings change
nothing, and GIT_CONFIG_NOSYSTEM=1 makes Git skip the computer's system-wide settings file too. Its ~/.gitconfig
holds only a name, an email address and init.defaultBranch = main, the settings the lesson on installing Git sets
up. In the output, "~" stands for that folder. Commit and tag times are fixed, because an object's ID covers the
time it was made: with the same files, messages, names and times, every run makes the same IDs. A command can still
set its own time, as in a shell: GIT_COMMITTER_DATE="@1767326400 +0530" git commit ...

Each line of a script is one /bin/sh command (macOS and Linux), except "cd", which the session does itself. A line
that starts with "# " is printed as a comment. A command written as "! command" is meant to fail: its exit status
is printed and the session goes on. Any other command that fails stops the session with exit status 1.

session() can also be given:
  user     the name and email address of the person at the keyboard (Asha Rao, asha@example.com by default);
  setup    commands that build the starting point first, without being shown (they are in the example's code);
  mask     (pattern, replacement) pairs for text that is new on every run, such as how long a command took;
  prepare  a function that receives the new home folder before any command runs, to put files into it;
  path     more folders in which to look for commands, such as the folder of a downloaded tool.
"""

import os
import re
import shutil
import subprocess
import sys
from pathlib import Path

HOME = Path("learner").resolve()
# One fixed moment in Git's own date format: seconds since 1970, then the time zone (India, UTC+05:30).
FIXED_TIME = "@1767240000 +0530"


def _environment(path):
    # Git's own folder comes first, so every command runs that Git directly (on macOS, /usr/bin/git is a small
    # launcher that looks Git up again on every call).
    git_folder = subprocess.run(["git", "--exec-path"], capture_output=True, text=True, check=True).stdout.strip()
    folders = [git_folder, *(str(Path(p).resolve()) for p in path), os.environ.get("PATH", "/usr/bin:/bin")]
    env = {
        "PATH": os.pathsep.join(folders),
        "HOME": str(HOME),
        "XDG_CONFIG_HOME": str(HOME / ".config"),
        "GIT_CONFIG_NOSYSTEM": "1",
        "GIT_AUTHOR_DATE": FIXED_TIME,
        "GIT_COMMITTER_DATE": FIXED_TIME,
        "LANG": "C",
        "LC_ALL": "C",
        "TERM": "dumb",
    }
    if "TMPDIR" in os.environ:
        env["TMPDIR"] = os.environ["TMPDIR"]
    return env


def _with_tilde(text):
    """Writes the home folder as "~", however the operating system spells its path."""
    for spelling in sorted({str(HOME), os.path.realpath(HOME)}, key=len, reverse=True):
        text = text.replace(spelling, "~")
    return text


def _commands(script):
    for line in script.strip().splitlines():
        if line.strip():
            yield line.strip()


def session(script, user=("Asha Rao", "asha@example.com"), setup="", mask=(), prepare=None, path=()):
    """Runs the script's commands one by one, starting in a new home folder."""
    shutil.rmtree(HOME, ignore_errors=True)
    HOME.mkdir()
    (HOME / ".gitconfig").write_text(f"[user]\n\tname = {user[0]}\n\temail = {user[1]}\n[init]\n\tdefaultBranch = main\n")
    if prepare:
        prepare(HOME)
    env = _environment(path)
    folder = HOME
    for command in _commands(setup):
        if command.startswith("# "):
            continue
        if command.startswith("cd "):
            folder = (folder / command[3:].strip()).resolve()
            continue
        done = subprocess.run(command, shell=True, cwd=folder, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
        if done.returncode != 0:
            sys.exit(f"setup command failed: {command}\n{_with_tilde(done.stdout)}")
    folder = HOME
    for command in _commands(script):
        if command.startswith("# "):
            print(command)
            continue
        meant_to_fail = command.startswith("! ")
        if meant_to_fail:
            command = command[2:]
        print("$", command)
        if command.startswith("cd "):
            folder = (folder / command[3:].strip()).resolve()
            continue
        done = subprocess.run(command, shell=True, cwd=folder, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
        output = _with_tilde(done.stdout)
        for pattern, replacement in mask:
            output = re.sub(pattern, replacement, output)
        if output and not output.endswith("\n"):
            output += "\n"
        print(output, end="")
        if done.returncode != 0:
            print(f"[exit status {done.returncode}]")
            if not meant_to_fail:
                sys.exit(1)
        elif meant_to_fail:
            sys.exit(f"{command} was meant to fail, but it worked")

"""Unpacks git-filter-repo, the history rewriting tool the Git project recommends instead of git filter-branch, from
its 2.47.0 release archive (MIT licence), so that the sessions can run it like an installed command.

The archive is checked first, against the SHA-256 it had when it was downloaded from the project's release page:
a tool that rewrites every commit you have should be exactly the file its authors published. On your own computer,
install it with a package manager instead (brew install git-filter-repo, or pipx install git-filter-repo), or put
the single git-filter-repo script from a release on your PATH.
"""

import hashlib
import tarfile
from pathlib import Path

ARCHIVE = Path(__file__).parent / "tools" / "git-filter-repo-2.47.0.tar.xz"
SHA256 = "4662cbe5918196a9f1b5b3e1211a32e61cff1812419c21df4f47c5439f09e902"


def unpack(into):
    """Checks the archive, unpacks it into the folder `into` and returns the folder that holds git-filter-repo."""
    if hashlib.sha256(ARCHIVE.read_bytes()).hexdigest() != SHA256:
        raise SystemExit(f"{ARCHIVE.name} is not the published 2.47.0 release archive")
    with tarfile.open(ARCHIVE) as archive:
        archive.extractall(into, filter="data")
    return Path(into) / "git-filter-repo-2.47.0"

"""The history that the rewrite sessions of this lesson start from: a shared repository, shop.git, that leaked a key.

The key is fake, like every key on this page. In order, Asha's commits are:
  1. Start the payments client          app/payments.py, which reads the key from the environment, and README.md
  2. Connect to the payment provider    config/payments.env, holding the key: this is the leak
  3. Add refunds                        then the annotated tag v1.0.0, so the tag points at history with the key
  4. Remove the key from the repository deletes config/payments.env, adds .gitignore and an example file
Asha pushed it all, with the tag, and Ravi cloned the repository afterwards.
"""

SECRET = "paydemo_live_FAKE0KEY0FOR0THIS0LESSON"

HISTORY = r"""
git init -q --bare shop.git
git clone -q shop.git asha
cd asha
mkdir app config
printf 'import os\nKEY = os.environ["PAYMENTS_API_KEY"]\n' > app/payments.py
printf '# Shop\n' > README.md
git add app README.md
git commit -q -m "Start the payments client"
printf 'PAYMENTS_API_KEY=paydemo_live_FAKE0KEY0FOR0THIS0LESSON\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env
git add config/payments.env
git commit -q -m "Connect to the payment provider"
printf 'def refund(order): ...\n' >> app/payments.py
git commit -q -am "Add refunds"
git tag -a v1.0.0 -m "shop 1.0.0"
printf 'PAYMENTS_API_KEY=\nPAYMENTS_URL=https://pay.example.com\n' > config/payments.env.example
git rm -q config/payments.env
printf 'config/*.env\n' > .gitignore
git add .gitignore config/payments.env.example
git commit -q -m "Remove the key from the repository"
git push -q --follow-tags origin main
cd ..
git clone -q shop.git ravi
# Ravi's clone names the shared repository by a relative path, so that merge messages are the same on every computer
git -C ravi remote set-url origin ../shop.git
"""

Key takeaways

• A committed secret is compromised: revoke or rotate it first, at its provider; that may be all you need.
• Deleting the file in a new commit removes nothing from history; git log -S and git show <commit>:<path> still find it.
• Search everything: git log --all -m -S for the commits that added or removed it, and git grep over git rev-list --all for every file that still holds it, on branches, tags and stashes.
• Rewrite with git filter-repo --sensitive-data-removal (2.47 or newer) in a fresh clone, using --replace-text or --invert-paths --path, verify, then git push --force --mirror.
• The host keeps old commits until it collects garbage, pull request refs and cached views; on GitHub, ask Support with the first changed commits. Forks belong to their owners.
• One old clone can push it all back: everyone re-clones, or rebases, expires the reflog and collects garbage.
• Prevent it with .gitignore and an example file, a pre-commit scanner such as gitleaks, and push protection.