How to Remove Sensitive Information from Git History

Introduction

We've all been there: you accidentally commit sensitive information like API keys, passwords, or credentials to your Git repository. Even if you quickly remove it in the next commit, the damage is done—the sensitive data remains in your Git history forever, accessible to anyone with repository access.

Fortunately, there's a way to permanently remove this sensitive information from your repository's history. In this comprehensive guide, I'll walk you through the process of using git filter-repo to scrub sensitive data from your Git history safely and effectively.

Why Is This Necessary?

When you commit a file to Git, it becomes part of the repository's permanent history. Simply deleting the file or removing the sensitive line in a subsequent commit doesn't actually remove it from the repository—it only removes it from the current state. The data remains accessible through:

• Git history (git log, git show)
• Historical commits and branches
• Repository clones that others may have
• GitHub's commit history and search features

This is why properly removing sensitive data requires rewriting the repository's entire history.

Prerequisites

Before starting, ensure you have:

• Git installed: Version 2.22 or higher
• git filter-repo: This powerful tool replaces the older git filter-branch
• Repository access: Admin rights to force-push to the remote repository
• Team coordination: All collaborators must be informed of the upcoming history rewrite

Installing git filter-repo

On most systems, you can install git filter-repo using:

# Using pip
pip install git-filter-repo

# On macOS with Homebrew
brew install git-filter-repo

# On Ubuntu/Debian
sudo apt-get install git-filter-repo

Step-by-Step Guide to Remove Sensitive Data

1Prepare a Fresh Clone

Always operate on a fresh clone to avoid accidental corruption of your working repository. This ensures you have a clean slate and can easily restart if something goes wrong.

git clone git@github.com:USERNAME/REPO.git
cd REPO

2Create a Secrets File

Create a file listing all the exact strings you want to remove, one per line. This file will be used by git filter-repo to identify and replace sensitive information throughout your repository's history.

# Example: If your secret is in task.txt
echo "password: testing git history removal" > secrets-to-remove.txt

# You can add multiple secrets
echo "API_KEY=abc123xyz789" >> secrets-to-remove.txt
echo "SECRET_TOKEN=super_secret_value" >> secrets-to-remove.txt

3Run git filter-repo

Now comes the main step: running git filter-repo to replace all occurrences of the sensitive data throughout your repository's entire history.

git filter-repo --replace-text secrets-to-remove.txt

By default, git filter-repo will replace matched strings with ***REMOVED***. If you want to specify custom replacement text, you can format your secrets file like this:

password: testing git history removal==>PASSWORD_REDACTED
API_KEY=abc123xyz789==>EXAMPLE_API_KEY

4Add the Remote Back

After running git filter-repo, you need to re-add your remote repository:

git remote add origin git@github.com:USERNAME/REPO.git

# Verify the remote was added correctly
git remote -v

You should see output like:

origin  git@github.com:USERNAME/REPO.git (fetch)
origin  git@github.com:USERNAME/REPO.git (push)

5Force Push the Cleaned History

Now it's time to push your cleaned repository history back to GitHub. This step requires force-pushing, which will overwrite the remote repository's history.

# Push all branches
git push origin --force --all

# Push all tags
git push origin --force --tags

The --all flag ensures all branches are pushed, and --tags ensures all tags are updated as well.

6Verify Removal

After pushing, it's crucial to verify that the sensitive data has been completely removed. Use Git's search functionality to look for any remaining traces:

git log -S"password: testing git history removal" --all

If the secret was successfully removed, this command should return no results. You can also search for other patterns:

# Search for specific patterns
git log -S"API_KEY" --all
git log -S"SECRET_TOKEN" --all

# Search in all file contents
git grep "password: testing" $(git rev-list --all)

7Inform Collaborators

All collaborators must reset their local repositories because the history has been rewritten. Share these instructions with your team:

# For collaborators: Reset your local repository
git fetch origin
git reset --hard origin/main

# If they were working on a feature branch
git fetch origin
git checkout feature-branch
git reset --hard origin/feature-branch

Additional Considerations

Rotating Compromised Credentials

Removing sensitive data from Git history is just the first step. You must also:

• Immediately rotate all exposed credentials (change passwords, regenerate API keys)
• Review access logs for any unauthorized usage
• Update any systems or services that use the compromised credentials
• Consider implementing secret scanning tools to prevent future incidents

Preventing Future Incidents

To avoid accidentally committing sensitive information in the future:

• Use .gitignore files to exclude sensitive configuration files
• Implement pre-commit hooks to scan for secrets before committing
• Use environment variables or secret management tools (AWS Secrets Manager, HashiCorp Vault)
• Enable GitHub Secret Scanning for your repositories
• Use git-secrets or truffleHog to detect secrets in commits
• Implement code review processes to catch sensitive data before merging

Alternative Approaches

Depending on your situation, you might consider:

• BFG Repo-Cleaner: A faster alternative to git filter-repo for large repositories
• Starting fresh: For repositories with extensive contamination, creating a new repository might be simpler
• GitHub Support: For public repositories, contact GitHub support to fully remove cached data

Common Pitfalls and Troubleshooting

Issue: Remote Rejection During Force Push

If you receive an error like "remote: error: GH006: Protected branch update failed", your branch has protection rules enabled. You'll need to:

• Temporarily disable branch protection rules in repository settings
• Perform the force push
• Re-enable branch protection rules

Issue: Large Repository Size After Cleanup

If your repository is still large after removing files, you may need to run garbage collection:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

Issue: Secrets Still Appearing in Search

GitHub caches repository data. After cleaning your history:

• Wait 24-48 hours for GitHub's cache to clear
• For immediate removal, contact GitHub Support
• Consider making the repository private temporarily

Conclusion

Accidentally committing sensitive information to a Git repository can be a serious security issue, but it's not irreversible. By following this comprehensive guide, you can permanently remove sensitive data from your Git history using git filter-repo.

Most importantly, implement preventive measures to ensure this never happens again. Use environment variables, secret management tools, and pre-commit hooks to catch sensitive data before it enters your repository.

Stay secure, and happy coding!