Rewriting a Git repo to remove secrets from the history#

I decided to make a GitHub repository public today that had previously been private. Unfortunately the revision history of that repository included some secret values, one of which I could not figure out a way to revoke.

I found a way to rewrite the entire repository from scratch to omit those values, using BFG Repo-Cleaner by Roberto Tyley. It’s a tool that is recommended on the GitHub help page.

Installing the tool#

BFG is a Java app. My Mac had Java installed, so installing it was just a case of downloading the latest .jar file from the download link on the website.

Running java -jar bfg-1.14.0.jar confirmed that this worked:

1
% java -jar bfg-1.14.0.jar
2
bfg 1.14.0
3
Usage: bfg [options] [<repo>]
4
...

Removing the secrets#

I made a list of secret strings that I wanted to remove and saved them in a file called secrets.txt.

Then I followed the BFG instructions to create a fresh clone of my repository:

1
cd /tmp
2
git clone ~/Dropbox/Development/my-repo --mirror

This gave me a /tmp/my-repo.git folder containing the git history but not any actually checked out files.

Then I ran this:

1
cd /tmp/my-repo.git
2
java -jar /tmp/bfg-1.14.0.jar --replace-text /tmp/secrets.txt

I got the following output:

1
Using repo : /private/tmp/my-repo.git
2

3
Found 11 objects to protect
4
Found 3 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/main
5

6
Protected commits
7
-----------------
8

9
These are your protected commits, and so their contents will NOT be altered:
10

11
 * commit 6badd000 (protected by 'HEAD')
12

13
Cleaning
14
--------
15

16
Found 11 commits
17
Cleaning commits:       100% (11/11)
18
Cleaning commits completed in 98 ms.
19

20
Updating 2 Refs
21
---------------
22

23
  Ref                        Before     After
24
  ----------------------------------------------
25
  refs/heads/main          | 6badd000 | 15881e3b
26
  refs/remotes/origin/main | 4af64070 | d404ebd0
27

28
Updating references:    100% (2/2)
29
...Ref update completed in 35 ms.
30

31
Commit Tree-Dirt History
32
------------------------
33

34
  Earliest      Latest
35
  |                  |
36
  D D D D DD D D D D m
37

38
  D = dirty commits (file tree fixed)
39
  m = modified commits (commit message or parents changed)
40
  . = clean commits (no changes to file tree)
41

42
                          Before     After
43
  -------------------------------------------
44
  First modified commit | ff14c49c | be204284
45
  Last dirty commit     | 4af64070 | d404ebd0
46

47
Changed files
48
-------------
49

50
  Filename      Before & After
51
  -----------------------------------------------------------
52
  __init__.py | d5936659 ⇒ 121d236c, d96951be ⇒ ac125def, ...
53

54

55
In total, 30 object ids were changed. Full details are logged here:
56

57
  /private/tmp/my-repo.git.bfg-report/2023-01-24/17-11-55
58

59
BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

And this had done the trick!

All of my commit hashes had changed, which was expected for a rewrite of the repository.

I knew that my first commit to the repo had a secret in it - so I ran git log --pretty=oneline | tail -n 1 to find that first commit hash, then ran git show be20428438bf42452f1a783d7b00dad84effc002 to see the commit… and sure enough it had this in it:

1
<p><label>JWT: <input name="jwt" value="***REMOVED***"></label></p>

That ***REMOVED*** used to be one of the secrets that I had wanted to removed.

Final steps#

As suggested in the output from the command, I ran this:

1
git reflog expire --expire=now --all && git gc --prune=now

I didn’t want to risk anything persisting on GitHub, so I deleted the entire repository and then created a new one with the same name. Then I pushed my repo to that:

1
git remote remove origin
2
git remote add origin git@github.com:simonw/my-repo.git
3
git push -u origin main

This was likely unnecessary: the GitHub documentation says that a force push should remove all traces of the old commits.