Backtestshugorun-20261013-011256#14937
Add build.cleanDestinationDir config option
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches add a `build.cleanDestinationDir` struct with an enable switch and glob-based retention matched against publishDir-relative paths. Both would let the reporter keep `.git`, `.domains`, `_redirects` and `foo/bar/.mysecretfile`. **Patch B** is the safer fix. - It adds a single `retain` list on top of the existing hardcoded protections, which still preserve `.gitignore`, `.gitattributes` and dot-directories. - The root `cleanDestinationDir` key and the `--cleanDestinationDir` flag keep working unchanged: the build decoder ORs them into `Enable`. - It also accepts the bool shorthand `build.cleanDestinationDir = true`, including from env vars. - Its glob compilation is explicit, uses '/' as the separator, normalizes leading slashes, and wraps errors with the option name. - Tests: - A unit test covers decoding (bool, bool-string and map forms), matching semantics (root-only versus `**` patterns, nested paths) and invalid-glob errors. - A new testscript covers the issue's exact scenario: enabled via config with no static dir, enabled via flag with a static dir, the bool form, and the disabled case. - Docs are updated. **Patch A** is a broader redesign that adds risk without adding value for the issue. - It deprecates the root `cleanDestinationDir` key. - It reroutes the CLI flag through `keyMap` to a nested dotted key, `build.cleanDestinationDir.enable`, which depends on nested Set semantics working for flags. - It replaces the hardcoded keep logic with default `keepFiles`/`keepDirs` globs. Any user who sets `keepFiles` silently loses the default protection for `.gitignore`/`.gitattributes` and dot-dirs; its own scenario 4 output shows `.gitignore` and `.hidden/` being deleted. That is a surprising behavior regression for anyone adopting the new option. - Its negation semantics (`! **.html` meaning 'keep everything except html') make the keep list double as a delete list, which is harder to reason about. - It does add `.git` files to the defaults, which helps the worktree case, and its testscript coverage is reasonable. - It has no unit tests and no docs. On balance, B resolves the issue with less risk, preserves backward compatibility, and has more thorough tests.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
1h 24m
Wall-clock time in the container
Full test suite
No new failures
2,077 passed · 42 skipped · 2,139 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
291vs 194 merged
Changed lines, ours vs merged
The issue
View on GitHub<!-- Describe this new feature. Think about if it really belongs in the Hugo core module; you may want to discuss it on https://discourse.gohugo.io/ first. -->
Usecase
I'm using Codeberg Pages to host my website, you can find the repostitory at https://codeberg.org/ilovelinux/pages
I have two branches:
main: hugo projectpages: hugo build (output ofpublic/)
Since I'm using Codeberg Pages with a custom domain, I also need a .domains file[^codebergDomains] to be pushed in the pages branch.
When I
- Have files in the
static/directory. - Switch to
mainbranch. - Create the
public/folder using a git worktree of thepagesbranch. - Run
hugo build --cleanDestinationDir.
I need
- To remove outdated content
- To retain:
.gitfile created bygit worktree[^gitWorktreeFile]..domainsfile used by Codeberg Pages in order to allow custom domains[^codebergDomains]._redirectsfile for redirects (however I'm not using it yet)[^codebergRedirects].
Instead
- The files I need to be retained are deleted.
Proposal
Add a configuration field to list relative file paths to public/ that should be retained when using --cleanDestinationDir.
Example
filesToRetain:
- .git
- .domains
- _redirects
- foo/bar/.mysecretfile
Files retained:
public/.gitpublic/.domainspublic/_redirectspublic/foo/bar/.mysecretfile
[!NOTE]
I'm not suggesting to usefilesToRetainas configuration field, but I couldn't think of a better name.
Notes
[!IMPORTANT]
--cleanDestinationDirdoes nothing when when thestatic/directory doesn't exist since its value is used inside thecopyStaticTofunction.
https://github.com/gohugoio/hugo/blob/c4bbc2805c7092deafabae01e71ff2dda37bd877/commands/hugobuilder.go#L481
Workaround
[!WARNING]
This approach is error-prone.
I implemented a workaround in my Makefile that:
- Repairs the worktree folder, as suggested in https://github.com/gohugoio/hugo/pull/6261#issuecomment-1692321365.
- Restores the
.domainsfile.
Related issues & PR
- https://github.com/gohugoio/hugo/issues/3202 was about
.gitfolder being deleted. Fixed thanks to: - https://github.com/spf13/fsync/pull/10
- https://github.com/gohugoio/hugo/pull/3210.
- https://github.com/gohugoio/hugo/issues/4546 was about
.gitfile of a submodule[^gitSubmoduleFile] being deleted. Closed because it was considered a discussion. - https://github.com/gohugoio/hugo/pull/6261 was introducing a fix for the
.gitfile in order to work correctly with git worktrees and submodules. Closed because of the Stale bot but it looks like it was still kept into consideration since @bep kept moving it among milestones after that. However this wouldn't solve my issue since.domainsand_redirectsfiles would still be deleted.
[^gitWorktreeFile]: https://git-scm.com/docs/git-worktree#_details
[^gitSubmoduleFile]: https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-absorbgitdirs
[^codebergDomains]: https://docs.codeberg.org/codeberg-pages/using-custom-domain/
[^codebergRedirects]: https://docs.codeberg.org/codeberg-pages/redirects/
The two patches
commands/hugobuilder.goindex a9cacaada..fb7677b78 100644@@ -467,14 +467,14 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint fs := &countingStatFs{Fs: sourceFs.Fs} syncer := fsync.NewSyncer()- var clean bool+ var clean config.CleanDestinationDir c.withConf(func(conf *commonConfig) { syncer.NoTimes = conf.configs.Base.NoTimes syncer.NoChmod = conf.configs.Base.NoChmod syncer.ChmodFilter = chmodFilter syncer.DestFs = conf.fs.PublishDirStatic- clean = conf.configs.Base.CleanDestinationDir+ clean = conf.configs.Base.Build.CleanDestinationDir }) syncer.SrcFs = fs@@ -491,10 +491,16 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint return 0, err } - if clean {+ if clean.Enable { infol.Logf("removing stale files from destination") - keep := func(f fsync.FileInfo) bool {+ keep := func(filename string, f fsync.FileInfo) bool {+ // Keep files and directories matching build.cleanDestinationDir.retain,+ // matched against the path relative to the publishDir.+ if clean.ShouldRetain(strings.TrimPrefix(filepath.ToSlash(filename), "/")) {+ return true+ }+ name := f.Name() // Keep git metadata files anywhere.@@ -522,7 +528,8 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint // removeStale removes entries in dstDir on dst that keep reports as false and // that don't exist in srcDir on src. Unlike fsync's delete pass, it applies // keep at every level, so kept files survive inside stale directories.-func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(fsync.FileInfo) bool) error {+// keep receives the entry's path on dst and its file info.+func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(string, fsync.FileInfo) bool) error { entries, err := afero.ReadDir(dst, dstDir) if err != nil { if herrors.IsNotExist(err) {@@ -531,10 +538,10 @@ func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(fsync.FileI return err } for _, entry := range entries {- if keep(entry) {+ dstPath := filepath.Join(dstDir, entry.Name())+ if keep(dstPath, entry) { continue }- dstPath := filepath.Join(dstDir, entry.Name()) srcPath := filepath.Join(srcDir, entry.Name()) if _, err := src.Stat(srcPath); err == nil { if entry.IsDir() {@@ -644,7 +651,7 @@ func (c *hugoBuilder) fullBuild(noBuildLock bool) error { // and it does so at the end of copyStatic() call. var cleanDestinationDir bool c.withConf(func(conf *commonConfig) {- cleanDestinationDir = conf.configs.Base.CleanDestinationDir+ cleanDestinationDir = conf.configs.Base.Build.CleanDestinationDir.Enable }) if cleanDestinationDir { if err := copyStaticFunc(); err != nil {config/allconfig/alldecoders.goindex eb453cfd7..cefbfbf0e 100644@@ -134,6 +134,11 @@ var allDecoderSetups = map[string]decodeWeight{ key: "build", decode: func(d decodeWeight, p decodeConfig) error { p.c.Build = config.DecodeBuildConfig(p.p)+ // The root cleanDestinationDir setting (also set by the+ // --cleanDestinationDir flag) enables the cleanup.+ if p.c.CleanDestinationDir {+ p.c.Build.CleanDestinationDir.Enable = true+ } return nil }, getCompiler: func(c *Config) configCompiler {config/commonConfig.goindex dae0ed905..aaa52c1d8 100644@@ -16,6 +16,7 @@ package config import ( "fmt" "net/http"+ "path/filepath" "regexp" "slices" "sort"@@ -112,6 +113,53 @@ type BuildConfig struct { // Can used to control how the resource cache gets evicted on rebuilds. CacheBusters []CacheBuster++ // Configures removal of stale files from the publishDir.+ // The root level cleanDestinationDir bool (and the --cleanDestinationDir+ // flag) is still supported and enables this.+ CleanDestinationDir CleanDestinationDir+}++// CleanDestinationDir configures removal of files from the publishDir+// that do not exist in the static directories.+type CleanDestinationDir struct {+ // Whether to remove stale files from the publishDir.+ Enable bool++ // A list of glob patterns matched against the slash-separated path+ // of each file or directory relative to the publishDir+ // (e.g. ".git", ".domains", "foo/bar/.mysecretfile" or "**.txt").+ // Matching files and directories are never removed.+ Retain []string++ compiledRetain []*glob.Pattern+}++// ShouldRetain reports whether the file or directory with the given+// slash-separated path relative to the publishDir should be retained.+func (c CleanDestinationDir) ShouldRetain(p string) bool {+ for _, g := range c.compiledRetain {+ if g.Match(p) {+ return true+ }+ }+ return false+}++func (c *CleanDestinationDir) compileConfig() error {+ c.compiledRetain = nil+ for _, pattern := range c.Retain {+ pattern = strings.Trim(filepath.ToSlash(pattern), "/")+ if pattern == "" {+ continue+ }+ g, err := glob.Compile(pattern, '/')+ if err != nil {+ return fmt.Errorf("failed to compile build.cleanDestinationDir.retain glob %q: %w", pattern, err)+ }+ c.compiledRetain = append(c.compiledRetain, g)+ }+ return nil } // BuildStats configures if and what to write to the hugo_stats.json file.@@ -131,6 +179,8 @@ func (w BuildStats) Enabled() bool { func (b BuildConfig) clone() BuildConfig { b.CacheBusters = slices.Clone(b.CacheBusters)+ b.CleanDestinationDir.Retain = slices.Clone(b.CleanDestinationDir.Retain)+ b.CleanDestinationDir.compiledRetain = slices.Clone(b.CleanDestinationDir.compiledRetain) return b } @@ -174,6 +224,9 @@ func (b *BuildConfig) CompileConfig(logger loggers.Logger) error { } b.CacheBusters[i] = cb }+ if err := b.CleanDestinationDir.compileConfig(); err != nil {+ return err+ } return nil } @@ -192,6 +245,14 @@ func DecodeBuildConfig(cfg Provider) BuildConfig { } } + // Allow build.cleanDestinationDir to be set as a bool+ // (or a bool string, e.g. from HUGO_BUILD_CLEANDESTINATIONDIR=true).+ if clean, ok := m["cleandestinationdir"]; ok {+ if bb, err := cast.ToBoolE(clean); err == nil {+ m["cleandestinationdir"] = CleanDestinationDir{Enable: bb}+ }+ }+ err := mapstructure.WeakDecode(m, &b) if err != nil { return bconfig/commonConfig_test.goindex 05ba185e3..285f04f60 100644@@ -195,3 +195,54 @@ func TestBuildConfigCacheBusterstTailwindSetup(t *testing.T) { c.Assert(err, qt.IsNil) c.Assert(m("css"), qt.IsTrue) }++func TestBuildConfigCleanDestinationDir(t *testing.T) {+ c := qt.New(t)++ v := New()+ b := DecodeBuildConfig(v)+ c.Assert(b.CleanDestinationDir.Enable, qt.IsFalse)+ c.Assert(b.CompileConfig(loggers.NewDefault()), qt.IsNil)+ c.Assert(b.CleanDestinationDir.ShouldRetain(".git"), qt.IsFalse)++ // Bool form.+ v.Set("build", map[string]any{+ "cleanDestinationDir": true,+ })+ b = DecodeBuildConfig(v)+ c.Assert(b.CleanDestinationDir.Enable, qt.IsTrue)++ // Bool string form, e.g. from HUGO_BUILD_CLEANDESTINATIONDIR=true.+ v.Set("build", map[string]any{+ "cleanDestinationDir": "true",+ })+ b = DecodeBuildConfig(v)+ c.Assert(b.CleanDestinationDir.Enable, qt.IsTrue)++ v.Set("build", map[string]any{+ "cleanDestinationDir": map[string]any{+ "enable": true,+ "retain": []string{".git", "/.domains", "_redirects", "foo/bar/.mysecretfile", "**.keep"},+ },+ })+ b = DecodeBuildConfig(v)+ c.Assert(b.CompileConfig(loggers.NewDefault()), qt.IsNil)+ clean := b.CleanDestinationDir+ c.Assert(clean.Enable, qt.IsTrue)+ c.Assert(clean.ShouldRetain(".git"), qt.IsTrue)+ c.Assert(clean.ShouldRetain(".domains"), qt.IsTrue)+ c.Assert(clean.ShouldRetain("_redirects"), qt.IsTrue)+ c.Assert(clean.ShouldRetain("foo/bar/.mysecretfile"), qt.IsTrue)+ c.Assert(clean.ShouldRetain("a/b/c.keep"), qt.IsTrue)+ c.Assert(clean.ShouldRetain("sub/.git"), qt.IsFalse)+ c.Assert(clean.ShouldRetain("foo/.mysecretfile"), qt.IsFalse)+ c.Assert(clean.ShouldRetain("index.html"), qt.IsFalse)++ v.Set("build", map[string]any{+ "cleanDestinationDir": map[string]any{+ "retain": []string{"[invalid"},+ },+ })+ b = DecodeBuildConfig(v)+ c.Assert(b.CompileConfig(loggers.NewDefault()), qt.ErrorMatches, ".*build.cleanDestinationDir.retain.*")+}docs/content/en/configuration/all.mdindex 842465254..839d2dc7d 100644@@ -46,7 +46,7 @@ aliases: [/getting-started/configuration/] : See [configure cascade][]. `cleanDestinationDir`-: (`bool`) Whether to remove files from the [`publishDir`](#publishdir) that do not exist in the [`staticDir`](#staticdir). Hugo performs this cleanup before rendering, so the current build's rendered output is unaffected. If the `staticDir` does not exist, the cleanup does not run. Note that `.gitignore` and `.gitattributes` files are always preserved, as are directories whose names begin with a dot, provided their parent directory also survives. Default is `false`.+: (`bool`) Whether to remove files from the [`publishDir`](#publishdir) that do not exist in the [`staticDir`](#staticdir). Hugo performs this cleanup before rendering, so the current build's rendered output is unaffected. Note that `.gitignore` and `.gitattributes` files are always preserved, as are directories whose names begin with a dot, provided their parent directory also survives. Default is `false`. See also [`build.cleanDestinationDir`](/configuration/build/#clean-destination-directory), which lets you retain additional files. `contentDir` : (`string`) The designated directory for content files. Default is `content`. {{% module-mounts-note %}}docs/content/en/configuration/build.mdindex 6a7924301..60d2bed33 100644@@ -17,6 +17,9 @@ This is the default configuration: `cachebusters` : See the [cache busters](#cache-busters) section below. +`cleanDestinationDir`+: See the [clean destination directory](#clean-destination-directory) section below.+ `noJSConfigInAssets` : (`bool`) Whether to disable writing a `jsconfig.json` in your `assets` directory with mapping of imports from running [js.Build][]. This file is intended to help with intellisense/navigation inside code editors such as [VS Code][]. Note that if you do not use `js.Build`, no file will be written. @@ -55,6 +58,26 @@ When `buildStats` is enabled, Hugo writes a `hugo_stats.json` file on each build `target` : (`string`) A [regular expression](g) matching the keys in the resource cache that should be expired when `source` changes. You can use the matching regexp groups from `source` in the expression, e.g. `$1`. +## Clean destination directory++{{< code-toggle config=build.cleanDestinationDir />}}++`enable`+: (`bool`) Whether to remove files from the [`publishDir`] that do not exist in the [`staticDir`]. Setting the root level [`cleanDestinationDir`] option to `true`, or passing the `--cleanDestinationDir` flag, also enables this. You can also set `build.cleanDestinationDir` to a `bool` as a shorthand for this option. Default is `false`.++`retain`+: (`[]string`) A list of [glob patterns](g) matched against the path of each file and directory relative to the [`publishDir`], using `/` as the path separator. Matching files and directories, including everything within a matching directory, are never removed. For example, `.domains` matches only `public/.domains`, while `**.domains` matches a `.domains` file at any depth. In a multihost configuration the path includes the language directory, for example `en/.domains`. Default is `[]`.++For example, to retain the `.git` file created by `git worktree`, the `.domains` and `_redirects` files used by some hosting services, and a file in a subdirectory:++{{< code-toggle file=hugo >}}+[build.cleanDestinationDir]+enable = true+retain = ['.git', '.domains', '_redirects', 'foo/bar/.mysecretfile']+{{< /code-toggle >}}++In addition to the files matching `retain`, Hugo always preserves `.gitignore` and `.gitattributes` files, as well as directories whose names begin with a dot.+ ## Build stats {{< code-toggle config=build.buildStats />}}@@ -80,6 +103,9 @@ Due to the nature of partial server builds, new HTML entities are added while th [Hugo Starter Tailwind Basic]: https://github.com/bep/hugo-starter-tailwind-basic [VS Code]: https://code.visualstudio.com/+[`cleanDestinationDir`]: /configuration/all/#cleandestinationdir [`config/production`]: /configuration/introduction/#configuration-directory+[`publishDir`]: /configuration/all/#publishdir+[`staticDir`]: /configuration/all/#staticdir [js.Build]: /functions/js/build/ [removing unused CSS]: /functions/templates/defer/#remove-unused-cssdocs/data/docs.yamlindex 6d2578339..3a2d7accf 100644@@ -1336,6 +1336,9 @@ config: cacheBusters: - source: '(postcss|tailwind)\.config\.(js|mjs|cjs)' target: (css|styles|scss|sass)+ cleanDestinationDir:+ enable: false+ retain: null noJSConfigInAssets: false useResourceCacheWhen: fallback buildDrafts: falsetestscripts/commands/hugo__clean_destination_dir_retain.txtnew file mode 100644index 000000000..14c1e147e@@ -0,0 +1,120 @@+# Test build.cleanDestinationDir.retain.++# --- Scenario 1: Enabled in config, no static directory ---++cpr golden_public public++hugo --config hugo.toml,enable.toml+tree public+cmp stdout clean_build.txt++# --- Scenario 2: Enabled via the flag, with a static directory ---++rm public++cpr golden_public public+cpr golden_static static++hugo --cleanDestinationDir+tree public+cmp stdout clean_build_with_static.txt++# --- Scenario 3: Enabled via the bool form of build.cleanDestinationDir ---++rm public static++cpr golden_public public++hugo --config enable_bool.toml+tree public+cmp stdout clean_build_no_retain.txt++# --- Scenario 4: Not enabled, nothing is removed ---++rm public++cpr golden_public public++hugo+tree public+cmp stdout default_build.txt++# Files++-- hugo.toml --+disableKinds = ['page','rss','section','sitemap','taxonomy','term']+[build.cleanDestinationDir]+retain = ['.git', '.domains', '/_redirects', 'foo/bar/.mysecretfile', '**.keep']+-- enable.toml --+[build.cleanDestinationDir]+enable = true+-- enable_bool.toml --+disableKinds = ['page','rss','section','sitemap','taxonomy','term']+[build]+cleanDestinationDir = true+-- content/_index.md --+---+title: home+---+-- layouts/home.html --+{{ .Title }}++-- golden_public/.git --+-- golden_public/.domains --+-- golden_public/_redirects --+-- golden_public/a.txt --+-- golden_public/foo/bar/.mysecretfile --+-- golden_public/foo/bar/b.txt --+-- golden_public/foo/c.txt --+-- golden_public/dir/.git --+-- golden_public/dir/.domains --+-- golden_public/dir/d.keep --+-- golden_public/dir/e.txt --++-- golden_static/f.txt --+-- golden_static/dir/g.txt --++-- default_build.txt --+.+└─.domains+└─.git+└─_redirects+└─a.txt+└─dir/+ └─.domains+ └─.git+ └─d.keep+ └─e.txt+└─foo/+ └─bar/+ └─.mysecretfile+ └─b.txt+ └─c.txt+└─index.html+-- clean_build.txt --+.+└─.domains+└─.git+└─_redirects+└─dir/+ └─d.keep+└─foo/+ └─bar/+ └─.mysecretfile+└─index.html+-- clean_build_no_retain.txt --+.+└─index.html+-- clean_build_with_static.txt --+.+└─.domains+└─.git+└─_redirects+└─dir/+ └─d.keep+ └─g.txt+└─f.txt+└─foo/+ └─bar/+ └─.mysecretfile+└─index.html
commands/helpers.goindex fbe1d362a..16555b6c4 100644@@ -76,9 +76,10 @@ func flagsToCfgWithAdditionalConfigBase(cd *simplecobra.Commandeer, cfg config.P // Flags with a different name in the config. keyMap := map[string]string{- "minify": "minify.minifyOutput",- "destination": "publishDir",- "editor": "newContentEditor",+ "minify": "minify.minifyOutput",+ "destination": "publishDir",+ "editor": "newContentEditor",+ "cleanDestinationDir": "build.cleanDestinationDir.enable", } // Flags that we for some reason don't want to expose in the project config.commands/hugobuilder.goindex a9cacaada..963db35c2 100644@@ -467,14 +467,14 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint fs := &countingStatFs{Fs: sourceFs.Fs} syncer := fsync.NewSyncer()- var clean bool+ var clean config.CleanDestinationDir c.withConf(func(conf *commonConfig) { syncer.NoTimes = conf.configs.Base.NoTimes syncer.NoChmod = conf.configs.Base.NoChmod syncer.ChmodFilter = chmodFilter syncer.DestFs = conf.fs.PublishDirStatic- clean = conf.configs.Base.CleanDestinationDir+ clean = conf.configs.Base.Build.CleanDestinationDir }) syncer.SrcFs = fs@@ -491,22 +491,9 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint return 0, err } - if clean {+ if clean.Enable { infol.Logf("removing stale files from destination")-- keep := func(f fsync.FileInfo) bool {- name := f.Name()-- // Keep git metadata files anywhere.- if name == ".gitignore" || name == ".gitattributes" {- return true- }-- // Keep dot-directories anywhere.- return f.IsDir() && strings.HasPrefix(name, ".")- }-- if err := removeStale(syncer.DestFs, sourceFs.Fs, publishDir, helpers.FilePathSeparator, keep); err != nil {+ if err := removeStale(syncer.DestFs, sourceFs.Fs, publishDir, helpers.FilePathSeparator, clean.Keep); err != nil { return 0, err } }@@ -522,7 +509,8 @@ func (c *hugoBuilder) copyStaticTo(sourceFs *filesystems.SourceFilesystem) (uint // removeStale removes entries in dstDir on dst that keep reports as false and // that don't exist in srcDir on src. Unlike fsync's delete pass, it applies // keep at every level, so kept files survive inside stale directories.-func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(fsync.FileInfo) bool) error {+// keep gets the slash separated path relative to srcDir's root, e.g. "dir/.gitignore".+func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(path string, isDir bool) bool) error { entries, err := afero.ReadDir(dst, dstDir) if err != nil { if herrors.IsNotExist(err) {@@ -531,11 +519,11 @@ func removeStale(dst, src afero.Fs, dstDir, srcDir string, keep func(fsync.FileI return err } for _, entry := range entries {- if keep(entry) {- continue- } dstPath := filepath.Join(dstDir, entry.Name()) srcPath := filepath.Join(srcDir, entry.Name())+ if keep(strings.TrimPrefix(filepath.ToSlash(srcPath), "/"), entry.IsDir()) {+ continue+ } if _, err := src.Stat(srcPath); err == nil { if entry.IsDir() { if err := removeStale(dst, src, dstPath, srcPath, keep); err != nil {@@ -644,7 +632,7 @@ func (c *hugoBuilder) fullBuild(noBuildLock bool) error { // and it does so at the end of copyStatic() call. var cleanDestinationDir bool c.withConf(func(conf *commonConfig) {- cleanDestinationDir = conf.configs.Base.CleanDestinationDir+ cleanDestinationDir = conf.configs.Base.Build.CleanDestinationDir.Enable }) if cleanDestinationDir { if err := copyStaticFunc(); err != nil {config/allconfig/allconfig.goindex 806cedc71..a84545edb 100644@@ -785,10 +785,6 @@ type RootConfig struct { // Don't sync modification time of files for the static mounts. NoChmod bool - // Clean the destination folder before a new build.- // This currently only handles static files.- CleanDestinationDir bool- // A Glob pattern of module paths to ignore in the _vendor folder. IgnoreVendorPaths string config/allconfig/load.goindex bc35af9d6..76eb65703 100644@@ -198,6 +198,13 @@ func (l configLoader) normalizeCfg(cfg config.Provider) error { } } + if b, ok := cfg.Get("cleanDestinationDir").(bool); ok {+ hugo.Deprecate("project config cleanDestinationDir", "Use build.cleanDestinationDir.enable instead.", "v0.167.0")+ if b {+ cfg.Set("build.cleanDestinationDir.enable", true)+ }+ }+ return nil } config/commonConfig.goindex dae0ed905..cadfcb5ef 100644@@ -25,7 +25,9 @@ import ( "github.com/gobwas/glob" "github.com/gohugoio/hugo/common/hstrings" "github.com/gohugoio/hugo/common/loggers"+ "github.com/gohugoio/hugo/common/predicate" "github.com/gohugoio/hugo/common/types"+ "github.com/gohugoio/hugo/hugofs/hglob" "github.com/gohugoio/hugo/common/herrors" "github.com/mitchellh/mapstructure"@@ -86,6 +88,10 @@ type LoadConfigResult struct { var defaultBuild = BuildConfig{ UseResourceCacheWhen: "fallback", BuildStats: BuildStats{},+ CleanDestinationDir: CleanDestinationDir{+ KeepFiles: []string{"{**/,}.{git,gitignore,gitattributes}"},+ KeepDirs: []string{"{**/,}.*"},+ }, CacheBusters: []CacheBuster{ {@@ -112,6 +118,42 @@ type BuildConfig struct { // Can used to control how the resource cache gets evicted on rebuilds. CacheBusters []CacheBuster++ // Removes files from the publish directory that are not found in the static source.+ CleanDestinationDir CleanDestinationDir+}++// CleanDestinationDir configures the removal of stale files from the publish directory.+type CleanDestinationDir struct {+ // Whether to remove stale files. Can also be enabled with the --cleanDestinationDir flag.+ Enable bool++ // Glob patterns of files to keep, matched against the slash separated path relative to the publish directory, e.g. "dir/.gitignore".+ // A pattern prefixed with "! " is a negation.+ KeepFiles []string++ // Glob patterns of directories to keep with all of their content.+ KeepDirs []string++ keepFiles predicate.P[string]+ keepDirs predicate.P[string]+}++// Keep reports whether path, relative to the publish directory, should be kept.+func (c CleanDestinationDir) Keep(path string, isDir bool) bool {+ if isDir {+ return c.keepDirs(path)+ }+ return c.keepFiles(path)+}++func (c *CleanDestinationDir) compile() error {+ var err error+ if c.keepFiles, err = predicate.NewStringPredicateFromGlobs(c.KeepFiles, hglob.GetGlob); err != nil {+ return err+ }+ c.keepDirs, err = predicate.NewStringPredicateFromGlobs(c.KeepDirs, hglob.GetGlob)+ return err } // BuildStats configures if and what to write to the hugo_stats.json file.@@ -131,6 +173,8 @@ func (w BuildStats) Enabled() bool { func (b BuildConfig) clone() BuildConfig { b.CacheBusters = slices.Clone(b.CacheBusters)+ b.CleanDestinationDir.KeepFiles = slices.Clone(b.CleanDestinationDir.KeepFiles)+ b.CleanDestinationDir.KeepDirs = slices.Clone(b.CleanDestinationDir.KeepDirs) return b } @@ -174,6 +218,9 @@ func (b *BuildConfig) CompileConfig(logger loggers.Logger) error { } b.CacheBusters[i] = cb }+ if err := b.CleanDestinationDir.compile(); err != nil {+ return fmt.Errorf("failed to compile build.cleanDestinationDir: %w", err)+ } return nil } testscripts/commands/hugo__clean_destination_dir.txtindex 9191c77e9..c3273227e 100644@@ -37,24 +37,76 @@ hugo --cleanDestinationDir tree public cmp stdout mismatch_clean_build.txt +# --- Scenario 4: Custom keepFiles and keepDirs ---++rm public static++cpr golden_public_keep public++hugo --config hugo_keep.toml+tree public+cmp stdout keep_clean_build.txt++# --- Scenario 5: Negated keepFiles ---++rm public++cpr golden_public_keep public++hugo --config hugo_keep_negate.toml+tree public+cmp stdout keep_negate_clean_build.txt++# --- Scenario 6: Deprecated root cleanDestinationDir key with default keep patterns ---++rm public++cpr golden_public_keep public++hugo --config hugo_legacy.toml+tree public+cmp stdout legacy_clean_build.txt+ # Files -- hugo.toml -- disableKinds = ['page','rss','section','sitemap','taxonomy','term']+-- hugo_keep.toml --+disableKinds = ['page','rss','section','sitemap','taxonomy','term']+[build.cleanDestinationDir]+enable = true+keepFiles = ['.domains', '_redirects']+keepDirs = ['keep-me']+-- hugo_keep_negate.toml --+disableKinds = ['page','rss','section','sitemap','taxonomy','term']+# Render home as JSON so no HTML gets rendered.+[outputs]+home = ['json']+[build.cleanDestinationDir]+enable = true+# Remove all HTML files, keep everything else.+keepFiles = ['! **.html']+-- hugo_legacy.toml --+disableKinds = ['page','rss','section','sitemap','taxonomy','term']+cleanDestinationDir = true -- content/_index.md -- --- title: home --- -- layouts/home.html -- {{ .Title }}+-- layouts/home.json --+{{ .Title | jsonify }} -- golden_public/.dir-a/a.txt -- -- golden_public/.b.txt --+-- golden_public/.git -- -- golden_public/.gitattributes -- -- golden_public/.gitignore -- -- golden_public/c.txt -- -- golden_public/dir-b/.dir-c/d.txt -- -- golden_public/dir-b/.e.txt --+-- golden_public/dir-b/.git -- -- golden_public/dir-b/.gitignore -- -- golden_public/dir-b/f.txt -- -- golden_public/dir-d/g.txt --@@ -68,6 +120,16 @@ title: home -- golden_static/dir-e/.m.txt -- -- golden_static/dir-e/n.txt -- +-- golden_public_keep/.domains --+-- golden_public_keep/.gitignore --+-- golden_public_keep/.hidden/x.txt --+-- golden_public_keep/_redirects --+-- golden_public_keep/a.html --+-- golden_public_keep/b.txt --+-- golden_public_keep/dir/.domains --+-- golden_public_keep/keep-me/x.html --+-- golden_public_keep/keep-me/y.txt --+ -- golden_public_mismatch/type-a/x.txt -- -- golden_public_mismatch/type-b -- @@ -79,6 +141,7 @@ title: home └─.b.txt └─.dir-a/ └─a.txt+└─.git └─.gitattributes └─.gitignore └─c.txt@@ -86,6 +149,7 @@ title: home └─.dir-c/ └─d.txt └─.e.txt+ └─.git └─.gitignore └─f.txt └─dir-d/@@ -101,11 +165,13 @@ title: home . └─.dir-a/ └─a.txt+└─.git └─.gitattributes └─.gitignore └─dir-b/ └─.dir-c/ └─d.txt+ └─.git └─.gitignore └─dir-e/ └─.dir-f/@@ -117,6 +183,7 @@ title: home └─.b.txt └─.dir-a/ └─a.txt+└─.git └─.gitattributes └─.gitignore └─.k.txt@@ -125,6 +192,7 @@ title: home └─.dir-c/ └─d.txt └─.e.txt+ └─.git └─.gitignore └─f.txt └─dir-d/@@ -143,12 +211,14 @@ title: home . └─.dir-a/ └─a.txt+└─.git └─.gitattributes └─.gitignore └─.k.txt └─dir-b/ └─.dir-c/ └─d.txt+ └─.git └─.gitignore └─dir-e/ └─.dir-f/@@ -164,3 +234,30 @@ title: home └─type-a └─type-b/ └─y.txt+-- keep_clean_build.txt --+.+└─.domains+└─_redirects+└─index.html+└─keep-me/+ └─x.html+ └─y.txt+-- keep_negate_clean_build.txt --+.+└─.domains+└─.gitignore+└─.hidden/+ └─x.txt+└─_redirects+└─b.txt+└─dir/+ └─.domains+└─index.json+└─keep-me/+ └─y.txt+-- legacy_clean_build.txt --+.+└─.gitignore+└─.hidden/+ └─x.txt+└─index.html