Backtestshugorun-20260917-015623#15199
related: Add option to tokenize headings
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both correctly add Tokenize gated to fragment headings while preserving exact matching for non-heading fragments. But Patch A has two genuine correctness defects: (1) it appends a doc to a token's posting list once per heading containing that token and duplicates query keywords likewise, so searchDate double-counts weight and inflates rank/reorders related results; (2) its FieldsFunc tokenizer emits garbage tokens from HTML entities in titles (e.g. 'amp','34') causing spurious matches. Patch B fixes both — deduplicating index- and query-side via seenKW and unescaping entities in splitWords — and backs each with a fail-before/pass-after test, alongside a much more comprehensive suite (fragments, entities, apply-filter, case-sensitivity, dedup, splitWords unit table). Patch A is tighter-scoped and includes the exact docs the issue requests (content page + docs.yaml) with cleaner DRY helpers, whereas B ships no docs, adds an out-of-scope undocumented MinTokenLength option, extends tokenization to basic indices beyond the documented fragments-only contract, and adds whitespace churn. Since correctness and test quality are the top two weighted criteria and B is meaningfully stronger on both, B wins despite A's superior scope and documentation.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
23m 41s22 turns
Wall-clock time in the container
Full test suite
No new failures
3,740 passed · 99 skipped · 3,845 total
Contamination audit
Unknown
See the audit notes for this run
Patch size
178vs 462 merged
Changed lines, ours vs merged
The issue
View on GitHubSee <https://discourse.gohugo.io/t/related-posts-based-on-heading/57467>.
I suggest using the name tokenize, documented as follows:
(
bool) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of typefragments; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; usecardinalityThresholdto prune common words and reduce build time. Default isfalse.
The two patches
docs/content/en/configuration/related-content.mdindex 92cf92e8a..25017402e 100644@@ -45,6 +45,9 @@ This is the default configuration: `pattern` : (`string`) This is currently only relevant for dates. When listing related content, we may want to list content that is also close in time. Setting "2006" (default value for date indexes) as the pattern for a date index will add weight to pages published in the same year. For busier blogs, "200601" (year and month) may be a better default. +`tokenize`+: (`bool`) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type `fragments`; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use `cardinalityThreshold` to prune common words and reduce build time. Default is `false`.+ `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. docs/data/docs.yamlindex 19b950dd5..831250fbc 100644@@ -2192,6 +2192,7 @@ config: name: keywords pattern: '' toLower: false+ tokenize: false type: basic weight: 100 - applyFilter: false@@ -2199,6 +2200,7 @@ config: name: date pattern: '' toLower: false+ tokenize: false type: basic weight: 10 - applyFilter: false@@ -2206,6 +2208,7 @@ config: name: tags pattern: '' toLower: false+ tokenize: false type: basic weight: 80 threshold: 80related/inverted_index.goindex c6c65d97f..e02d1a118 100644@@ -24,6 +24,7 @@ import ( "strings" "sync" "time"+ "unicode" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +126,15 @@ type IndexConfig struct { // Will lower case all string values in and queries tothis index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords for+ // indexing and matching. Currently applies to heading titles in indices+ // of type "fragments"; non-heading fragments such as definition terms+ // continue to use exact-identifier matching regardless of this setting.+ // This increases index size in proportion to the average number of words+ // per heading; use CardinalityThreshold to prune common words and reduce+ // build time. Default is false.+ Tokenize bool } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,9 +230,9 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ for _, keyword := range config.fragmentsToKeywords(fp.Fragments(ctx)) { added = true- setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc)+ setm[keyword] = append(setm[keyword], doc) } } }@@ -377,9 +387,7 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {- keywords = append(keywords, FragmentKeyword(fragment))- }+ keywords = append(keywords, cfg.fragmentsToKeywords(fp.Fragments(ctx))...) } } @@ -423,6 +431,39 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +// tokenizeTitle splits a heading title into individual, normalized word tokens.+func (cfg IndexConfig) tokenizeTitle(title string) []string {+ tokens := strings.FieldsFunc(title, func(r rune) bool {+ return !unicode.IsLetter(r) && !unicode.IsNumber(r)+ })+ if cfg.ToLower {+ for i, t := range tokens {+ tokens[i] = strings.ToLower(t)+ }+ }+ return tokens+}++// fragmentsToKeywords returns the keywords used to index or query a fragments+// index for the given fragments. When Tokenize is enabled, heading titles are+// split into individual word tokens; non-heading fragments (and all fragments+// when Tokenize is disabled) use the exact identifier.+func (cfg IndexConfig) fragmentsToKeywords(fragments *tableofcontents.Fragments) []Keyword {+ var keywords []Keyword+ for _, id := range fragments.Identifiers {+ if cfg.Tokenize {+ if h, ok := fragments.HeadingsMap[id]; ok && h.Title != "" {+ for _, t := range cfg.tokenizeTitle(h.Title) {+ keywords = append(keywords, FragmentKeyword(t))+ }+ continue+ }+ }+ keywords = append(keywords, FragmentKeyword(id))+ }+ return keywords+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword@@ -467,6 +508,7 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero() var fragmentsFilter collections.SortedStringSlice+ var fragmentsFilterConfig IndexConfig for _, el := range query { setm, found := idx.index[el.Index]@@ -496,6 +538,7 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok { fragmentsFilter = append(fragmentsFilter, string(fkw))+ fragmentsFilterConfig = config } } @@ -538,6 +581,14 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if len(fragmentsFilter) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {+ if fragmentsFilterConfig.Tokenize {+ for _, t := range fragmentsFilterConfig.tokenizeTitle(h.Title) {+ if fragmentsFilter.Contains(t) {+ return true+ }+ }+ return false+ } return fragmentsFilter.Contains(h.ID) }) }related/related_tokenize_integration_test.gonew file mode 100644index 000000000..a529b3a5e@@ -0,0 +1,111 @@+// Copyright 2024 The Hugo Authors. All rights reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++package related_test++import (+ "testing"++ "github.com/gohugoio/hugo/hugolib"+)++func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ t.Run("tokenize enabled", func(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/a.md --+---+title: pagea+---++## Delicious apple pie+-- content/b.md --+---+title: pageb+---++## Delicious banana bread+-- content/c.md --+---+title: pagec+---++## Zebra crossing+-- layouts/single.html --+Related:{{ range site.RegularPages.Related . }} {{ .Title }}{{ end }}::END+`++ b := hugolib.Test(t, files)++ b.AssertFileContent("public/a/index.html", "Related: pageb::END")+ b.AssertFileContent("public/b/index.html", "Related: pagea::END")+ })++ t.Run("tokenize disabled", func(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ weight = 100+ tokenize = false+-- content/a.md --+---+title: pagea+---++## Delicious apple pie+-- content/b.md --+---+title: pageb+---++## Delicious banana bread+-- content/c.md --+---+title: pagec+---++## Zebra crossing+-- layouts/single.html --+Related:{{ range site.RegularPages.Related . }} {{ .Title }}{{ end }}::END+`++ b := hugolib.Test(t, files)++ b.AssertFileContent("public/a/index.html", "Related:::END")+ })+}
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode"+ "unicode/utf8" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { // used in more than 50% of the documents in the index. CardinalityThreshold int - // Will lower case all string values in and queries tothis index.+ // Will lower case all string values in and queries to this index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords.+ Tokenize bool++ // Minimum token length in Unicode code points when tokenizing.+ MinTokenLength int } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers { added = true+ if config.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !config.shouldSkipToken(word) {+ kw := config.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ setm[kw] = append(setm[kw], doc)+ }+ }+ }+ continue+ }+ } setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) } }@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers {+ if cfg.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !cfg.shouldSkipToken(word) {+ kw := cfg.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ keywords = append(keywords, kw)+ }+ }+ }+ continue+ }+ } keywords = append(keywords, FragmentKeyword(fragment)) } }@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+ return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ if cfg.Tokenize {+ for _, word := range splitWords(vv) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } else {+ keywords = append(keywords, cfg.stringToKeyword(vv))+ } case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])+ if !cfg.Tokenize {+ kw := make([]Keyword, len(vv))+ for i, s := range vv {+ kw[i] = cfg.stringToKeyword(s)+ }+ keywords = append(keywords, kw...)+ } else {+ for _, s := range vv {+ for _, word := range splitWords(s) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } }- keywords = append(keywords, vvv...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ // Bundled into one struct so the closure below causes only one heap escape.+ var ff struct {+ ids collections.SortedStringSlice+ words collections.SortedStringSlice+ toLower bool+ } for _, el := range query { setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return []Document{}, fmt.Errorf("index config for %q not found", el.Index) } + if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+ ff.toLower = config.ToLower+ }+ for _, kw := range el.Keywords { if docs, found := setm[kw]; found { for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok {- fragmentsFilter = append(fragmentsFilter, string(fkw))+ if config.Tokenize {+ ff.words = append(ff.words, string(fkw))+ } else {+ ff.ids = append(ff.ids, string(fkw))+ } } }- r, found := matchm[doc] if !found { r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } sort.Stable(matches)- sort.Strings(fragmentsFilter)+ sort.Strings(ff.ids)+ sort.Strings(ff.words) result := make([]Document, len(matches)) for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(ff.ids) > 0 || len(ff.words) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if ff.ids.Contains(h.ID) {+ return true+ }+ if len(ff.words) > 0 {+ for _, word := range splitWords(h.Title) {+ if ff.toLower {+ word = strings.ToLower(word)+ }+ if ff.words.Contains(word) {+ return true+ }+ }+ }+ return false }) } }@@ -618,6 +707,31 @@ type Keyword interface { String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+ // Fast path: no whitespace or HTML entities means at most one token.+ if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+ title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if title == "" {+ return nil+ }+ return []string{title}+ }+ raw := strings.Fields(html.UnescapeString(title))+ words := raw[:0]+ for _, w := range raw {+ w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if w != "" {+ words = append(words, w)+ }+ }+ if len(words) == 0 {+ return nil+ }+ return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { }) } +func TestSplitWords(t *testing.T) {+ c := qt.New(t)++ tests := []struct {+ in string+ want []string+ }{+ {"", nil},+ {" ", nil},+ {"hello world", []string{"hello", "world"}},+ // leading/trailing punctuation stripped+ {"...hello...", []string{"hello"}},+ {",hello, world!", []string{"hello", "world"}},+ // digits preserved+ {"go1.24", []string{"go1.24"}},+ // HTML entities decoded+ {"foo &bar", []string{"foo", "bar"}},+ // fast path: single token, no whitespace+ {"&foo&", []string{"foo"}},+ // and similar entities that decode to whitespace must not use the fast path+ {"Apples and oranges!", []string{"Apples", "and", "oranges"}},+ // punctuation-only tokens dropped+ {"hello --- world", []string{"hello", "world"}},+ // mixed spaces+ {" foo bar ", []string{"foo", "bar"}},+ }++ for _, tt := range tests {+ c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+ }+}+ func BenchmarkRelatedNewIndex(b *testing.B) { pages := make([]*testDoc, 100) numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] weight = 90 [[related.indices]] name = 'keywords'- weight = 80 + weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} - + ` b := hugolib.Test(t, files) expect := `-P1 Fragments: [p1-title] +P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 b.AssertFileContent("public/"+p+"/index.html", expect) } - b.AssertFileContent("public/p1/index.html",+ b.AssertFileContent(+ "public/p1/index.html", "Related 1: 0: p2: h0: First title|ref1|::END", "Related 2: 0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", ) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+ b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ applyFilter = true+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+ b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = false+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // toLower=false: "Hugo" != "hugo", so no match.+ b.AssertFileContent("public/p1/index.html", "Related: ")+ b.AssertFileContent("public/p2/index.html", "Related: ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+ minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+ b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+ b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+ // All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+[[related.indices]]+ name = 'keywords'+ weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p3 share the same unique fragment tokens; p1 also matches on+ // keywords ("fruit"). Without deduplication, p3's repeated headings+ // inflate its weight above p1's combined fragment+keywords weight.+ b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] toLower = false [[related.indices]] name = 'keywords'- weight = 70 + weight = 70 [[related.indices]] name = 'pagerefs' type = 'fragments'- weight = 30 + weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)
What the agent reported
The reviewer returned a clean PASS, and appropriately flagged and ignored some prompt-injection-style text it noticed in docs/AGENTS.md (unrelated to this task; I'm disregarding it too). All stages complete.
Summary
What was requested: Add a tokenize option to Hugo's related-content feature so that multi-word heading titles in fragments-type indices are split into individual keywords for indexing and matching, while non-heading fragments (e.g. definition terms) keep exact-identifier matching. Default false.
Root situation: For type = "fragments" indices, documents were only ever indexed/queried under their exact heading identifiers (slugified IDs) as FragmentKeywords, so two pages could relate only if they shared a full heading ID. There was no mechanism to relate pages that merely share a word within a heading.
What I changed (all in related/inverted_index.go, plus docs):
- Added
Tokenize booltoIndexConfig(decoded automatically viamapstructure.WeakDecode; defaultfalse). - Added two helpers:
tokenizeTitle(splits a title into normalized word tokens, honoringToLower) andfragmentsToKeywords(tokenizes a heading's title intoFragmentKeywordtokens only whenTokenizeis on and the identifier is a real heading inHeadingsMapwith a non-empty title; otherwise emits the exactFragmentKeyword(id)). - Routed both
Add(indexing) andSearch(query building) throughfragmentsToKeywords, keeping them symmetric. Explicit user-suppliedopts.Fragmentsstay exact. - Made the
ApplyFilter/HeadingsFilteredpredicate insearchDatetokenize-aware, while leaving the non-tokenize path byte-identical (fragmentsFilter.Contains(h.ID)). - Documented the option in
docs/content/en/configuration/related-content.mdanddocs/data/docs.yaml. - Added regression test
related/related_tokenize_integration_test.go.
Test evidence:
- New
TestRelatedFragmentsTokenizewas verified to fail before the production change (tokenize=true found no relation) and pass after; the tokenize=false sub-test passes in both states. - Gate run (
./bt-test -run 'TestRelated|TestCardinalityThreshold|TestDecodeConfig|TestToKeywords'): 34 passed, 0 failed, including the previously-passingrelated TestRelatedFragments,TestCardinalityThreshold,TestDecodeConfig,TestToKeywords*, andpage TestRelated, plus the newTestRelatedFragmentsTokenize. The broad./...compile in that run succeeded, confirming the production code builds.
No existing tests were skipped, weakened, or mocked. The AGENTS.md/CLAUDE.md deletions shown in git status were pre-existing before this task and are not part of this change. All changes are left uncommitted in the working tree.