Backtestshugorun-20260930-140149#15199

Issue #15199 Merged fix PR # Run 30 Sep 2026 · run-20260930-140149 gohugoio/hugo

related: Add option to tokenize headings

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Correctness decides this. Patch A indexes and queries tokenized heading words as StringKeyword, because it routes them through config.stringToKeyword. In searchDate, the applyFilter path only collects keywords that pass `kw.(FragmentKeyword)`, so ff.words is never populated for tokenized matches. With tokenize and applyFilter both on, no heading filter gets applied, and HeadingsFiltered would not return the matched headings. A's own TestRelatedFragmentsTokenizeApplyFilter expects filtered headings, so it likely fails. A also widens scope: it tokenizes ToKeywords for basic indices and adds an undocumented MinTokenLength option. The issue scopes the feature to fragment heading titles, so this adds behaviour beyond the ask. Patch B keeps tokenized words as FragmentKeyword and applies ToLower inside tokenize. Its filter closure re-tokenizes heading titles with the same config, so indexing, querying and applyFilter stay consistent. It falls back to the exact identifier for non-heading fragments and headings with no words, as the issue specifies. It also tokenizes opts.Fragments while keeping the exact id, strips HTML tags and entities, and adds the requested documentation text verbatim. B adds no tests, which is a real weakness, but A's tests exercise an apparently broken path. B is the correct, well-scoped implementation.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

21m 33s15 turns

Wall-clock time in the container

Full test suite

No new failures

3,741 passed · 99 skipped · 3,842 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

97vs 462 merged

Changed lines, ours vs merged

See <https://discourse.gohugo.io/t/related-posts-based-on-heading/57467>.

I suggest using the name tokenize, documented as follows:

(bool) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type fragments; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use cardinalityThreshold to prune common words and reduce build time. Default is false.

The two patches

Our patch+91 −6
docs/content/en/configuration/related-content.mdindex 92cf92e8a..496c33b08 100644@@ -48,6 +48,9 @@ This is the default configuration: `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. +`tokenize`+: (`bool`) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type `fragments`; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use `cardinalityThreshold` to prune common words and reduce build time. Default is `false`.+ `type` : (`string`) One of `basic` or `fragments`. Default is `basic`. related/inverted_index.goindex c6c65d97f..4567b2dae 100644@@ -18,12 +18,15 @@ import ( 	"context" 	"errors" 	"fmt"+	"html" 	"maps" 	"math"+	"regexp" 	"sort" 	"strings" 	"sync" 	"time"+	"unicode"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +128,11 @@ type IndexConfig struct { 	// Will lower case all string values in and queries tothis index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Whether to split multi-word values into individual keywords for indexing and matching.+	// Currently applies to heading titles in indices of type "fragments"; non-heading+	// fragments such as definition terms continue to use exact-identifier matching.+	Tokenize bool }  // Document is the interface an indexable document in Hugo must fulfill.@@ -220,9 +228,9 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error {  			if config.Type == TypeFragments { 				if fp, ok := doc.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					for _, keyword := range config.fragmentKeywords(fp.Fragments(ctx)) { 						added = true-						setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc)+						setm[keyword] = append(setm[keyword], doc) 					} 				} 			}@@ -374,12 +382,16 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 		if cfg.Type == TypeFragments { 			for _, fragment := range opts.Fragments { 				keywords = append(keywords, FragmentKeyword(fragment))+				if cfg.Tokenize {+					// Keep the exact identifier above to match non-heading fragments.+					for _, word := range cfg.tokenize(fragment) {+						keywords = append(keywords, FragmentKeyword(word))+					}+				} 			} 			if opts.Document != nil { 				if fp, ok := opts.Document.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {-						keywords = append(keywords, FragmentKeyword(fragment))-					}+					keywords = append(keywords, cfg.fragmentKeywords(fp.Fragments(ctx))...) 				} 			} @@ -413,6 +425,62 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 	return idx.search(ctx, queryElements...) } +var htmlTagRe = regexp.MustCompile(`<[^>]*>`)++// tokenize splits s into words, lower casing them if configured.+// Any HTML tags and entities in s are removed before splitting.+func (cfg IndexConfig) tokenize(s string) []string {+	if strings.ContainsAny(s, "<&") {+		s = html.UnescapeString(htmlTagRe.ReplaceAllString(s, " "))+	}+	words := strings.FieldsFunc(s, func(r rune) bool {+		return !unicode.IsLetter(r) && !unicode.IsNumber(r)+	})+	if cfg.ToLower {+		for i, w := range words {+			words[i] = strings.ToLower(w)+		}+	}+	return words+}++// fragmentKeywords returns the keywords to use for the given fragments.+func (cfg IndexConfig) fragmentKeywords(f *tableofcontents.Fragments) []Keyword {+	if f == nil {+		return nil+	}+	keywords := make([]Keyword, 0, len(f.Identifiers))+	if !cfg.Tokenize {+		for _, id := range f.Identifiers {+			keywords = append(keywords, FragmentKeyword(id))+		}+		return keywords+	}++	seen := make(map[FragmentKeyword]bool)+	add := func(s string) {+		kw := FragmentKeyword(s)+		if !seen[kw] {+			seen[kw] = true+			keywords = append(keywords, kw)+		}+	}+	for _, id := range f.Identifiers {+		var words []string+		if h, found := f.HeadingsMap[id]; found && h != nil {+			words = cfg.tokenize(h.Title)+		}+		if len(words) == 0 {+			add(id)+			continue+		}+		for _, word := range words {+			add(word)+		}+	}+	return keywords+}+ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	if cfg.ToLower { 		s = strings.ToLower(s)@@ -467,6 +535,7 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa  	applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero() 	var fragmentsFilter collections.SortedStringSlice+	var tokenizeCfg *IndexConfig  	for _, el := range query { 		setm, found := idx.index[el.Index]@@ -496,6 +565,9 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 					if config.Type == TypeFragments && config.ApplyFilter { 						if fkw, ok := kw.(FragmentKeyword); ok { 							fragmentsFilter = append(fragmentsFilter, string(fkw))+							if config.Tokenize && tokenizeCfg == nil {+								tokenizeCfg = &config+							} 						} 					} @@ -538,7 +610,17 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 		if len(fragmentsFilter) > 0 { 			if dp, ok := result[i].(FragmentProvider); ok { 				result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {-					return fragmentsFilter.Contains(h.ID)+					if fragmentsFilter.Contains(h.ID) {+						return true+					}+					if tokenizeCfg != nil {+						for _, word := range tokenizeCfg.tokenize(h.Title) {+							if fragmentsFilter.Contains(word) {+								return true+							}+						}+					}+					return false 				}) 			} 		} 
Merged fix · PR #+442 −20
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( 	"context" 	"errors" 	"fmt"+	"html" 	"maps" 	"math" 	"sort" 	"strings" 	"sync" 	"time"+	"unicode"+	"unicode/utf8"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { 	// used in more than 50% of the documents in the index. 	CardinalityThreshold int -	// Will lower case all string values in and queries tothis index.+	// Will lower case all string values in and queries to this index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Whether to split multi-word values into individual keywords.+	Tokenize bool++	// Minimum token length in Unicode code points when tokenizing.+	MinTokenLength int }  // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error {  			if config.Type == TypeFragments { 				if fp, ok := doc.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers { 						added = true+						if config.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !config.shouldSkipToken(word) {+										kw := config.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											setm[kw] = append(setm[kw], doc)+										}+									}+								}+								continue+							}+						} 						setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) 					} 				}@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 			} 			if opts.Document != nil { 				if fp, ok := opts.Document.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers {+						if cfg.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !cfg.shouldSkipToken(word) {+										kw := cfg.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											keywords = append(keywords, kw)+										}+									}+								}+								continue+							}+						} 						keywords = append(keywords, FragmentKeyword(fragment)) 					} 				}@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+	return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { 	var keywords []Keyword  	switch vv := v.(type) { 	case string:-		keywords = append(keywords, cfg.stringToKeyword(vv))+		if cfg.Tokenize {+			for _, word := range splitWords(vv) {+				if !cfg.shouldSkipToken(word) {+					keywords = append(keywords, cfg.stringToKeyword(word))+				}+			}+		} else {+			keywords = append(keywords, cfg.stringToKeyword(vv))+		} 	case []string:-		vvv := make([]Keyword, len(vv))-		for i := range vvv {-			vvv[i] = cfg.stringToKeyword(vv[i])+		if !cfg.Tokenize {+			kw := make([]Keyword, len(vv))+			for i, s := range vv {+				kw[i] = cfg.stringToKeyword(s)+			}+			keywords = append(keywords, kw...)+		} else {+			for _, s := range vv {+				for _, word := range splitWords(s) {+					if !cfg.shouldSkipToken(word) {+						keywords = append(keywords, cfg.stringToKeyword(word))+					}+				}+			} 		}-		keywords = append(keywords, vvv...) 	case []any: 		return cfg.ToKeywords(cast.ToStringSlice(vv)) 	case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}()  	applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()-	var fragmentsFilter collections.SortedStringSlice+	// Bundled into one struct so the closure below causes only one heap escape.+	var ff struct {+		ids     collections.SortedStringSlice+		words   collections.SortedStringSlice+		toLower bool+	}  	for _, el := range query { 		setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 			return []Document{}, fmt.Errorf("index config for %q not found", el.Index) 		} +		if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+			ff.toLower = config.ToLower+		}+ 		for _, kw := range el.Keywords { 			if docs, found := setm[kw]; found { 				for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa  					if config.Type == TypeFragments && config.ApplyFilter { 						if fkw, ok := kw.(FragmentKeyword); ok {-							fragmentsFilter = append(fragmentsFilter, string(fkw))+							if config.Tokenize {+								ff.words = append(ff.words, string(fkw))+							} else {+								ff.ids = append(ff.ids, string(fkw))+							} 						} 					}- 					r, found := matchm[doc] 					if !found { 						r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}  	sort.Stable(matches)-	sort.Strings(fragmentsFilter)+	sort.Strings(ff.ids)+	sort.Strings(ff.words)  	result := make([]Document, len(matches))  	for i, m := range matches { 		result[i] = m.Doc -		if len(fragmentsFilter) > 0 {+		if len(ff.ids) > 0 || len(ff.words) > 0 { 			if dp, ok := result[i].(FragmentProvider); ok { 				result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {-					return fragmentsFilter.Contains(h.ID)+					if ff.ids.Contains(h.ID) {+						return true+					}+					if len(ff.words) > 0 {+						for _, word := range splitWords(h.Title) {+							if ff.toLower {+								word = strings.ToLower(word)+							}+							if ff.words.Contains(word) {+								return true+							}+						}+					}+					return false 				}) 			} 		}@@ -618,6 +707,31 @@ type Keyword interface { 	String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+	// Fast path: no whitespace or HTML entities means at most one token.+	if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+		title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if title == "" {+			return nil+		}+		return []string{title}+	}+	raw := strings.Fields(html.UnescapeString(title))+	words := raw[:0]+	for _, w := range raw {+		w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if w != "" {+			words = append(words, w)+		}+	}+	if len(words) == 0 {+		return nil+	}+	return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { 	kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { 	}) } +func TestSplitWords(t *testing.T) {+	c := qt.New(t)++	tests := []struct {+		in   string+		want []string+	}{+		{"", nil},+		{"   ", nil},+		{"hello world", []string{"hello", "world"}},+		// leading/trailing punctuation stripped+		{"...hello...", []string{"hello"}},+		{",hello, world!", []string{"hello", "world"}},+		// digits preserved+		{"go1.24", []string{"go1.24"}},+		// HTML entities decoded+		{"foo &amp;bar", []string{"foo", "bar"}},+		// fast path: single token, no whitespace+		{"&amp;foo&amp;", []string{"foo"}},+		// &nbsp; and similar entities that decode to whitespace must not use the fast path+		{"Apples&nbsp;and&nbsp;oranges!", []string{"Apples", "and", "oranges"}},+		// punctuation-only tokens dropped+		{"hello --- world", []string{"hello", "world"}},+		// mixed spaces+		{"  foo   bar  ", []string{"foo", "bar"}},+	}++	for _, tt := range tests {+		c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+	}+}+ func BenchmarkRelatedNewIndex(b *testing.B) { 	pages := make([]*testDoc, 100) 	numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   weight = 90 [[related.indices]]   name = 'keywords'-  weight = 80	+  weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} -	+ `  	b := hugolib.Test(t, files)  	expect := `-P1 Fragments: [p1-title]	+P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 		b.AssertFileContent("public/"+p+"/index.html", expect) 	} -	b.AssertFileContent("public/p1/index.html",+	b.AssertFileContent(+		"public/p1/index.html", 		"Related 1:  0: p2: h0: First title|ref1|::END", 		"Related 2:  0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", 	) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+	b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  applyFilter = true+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+	b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = false+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// toLower=false: "Hugo" != "hugo", so no match.+	b.AssertFileContent("public/p1/index.html", "Related:  ")+	b.AssertFileContent("public/p2/index.html", "Related:  ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+  minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+	b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+	b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+	// All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+[[related.indices]]+  name = 'keywords'+  weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p3 share the same unique fragment tokens; p1 also matches on+	// keywords ("fruit"). Without deduplication, p3's repeated headings+	// inflate its weight above p1's combined fragment+keywords weight.+	b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { 	var files strings.Builder 	files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   toLower = false [[related.indices]]   name = 'keywords'-  weight = 70	+  weight = 70 [[related.indices]]   name = 'pagerefs'   type = 'fragments'-  weight = 30	+  weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `) 

What the agent reported

I added the tokenize option, but it is untested: no Go code could be compiled or run here. ./bt-test ./related/ stopped at setup with FAIL github.com/gohugoio/hugo/related [setup failed], because Go modules can't be downloaded (proxy.golang.org fails to resolve, the local module cache is empty and there's no vendor/). go vet failed the same way, and only gofmt could run (it reports no problems).

What was wrong: related-content indices of type fragments only matched whole heading IDs. Two headings that share words, such as "Installing Hugo on Linux" and "Hugo Installation Guide", never matched each other.

What I changed (uncommitted, in related/inverted_index.go and docs/content/en/configuration/related-content.md):

  • New option: IndexConfig.Tokenize (tokenize). The config decoder ignores case, so no decoding change was needed.
  • Splitting: when it's on, heading titles are split into words (on anything that isn't a letter or digit), lower-cased when toLower is set. HTML tags and entities are removed first, because goldmark heading titles can contain markup.
  • Non-heading fragments: identifiers that aren't headings, such as definition terms, still match by their exact ID. So does a heading whose title produces no words.
  • One entry per word: each word is recorded once per page, so a page isn't counted twice for the same word.
  • Indexing and page queries: both use the same logic.
  • Fragments you pass in a search: each value is still sent whole, so exact IDs keep matching, and its words are added when tokenize is on.
  • applyFilter: a heading is kept if its ID matches, or, with tokenize, if any word of its title matches.
  • With tokenize = false: behaviour is the same as before.
  • Docs: the tokenize entry is under "Per-index settings", with your wording.

Test evidence: none that ran.

  • Test dropped: a subagent wrote TestRelatedFragmentsTokenize in related/related_integration_test.go, with a tokenize=true and a tokenize=false case. It checked that p1 and p2 find each other through the word "hugo", that p3 doesn't match, and that .HeadingsFiltered keeps only the matching heading. I never saw it run, so I removed it; the final diff contains no test.
  • Review: a reviewer traced the code and that test by hand. It found no compile errors and said the expected output looked exact. It also found that splitting the fragments passed to a search broke exact definition-term matching, and that HTML in titles was being indexed as words. I fixed both after the review, and those fixes were not reviewed again.
  • Before merging: re-add a test like that one and run ./bt-test ./related/.

Blast radius: the indexing and query loops for fragments indices now go through one shared function. With tokenize off it produces the same IDs as before, and the applyFilter check is the same too. The existing TestRelatedFragments would confirm this, but it couldn't run either.

Known limitations:

  • If a site has more than one fragments index, the applyFilter word check uses the settings of the first index with tokenize on.
  • There's no stop-word list; common words have to be pruned with cardinalityThreshold, as the docs say.

Requirements:

#RequirementStatus
1Add an option named tokenizeDone (Tokenize field), not compiled
2It applies to heading titles in fragments indicesDone, not tested
3Definition terms and other non-heading fragments keep exact-ID matchingDone, including search-time fragments, not tested
4Default is falseDone: the zero value, and the old path is unchanged
5Document it with the given textDone: exact wording, alphabetical order
6Mention that cardinalityThreshold prunes common wordsIn the docs text; cardinalityThreshold itself is unchanged and applies to the new word entries