Backtestshugorun-20261002-200313#15199

Issue #15199 Merged fix PR # Run 2 Oct 2026 · run-20261002-200313 gohugoio/hugo

related: Add option to tokenize headings

Better

Our patch was judged better than the merged fix.

In this judgment, Patch A = our agent, Patch B = the merged fix.

Patch A is the better fix, mainly on correctness and scope.

**Correctness**
- **ID matching.** A keeps exact-identifier matching and adds heading-title tokens on top of it. B replaces a heading's identifier with its tokens when `tokenize` is on. Under B, heading IDs are no longer indexed, so explicit `fragments` queries by heading ID (e.g. `Related (dict "fragments" ...)`) stop matching.
- **ApplyFilter.** B puts every tokenized-index FragmentKeyword, including query IDs, into `ff.words`. It then compares those only against title words, so ID-based heading filtering breaks for tokenized indices.
- **Query tokens.** A also tokenizes `opts.Fragments` query strings and deduplicates keywords, so a shared word counts once per document. Its ApplyFilter path still checks IDs first and then title tokens, honouring ToLower.
- **HTML in titles.** A's tokenizer strips HTML tags (heading titles can contain `<code>`) and decodes entities. B's `splitWords` decodes entities but leaves tags in place, so tag fragments can leak into tokens.

**Scope**
- B extends tokenization to basic indices via `ToKeywords`. The issue explicitly limits it to heading titles in fragments indices, and this change also alters keyword query behaviour.
- B adds an unrequested `MinTokenLength` option and makes whitespace churn in existing tests.
- A sticks to the issue's specification and adds the requested documentation text verbatim.

**Tests**
- Both add meaningful tests.
- A's integration test covers default vs. tokenize vs. toLower, applyFilter output, definition terms staying exact, inline code in titles, and query strings.
- A also has unit tests for the tokenizer, keyword generation, config decoding and duplicate-weight avoidance.
- B's tests are reasonable, but several target behaviour outside the issue's scope (basic-index tokenization, minTokenLength).

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

1h 57m

Wall-clock time in the container

Full test suite

No new failures

3,749 passed · 99 skipped · 3,850 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

336vs 462 merged

Changed lines, ours vs merged

See <https://discourse.gohugo.io/t/related-posts-based-on-heading/57467>.

I suggest using the name tokenize, documented as follows:

(bool) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type fragments; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use cardinalityThreshold to prune common words and reduce build time. Default is false.

The two patches

Our patch+328 −8
docs/content/en/configuration/related-content.mdindex 92cf92e8a..496c33b08 100644@@ -48,6 +48,9 @@ This is the default configuration: `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. +`tokenize`+: (`bool`) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type `fragments`; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use `cardinalityThreshold` to prune common words and reduce build time. Default is `false`.+ `type` : (`string`) One of `basic` or `fragments`. Default is `basic`. docs/data/docs.yamlindex 19b950dd5..831250fbc 100644@@ -2192,6 +2192,7 @@ config:       name: keywords       pattern: ''       toLower: false+      tokenize: false       type: basic       weight: 100     - applyFilter: false@@ -2199,6 +2200,7 @@ config:       name: date       pattern: ''       toLower: false+      tokenize: false       type: basic       weight: 10     - applyFilter: false@@ -2206,6 +2208,7 @@ config:       name: tags       pattern: ''       toLower: false+      tokenize: false       type: basic       weight: 80     threshold: 80related/inverted_index.goindex c6c65d97f..3dbbd2da4 100644@@ -18,12 +18,14 @@ import ( 	"context" 	"errors" 	"fmt"+	"html" 	"maps" 	"math" 	"sort" 	"strings" 	"sync" 	"time"+	"unicode"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +127,11 @@ type IndexConfig struct { 	// Will lower case all string values in and queries tothis index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Whether to split multi-word values into individual keywords for indexing and matching.+	// This is currently only used for heading titles in the "fragments" type;+	// non-heading fragments (e.g. definition terms) still use exact identifier matching.+	Tokenize bool }  // Document is the interface an indexable document in Hugo must fulfill.@@ -220,9 +227,9 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error {  			if config.Type == TypeFragments { 				if fp, ok := doc.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					for _, fragment := range config.fragmentKeywords(fp.Fragments(ctx)) { 						added = true-						setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc)+						setm[fragment] = append(setm[fragment], doc) 					} 				} 			}@@ -374,15 +381,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 		if cfg.Type == TypeFragments { 			for _, fragment := range opts.Fragments { 				keywords = append(keywords, FragmentKeyword(fragment))+				if cfg.Tokenize {+					for _, token := range tokenize(fragment) {+						if kw := cfg.stringToKeyword(token); kw.String() != fragment {+							keywords = append(keywords, kw)+						}+					}+				} 			} 			if opts.Document != nil { 				if fp, ok := opts.Document.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {-						keywords = append(keywords, FragmentKeyword(fragment))-					}+					keywords = append(keywords, cfg.fragmentKeywords(fp.Fragments(ctx))...) 				} 			}-+			if cfg.Tokenize {+				// Avoid counting a shared word more than once per document.+				keywords = uniqueKeywords(keywords)+			} 		} 		queryElements = append(queryElements, newQueryElement(cfg.Name, keywords...)) 	}@@ -413,6 +428,88 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 	return idx.search(ctx, queryElements...) } +// fragmentKeywords returns the keywords for the given fragments in a "fragments" index.+// The identifiers are always matched exactly. If Tokenize is enabled, the+// heading titles are also split into individual words, and the result is+// deduplicated so a document is counted once per keyword.+func (cfg IndexConfig) fragmentKeywords(fragments *tableofcontents.Fragments) []Keyword {+	if fragments == nil {+		return nil+	}+	if !cfg.Tokenize {+		keywords := make([]Keyword, len(fragments.Identifiers))+		for i, fragment := range fragments.Identifiers {+			keywords[i] = FragmentKeyword(fragment)+		}+		return keywords+	}++	var keywords []Keyword+	seen := make(map[Keyword]bool)+	add := func(kw Keyword) {+		if !seen[kw] {+			seen[kw] = true+			keywords = append(keywords, kw)+		}+	}+	for _, fragment := range fragments.Identifiers {+		add(FragmentKeyword(fragment))+	}+	var walk func(headings tableofcontents.Headings)+	walk = func(headings tableofcontents.Headings) {+		for _, h := range headings {+			if h == nil {+				continue+			}+			for _, token := range tokenize(h.Title) {+				add(cfg.stringToKeyword(token))+			}+			walk(h.Headings)+		}+	}+	walk(fragments.Headings)++	return keywords+}++func uniqueKeywords(keywords []Keyword) []Keyword {+	seen := make(map[Keyword]bool, len(keywords))+	unique := keywords[:0]+	for _, kw := range keywords {+		if !seen[kw] {+			seen[kw] = true+			unique = append(unique, kw)+		}+	}+	return unique+}++// tokenize splits s into words, ignoring any HTML markup and punctuation.+func tokenize(s string) []string {+	if strings.IndexByte(s, '<') != -1 {+		var sb strings.Builder+		inTag := false+		for _, r := range s {+			switch {+			case r == '<':+				inTag = true+			case r == '>' && inTag:+				inTag = false+				sb.WriteRune(' ')+			case !inTag:+				sb.WriteRune(r)+			}+		}+		s = sb.String()+	}+	if strings.IndexByte(s, '&') != -1 {+		s = html.UnescapeString(s)+	}+	return strings.FieldsFunc(s, func(r rune) bool {+		return !unicode.IsLetter(r) && !unicode.IsNumber(r) && !unicode.IsMark(r)+	})+}+ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	if cfg.ToLower { 		s = strings.ToLower(s)@@ -466,7 +563,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}()  	applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()-	var fragmentsFilter collections.SortedStringSlice+	var (+		fragmentsFilter collections.SortedStringSlice+		// Set if any of the filter keywords come from a tokenized index.+		filterTokens        bool+		filterTokensToLower bool+	)  	for _, el := range query { 		setm, found := idx.index[el.Index]@@ -496,6 +598,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 					if config.Type == TypeFragments && config.ApplyFilter { 						if fkw, ok := kw.(FragmentKeyword); ok { 							fragmentsFilter = append(fragmentsFilter, string(fkw))+							if config.Tokenize {+								filterTokens = true+								filterTokensToLower = filterTokensToLower || config.ToLower+							} 						} 					} @@ -538,7 +644,20 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 		if len(fragmentsFilter) > 0 { 			if dp, ok := result[i].(FragmentProvider); ok { 				result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {-					return fragmentsFilter.Contains(h.ID)+					if fragmentsFilter.Contains(h.ID) {+						return true+					}+					if filterTokens {+						for _, token := range tokenize(h.Title) {+							if fragmentsFilter.Contains(token) {+								return true+							}+							if filterTokensToLower && fragmentsFilter.Contains(strings.ToLower(token)) {+								return true+							}+						}+					}+					return false 				}) 			} 		}related/inverted_index_test.goindex b33d42955..42b1d9159 100644@@ -23,6 +23,7 @@ import (  	qt "github.com/frankban/quicktest" 	"github.com/gohugoio/hugo/config"+	"github.com/gohugoio/hugo/markup/tableofcontents" )  type testDoc struct {@@ -296,6 +297,58 @@ func TestDecodeConfig(t *testing.T) { 	c.Assert(first.Name, qt.Equals, "keywords") } +func TestTokenize(t *testing.T) {+	c := qt.New(t)+	c.Assert(tokenize(""), qt.HasLen, 0)+	c.Assert(tokenize("Installing Hugo on Linux"), qt.DeepEquals, []string{"Installing", "Hugo", "on", "Linux"})+	c.Assert(tokenize("Hugo's  themes: an overview!"), qt.DeepEquals, []string{"Hugo", "s", "themes", "an", "overview"})+	c.Assert(tokenize("Using <code>linux</code> commands &amp; more"), qt.DeepEquals, []string{"Using", "linux", "commands", "more"})+	c.Assert(tokenize("Über 2 Größen"), qt.DeepEquals, []string{"Über", "2", "Größen"})+	// Combining marks are part of the word.+	c.Assert(tokenize("हिन्दी भाषा"), qt.DeepEquals, []string{"हिन्दी", "भाषा"})+}++func TestFragmentKeywordsTokenize(t *testing.T) {+	c := qt.New(t)++	var b tableofcontents.Builder+	b.AddAt(&tableofcontents.Heading{ID: "hugo-themes", Title: "Hugo Themes", Level: 2}, 0, 0)+	b.AddAt(&tableofcontents.Heading{ID: "more-hugo", Title: "More Hugo", Level: 3}, 0, 1)+	fragments := b.Build()++	cfg := IndexConfig{Name: "fragments", Type: TypeFragments}+	c.Assert(cfg.fragmentKeywords(fragments), qt.DeepEquals, []Keyword{FragmentKeyword("hugo-themes"), FragmentKeyword("more-hugo")})++	cfg.Tokenize = true+	c.Assert(cfg.fragmentKeywords(fragments), qt.DeepEquals, []Keyword{+		FragmentKeyword("hugo-themes"), FragmentKeyword("more-hugo"),+		FragmentKeyword("Hugo"), FragmentKeyword("Themes"), FragmentKeyword("More"),+	})++	cfg.ToLower = true+	c.Assert(cfg.fragmentKeywords(fragments), qt.DeepEquals, []Keyword{+		FragmentKeyword("hugo-themes"), FragmentKeyword("more-hugo"),+		FragmentKeyword("hugo"), FragmentKeyword("themes"), FragmentKeyword("more"),+	})+}++func TestDecodeConfigTokenize(t *testing.T) {+	c := qt.New(t)+	m, err := config.FromConfigString(`+[related]+threshold = 80+[[related.indices]]+name = 'fragments'+type = 'fragments'+tokenize = true+weight = 80+`, "toml")+	c.Assert(err, qt.IsNil)+	conf, err := DecodeConfig(m.GetParams("related"))+	c.Assert(err, qt.IsNil)+	c.Assert(conf.Indices[0].Tokenize, qt.IsTrue)+}+ func TestToKeywordsAnySlice(t *testing.T) { 	c := qt.New(t) 	var config IndexConfig@@ -412,3 +465,51 @@ func BenchmarkRelatedMatchesIn(b *testing.B) { 		} 	} }++type testFragmentDoc struct {+	*testDoc+	fragments *tableofcontents.Fragments+}++func (d *testFragmentDoc) Fragments(context.Context) *tableofcontents.Fragments {+	return d.fragments+}++func (d *testFragmentDoc) ApplyFilterToHeadings(context.Context, func(*tableofcontents.Heading) bool) Document {+	return d+}++func newTestFragmentDoc(name string, date time.Time, titles ...string) *testFragmentDoc {+	var b tableofcontents.Builder+	for i, title := range titles {+		b.AddAt(&tableofcontents.Heading{ID: fmt.Sprintf("%s-%d", name, i), Title: title, Level: 2}, 0, i)+	}+	d := newTestDocWithDate(name, date)+	d.name = name+	return &testFragmentDoc{testDoc: d, fragments: b.Build()}+}++// A word shared by several query fragments must only be counted once.+func TestSearchFragmentsTokenizeNoDuplicateWeight(t *testing.T) {+	c := qt.New(t)+	ctx := context.Background()++	idx := NewInvertedIndex(Config{+		Threshold: 0,+		Indices: IndicesConfig{+			IndexConfig{Name: "keywords", Weight: 1},+			IndexConfig{Name: "fragments", Type: TypeFragments, Weight: 10, Tokenize: true},+		},+	})++	date := time.Now()+	a := newTestFragmentDoc("a", date, "Hugo")+	b := newTestFragmentDoc("b", date, "Themes Guide")+	c.Assert(idx.Add(ctx, a, b), qt.IsNil)++	m, err := idx.Search(ctx, SearchOpts{Fragments: []string{"Hugo Themes", "Hugo Guide"}})+	c.Assert(err, qt.IsNil)+	c.Assert(m, qt.HasLen, 2)+	c.Assert(m[0], qt.Equals, Document(b))+	c.Assert(m[1], qt.Equals, Document(a))+}related/related_integration_test.goindex e1387d3c5..4696effbd 100644@@ -134,6 +134,100 @@ Related 2: 2 	) } +func TestRelatedFragmentsTokenize(t *testing.T) {+	t.Parallel()++	filesTemplate := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[markup.goldmark.parser]+autoDefinitionTermID = true+[related]+  includeNewer = true+  threshold = 20+  toLower = TOLOWER+[[related.indices]]+  name = 'fragments'+  type = 'fragments'+  applyFilter = true+  tokenize = TOKENIZE+  weight = 100+-- content/p1.md --+---+title: p1+---+## Installing Hugo on Linux++Some text.++Linux kernel+: A definition.+-- content/p2.md --+---+title: p2+---+## Hugo themes overview++## Unrelated section+-- content/p3.md --+---+title: p3+---+## Using ` + "`linux`" + ` commands++## Another unrelated section+-- content/p4.md --+---+title: p4+---+## Something else++Linux kernel+: Another definition.+-- layouts/single.html --+{{ $related := site.RegularPages.Related . }}+Related: {{ range $related }}{{ .Title }}: {{ range .HeadingsFiltered }}{{ .Title }}|{{ .ID }}|{{ end }}::{{ end }}END+{{ $q := site.RegularPages.Related (dict "indices" (slice "fragments") "fragments" (slice "Themes overview")) }}+Query: {{ range $q }}{{ .Title }}: {{ range .HeadingsFiltered }}{{ .ID }}|{{ end }}::{{ end }}END+`++	files := func(tokenize, toLower bool) string {+		return strings.NewReplacer("TOKENIZE", fmt.Sprint(tokenize), "TOLOWER", fmt.Sprint(toLower)).Replace(filesTemplate)+	}++	t.Run("Default", func(t *testing.T) {+		t.Parallel()+		b := hugolib.Test(t, files(false, false))+		// Only exact identifier matches (the definition term).+		b.AssertFileContent("public/p1/index.html", "Related: p4: ::END", "Query: END")+		b.AssertFileContent("public/p2/index.html", "Related: END")+	})++	t.Run("Tokenize", func(t *testing.T) {+		t.Parallel()+		b := hugolib.Test(t, files(true, false))+		// "Hugo" in heading titles of p1 and p2, definition term matches p4 exactly.+		b.AssertFileContent("public/p1/index.html", "Related: ", "p2: Hugo themes overview|hugo-themes-overview|::", "p4: ::")+		b.AssertFileContent("public/p1/index.html", "! p3:")+		b.AssertFileContent("public/p2/index.html", "Related: p1: Installing Hugo on Linux|installing-hugo-on-linux|::p3: Another unrelated section|another-unrelated-section|::END")+		// Unrelated sections share the "unrelated" and "section" words.+		b.AssertFileContent("public/p3/index.html", "Related: p2: Unrelated section|unrelated-section|::END")+		// The non-heading identifier is not tokenized, so "kernel" etc. does not match p3.+		b.AssertFileContent("public/p4/index.html", "Related: p1: ::END")+		// Query with a multi-word fragment.+		b.AssertFileContent("public/p1/index.html", "Query: p2: hugo-themes-overview|::END")+	})++	t.Run("TokenizeToLower", func(t *testing.T) {+		t.Parallel()+		b := hugolib.Test(t, files(true, true))+		// "Linux" in p1 and "linux" (inside code) in p3 now match.+		b.AssertFileContent("public/p3/index.html", "p1: Installing Hugo on Linux|installing-hugo-on-linux|::", "p2: Unrelated section|unrelated-section|::")+		b.AssertFileContent("public/p1/index.html", "p3: Using &lt;code&gt;linux&lt;/code&gt; commands|using-linux-commands|::")+	})+}+ func BenchmarkRelatedSite(b *testing.B) { 	var files strings.Builder 	files.WriteString(` 
Merged fix · PR #+442 −20
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( 	"context" 	"errors" 	"fmt"+	"html" 	"maps" 	"math" 	"sort" 	"strings" 	"sync" 	"time"+	"unicode"+	"unicode/utf8"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { 	// used in more than 50% of the documents in the index. 	CardinalityThreshold int -	// Will lower case all string values in and queries tothis index.+	// Will lower case all string values in and queries to this index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Whether to split multi-word values into individual keywords.+	Tokenize bool++	// Minimum token length in Unicode code points when tokenizing.+	MinTokenLength int }  // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error {  			if config.Type == TypeFragments { 				if fp, ok := doc.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers { 						added = true+						if config.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !config.shouldSkipToken(word) {+										kw := config.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											setm[kw] = append(setm[kw], doc)+										}+									}+								}+								continue+							}+						} 						setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) 					} 				}@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 			} 			if opts.Document != nil { 				if fp, ok := opts.Document.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers {+						if cfg.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !cfg.shouldSkipToken(word) {+										kw := cfg.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											keywords = append(keywords, kw)+										}+									}+								}+								continue+							}+						} 						keywords = append(keywords, FragmentKeyword(fragment)) 					} 				}@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+	return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { 	var keywords []Keyword  	switch vv := v.(type) { 	case string:-		keywords = append(keywords, cfg.stringToKeyword(vv))+		if cfg.Tokenize {+			for _, word := range splitWords(vv) {+				if !cfg.shouldSkipToken(word) {+					keywords = append(keywords, cfg.stringToKeyword(word))+				}+			}+		} else {+			keywords = append(keywords, cfg.stringToKeyword(vv))+		} 	case []string:-		vvv := make([]Keyword, len(vv))-		for i := range vvv {-			vvv[i] = cfg.stringToKeyword(vv[i])+		if !cfg.Tokenize {+			kw := make([]Keyword, len(vv))+			for i, s := range vv {+				kw[i] = cfg.stringToKeyword(s)+			}+			keywords = append(keywords, kw...)+		} else {+			for _, s := range vv {+				for _, word := range splitWords(s) {+					if !cfg.shouldSkipToken(word) {+						keywords = append(keywords, cfg.stringToKeyword(word))+					}+				}+			} 		}-		keywords = append(keywords, vvv...) 	case []any: 		return cfg.ToKeywords(cast.ToStringSlice(vv)) 	case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}()  	applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()-	var fragmentsFilter collections.SortedStringSlice+	// Bundled into one struct so the closure below causes only one heap escape.+	var ff struct {+		ids     collections.SortedStringSlice+		words   collections.SortedStringSlice+		toLower bool+	}  	for _, el := range query { 		setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 			return []Document{}, fmt.Errorf("index config for %q not found", el.Index) 		} +		if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+			ff.toLower = config.ToLower+		}+ 		for _, kw := range el.Keywords { 			if docs, found := setm[kw]; found { 				for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa  					if config.Type == TypeFragments && config.ApplyFilter { 						if fkw, ok := kw.(FragmentKeyword); ok {-							fragmentsFilter = append(fragmentsFilter, string(fkw))+							if config.Tokenize {+								ff.words = append(ff.words, string(fkw))+							} else {+								ff.ids = append(ff.ids, string(fkw))+							} 						} 					}- 					r, found := matchm[doc] 					if !found { 						r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}  	sort.Stable(matches)-	sort.Strings(fragmentsFilter)+	sort.Strings(ff.ids)+	sort.Strings(ff.words)  	result := make([]Document, len(matches))  	for i, m := range matches { 		result[i] = m.Doc -		if len(fragmentsFilter) > 0 {+		if len(ff.ids) > 0 || len(ff.words) > 0 { 			if dp, ok := result[i].(FragmentProvider); ok { 				result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {-					return fragmentsFilter.Contains(h.ID)+					if ff.ids.Contains(h.ID) {+						return true+					}+					if len(ff.words) > 0 {+						for _, word := range splitWords(h.Title) {+							if ff.toLower {+								word = strings.ToLower(word)+							}+							if ff.words.Contains(word) {+								return true+							}+						}+					}+					return false 				}) 			} 		}@@ -618,6 +707,31 @@ type Keyword interface { 	String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+	// Fast path: no whitespace or HTML entities means at most one token.+	if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+		title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if title == "" {+			return nil+		}+		return []string{title}+	}+	raw := strings.Fields(html.UnescapeString(title))+	words := raw[:0]+	for _, w := range raw {+		w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if w != "" {+			words = append(words, w)+		}+	}+	if len(words) == 0 {+		return nil+	}+	return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { 	kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { 	}) } +func TestSplitWords(t *testing.T) {+	c := qt.New(t)++	tests := []struct {+		in   string+		want []string+	}{+		{"", nil},+		{"   ", nil},+		{"hello world", []string{"hello", "world"}},+		// leading/trailing punctuation stripped+		{"...hello...", []string{"hello"}},+		{",hello, world!", []string{"hello", "world"}},+		// digits preserved+		{"go1.24", []string{"go1.24"}},+		// HTML entities decoded+		{"foo &amp;bar", []string{"foo", "bar"}},+		// fast path: single token, no whitespace+		{"&amp;foo&amp;", []string{"foo"}},+		// &nbsp; and similar entities that decode to whitespace must not use the fast path+		{"Apples&nbsp;and&nbsp;oranges!", []string{"Apples", "and", "oranges"}},+		// punctuation-only tokens dropped+		{"hello --- world", []string{"hello", "world"}},+		// mixed spaces+		{"  foo   bar  ", []string{"foo", "bar"}},+	}++	for _, tt := range tests {+		c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+	}+}+ func BenchmarkRelatedNewIndex(b *testing.B) { 	pages := make([]*testDoc, 100) 	numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   weight = 90 [[related.indices]]   name = 'keywords'-  weight = 80	+  weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} -	+ `  	b := hugolib.Test(t, files)  	expect := `-P1 Fragments: [p1-title]	+P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 		b.AssertFileContent("public/"+p+"/index.html", expect) 	} -	b.AssertFileContent("public/p1/index.html",+	b.AssertFileContent(+		"public/p1/index.html", 		"Related 1:  0: p2: h0: First title|ref1|::END", 		"Related 2:  0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", 	) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+	b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  applyFilter = true+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+	b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = false+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// toLower=false: "Hugo" != "hugo", so no match.+	b.AssertFileContent("public/p1/index.html", "Related:  ")+	b.AssertFileContent("public/p2/index.html", "Related:  ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+  minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+	b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+	b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+	// All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+[[related.indices]]+  name = 'keywords'+  weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p3 share the same unique fragment tokens; p1 also matches on+	// keywords ("fruit"). Without deduplication, p3's repeated headings+	// inflate its weight above p1's combined fragment+keywords weight.+	b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { 	var files strings.Builder 	files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   toLower = false [[related.indices]]   name = 'keywords'-  weight = 70	+  weight = 70 [[related.indices]]   name = 'pagerefs'   type = 'fragments'-  weight = 30	+  weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)