Backtestshugorun-20261007-020926#7515

Issue #7515 Merged fix PR #15238 Run 7 Oct 2026 · run-20261007-020926 gohugoio/hugo

Related content: Partial-text indexing

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Patch B resolves the issue directly and is the better fix. It adds an opt-in `splitWords` flag for string indices such as title. Values are split on characters that are not letters, numbers or marks, which is Unicode-aware. Words are deduplicated per document, so repeated words cannot inflate rank. It also adds configurable `stopWords`, which the issue explicitly suggests for ignoring common words in an i18n-friendly way. The routing goes through the existing StringsToKeywords/ToKeywords paths, so both indexing and querying are consistent. It rejects the unsupported fragments combination at config decode time with a clear error and documents the new options. Its tests are meaningful and would fail before the change: unit tests cover splitting, Unicode, stop words and dedup; search tests cover named slices, document search and stop-word exclusion; there is a config decode test including the fragments error; and an integration test compares results with and without splitWords.

Patch A adds `tokenize` and `minTokenLength` and extends tokenization to fragment heading titles, which adds complexity and risk:
- For fragments, tokenized heading words become `StringKeyword` (via stringToKeyword), not `FragmentKeyword`. The ApplyFilter branch only collects words when `kw.(FragmentKeyword)` succeeds, so `ff.words` appears never to be populated. The tokenized applyFilter path therefore looks broken, which casts doubt on its own ApplyFilter test.
- The plain string/[]string tokenize path does not deduplicate repeated words within a value, so rank can be inflated. Dedup was added only for fragments.
- It has no stop-word support; minTokenLength is a cruder substitute.
- It mixes in unrelated whitespace churn in an existing test file.
- It has no docs and no config validation.

Overall, B is more correct, better scoped, better tested and better documented.

The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.

Solve

43m 53s

Wall-clock time in the container

Full test suite

No new failures

3,745 passed · 99 skipped · 3,846 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

235vs 462 merged

Changed lines, ours vs merged

When specifying the indices for related content, it would be useful to specify that some of them were split by word, so that their text could be matched partially. This would allow the title to be used for more in-depth indexing, rather than requiring the user to specify tags / keywords.

This would likely increase memory usage and decrease performance, so should definitely be opt-in!

I suspect common words (a, an, the etc) might want to be ignored? Although doing that in a sane, i18n-friendly manner might be a challenge!

The two patches

Our patch+222 −13
docs/content/en/configuration/related-content.mdindex 92cf92e8a..cb81020d9 100644@@ -45,6 +45,12 @@ This is the default configuration: `pattern` : (`string`) This is currently only relevant for dates. When listing related content, we may want to list content that is also close in time. Setting "2006" (default value for date indexes) as the pattern for a date index will add weight to pages published in the same year. For busier blogs, "200601" (year and month) may be a better default. +`splitWords`+: (`bool`) Whether to split string values into words before indexing them, so that text such as the page `title` can be matched partially. Words are separated by any character that is not a letter, a number, or a mark. This increases memory usage and indexing time. Not supported for the `fragments` type. Default is `false`.++`stopWords`+: (`[]string`) Words to ignore when `splitWords` is enabled, for example `['a', 'an', 'the']`. Matched case-insensitively. Default is none.+ `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. related/inverted_index.goindex c6c65d97f..18f937bbd 100644@@ -24,6 +24,7 @@ import ( 	"strings" 	"sync" 	"time"+	"unicode"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +126,17 @@ type IndexConfig struct { 	// Will lower case all string values in and queries tothis index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Enable to split string values into words, so text (e.g. the title)+	// can be matched partially. Words are separated by any character that is+	// not a letter, a number or a mark.+	// Note that this increases the size of the index.+	// Not supported for the "fragments" type.+	SplitWords bool++	// Words to ignore when SplitWords is enabled, e.g. "a", "an" and "the".+	// Matched case-insensitively.+	StopWords []string }  // Document is the interface an indexable document in Hugo must fulfill.@@ -423,19 +435,59 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	return StringKeyword(s) } +// stringsToKeywords converts the given strings to keywords, splitting+// them into unique words if SplitWords is enabled.+func (cfg IndexConfig) stringsToKeywords(s ...string) []Keyword {+	if !cfg.SplitWords {+		kw := make([]Keyword, len(s))+		for i := range s {+			kw[i] = cfg.stringToKeyword(s[i])+		}+		return kw+	}++	var kw []Keyword+	seen := make(map[Keyword]bool)+	for _, v := range s {+		for _, word := range splitWords(v) {+			if cfg.isStopWord(word) {+				continue+			}+			k := cfg.stringToKeyword(word)+			if seen[k] {+				continue+			}+			seen[k] = true+			kw = append(kw, k)+		}+	}+	return kw+}++func (cfg IndexConfig) isStopWord(word string) bool {+	for _, sw := range cfg.StopWords {+		if strings.EqualFold(sw, word) {+			return true+		}+	}+	return false+}++func splitWords(s string) []string {+	return strings.FieldsFunc(s, func(r rune) bool {+		return !(unicode.IsLetter(r) || unicode.IsNumber(r) || unicode.IsMark(r))+	})+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { 	var keywords []Keyword  	switch vv := v.(type) { 	case string:-		keywords = append(keywords, cfg.stringToKeyword(vv))+		keywords = append(keywords, cfg.stringsToKeywords(vv)...) 	case []string:-		vvv := make([]Keyword, len(vv))-		for i := range vvv {-			vvv[i] = cfg.stringToKeyword(vv[i])-		}-		keywords = append(keywords, vvv...)+		keywords = append(keywords, cfg.stringsToKeywords(vv...)...) 	case []any: 		return cfg.ToKeywords(cast.ToStringSlice(vv)) 	case time.Time:@@ -591,6 +643,9 @@ func DecodeConfig(m hmaps.Params) (Config, error) { 		if !validTypes[c.Indices[i].Type] { 			return c, fmt.Errorf("invalid index type %q. Must be one of %v", c.Indices[i].Type, maps.Keys(validTypes)) 		}+		if icfg.SplitWords && c.Indices[i].Type == TypeFragments {+			return Config{}, fmt.Errorf("splitWords is not supported for index %q of type %q", icfg.Name, TypeFragments)+		} 		if icfg.CardinalityThreshold < 0 || icfg.CardinalityThreshold > 100 { 			return Config{}, errors.New("cardinalityThreshold threshold must be between 0 and 100") 		}@@ -620,11 +675,5 @@ type Keyword interface {  // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword {-	kw := make([]Keyword, len(s))--	for i := range s {-		kw[i] = cfg.stringToKeyword(s[i])-	}--	return kw+	return cfg.stringsToKeywords(s...) }related/inverted_index_test.goindex b33d42955..f14da89fe 100644@@ -22,6 +22,7 @@ import ( 	"time"  	qt "github.com/frankban/quicktest"+	"github.com/gohugoio/hugo/common/types" 	"github.com/gohugoio/hugo/config" ) @@ -252,6 +253,108 @@ func TestToKeywordsToLower(t *testing.T) { 	}) } +func TestToKeywordsSplitWords(t *testing.T) {+	c := qt.New(t)+	config := IndexConfig{Name: "title", ToLower: true, SplitWords: true, StopWords: []string{"The", "a"}}+	keywords, err := config.ToKeywords("The Go Gopher: a go-to guide, café ünïcode 2024! Go")+	c.Assert(err, qt.IsNil)+	c.Assert(keywords, qt.DeepEquals, config.StringsToKeywords("go", "gopher", "to", "guide", "café", "ünïcode", "2024"))+	c.Assert(keywords, qt.DeepEquals, []Keyword{+		StringKeyword("go"),+		StringKeyword("gopher"),+		StringKeyword("to"),+		StringKeyword("guide"),+		StringKeyword("café"),+		StringKeyword("ünïcode"),+		StringKeyword("2024"),+	})++	// Not enabled: whole values.+	config = IndexConfig{Name: "title", StopWords: []string{"the"}}+	keywords, err = config.ToKeywords([]any{"The Go Gopher"})+	c.Assert(err, qt.IsNil)+	c.Assert(keywords, qt.DeepEquals, []Keyword{StringKeyword("The Go Gopher")})+}++type textDoc struct {+	name  string+	title string+	date  time.Time+}++func (d *textDoc) Name() string           { return d.name }+func (d *textDoc) PublishDate() time.Time { return d.date }+func (d *textDoc) RelatedKeywords(cfg IndexConfig) ([]Keyword, error) {+	return cfg.ToKeywords(d.title)+}++func TestSearchSplitWords(t *testing.T) {+	c := qt.New(t)+	config := Config{+		Threshold: 20,+		Indices: IndicesConfig{+			IndexConfig{Name: "title", Weight: 100, ToLower: true, SplitWords: true, StopWords: []string{"the", "a"}},+		},+	}+	idx := NewInvertedIndex(config)+	date := time.Now()+	docs := []*textDoc{+		{name: "p1", title: "The Hugo Static Site Generator", date: date},+		{name: "p2", title: "Getting started with Hugo", date: date.Add(-time.Hour)},+		{name: "p3", title: "The art of a good coffee", date: date},+	}+	for _, d := range docs {+		c.Assert(idx.Add(context.Background(), d), qt.IsNil)+	}++	m, err := idx.Search(context.Background(), SearchOpts{NamedSlices: []types.KeyValues{{Key: "title", Values: []any{"Static sites with HUGO"}}}})+	c.Assert(err, qt.IsNil)+	c.Assert(m, qt.HasLen, 2)+	c.Assert(m[0], qt.Equals, docs[0])+	c.Assert(m[1], qt.Equals, docs[1])++	// Search using a document: p1 shares "hugo" with p2.+	m, err = idx.Search(context.Background(), SearchOpts{Document: docs[0]})+	c.Assert(err, qt.IsNil)+	c.Assert(m, qt.HasLen, 1)+	c.Assert(m[0], qt.Equals, docs[1])++	// Stop words are not indexed.+	m, err = idx.Search(context.Background(), SearchOpts{NamedSlices: []types.KeyValues{{Key: "title", Values: []any{"the"}}}})+	c.Assert(err, qt.IsNil)+	c.Assert(m, qt.HasLen, 0)+}++func TestDecodeConfigSplitWords(t *testing.T) {+	c := qt.New(t)++	m, err := config.FromConfigString(`+[related]+  [[related.indices]]+    name = 'title'+    weight = 100+    splitWords = true+    stopWords = ['the', 'a']+`, "toml")+	c.Assert(err, qt.IsNil)+	conf, err := DecodeConfig(m.GetParams("related"))+	c.Assert(err, qt.IsNil)+	c.Assert(conf.Indices[0].SplitWords, qt.IsTrue)+	c.Assert(conf.Indices[0].StopWords, qt.DeepEquals, []string{"the", "a"})++	m, err = config.FromConfigString(`+[related]+  [[related.indices]]+    name = 'refs'+    type = 'fragments'+    weight = 100+    splitWords = true+`, "toml")+	c.Assert(err, qt.IsNil)+	_, err = DecodeConfig(m.GetParams("related"))+	c.Assert(err, qt.ErrorMatches, `splitWords is not supported for index "refs" of type "fragments"`)+}+ func TestDecodeConfig(t *testing.T) { 	c := qt.New(t) related/related_integration_test.goindex e1387d3c5..e6a0264ec 100644@@ -134,6 +134,57 @@ Related 2: 2 	) } +func TestRelatedSplitWords(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+  includeNewer = true+  threshold = 20+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  splitWords = true+  stopWords = ['the', 'a', 'of']+-- content/p1.md --+---+title: "The Hugo Static Site Generator"+date: 2024-01-03+---+-- content/p2.md --+---+title: "Getting started with Hugo"+date: 2024-01-02+---+-- content/p3.md --+---+title: "The Art of a Good Coffee"+date: 2024-01-01+---+-- content/p4.md --+---+title: "Static Site Hosting"+date: 2024-01-01+---+-- layouts/single.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}|{{ end }}$+`++	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: Static Site Hosting|Getting started with Hugo|$")+	b.AssertFileContent("public/p2/index.html", "Related: The Hugo Static Site Generator|$")+	b.AssertFileContent("public/p3/index.html", "Related: $")++	// Without splitWords, the whole title must match.+	b = hugolib.Test(t, strings.Replace(files, "splitWords = true", "splitWords = false", 1))+	b.AssertFileContent("public/p1/index.html", "Related: $")+}+ func BenchmarkRelatedSite(b *testing.B) { 	var files strings.Builder 	files.WriteString(` 
Merged fix · PR #15238+442 −20
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( 	"context" 	"errors" 	"fmt"+	"html" 	"maps" 	"math" 	"sort" 	"strings" 	"sync" 	"time"+	"unicode"+	"unicode/utf8"  	"github.com/gohugoio/hugo/common/collections" 	"github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { 	// used in more than 50% of the documents in the index. 	CardinalityThreshold int -	// Will lower case all string values in and queries tothis index.+	// Will lower case all string values in and queries to this index. 	// May get better accurate results, but at a slight performance cost. 	ToLower bool++	// Whether to split multi-word values into individual keywords.+	Tokenize bool++	// Minimum token length in Unicode code points when tokenizing.+	MinTokenLength int }  // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error {  			if config.Type == TypeFragments { 				if fp, ok := doc.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers { 						added = true+						if config.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !config.shouldSkipToken(word) {+										kw := config.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											setm[kw] = append(setm[kw], doc)+										}+									}+								}+								continue+							}+						} 						setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) 					} 				}@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume 			} 			if opts.Document != nil { 				if fp, ok := opts.Document.(FragmentProvider); ok {-					for _, fragment := range fp.Fragments(ctx).Identifiers {+					frags := fp.Fragments(ctx)+					seenKW := make(map[Keyword]bool)+					for _, fragment := range frags.Identifiers {+						if cfg.Tokenize {+							if h, found := frags.HeadingsMap[fragment]; found {+								for _, word := range splitWords(h.Title) {+									if !cfg.shouldSkipToken(word) {+										kw := cfg.stringToKeyword(word)+										if !seenKW[kw] {+											seenKW[kw] = true+											keywords = append(keywords, kw)+										}+									}+								}+								continue+							}+						} 						keywords = append(keywords, FragmentKeyword(fragment)) 					} 				}@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { 	return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+	return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { 	var keywords []Keyword  	switch vv := v.(type) { 	case string:-		keywords = append(keywords, cfg.stringToKeyword(vv))+		if cfg.Tokenize {+			for _, word := range splitWords(vv) {+				if !cfg.shouldSkipToken(word) {+					keywords = append(keywords, cfg.stringToKeyword(word))+				}+			}+		} else {+			keywords = append(keywords, cfg.stringToKeyword(vv))+		} 	case []string:-		vvv := make([]Keyword, len(vv))-		for i := range vvv {-			vvv[i] = cfg.stringToKeyword(vv[i])+		if !cfg.Tokenize {+			kw := make([]Keyword, len(vv))+			for i, s := range vv {+				kw[i] = cfg.stringToKeyword(s)+			}+			keywords = append(keywords, kw...)+		} else {+			for _, s := range vv {+				for _, word := range splitWords(s) {+					if !cfg.shouldSkipToken(word) {+						keywords = append(keywords, cfg.stringToKeyword(word))+					}+				}+			} 		}-		keywords = append(keywords, vvv...) 	case []any: 		return cfg.ToKeywords(cast.ToStringSlice(vv)) 	case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}()  	applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()-	var fragmentsFilter collections.SortedStringSlice+	// Bundled into one struct so the closure below causes only one heap escape.+	var ff struct {+		ids     collections.SortedStringSlice+		words   collections.SortedStringSlice+		toLower bool+	}  	for _, el := range query { 		setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 			return []Document{}, fmt.Errorf("index config for %q not found", el.Index) 		} +		if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+			ff.toLower = config.ToLower+		}+ 		for _, kw := range el.Keywords { 			if docs, found := setm[kw]; found { 				for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa  					if config.Type == TypeFragments && config.ApplyFilter { 						if fkw, ok := kw.(FragmentKeyword); ok {-							fragmentsFilter = append(fragmentsFilter, string(fkw))+							if config.Tokenize {+								ff.words = append(ff.words, string(fkw))+							} else {+								ff.ids = append(ff.ids, string(fkw))+							} 						} 					}- 					r, found := matchm[doc] 					if !found { 						r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa 	}  	sort.Stable(matches)-	sort.Strings(fragmentsFilter)+	sort.Strings(ff.ids)+	sort.Strings(ff.words)  	result := make([]Document, len(matches))  	for i, m := range matches { 		result[i] = m.Doc -		if len(fragmentsFilter) > 0 {+		if len(ff.ids) > 0 || len(ff.words) > 0 { 			if dp, ok := result[i].(FragmentProvider); ok { 				result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {-					return fragmentsFilter.Contains(h.ID)+					if ff.ids.Contains(h.ID) {+						return true+					}+					if len(ff.words) > 0 {+						for _, word := range splitWords(h.Title) {+							if ff.toLower {+								word = strings.ToLower(word)+							}+							if ff.words.Contains(word) {+								return true+							}+						}+					}+					return false 				}) 			} 		}@@ -618,6 +707,31 @@ type Keyword interface { 	String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+	// Fast path: no whitespace or HTML entities means at most one token.+	if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+		title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if title == "" {+			return nil+		}+		return []string{title}+	}+	raw := strings.Fields(html.UnescapeString(title))+	words := raw[:0]+	for _, w := range raw {+		w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+		if w != "" {+			words = append(words, w)+		}+	}+	if len(words) == 0 {+		return nil+	}+	return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { 	kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { 	}) } +func TestSplitWords(t *testing.T) {+	c := qt.New(t)++	tests := []struct {+		in   string+		want []string+	}{+		{"", nil},+		{"   ", nil},+		{"hello world", []string{"hello", "world"}},+		// leading/trailing punctuation stripped+		{"...hello...", []string{"hello"}},+		{",hello, world!", []string{"hello", "world"}},+		// digits preserved+		{"go1.24", []string{"go1.24"}},+		// HTML entities decoded+		{"foo &amp;bar", []string{"foo", "bar"}},+		// fast path: single token, no whitespace+		{"&amp;foo&amp;", []string{"foo"}},+		// &nbsp; and similar entities that decode to whitespace must not use the fast path+		{"Apples&nbsp;and&nbsp;oranges!", []string{"Apples", "and", "oranges"}},+		// punctuation-only tokens dropped+		{"hello --- world", []string{"hello", "world"}},+		// mixed spaces+		{"  foo   bar  ", []string{"foo", "bar"}},+	}++	for _, tt := range tests {+		c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+	}+}+ func BenchmarkRelatedNewIndex(b *testing.B) { 	pages := make([]*testDoc, 100) 	numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   weight = 90 [[related.indices]]   name = 'keywords'-  weight = 80	+  weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} -	+ `  	b := hugolib.Test(t, files)  	expect := `-P1 Fragments: [p1-title]	+P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 		b.AssertFileContent("public/"+p+"/index.html", expect) 	} -	b.AssertFileContent("public/p1/index.html",+	b.AssertFileContent(+		"public/p1/index.html", 		"Related 1:  0: p2: h0: First title|ref1|::END", 		"Related 2:  0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", 	) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2")+	b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+	b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  applyFilter = true+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+	b := hugolib.Test(t, files)++	b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+	b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = false+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// toLower=false: "Hugo" != "hugo", so no match.+	b.AssertFileContent("public/p1/index.html", "Related:  ")+	b.AssertFileContent("public/p2/index.html", "Related:  ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  name = 'title'+  weight = 100+  tokenize = true+  minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+	b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+	b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+	// All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+	b.AssertFileContent("public/p3/index.html", "Related:  ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+	t.Parallel()++	files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+  includeNewer = true+  threshold = 80+  toLower = true+[[related.indices]]+  type = 'fragments'+  weight = 100+  tokenize = true+[[related.indices]]+  name = 'keywords'+  weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+	b := hugolib.Test(t, files)++	// p1 and p3 share the same unique fragment tokens; p1 also matches on+	// keywords ("fruit"). Without deduplication, p3's repeated headings+	// inflate its weight above p1's combined fragment+keywords weight.+	b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { 	var files strings.Builder 	files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]   toLower = false [[related.indices]]   name = 'keywords'-  weight = 70	+  weight = 70 [[related.indices]]   name = 'pagerefs'   type = 'fragments'-  weight = 30	+  weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)