Backtestshugorun-20261024-162305#7515
Related content: Partial-text indexing
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches add an opt-in word-splitting flag on string indices and both pass all independent tests, so the decision rests on the finer points of the title-index path the issue actually asks about. Patch B routes everything through StringsToKeywords/wordsToKeywords, which deduplicates tokens per value and per query. Patch A deduplicates only on its fragments path; in ToKeywords (the path the title index uses) a title like "Rings Rings Rings" appends the document to the same posting list three times and a repeated query word is likewise counted multiple times, inflating rank for the exact scenario the issue describes. B also answers the issue's second ask more directly: a user-supplied, case-insensitive stopWords list is language-neutral and matches the issue's "a, an, the" example, whereas A's minTokenLength is a coarser proxy that cannot drop long common words or keep short meaningful ones. B validates the config (rejects splitWords on fragments with a clear error, with a decode test), documents the two new settings including the apostrophe caveat, and adds a real inverted-index unit test plus DecodeConfig and integration tests. A ships no docs, touches unrelated whitespace in existing tests, and spreads near-duplicate tokenizing logic across Add, Search, ToKeywords and the ApplyFilter closure, which raises risk without being required by the issue. A's splitWords does handle HTML entities and keeps internal punctuation (useful for headings), and its fragments support is a genuine extra, but that is beyond the ask and does not offset the missing dedup on the primary path. B's splitter has its own tradeoff (apostrophes and hyphens become separators), but it is simpler, Unicode-aware via IsLetter/IsNumber/IsMark, and the behaviour is documented. Overall B is the more correct and better-scoped fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
37m 19s
Wall-clock time in the container
Full test suite
No new failures
902 passed · 1 skipped · 908 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
239vs 462 merged
Changed lines, ours vs merged
The issue
View on GitHubWhen specifying the indices for related content, it would be useful to specify that some of them were split by word, so that their text could be matched partially. This would allow the title to be used for more in-depth indexing, rather than requiring the user to specify tags / keywords.
This would likely increase memory usage and decrease performance, so should definitely be opt-in!
I suspect common words (a, an, the etc) might want to be ignored? Although doing that in a sane, i18n-friendly manner might be a challenge!
The two patches
docs/content/en/configuration/related-content.mdindex 92cf92e8a..731c58672 100644@@ -45,6 +45,12 @@ This is the default configuration: `pattern` : (`string`) This is currently only relevant for dates. When listing related content, we may want to list content that is also close in time. Setting "2006" (default value for date indexes) as the pattern for a date index will add weight to pages published in the same year. For busier blogs, "200601" (year and month) may be a better default. +`splitWords`+: (`bool`) Whether to split string values into words, in both the indexes and the queries, so that text such as the `title` can be partially matched by the words it contains. Any character that is not a letter, a number or a mark is treated as a word separator, so this will not split text written without spaces between words (e.g. Chinese or Japanese). This increases memory usage and indexing time. Not supported for the `fragments` type. Default is `false`.++`stopWords`+: (`[]string`) A list of words to ignore when `splitWords` is enabled, e.g. `['a', 'an', 'the']` for English. Words are matched case-insensitively. Note that apostrophes are word separators, so e.g. `fox's` gives the words `fox` and `s`; you may want to add such fragments (e.g. `s`, `t`) to this list. See also `cardinalityThreshold` to automatically remove words used in many documents.+ `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. related/inverted_index.goindex c6c65d97f..bd427396f 100644@@ -24,6 +24,7 @@ import ( "strings" "sync" "time"+ "unicode" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +126,16 @@ type IndexConfig struct { // Will lower case all string values in and queries tothis index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Enable to split string values into words, so e.g. a title can be+ // partially matched by the words it contains.+ // This increases memory usage and indexing time, so it's opt-in.+ // Not supported for the "fragments" type.+ SplitWords bool++ // Words to ignore (case insensitive) when SplitWords is enabled,+ // e.g. "a", "an" and "the" in English.+ StopWords []string } // Document is the interface an indexable document in Hugo must fulfill.@@ -429,13 +440,9 @@ func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ keywords = append(keywords, cfg.StringsToKeywords(vv)...) case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])- }- keywords = append(keywords, vvv...)+ keywords = append(keywords, cfg.StringsToKeywords(vv...)...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -591,6 +598,9 @@ func DecodeConfig(m hmaps.Params) (Config, error) { if !validTypes[c.Indices[i].Type] { return c, fmt.Errorf("invalid index type %q. Must be one of %v", c.Indices[i].Type, maps.Keys(validTypes)) }+ if icfg.SplitWords && c.Indices[i].Type == TypeFragments {+ return Config{}, fmt.Errorf("splitWords is not supported for index %q of type %q", c.Indices[i].Name, TypeFragments)+ } if icfg.CardinalityThreshold < 0 || icfg.CardinalityThreshold > 100 { return Config{}, errors.New("cardinalityThreshold threshold must be between 0 and 100") }@@ -619,7 +629,13 @@ type Keyword interface { } // StringsToKeywords converts the given slice of strings to a slice of Keyword.+// If SplitWords is enabled, each string is split into words, stop words are+// removed and duplicates are dropped. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword {+ if cfg.SplitWords {+ return cfg.wordsToKeywords(s...)+ }+ kw := make([]Keyword, len(s)) for i := range s {@@ -628,3 +644,39 @@ func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { return kw }++func (cfg IndexConfig) wordsToKeywords(s ...string) []Keyword {+ var kw []Keyword+ seen := make(map[Keyword]bool)+ for _, ss := range s {+ for _, word := range splitWords(ss) {+ if cfg.isStopWord(word) {+ continue+ }+ k := cfg.stringToKeyword(word)+ if seen[k] {+ continue+ }+ seen[k] = true+ kw = append(kw, k)+ }+ }+ return kw+}++func (cfg IndexConfig) isStopWord(word string) bool {+ for _, sw := range cfg.StopWords {+ if strings.EqualFold(sw, word) {+ return true+ }+ }+ return false+}++// splitWords splits s into words, using any rune that's not a letter,+// a number or a mark (e.g. a diacritic) as a separator.+func splitWords(s string) []string {+ return strings.FieldsFunc(s, func(r rune) bool {+ return !unicode.IsLetter(r) && !unicode.IsNumber(r) && !unicode.IsMark(r)+ })+}related/inverted_index_test.goindex b33d42955..7459ab386 100644@@ -252,6 +252,127 @@ func TestToKeywordsToLower(t *testing.T) { }) } +func TestToKeywordsSplitWords(t *testing.T) {+ c := qt.New(t)++ config := IndexConfig{SplitWords: true, ToLower: true, StopWords: []string{"THE", "a"}}+ keywords, err := config.ToKeywords("The Quick, brown fox: a fox's café-story 2024!")+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{+ StringKeyword("quick"),+ StringKeyword("brown"),+ StringKeyword("fox"),+ StringKeyword("s"),+ StringKeyword("café"),+ StringKeyword("story"),+ StringKeyword("2024"),+ })++ keywords, err = config.ToKeywords([]any{"Hugo Rocks", "hugo docs", 32})+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{+ StringKeyword("hugo"),+ StringKeyword("rocks"),+ StringKeyword("docs"),+ StringKeyword("32"),+ })++ // Without ToLower, case is preserved but stop words are still matched case insensitively.+ config = IndexConfig{SplitWords: true, StopWords: []string{"the"}}+ c.Assert(config.StringsToKeywords("The Hugo the hugo"), qt.DeepEquals, []Keyword{+ StringKeyword("Hugo"),+ StringKeyword("hugo"),+ })++ // Default is to keep the string as is.+ config = IndexConfig{}+ c.Assert(config.StringsToKeywords("The Hugo"), qt.DeepEquals, []Keyword{+ StringKeyword("The Hugo"),+ })+}++// rawDoc converts its raw values to keywords using the index config, as Hugo's pages do.+type rawDoc struct {+ name string+ values map[string]any+}++func (d *rawDoc) RelatedKeywords(cfg IndexConfig) ([]Keyword, error) {+ return cfg.ToKeywords(d.values[cfg.Name])+}++func (d *rawDoc) PublishDate() time.Time {+ return time.Time{}+}++func (d *rawDoc) Name() string {+ return d.name+}++func TestSearchSplitWords(t *testing.T) {+ c := qt.New(t)+ cfg := Config{+ Threshold: 20,+ Indices: IndicesConfig{+ IndexConfig{Name: "title", Weight: 100, SplitWords: true, ToLower: true, StopWords: []string{"the", "of"}},+ },+ }+ icfg := cfg.Indices[0]++ idx := NewInvertedIndex(cfg)+ docs := []*rawDoc{+ {name: "lotr", values: map[string]any{"title": "The Lord of the Rings"}},+ {name: "hobbit", values: map[string]any{"title": "The Hobbit"}},+ {name: "rop", values: map[string]any{"title": "Rings of Power"}},+ }+ for _, d := range docs {+ c.Assert(idx.Add(context.Background(), d), qt.IsNil)+ }++ m, err := idx.search(context.Background(), newQueryElement("title", icfg.StringsToKeywords("The rings")...))+ c.Assert(err, qt.IsNil)+ c.Assert(m, qt.HasLen, 2)+ c.Assert(m, qt.Contains, Document(docs[0]))+ c.Assert(m, qt.Contains, Document(docs[2]))++ // Stop words alone match nothing.+ m, err = idx.search(context.Background(), newQueryElement("title", icfg.StringsToKeywords("The Of")...))+ c.Assert(err, qt.IsNil)+ c.Assert(m, qt.HasLen, 0)+}++func TestDecodeConfigSplitWords(t *testing.T) {+ c := qt.New(t)++ m, err := config.FromConfigString(`+[related]+ threshold = 80+ [[related.indices]]+ name = 'title'+ weight = 100+ splitWords = true+ stopWords = ['a', 'the']+`, "toml")+ c.Assert(err, qt.IsNil)+ conf, err := DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.IsNil)+ c.Assert(conf.Indices[0].SplitWords, qt.IsTrue)+ c.Assert(conf.Indices[0].StopWords, qt.DeepEquals, []string{"a", "the"})++ m, err = config.FromConfigString(`+[related]+ threshold = 80+ [[related.indices]]+ name = 'pagerefs'+ type = 'fragments'+ weight = 100+ splitWords = true+`, "toml")+ c.Assert(err, qt.IsNil)+ _, err = DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.ErrorMatches, `splitWords is not supported for index "pagerefs" of type "fragments"`)+}+ func TestDecodeConfig(t *testing.T) { c := qt.New(t) related/related_integration_test.goindex e1387d3c5..82a6e3f26 100644@@ -134,6 +134,54 @@ Related 2: 2 ) } +func TestRelatedSplitWords(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 20+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ splitWords = true+ stopWords = ['the', 'of', 'a']+-- content/p1.md --+---+title: "The Lord of the Rings"+---+-- content/p2.md --+---+title: "Rings of Power"+---+-- content/p3.md --+---+title: "The Hobbit"+---+-- content/p4.md --+---+title: "A Tale of Two Cities"+---+-- layouts/single.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}|{{ end }}$+-- layouts/home.html --+Hobbit: {{ range site.RegularPages.Related (dict "namedSlices" (slice (keyVals "title" "the hobbit returns"))) }}{{ .Title }}|{{ end }}$+Stop: {{ len (site.RegularPages.Related (dict "namedSlices" (slice (keyVals "title" "The Of A")))) }}$+`++ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Rings of Power|$")+ b.AssertFileContent("public/p2/index.html", "Related: The Lord of the Rings|$")+ b.AssertFileContent("public/p3/index.html", "Related: $")+ b.AssertFileContent("public/p4/index.html", "Related: $")+ b.AssertFileContent("public/index.html", "Hobbit: The Hobbit|$", "Stop: 0$")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode"+ "unicode/utf8" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { // used in more than 50% of the documents in the index. CardinalityThreshold int - // Will lower case all string values in and queries tothis index.+ // Will lower case all string values in and queries to this index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords.+ Tokenize bool++ // Minimum token length in Unicode code points when tokenizing.+ MinTokenLength int } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers { added = true+ if config.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !config.shouldSkipToken(word) {+ kw := config.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ setm[kw] = append(setm[kw], doc)+ }+ }+ }+ continue+ }+ } setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) } }@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers {+ if cfg.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !cfg.shouldSkipToken(word) {+ kw := cfg.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ keywords = append(keywords, kw)+ }+ }+ }+ continue+ }+ } keywords = append(keywords, FragmentKeyword(fragment)) } }@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+ return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ if cfg.Tokenize {+ for _, word := range splitWords(vv) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } else {+ keywords = append(keywords, cfg.stringToKeyword(vv))+ } case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])+ if !cfg.Tokenize {+ kw := make([]Keyword, len(vv))+ for i, s := range vv {+ kw[i] = cfg.stringToKeyword(s)+ }+ keywords = append(keywords, kw...)+ } else {+ for _, s := range vv {+ for _, word := range splitWords(s) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } }- keywords = append(keywords, vvv...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ // Bundled into one struct so the closure below causes only one heap escape.+ var ff struct {+ ids collections.SortedStringSlice+ words collections.SortedStringSlice+ toLower bool+ } for _, el := range query { setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return []Document{}, fmt.Errorf("index config for %q not found", el.Index) } + if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+ ff.toLower = config.ToLower+ }+ for _, kw := range el.Keywords { if docs, found := setm[kw]; found { for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok {- fragmentsFilter = append(fragmentsFilter, string(fkw))+ if config.Tokenize {+ ff.words = append(ff.words, string(fkw))+ } else {+ ff.ids = append(ff.ids, string(fkw))+ } } }- r, found := matchm[doc] if !found { r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } sort.Stable(matches)- sort.Strings(fragmentsFilter)+ sort.Strings(ff.ids)+ sort.Strings(ff.words) result := make([]Document, len(matches)) for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(ff.ids) > 0 || len(ff.words) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if ff.ids.Contains(h.ID) {+ return true+ }+ if len(ff.words) > 0 {+ for _, word := range splitWords(h.Title) {+ if ff.toLower {+ word = strings.ToLower(word)+ }+ if ff.words.Contains(word) {+ return true+ }+ }+ }+ return false }) } }@@ -618,6 +707,31 @@ type Keyword interface { String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+ // Fast path: no whitespace or HTML entities means at most one token.+ if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+ title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if title == "" {+ return nil+ }+ return []string{title}+ }+ raw := strings.Fields(html.UnescapeString(title))+ words := raw[:0]+ for _, w := range raw {+ w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if w != "" {+ words = append(words, w)+ }+ }+ if len(words) == 0 {+ return nil+ }+ return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { }) } +func TestSplitWords(t *testing.T) {+ c := qt.New(t)++ tests := []struct {+ in string+ want []string+ }{+ {"", nil},+ {" ", nil},+ {"hello world", []string{"hello", "world"}},+ // leading/trailing punctuation stripped+ {"...hello...", []string{"hello"}},+ {",hello, world!", []string{"hello", "world"}},+ // digits preserved+ {"go1.24", []string{"go1.24"}},+ // HTML entities decoded+ {"foo &bar", []string{"foo", "bar"}},+ // fast path: single token, no whitespace+ {"&foo&", []string{"foo"}},+ // and similar entities that decode to whitespace must not use the fast path+ {"Apples and oranges!", []string{"Apples", "and", "oranges"}},+ // punctuation-only tokens dropped+ {"hello --- world", []string{"hello", "world"}},+ // mixed spaces+ {" foo bar ", []string{"foo", "bar"}},+ }++ for _, tt := range tests {+ c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+ }+}+ func BenchmarkRelatedNewIndex(b *testing.B) { pages := make([]*testDoc, 100) numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] weight = 90 [[related.indices]] name = 'keywords'- weight = 80 + weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} - + ` b := hugolib.Test(t, files) expect := `-P1 Fragments: [p1-title] +P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 b.AssertFileContent("public/"+p+"/index.html", expect) } - b.AssertFileContent("public/p1/index.html",+ b.AssertFileContent(+ "public/p1/index.html", "Related 1: 0: p2: h0: First title|ref1|::END", "Related 2: 0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", ) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+ b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ applyFilter = true+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+ b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = false+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // toLower=false: "Hugo" != "hugo", so no match.+ b.AssertFileContent("public/p1/index.html", "Related: ")+ b.AssertFileContent("public/p2/index.html", "Related: ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+ minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+ b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+ b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+ // All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+[[related.indices]]+ name = 'keywords'+ weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p3 share the same unique fragment tokens; p1 also matches on+ // keywords ("fruit"). Without deduplication, p3's repeated headings+ // inflate its weight above p1's combined fragment+keywords weight.+ b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] toLower = false [[related.indices]] name = 'keywords'- weight = 70 + weight = 70 [[related.indices]] name = 'pagerefs' type = 'fragments'- weight = 30 + weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)