Backtestshugorun-20261013-011256#7515
Related content: Partial-text indexing
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Patch B is the better fix. **Correctness** - B adds an opt-in `splitWords` option to the basic index type, which is exactly what the issue asks for (partial matching on e.g. `title`). - Splitting uses `unicode.IsLetter`, `IsNumber` and `IsMark`, so non-ASCII text such as accented words and CJK is handled correctly. - Keywords are deduplicated per document, so repeated words cannot inflate rank. - It adds a `stopWords` list, which directly answers the issue's question about ignoring common words. The list can be configured per language, which addresses the i18n concern. - `DecodeConfig` rejects `splitWords` on non-basic index types instead of misbehaving silently. Patch A has a broader scope (tokenized fragments plus an `ApplyFilter` rewrite) with more risk: - Tokenized fragment keywords are built through `stringToKeyword`, which returns a `StringKeyword`, not a `FragmentKeyword`. - In `searchDate`, the `kw.(FragmentKeyword)` type assertion therefore never succeeds for tokenized words. `ff.words` stays empty, and the new heading filter appears to be dead code. Its `ApplyFilter` test looks likely to fail or to pass only by accident. - On the basic path, A's `ToKeywords` does not deduplicate tokens, so repeated words over-weight a document. - A offers only `MinTokenLength` as a crude stand-in for stop words. - Whitespace-based splitting with punctuation trimming leaves hyphenated or joined words unsplit. **Tests** - B has focused unit tests (keyword splitting, stop words, non-ASCII input, search, config validation) and an integration test that checks a split index alongside an unsplit one. - A has more integration tests, but they cover the riskier fragments path, including the questionable `ApplyFilter` behavior. **Scope and craft** - A includes unrelated whitespace churn in existing tests. - B updates the docs and keeps the change small and coherent by routing through a single `stringsToKeywords` helper.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
1h 18m
Wall-clock time in the container
Full test suite
No new failures
853 passed · 25 skipped · 882 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
273vs 462 merged
Changed lines, ours vs merged
The issue
View on GitHubWhen specifying the indices for related content, it would be useful to specify that some of them were split by word, so that their text could be matched partially. This would allow the title to be used for more in-depth indexing, rather than requiring the user to specify tags / keywords.
This would likely increase memory usage and decrease performance, so should definitely be opt-in!
I suspect common words (a, an, the etc) might want to be ignored? Although doing that in a sane, i18n-friendly manner might be a challenge!
The two patches
docs/content/en/configuration/related-content.mdindex 92cf92e8a..ef2cfa3a2 100644@@ -45,6 +45,14 @@ This is the default configuration: `pattern` : (`string`) This is currently only relevant for dates. When listing related content, we may want to list content that is also close in time. Setting "2006" (default value for date indexes) as the pattern for a date index will add weight to pages published in the same year. For busier blogs, "200601" (year and month) may be a better default. +`splitWords`+: {{< new-in 0.166.0 />}}+: (`bool`) Whether to split string values in both the index and the queries into words, allowing partial matches on e.g. the page `title`. Any character that is not a letter, a number, or a combining mark is treated as a word separator. This increases memory usage and the time needed to build the index. Only supported for the `basic` type. Default is `false`.++`stopWords`+: {{< new-in 0.166.0 />}}+: (`[]string`) A list of words to ignore when `splitWords` is enabled, e.g. `['a', 'an', 'the']`. Words are compared case-insensitively. Configure this per language to match the language of your content. Default is `[]`.+ `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. docs/data/docs.yamlindex 19b950dd5..e76499e1b 100644@@ -2191,6 +2191,8 @@ config: cardinalityThreshold: 0 name: keywords pattern: ''+ splitWords: false+ stopWords: null toLower: false type: basic weight: 100@@ -2198,6 +2200,8 @@ config: cardinalityThreshold: 0 name: date pattern: ''+ splitWords: false+ stopWords: null toLower: false type: basic weight: 10@@ -2205,6 +2209,8 @@ config: cardinalityThreshold: 0 name: tags pattern: ''+ splitWords: false+ stopWords: null toLower: false type: basic weight: 80related/inverted_index.goindex c6c65d97f..7571bf5e4 100644@@ -24,6 +24,7 @@ import ( "strings" "sync" "time"+ "unicode" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +126,17 @@ type IndexConfig struct { // Will lower case all string values in and queries tothis index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Will split string values in and queries to this index into words,+ // allowing e.g. titles to be matched partially.+ // This is only supported for the "basic" type.+ // Note that this increases memory usage and the time used to build the index.+ SplitWords bool++ // A list of words to exclude from the index and queries when SplitWords is enabled,+ // e.g. common words like "a", "an" and "the".+ // The words are compared case-insensitively.+ StopWords []string } // Document is the interface an indexable document in Hugo must fulfill.@@ -423,19 +435,61 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +// stringsToKeywords converts the given strings to keywords, splitting+// them into words if configured to do so.+func (cfg IndexConfig) stringsToKeywords(s ...string) []Keyword {+ if !cfg.SplitWords {+ kw := make([]Keyword, len(s))+ for i := range s {+ kw[i] = cfg.stringToKeyword(s[i])+ }+ return kw+ }++ var kw []Keyword+ seen := make(map[Keyword]bool)+ for _, ss := range s {+ for _, word := range splitWords(ss) {+ if cfg.isStopWord(word) {+ continue+ }+ k := cfg.stringToKeyword(word)+ if seen[k] {+ continue+ }+ seen[k] = true+ kw = append(kw, k)+ }+ }+ return kw+}++func (cfg IndexConfig) isStopWord(word string) bool {+ for _, sw := range cfg.StopWords {+ if strings.EqualFold(sw, word) {+ return true+ }+ }+ return false+}++// splitWords splits s into words, treating any rune that is not a letter,+// a number or a combining mark as a separator.+func splitWords(s string) []string {+ return strings.FieldsFunc(s, func(r rune) bool {+ return !unicode.IsLetter(r) && !unicode.IsNumber(r) && !unicode.IsMark(r)+ })+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ keywords = append(keywords, cfg.stringsToKeywords(vv)...) case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])- }- keywords = append(keywords, vvv...)+ keywords = append(keywords, cfg.stringsToKeywords(vv...)...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -594,6 +648,9 @@ func DecodeConfig(m hmaps.Params) (Config, error) { if icfg.CardinalityThreshold < 0 || icfg.CardinalityThreshold > 100 { return Config{}, errors.New("cardinalityThreshold threshold must be between 0 and 100") }+ if icfg.SplitWords && c.Indices[i].Type != TypeBasic {+ return Config{}, fmt.Errorf("splitWords is not supported for index %q of type %q", c.Indices[i].Name, c.Indices[i].Type)+ } } return c, nil@@ -620,11 +677,5 @@ type Keyword interface { // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword {- kw := make([]Keyword, len(s))-- for i := range s {- kw[i] = cfg.stringToKeyword(s[i])- }-- return kw+ return cfg.stringsToKeywords(s...) }related/inverted_index_test.goindex b33d42955..75fd0b239 100644@@ -22,6 +22,7 @@ import ( "time" qt "github.com/frankban/quicktest"+ "github.com/gohugoio/hugo/common/types" "github.com/gohugoio/hugo/config" ) @@ -296,6 +297,129 @@ func TestDecodeConfig(t *testing.T) { c.Assert(first.Name, qt.Equals, "keywords") } +func TestToKeywordsSplitWords(t *testing.T) {+ c := qt.New(t)++ config := IndexConfig{SplitWords: true, ToLower: true, StopWords: []string{"a", "The", "OF"}}++ keywords, err := config.ToKeywords("The Lord of the Rings: The Return of the King")+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{+ StringKeyword("lord"),+ StringKeyword("rings"),+ StringKeyword("return"),+ StringKeyword("king"),+ })++ keywords, err = config.ToKeywords([]any{"Hugo is fast", "Go-templates, a.k.a. Go"})+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{+ StringKeyword("hugo"),+ StringKeyword("is"),+ StringKeyword("fast"),+ StringKeyword("go"),+ StringKeyword("templates"),+ StringKeyword("k"),+ })++ // Non-ASCII text.+ keywords, err = IndexConfig{SplitWords: true}.ToKeywords("Größe und café, 日本語")+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{+ StringKeyword("Größe"),+ StringKeyword("und"),+ StringKeyword("café"),+ StringKeyword("日本語"),+ })++ // Not enabled: the full string is the keyword.+ keywords, err = IndexConfig{StopWords: []string{"the"}}.ToKeywords("The Lord of the Rings")+ c.Assert(err, qt.IsNil)+ c.Assert(keywords, qt.DeepEquals, []Keyword{StringKeyword("The Lord of the Rings")})+}++func TestSearchSplitWords(t *testing.T) {+ c := qt.New(t)+ cfg := Config{+ Threshold: 20,+ Indices: IndicesConfig{+ IndexConfig{Name: "title", Weight: 100, ToLower: true, SplitWords: true, StopWords: []string{"the", "a"}},+ },+ }++ idx := NewInvertedIndex(cfg)+ docs := []Document{+ &testTitleDoc{title: "The Lord of the Rings"},+ &testTitleDoc{title: "A Song of Ice and Fire"},+ &testTitleDoc{title: "The Hobbit"},+ }+ c.Assert(idx.Add(context.Background(), docs...), qt.IsNil)++ m, err := idx.Search(context.Background(), SearchOpts{+ NamedSlices: []types.KeyValues{types.NewKeyValuesStrings("title", "Rings of Power")},+ })+ c.Assert(err, qt.IsNil)+ c.Assert(m, qt.HasLen, 2)+ c.Assert(m[0], qt.Equals, docs[0])+ c.Assert(m[1], qt.Equals, docs[1])++ // Stop words never match.+ m, err = idx.Search(context.Background(), SearchOpts{+ NamedSlices: []types.KeyValues{types.NewKeyValuesStrings("title", "The")},+ })+ c.Assert(err, qt.IsNil)+ c.Assert(m, qt.HasLen, 0)+}++// testTitleDoc is a Document that converts its title to keywords using the index config.+type testTitleDoc struct {+ title string+}++func (d *testTitleDoc) RelatedKeywords(cfg IndexConfig) ([]Keyword, error) {+ return cfg.ToKeywords(d.title)+}++func (d *testTitleDoc) PublishDate() time.Time {+ return time.Time{}+}++func (d *testTitleDoc) Name() string {+ return d.title+}++func TestDecodeConfigSplitWords(t *testing.T) {+ c := qt.New(t)++ m, err := config.FromConfigString(`+[related]+ threshold = 80+ [[related.indices]]+ name = 'title'+ splitWords = true+ stopWords = ['a', 'the']+ weight = 100+`, "toml")+ c.Assert(err, qt.IsNil)+ conf, err := DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.IsNil)+ c.Assert(conf.Indices[0].SplitWords, qt.IsTrue)+ c.Assert(conf.Indices[0].StopWords, qt.DeepEquals, []string{"a", "the"})++ m, err = config.FromConfigString(`+[related]+ threshold = 80+ [[related.indices]]+ name = 'refs'+ type = 'fragments'+ splitWords = true+ weight = 100+`, "toml")+ c.Assert(err, qt.IsNil)+ _, err = DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.ErrorMatches, `splitWords is not supported for index "refs" of type "fragments"`)+}+ func TestToKeywordsAnySlice(t *testing.T) { c := qt.New(t) var config IndexConfigrelated/related_integration_test.goindex e1387d3c5..e2f25b31c 100644@@ -134,6 +134,64 @@ Related 2: 2 ) } +func TestRelatedSplitWords(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 20+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ splitWords = true+ stopWords = ['a', 'the', 'of', 'and']+[[related.indices]]+ name = 'description'+ weight = 50+-- content/p1.md --+---+title: "The Lord of the Rings"+description: "The Lord of the Rings"+---+-- content/p2.md --+---+title: "Rings of Power"+description: "Rings of Power"+---+-- content/p3.md --+---+title: "A Song of Ice and Fire"+description: "A Song of Ice and Fire"+---+-- content/p4.md --+---+title: "Fire and Blood"+description: "Fire and Blood"+---+-- layouts/single.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}|{{ end }}$+Query: {{ range site.RegularPages.Related (dict "namedSlices" (slice (keyVals "title" "Ice Age"))) }}{{ .Title }}|{{ end }}$+Description: {{ range site.RegularPages.Related (dict "namedSlices" (slice (keyVals "description" "Fire"))) }}{{ .Title }}|{{ end }}$+`++ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Rings of Power|$")+ b.AssertFileContent("public/p2/index.html", "Related: The Lord of the Rings|$")+ b.AssertFileContent("public/p3/index.html",+ "Related: Fire and Blood|$",+ "Query: A Song of Ice and Fire|$",+ // The description index is not split, so only full matches.+ "Description: $",+ )+ b.AssertFileContent("public/p4/index.html", "Related: A Song of Ice and Fire|$")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode"+ "unicode/utf8" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { // used in more than 50% of the documents in the index. CardinalityThreshold int - // Will lower case all string values in and queries tothis index.+ // Will lower case all string values in and queries to this index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords.+ Tokenize bool++ // Minimum token length in Unicode code points when tokenizing.+ MinTokenLength int } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers { added = true+ if config.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !config.shouldSkipToken(word) {+ kw := config.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ setm[kw] = append(setm[kw], doc)+ }+ }+ }+ continue+ }+ } setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) } }@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers {+ if cfg.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !cfg.shouldSkipToken(word) {+ kw := cfg.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ keywords = append(keywords, kw)+ }+ }+ }+ continue+ }+ } keywords = append(keywords, FragmentKeyword(fragment)) } }@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+ return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ if cfg.Tokenize {+ for _, word := range splitWords(vv) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } else {+ keywords = append(keywords, cfg.stringToKeyword(vv))+ } case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])+ if !cfg.Tokenize {+ kw := make([]Keyword, len(vv))+ for i, s := range vv {+ kw[i] = cfg.stringToKeyword(s)+ }+ keywords = append(keywords, kw...)+ } else {+ for _, s := range vv {+ for _, word := range splitWords(s) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } }- keywords = append(keywords, vvv...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ // Bundled into one struct so the closure below causes only one heap escape.+ var ff struct {+ ids collections.SortedStringSlice+ words collections.SortedStringSlice+ toLower bool+ } for _, el := range query { setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return []Document{}, fmt.Errorf("index config for %q not found", el.Index) } + if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+ ff.toLower = config.ToLower+ }+ for _, kw := range el.Keywords { if docs, found := setm[kw]; found { for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok {- fragmentsFilter = append(fragmentsFilter, string(fkw))+ if config.Tokenize {+ ff.words = append(ff.words, string(fkw))+ } else {+ ff.ids = append(ff.ids, string(fkw))+ } } }- r, found := matchm[doc] if !found { r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } sort.Stable(matches)- sort.Strings(fragmentsFilter)+ sort.Strings(ff.ids)+ sort.Strings(ff.words) result := make([]Document, len(matches)) for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(ff.ids) > 0 || len(ff.words) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if ff.ids.Contains(h.ID) {+ return true+ }+ if len(ff.words) > 0 {+ for _, word := range splitWords(h.Title) {+ if ff.toLower {+ word = strings.ToLower(word)+ }+ if ff.words.Contains(word) {+ return true+ }+ }+ }+ return false }) } }@@ -618,6 +707,31 @@ type Keyword interface { String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+ // Fast path: no whitespace or HTML entities means at most one token.+ if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+ title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if title == "" {+ return nil+ }+ return []string{title}+ }+ raw := strings.Fields(html.UnescapeString(title))+ words := raw[:0]+ for _, w := range raw {+ w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if w != "" {+ words = append(words, w)+ }+ }+ if len(words) == 0 {+ return nil+ }+ return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { }) } +func TestSplitWords(t *testing.T) {+ c := qt.New(t)++ tests := []struct {+ in string+ want []string+ }{+ {"", nil},+ {" ", nil},+ {"hello world", []string{"hello", "world"}},+ // leading/trailing punctuation stripped+ {"...hello...", []string{"hello"}},+ {",hello, world!", []string{"hello", "world"}},+ // digits preserved+ {"go1.24", []string{"go1.24"}},+ // HTML entities decoded+ {"foo &bar", []string{"foo", "bar"}},+ // fast path: single token, no whitespace+ {"&foo&", []string{"foo"}},+ // and similar entities that decode to whitespace must not use the fast path+ {"Apples and oranges!", []string{"Apples", "and", "oranges"}},+ // punctuation-only tokens dropped+ {"hello --- world", []string{"hello", "world"}},+ // mixed spaces+ {" foo bar ", []string{"foo", "bar"}},+ }++ for _, tt := range tests {+ c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+ }+}+ func BenchmarkRelatedNewIndex(b *testing.B) { pages := make([]*testDoc, 100) numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] weight = 90 [[related.indices]] name = 'keywords'- weight = 80 + weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} - + ` b := hugolib.Test(t, files) expect := `-P1 Fragments: [p1-title] +P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 b.AssertFileContent("public/"+p+"/index.html", expect) } - b.AssertFileContent("public/p1/index.html",+ b.AssertFileContent(+ "public/p1/index.html", "Related 1: 0: p2: h0: First title|ref1|::END", "Related 2: 0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", ) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+ b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ applyFilter = true+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+ b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = false+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // toLower=false: "Hugo" != "hugo", so no match.+ b.AssertFileContent("public/p1/index.html", "Related: ")+ b.AssertFileContent("public/p2/index.html", "Related: ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+ minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+ b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+ b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+ // All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+[[related.indices]]+ name = 'keywords'+ weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p3 share the same unique fragment tokens; p1 also matches on+ // keywords ("fruit"). Without deduplication, p3's repeated headings+ // inflate its weight above p1's combined fragment+keywords weight.+ b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] toLower = false [[related.indices]] name = 'keywords'- weight = 70 + weight = 70 [[related.indices]] name = 'pagerefs' type = 'fragments'- weight = 30 + weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)