Backtestshugorun-20261009-153657#15199
related: Add option to tokenize headings
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Patch A is the more correct fix. **Patch A** - It adds `Tokenize` as a purely additive feature for `fragments` indices. - Heading identifiers are still indexed as `FragmentKeyword`, so exact front-matter references keep working. Its test checks this with p4 referencing p1's heading ID. - Heading word tokens go in under a distinct `HeadingTokenKeyword` type. - Titles have HTML stripped (`tpl.StripHTML`), are unescaped, honor `ToLower`, and are split on non-letter/digit runes. - `applyFilter` handles the new keyword type and re-tokenizes heading titles with the matching `ToLower` setting. It even covers multiple indices whose `ToLower` settings differ. - It updates the docs exactly as the issue proposes. - Tests cover unit tokenization, config decoding, the enabled/disabled integration paths, multiple indices with mixed `ToLower`, and `cardinalityThreshold` pruning. **Patch B has real correctness problems** 1. When `Tokenize` is on, heading identifiers are replaced by tokens and never indexed as `FragmentKeyword`. Front-matter fragment references to heading IDs therefore stop matching. 2. Tokens are created through `stringToKeyword`, which yields `StringKeyword`. The `applyFilter` branch only checks for `kw.(FragmentKeyword)`, so `ff.words` is never filled. Heading filtering on token matches cannot work, and its own `ApplyFilter` test looks like it would fail. 3. `splitWords` does not strip HTML tags, so headings with markup (e.g. `<em>`) produce polluted tokens. 4. It widens scope with tokenization of basic-index `ToKeywords` and an unrequested `MinTokenLength` option. These change semantics beyond the issue and add risk. 5. It also includes whitespace churn in existing tests. Patch A is correct, tightly scoped, documented and well tested; Patch B breaks existing behavior and has a broken filter path.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
32m 41s
Wall-clock time in the container
Full test suite
No new failures
860 passed · 25 skipped · 885 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
324vs 462 merged
Changed lines, ours vs merged
The issue
View on GitHubSee <https://discourse.gohugo.io/t/related-posts-based-on-heading/57467>.
I suggest using the name tokenize, documented as follows:
(
bool) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of typefragments; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; usecardinalityThresholdto prune common words and reduce build time. Default isfalse.
The two patches
docs/content/en/configuration/related-content.mdindex 92cf92e8a..945523696 100644@@ -45,6 +45,10 @@ This is the default configuration: `pattern` : (`string`) This is currently only relevant for dates. When listing related content, we may want to list content that is also close in time. Setting "2006" (default value for date indexes) as the pattern for a date index will add weight to pages published in the same year. For busier blogs, "200601" (year and month) may be a better default. +`tokenize`+: {{< new-in 0.166.0 />}}+: (`bool`) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type `fragments`; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use `cardinalityThreshold` to prune common words and reduce build time. Default is `false`.+ `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. docs/data/docs.yamlindex 19b950dd5..7fc7c7b8e 100644@@ -2191,6 +2191,7 @@ config: cardinalityThreshold: 0 name: keywords pattern: ''+ tokenize: false toLower: false type: basic weight: 100@@ -2198,6 +2199,7 @@ config: cardinalityThreshold: 0 name: date pattern: ''+ tokenize: false toLower: false type: basic weight: 10@@ -2205,6 +2207,7 @@ config: cardinalityThreshold: 0 name: tags pattern: ''+ tokenize: false toLower: false type: basic weight: 80related/inverted_index.goindex c6c65d97f..5ff257e81 100644@@ -18,17 +18,20 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps" "github.com/gohugoio/hugo/compare" "github.com/gohugoio/hugo/markup/tableofcontents"+ "github.com/gohugoio/hugo/tpl" "github.com/spf13/cast" "github.com/gohugoio/hugo/common/types"@@ -125,6 +128,11 @@ type IndexConfig struct { // Will lower case all string values in and queries tothis index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords for indexing and matching.+ // This is currently only used for heading titles in the "fragments" type;+ // non-heading fragments (e.g. definition terms) always use exact identifier matching.+ Tokenize bool } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,10 +228,15 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ fragments := fp.Fragments(ctx)+ for _, fragment := range fragments.Identifiers { added = true setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) }+ for _, keyword := range config.headingTokenKeywords(fragments) {+ added = true+ setm[keyword] = append(setm[keyword], doc)+ } } } @@ -377,9 +390,11 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ fragments := fp.Fragments(ctx)+ for _, fragment := range fragments.Identifiers { keywords = append(keywords, FragmentKeyword(fragment)) }+ keywords = append(keywords, cfg.headingTokenKeywords(fragments)...) } } @@ -453,6 +468,41 @@ func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { return keywords, nil } +// headingTokenKeywords returns the unique words in the heading titles in f+// as keywords, or nil if this index is not configured to tokenize headings.+func (cfg IndexConfig) headingTokenKeywords(f *tableofcontents.Fragments) []Keyword {+ if !cfg.Tokenize || cfg.Type != TypeFragments || f == nil {+ return nil+ }+ seen := make(map[string]bool)+ var keywords []Keyword+ f.Headings.FilterBy(func(h *tableofcontents.Heading) bool {+ for _, token := range cfg.tokenizeHeadingTitle(h.Title) {+ if !seen[token] {+ seen[token] = true+ keywords = append(keywords, HeadingTokenKeyword(token))+ }+ }+ return false+ })+ return keywords+}++// tokenizeHeadingTitle splits the given heading title, which may contain HTML,+// into words.+func (cfg IndexConfig) tokenizeHeadingTitle(title string) []string {+ if title == "" {+ return nil+ }+ title = html.UnescapeString(tpl.StripHTML(title))+ if cfg.ToLower {+ title = strings.ToLower(title)+ }+ return strings.FieldsFunc(title, func(r rune) bool {+ return !unicode.IsLetter(r) && !unicode.IsDigit(r)+ })+}+ func (idx *InvertedIndex) search(ctx context.Context, query ...queryElement) ([]Document, error) { return idx.searchDate(ctx, nil, zeroDate, query...) }@@ -466,7 +516,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ var (+ fragmentsFilter collections.SortedStringSlice+ // Matched heading tokens keyed by the ToLower setting of the index they matched in,+ // so the headings get tokenized the same way when filtering.+ headingTokensFilter map[bool]map[string]bool+ ) for _, el := range query { setm, found := idx.index[el.Index]@@ -494,8 +549,17 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } if config.Type == TypeFragments && config.ApplyFilter {- if fkw, ok := kw.(FragmentKeyword); ok {+ switch fkw := kw.(type) {+ case FragmentKeyword: fragmentsFilter = append(fragmentsFilter, string(fkw))+ case HeadingTokenKeyword:+ if headingTokensFilter == nil {+ headingTokensFilter = make(map[bool]map[string]bool)+ }+ if headingTokensFilter[config.ToLower] == nil {+ headingTokensFilter[config.ToLower] = make(map[string]bool)+ }+ headingTokensFilter[config.ToLower][string(fkw)] = true } } @@ -535,10 +599,20 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(fragmentsFilter) > 0 || len(headingTokensFilter) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if fragmentsFilter.Contains(h.ID) {+ return true+ }+ for toLower, tokens := range headingTokensFilter {+ for _, token := range (IndexConfig{ToLower: toLower}).tokenizeHeadingTitle(h.Title) {+ if tokens[token] {+ return true+ }+ }+ }+ return false }) } }@@ -613,6 +687,14 @@ func (f FragmentKeyword) String() string { return string(f) } +// HeadingTokenKeyword represents a single word in a heading title.+// It is used by indices of type "fragments" with Tokenize enabled.+type HeadingTokenKeyword string++func (f HeadingTokenKeyword) String() string {+ return string(f)+}+ // Keyword is the interface a keyword in the search index must implement. type Keyword interface { String() stringrelated/inverted_index_test.goindex b33d42955..37511e281 100644@@ -23,6 +23,7 @@ import ( qt "github.com/frankban/quicktest" "github.com/gohugoio/hugo/config"+ "github.com/gohugoio/hugo/markup/tableofcontents" ) type testDoc struct {@@ -252,6 +253,60 @@ func TestToKeywordsToLower(t *testing.T) { }) } +func TestHeadingTokenKeywords(t *testing.T) {+ c := qt.New(t)++ var b tableofcontents.Builder+ b.AddAt(&tableofcontents.Heading{ID: "installing-hugo", Level: 2, Title: "Installing <em>Hugo</em> & Go"}, 0, 1)+ b.AddAt(&tableofcontents.Heading{ID: "hugo-modules", Level: 3, Title: "Hugo modules"}, 0, 2)+ // A non-heading identifier, e.g. a definition term.+ b.SetIdentifiers([]string{"installing-hugo", "hugo-modules", "some-term"})+ fragments := b.Build()++ cfg := IndexConfig{Name: "fragmentrefs", Type: TypeFragments, Tokenize: true}+ c.Assert(cfg.headingTokenKeywords(fragments), qt.DeepEquals, []Keyword{+ HeadingTokenKeyword("Installing"),+ HeadingTokenKeyword("Hugo"),+ HeadingTokenKeyword("Go"),+ HeadingTokenKeyword("modules"),+ })++ cfg.ToLower = true+ c.Assert(cfg.headingTokenKeywords(fragments), qt.DeepEquals, []Keyword{+ HeadingTokenKeyword("installing"),+ HeadingTokenKeyword("hugo"),+ HeadingTokenKeyword("go"),+ HeadingTokenKeyword("modules"),+ })++ cfg.Tokenize = false+ c.Assert(cfg.headingTokenKeywords(fragments), qt.IsNil)++ cfg = IndexConfig{Name: "tags", Type: TypeBasic, Tokenize: true}+ c.Assert(cfg.headingTokenKeywords(fragments), qt.IsNil)+}++func TestDecodeConfigTokenize(t *testing.T) {+ c := qt.New(t)++ m, err := config.FromConfigString(`+[related]+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ tokenize = true+ weight = 80+[[related.indices]]+ name = 'keywords'+ weight = 100+`, "toml")+ c.Assert(err, qt.IsNil)+ conf, err := DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.IsNil)+ c.Assert(conf.Indices[0].Tokenize, qt.IsTrue)+ c.Assert(conf.Indices[1].Tokenize, qt.IsFalse)+}+ func TestDecodeConfig(t *testing.T) { c := qt.New(t) related/related_integration_test.goindex e1387d3c5..840d7f997 100644@@ -134,6 +134,174 @@ Related 2: 2 ) } +func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ filesTemplate := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 0+ toLower = true+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ applyFilter = true+ tokenize = TOKENIZE+ weight = 80+-- content/p1.md --+---+title: p1+---+## Installing *Hugo* on Linux++-- content/p2.md --+---+title: p2+---+## Upgrade hugo++## Something else++-- content/p3.md --+---+title: p3+---+## Unrelated topic++-- content/p4.md --+---+title: p4+fragmentrefs: ['installing-hugo-on-linux']+---+## Nothing in common++-- layouts/single.html --+{{ $related := site.RegularPages.Related . }}+Related: {{ range $related }}{{ .Title }}: {{ range .HeadingsFiltered }}{{ .Title | safeHTML }}|{{ .ID }}|{{ end }};{{ end }}:END+`++ t.Run("enabled", func(t *testing.T) {+ t.Parallel()+ b := hugolib.Test(t, strings.ReplaceAll(filesTemplate, "TOKENIZE", "true"))++ // "Installing <em>Hugo</em> on Linux" and "Upgrade hugo" share the word "hugo".+ // p4 references p1's heading identifier in front matter.+ b.AssertFileContent("public/p1/index.html", "Related: p2: Upgrade hugo|upgrade-hugo|;p4: ;:END")+ b.AssertFileContent("public/p2/index.html", "Related: p1: Installing <em>Hugo</em> on Linux|installing-hugo-on-linux|;:END")+ b.AssertFileContent("public/p3/index.html", "Related: :END")+ // Front matter references still match heading identifiers exactly.+ b.AssertFileContent("public/p4/index.html", "Related: p1: Installing <em>Hugo</em> on Linux|installing-hugo-on-linux|;:END")+ })++ t.Run("disabled", func(t *testing.T) {+ t.Parallel()+ b := hugolib.Test(t, strings.ReplaceAll(filesTemplate, "TOKENIZE", "false"))++ b.AssertFileContent("public/p1/index.html", "Related: p4: ;:END")+ b.AssertFileContent("public/p2/index.html", "Related: :END")+ b.AssertFileContent("public/p4/index.html", "Related: p1: Installing <em>Hugo</em> on Linux|installing-hugo-on-linux|;:END")+ })+}++func TestRelatedFragmentsTokenizeMultipleIndicesToLower(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 0+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ applyFilter = true+ tokenize = true+ weight = 80+[[related.indices]]+ name = 'fragmentrefslower'+ type = 'fragments'+ applyFilter = true+ tokenize = true+ toLower = true+ weight = 80+-- content/p1.md --+---+title: p1+---+## Hugo rocks++## Go+-- content/p2.md --+---+title: p2+---+## Hugo things++## GO stuff+-- layouts/single.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}: {{ range .HeadingsFiltered }}{{ .Title }}|{{ end }};{{ end }}:END+`++ b := hugolib.Test(t, files)++ // "Hugo" matches in both indices, "go" only in the lower case index.+ b.AssertFileContent("public/p1/index.html", "Related: p2: Hugo things|GO stuff|;:END")+ b.AssertFileContent("public/p2/index.html", "Related: p1: Hugo rocks|Go|;:END")+}++func TestRelatedFragmentsTokenizeCardinalityThreshold(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[related]+ includeNewer = true+ threshold = 0+ toLower = true+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ tokenize = true+ cardinalityThreshold = 60+ weight = 80+-- content/p1.md --+---+title: p1+---+## The apple+-- content/p2.md --+---+title: p2+---+## The apple pie+-- content/p3.md --+---+title: p3+---+## The banana+-- content/p4.md --+---+title: p4+---+## The cherry+-- layouts/single.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }};{{ end }}:END+`++ b := hugolib.Test(t, files)++ // "the" is used in all pages and is pruned from the index, "apple" is used in 50% of the pages.+ b.AssertFileContent("public/p1/index.html", "Related: p2;:END")+ b.AssertFileContent("public/p3/index.html", "Related: :END")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode"+ "unicode/utf8" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { // used in more than 50% of the documents in the index. CardinalityThreshold int - // Will lower case all string values in and queries tothis index.+ // Will lower case all string values in and queries to this index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords.+ Tokenize bool++ // Minimum token length in Unicode code points when tokenizing.+ MinTokenLength int } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers { added = true+ if config.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !config.shouldSkipToken(word) {+ kw := config.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ setm[kw] = append(setm[kw], doc)+ }+ }+ }+ continue+ }+ } setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) } }@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers {+ if cfg.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !cfg.shouldSkipToken(word) {+ kw := cfg.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ keywords = append(keywords, kw)+ }+ }+ }+ continue+ }+ } keywords = append(keywords, FragmentKeyword(fragment)) } }@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+ return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ if cfg.Tokenize {+ for _, word := range splitWords(vv) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } else {+ keywords = append(keywords, cfg.stringToKeyword(vv))+ } case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])+ if !cfg.Tokenize {+ kw := make([]Keyword, len(vv))+ for i, s := range vv {+ kw[i] = cfg.stringToKeyword(s)+ }+ keywords = append(keywords, kw...)+ } else {+ for _, s := range vv {+ for _, word := range splitWords(s) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } }- keywords = append(keywords, vvv...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ // Bundled into one struct so the closure below causes only one heap escape.+ var ff struct {+ ids collections.SortedStringSlice+ words collections.SortedStringSlice+ toLower bool+ } for _, el := range query { setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return []Document{}, fmt.Errorf("index config for %q not found", el.Index) } + if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+ ff.toLower = config.ToLower+ }+ for _, kw := range el.Keywords { if docs, found := setm[kw]; found { for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok {- fragmentsFilter = append(fragmentsFilter, string(fkw))+ if config.Tokenize {+ ff.words = append(ff.words, string(fkw))+ } else {+ ff.ids = append(ff.ids, string(fkw))+ } } }- r, found := matchm[doc] if !found { r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } sort.Stable(matches)- sort.Strings(fragmentsFilter)+ sort.Strings(ff.ids)+ sort.Strings(ff.words) result := make([]Document, len(matches)) for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(ff.ids) > 0 || len(ff.words) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if ff.ids.Contains(h.ID) {+ return true+ }+ if len(ff.words) > 0 {+ for _, word := range splitWords(h.Title) {+ if ff.toLower {+ word = strings.ToLower(word)+ }+ if ff.words.Contains(word) {+ return true+ }+ }+ }+ return false }) } }@@ -618,6 +707,31 @@ type Keyword interface { String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+ // Fast path: no whitespace or HTML entities means at most one token.+ if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+ title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if title == "" {+ return nil+ }+ return []string{title}+ }+ raw := strings.Fields(html.UnescapeString(title))+ words := raw[:0]+ for _, w := range raw {+ w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if w != "" {+ words = append(words, w)+ }+ }+ if len(words) == 0 {+ return nil+ }+ return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { }) } +func TestSplitWords(t *testing.T) {+ c := qt.New(t)++ tests := []struct {+ in string+ want []string+ }{+ {"", nil},+ {" ", nil},+ {"hello world", []string{"hello", "world"}},+ // leading/trailing punctuation stripped+ {"...hello...", []string{"hello"}},+ {",hello, world!", []string{"hello", "world"}},+ // digits preserved+ {"go1.24", []string{"go1.24"}},+ // HTML entities decoded+ {"foo &bar", []string{"foo", "bar"}},+ // fast path: single token, no whitespace+ {"&foo&", []string{"foo"}},+ // and similar entities that decode to whitespace must not use the fast path+ {"Apples and oranges!", []string{"Apples", "and", "oranges"}},+ // punctuation-only tokens dropped+ {"hello --- world", []string{"hello", "world"}},+ // mixed spaces+ {" foo bar ", []string{"foo", "bar"}},+ }++ for _, tt := range tests {+ c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+ }+}+ func BenchmarkRelatedNewIndex(b *testing.B) { pages := make([]*testDoc, 100) numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] weight = 90 [[related.indices]] name = 'keywords'- weight = 80 + weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} - + ` b := hugolib.Test(t, files) expect := `-P1 Fragments: [p1-title] +P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 b.AssertFileContent("public/"+p+"/index.html", expect) } - b.AssertFileContent("public/p1/index.html",+ b.AssertFileContent(+ "public/p1/index.html", "Related 1: 0: p2: h0: First title|ref1|::END", "Related 2: 0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", ) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+ b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ applyFilter = true+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+ b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = false+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // toLower=false: "Hugo" != "hugo", so no match.+ b.AssertFileContent("public/p1/index.html", "Related: ")+ b.AssertFileContent("public/p2/index.html", "Related: ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+ minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+ b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+ b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+ // All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+[[related.indices]]+ name = 'keywords'+ weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p3 share the same unique fragment tokens; p1 also matches on+ // keywords ("fruit"). Without deduplication, p3's repeated headings+ // inflate its weight above p1's combined fragment+keywords weight.+ b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] toLower = false [[related.indices]] name = 'keywords'- weight = 70 + weight = 70 [[related.indices]] name = 'pagerefs' type = 'fragments'- weight = 30 + weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)