Backtestshugorun-20261007-020926#15199
related: Add option to tokenize headings
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Patch B is the more correct and better-contained fix. Patch B: - Keeps the existing identifier-based `FragmentKeyword` entries and adds heading-word tokens alongside them. Exact identifier matching still works, including explicit `fragments` in search options and `fragmentrefs` pointing at heading IDs; its test covers this with p4/p5. - Uses a distinct `headingTokenKeyword` type, so a word can never collide with a fragment ID. - Makes `applyFilter` work with tokens: the type switch in `searchDate` collects token keywords, and `headingMatchesTokens` re-tokenizes heading titles for the filter. - Tokenizes only heading titles, as the issue specifies. Definition terms keep exact matching, and a test asserts this. - Handles HTML in titles by stripping tags and unescaping entities, and keeps Unicode combining marks inside words. - Updates the docs with the exact wording from the issue. - Tests cover both tokenize=true and tokenize=false, plus config decoding and the tokenizer itself. Patch A has real correctness problems: - When tokenize is on, it `continue`s past headings, so it no longer indexes their fragment identifiers. This silently breaks exact fragment-ID matching, for example `Related` with `fragments` options or `fragmentrefs` targeting headings. - Its tokens are created with `stringToKeyword`, which produces `StringKeyword`. But the `applyFilter` path in `searchDate` only collects `kw.(FragmentKeyword)`, so `ff.words` is never populated. Heading filtering with tokenize therefore does not work as intended, and its own ApplyFilter test looks unlikely to pass for the right reason. - It expands scope beyond the issue: tokenization for basic indices through `ToKeywords`, plus a new `MinTokenLength` option. This adds risk without being asked for. - It adds no documentation and includes whitespace churn in an existing test file. On correctness, scope and tests, B is clearly better.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
1h
Wall-clock time in the container
Full test suite
No new failures
3,746 passed · 99 skipped · 3,847 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
246vs 462 merged
Changed lines, ours vs merged
The issue
View on GitHubSee <https://discourse.gohugo.io/t/related-posts-based-on-heading/57467>.
I suggest using the name tokenize, documented as follows:
(
bool) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of typefragments; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; usecardinalityThresholdto prune common words and reduce build time. Default isfalse.
The two patches
docs/content/en/configuration/related-content.mdindex 92cf92e8a..496c33b08 100644@@ -48,6 +48,9 @@ This is the default configuration: `toLower` : (`bool`) Whether to transform keywords in both the indexes and the queries to lower case. This may give more accurate results at a slight performance penalty. Default is `false`. +`tokenize`+: (`bool`) Whether to split multi-word values into individual keywords for indexing and matching. Currently applies to heading titles in indices of type `fragments`; non-heading fragments such as definition terms continue to use exact-identifier matching regardless of this setting. This increases index size in proportion to the average number of words per heading; use `cardinalityThreshold` to prune common words and reduce build time. Default is `false`.+ `type` : (`string`) One of `basic` or `fragments`. Default is `basic`. docs/data/docs.yamlindex 19b950dd5..831250fbc 100644@@ -2192,6 +2192,7 @@ config: name: keywords pattern: '' toLower: false+ tokenize: false type: basic weight: 100 - applyFilter: false@@ -2199,6 +2200,7 @@ config: name: date pattern: '' toLower: false+ tokenize: false type: basic weight: 10 - applyFilter: false@@ -2206,6 +2208,7 @@ config: name: tags pattern: '' toLower: false+ tokenize: false type: basic weight: 80 threshold: 80markup/tableofcontents/tableofcontents.goindex 6c40c9a59..d03e351a3 100644@@ -174,6 +174,12 @@ func (toc *Fragments) ToHTML(startLevel, stopLevel any, ordered bool) (template. return template.HTML(b.s.String()), nil } +// Walk calls fn for every heading in the ToC, including nested headings.+// For internal use only.+func (toc *Fragments) Walk(fn func(*Heading)) {+ toc.walk(fn)+}+ func (toc Fragments) walk(fn func(*Heading)) { for _, h := range toc.Headings { h.walk(fn)related/inverted_index.goindex c6c65d97f..a69cd905b 100644@@ -18,12 +18,14 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -125,6 +127,11 @@ type IndexConfig struct { // Will lower case all string values in and queries tothis index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords for indexing and matching.+ // This is currently only used for heading titles in the "fragments" type;+ // non-heading fragments (e.g. definition terms) always use exact identifier matching.+ Tokenize bool } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,10 +227,15 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ fragments := fp.Fragments(ctx)+ for _, fragment := range fragments.Identifiers { added = true setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) }+ for _, keyword := range config.headingTokenKeywords(fragments) {+ added = true+ setm[keyword] = append(setm[keyword], doc)+ } } } @@ -377,9 +389,11 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ fragments := fp.Fragments(ctx)+ for _, fragment := range fragments.Identifiers { keywords = append(keywords, FragmentKeyword(fragment)) }+ keywords = append(keywords, cfg.headingTokenKeywords(fragments)...) } } @@ -413,6 +427,61 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume return idx.search(ctx, queryElements...) } +// headingTokenKeywords returns the unique keywords created by splitting the+// heading titles in f into words.+// It returns nil if tokenization is not enabled for this index.+func (cfg IndexConfig) headingTokenKeywords(f *tableofcontents.Fragments) []Keyword {+ if !cfg.Tokenize || cfg.Type != TypeFragments || f == nil {+ return nil+ }+ var keywords []Keyword+ seen := make(map[headingTokenKeyword]bool)+ f.Walk(func(h *tableofcontents.Heading) {+ for _, token := range tokenizeHeadingTitle(h.Title, cfg.ToLower) {+ kw := headingTokenKeyword{token: token, toLower: cfg.ToLower}+ if !seen[kw] {+ seen[kw] = true+ keywords = append(keywords, kw)+ }+ }+ })+ return keywords+}++// tokenizeHeadingTitle splits a heading title into words.+// The title may contain (sanitized) HTML, so tags are removed and+// entities unescaped before splitting on anything not a letter, a mark or a number.+func tokenizeHeadingTitle(title string, toLower bool) []string {+ if title == "" {+ return nil+ }+ if strings.IndexByte(title, '<') != -1 {+ var b strings.Builder+ inTag := false+ for _, r := range title {+ switch {+ case r == '<':+ inTag = true+ // Make sure that words on either side of a tag are kept apart.+ b.WriteByte(' ')+ case r == '>' && inTag:+ inTag = false+ case !inTag:+ b.WriteRune(r)+ }+ }+ title = b.String()+ }+ title = html.UnescapeString(title)+ if toLower {+ title = strings.ToLower(title)+ }+ return strings.FieldsFunc(title, func(r rune) bool {+ // Combining marks (e.g. Devanagari vowel signs) are part of the word.+ return !unicode.IsLetter(r) && !unicode.IsMark(r) && !unicode.IsNumber(r)+ })+}+ func (cfg IndexConfig) stringToKeyword(s string) Keyword { if cfg.ToLower { s = strings.ToLower(s)@@ -467,6 +536,7 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero() var fragmentsFilter collections.SortedStringSlice+ var tokensFilter map[headingTokenKeyword]bool for _, el := range query { setm, found := idx.index[el.Index]@@ -494,8 +564,14 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } if config.Type == TypeFragments && config.ApplyFilter {- if fkw, ok := kw.(FragmentKeyword); ok {+ switch fkw := kw.(type) {+ case FragmentKeyword: fragmentsFilter = append(fragmentsFilter, string(fkw))+ case headingTokenKeyword:+ if tokensFilter == nil {+ tokensFilter = make(map[headingTokenKeyword]bool)+ }+ tokensFilter[fkw] = true } } @@ -535,10 +611,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(fragmentsFilter) > 0 || len(tokensFilter) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ return fragmentsFilter.Contains(h.ID) || headingMatchesTokens(h, tokensFilter) }) } }@@ -547,6 +623,21 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return result, nil } +// headingMatchesTokens reports whether any word in the title of h is in tokens.+func headingMatchesTokens(h *tableofcontents.Heading, tokens map[headingTokenKeyword]bool) bool {+ if len(tokens) == 0 {+ return false+ }+ for _, toLower := range []bool{false, true} {+ for _, token := range tokenizeHeadingTitle(h.Title, toLower) {+ if tokens[headingTokenKeyword{token: token, toLower: toLower}] {+ return true+ }+ }+ }+ return false+}+ // normalizes num to a number between 0 and 100. func norm(num, min, max int) int { if min > max {@@ -613,6 +704,17 @@ func (f FragmentKeyword) String() string { return string(f) } +// headingTokenKeyword represents a single word in a heading title.+// It is kept separate from FragmentKeyword so a word never matches a fragment identifier.+type headingTokenKeyword struct {+ token string+ toLower bool+}++func (h headingTokenKeyword) String() string {+ return h.token+}+ // Keyword is the interface a keyword in the search index must implement. type Keyword interface { String() stringrelated/inverted_index_test.goindex b33d42955..dd4e2d002 100644@@ -296,6 +296,44 @@ func TestDecodeConfig(t *testing.T) { c.Assert(first.Name, qt.Equals, "keywords") } +func TestDecodeConfigTokenize(t *testing.T) {+ c := qt.New(t)++ configToml := `+[related]+ threshold = 80+ [[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ tokenize = true+ weight = 80+ [[related.indices]]+ name = 'tags'+ weight = 60+`++ m, err := config.FromConfigString(configToml, "toml")+ c.Assert(err, qt.IsNil)+ conf, err := DecodeConfig(m.GetParams("related"))+ c.Assert(err, qt.IsNil)+ c.Assert(conf.Indices[0].Tokenize, qt.IsTrue)+ c.Assert(conf.Indices[1].Tokenize, qt.IsFalse)+}++func TestTokenizeHeadingTitle(t *testing.T) {+ c := qt.New(t)++ c.Assert(tokenizeHeadingTitle("", false), qt.HasLen, 0)+ c.Assert(tokenizeHeadingTitle("Install Hugo on Linux", false), qt.DeepEquals, []string{"Install", "Hugo", "on", "Linux"})+ c.Assert(tokenizeHeadingTitle("Install Hugo on Linux", true), qt.DeepEquals, []string{"install", "hugo", "on", "linux"})+ c.Assert(tokenizeHeadingTitle("Common p3, p4, p5!", false), qt.DeepEquals, []string{"Common", "p3", "p4", "p5"})+ c.Assert(tokenizeHeadingTitle("Use <code>hugo</code>&<em>Go</em>", false), qt.DeepEquals, []string{"Use", "hugo", "Go"})+ c.Assert(tokenizeHeadingTitle("Ærlig talt café", true), qt.DeepEquals, []string{"ærlig", "talt", "café"})+ // Combining marks belong to the word they modify.+ c.Assert(tokenizeHeadingTitle("हिन्दी भाषा", false), qt.DeepEquals, []string{"हिन्दी", "भाषा"})+ c.Assert(tokenizeHeadingTitle("cafe\u0301 au lait", false), qt.DeepEquals, []string{"cafe\u0301", "au", "lait"})+}+ func TestToKeywordsAnySlice(t *testing.T) { c := qt.New(t) var config IndexConfigrelated/related_integration_test.goindex e1387d3c5..5c2148adb 100644@@ -134,6 +134,90 @@ Related 2: 2 ) } +func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ filesTemplate := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"]+[markup.goldmark.parser]+ autoDefinitionTermID = true+[related]+ includeNewer = true+ threshold = 20+ toLower = true+[[related.indices]]+ name = 'fragmentrefs'+ type = 'fragments'+ applyFilter = true+ tokenize = TOKENIZE+ weight = 80+-- content/p1.md --+---+title: p1+---+## Install Hugo on Linux++## Something else+-- content/p2.md --+---+title: p2+---+## Hugo installation for macOS++## Unrelated heading+-- content/p3.md --+---+title: p3+---+## Cooking pasta++Linux term+: A definition+-- content/p4.md --+---+title: p4+---+## Exact {#ref1}+-- content/p5.md --+---+title: p5+fragmentrefs: ['ref1']+---+## Nothing in common+-- layouts/single.html --+{{ $related := site.RegularPages.Related . }}+Related: {{ range $related }}{{ .Title }}: {{ range .HeadingsFiltered }}{{ .Title }}|{{ end }};{{ end }}:END+{{ $opts := dict "document" . "fragments" (slice "linux-term") }}+RelatedOpts: {{ range site.RegularPages.Related $opts }}{{ .Title }};{{ end }}:END+`++ t.Run("Enabled", func(t *testing.T) {+ t.Parallel()+ b := hugolib.Test(t, strings.ReplaceAll(filesTemplate, "TOKENIZE", "true"))++ // The word "hugo" is shared between the two headings.+ b.AssertFileContent("public/p1/index.html", "Related: p2: Hugo installation for macOS|;:END")+ b.AssertFileContent("public/p2/index.html", "Related: p1: Install Hugo on Linux|;:END")+ // The definition term is not tokenized, so "Linux term" does not match+ // the "Linux" heading in p1; the term only matches its exact identifier.+ b.AssertFileContent("public/p3/index.html", "Related: :END", "RelatedOpts: :END")+ b.AssertFileContent("public/p1/index.html", "RelatedOpts: p2;p3;:END")+ // Exact identifier matching still works.+ b.AssertFileContent("public/p5/index.html", "Related: p4: Exact|;:END")+ })++ t.Run("Disabled", func(t *testing.T) {+ t.Parallel()+ b := hugolib.Test(t, strings.ReplaceAll(filesTemplate, "TOKENIZE", "false"))++ b.AssertFileContent("public/p1/index.html", "Related: :END", "RelatedOpts: p3;:END")+ b.AssertFileContent("public/p2/index.html", "Related: :END")+ b.AssertFileContent("public/p5/index.html", "Related: p4: Exact|;:END")+ })+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`
related/inverted_index.goindex c6c65d97f..568880107 100644@@ -18,12 +18,15 @@ import ( "context" "errors" "fmt"+ "html" "maps" "math" "sort" "strings" "sync" "time"+ "unicode"+ "unicode/utf8" "github.com/gohugoio/hugo/common/collections" "github.com/gohugoio/hugo/common/hmaps"@@ -122,9 +125,15 @@ type IndexConfig struct { // used in more than 50% of the documents in the index. CardinalityThreshold int - // Will lower case all string values in and queries tothis index.+ // Will lower case all string values in and queries to this index. // May get better accurate results, but at a slight performance cost. ToLower bool++ // Whether to split multi-word values into individual keywords.+ Tokenize bool++ // Minimum token length in Unicode code points when tokenizing.+ MinTokenLength int } // Document is the interface an indexable document in Hugo must fulfill.@@ -220,8 +229,24 @@ func (idx *InvertedIndex) Add(ctx context.Context, docs ...Document) error { if config.Type == TypeFragments { if fp, ok := doc.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers { added = true+ if config.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !config.shouldSkipToken(word) {+ kw := config.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ setm[kw] = append(setm[kw], doc)+ }+ }+ }+ continue+ }+ } setm[FragmentKeyword(fragment)] = append(setm[FragmentKeyword(fragment)], doc) } }@@ -377,7 +402,23 @@ func (idx *InvertedIndex) Search(ctx context.Context, opts SearchOpts) ([]Docume } if opts.Document != nil { if fp, ok := opts.Document.(FragmentProvider); ok {- for _, fragment := range fp.Fragments(ctx).Identifiers {+ frags := fp.Fragments(ctx)+ seenKW := make(map[Keyword]bool)+ for _, fragment := range frags.Identifiers {+ if cfg.Tokenize {+ if h, found := frags.HeadingsMap[fragment]; found {+ for _, word := range splitWords(h.Title) {+ if !cfg.shouldSkipToken(word) {+ kw := cfg.stringToKeyword(word)+ if !seenKW[kw] {+ seenKW[kw] = true+ keywords = append(keywords, kw)+ }+ }+ }+ continue+ }+ } keywords = append(keywords, FragmentKeyword(fragment)) } }@@ -423,19 +464,41 @@ func (cfg IndexConfig) stringToKeyword(s string) Keyword { return StringKeyword(s) } +func (cfg IndexConfig) shouldSkipToken(word string) bool {+ return cfg.MinTokenLength > 0 && utf8.RuneCountInString(word) < cfg.MinTokenLength+}+ // ToKeywords returns a Keyword slice of the given input. func (cfg IndexConfig) ToKeywords(v any) ([]Keyword, error) { var keywords []Keyword switch vv := v.(type) { case string:- keywords = append(keywords, cfg.stringToKeyword(vv))+ if cfg.Tokenize {+ for _, word := range splitWords(vv) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } else {+ keywords = append(keywords, cfg.stringToKeyword(vv))+ } case []string:- vvv := make([]Keyword, len(vv))- for i := range vvv {- vvv[i] = cfg.stringToKeyword(vv[i])+ if !cfg.Tokenize {+ kw := make([]Keyword, len(vv))+ for i, s := range vv {+ kw[i] = cfg.stringToKeyword(s)+ }+ keywords = append(keywords, kw...)+ } else {+ for _, s := range vv {+ for _, word := range splitWords(s) {+ if !cfg.shouldSkipToken(word) {+ keywords = append(keywords, cfg.stringToKeyword(word))+ }+ }+ } }- keywords = append(keywords, vvv...) case []any: return cfg.ToKeywords(cast.ToStringSlice(vv)) case time.Time:@@ -466,7 +529,12 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa }() applyDateFilter := !idx.cfg.IncludeNewer && !upperDate.IsZero()- var fragmentsFilter collections.SortedStringSlice+ // Bundled into one struct so the closure below causes only one heap escape.+ var ff struct {+ ids collections.SortedStringSlice+ words collections.SortedStringSlice+ toLower bool+ } for _, el := range query { setm, found := idx.index[el.Index]@@ -479,6 +547,10 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa return []Document{}, fmt.Errorf("index config for %q not found", el.Index) } + if config.Type == TypeFragments && config.ApplyFilter && config.Tokenize {+ ff.toLower = config.ToLower+ }+ for _, kw := range el.Keywords { if docs, found := setm[kw]; found { for _, doc := range docs {@@ -495,10 +567,13 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa if config.Type == TypeFragments && config.ApplyFilter { if fkw, ok := kw.(FragmentKeyword); ok {- fragmentsFilter = append(fragmentsFilter, string(fkw))+ if config.Tokenize {+ ff.words = append(ff.words, string(fkw))+ } else {+ ff.ids = append(ff.ids, string(fkw))+ } } }- r, found := matchm[doc] if !found { r = getRank(doc, config.Weight)@@ -528,17 +603,31 @@ func (idx *InvertedIndex) searchDate(ctx context.Context, self Document, upperDa } sort.Stable(matches)- sort.Strings(fragmentsFilter)+ sort.Strings(ff.ids)+ sort.Strings(ff.words) result := make([]Document, len(matches)) for i, m := range matches { result[i] = m.Doc - if len(fragmentsFilter) > 0 {+ if len(ff.ids) > 0 || len(ff.words) > 0 { if dp, ok := result[i].(FragmentProvider); ok { result[i] = dp.ApplyFilterToHeadings(ctx, func(h *tableofcontents.Heading) bool {- return fragmentsFilter.Contains(h.ID)+ if ff.ids.Contains(h.ID) {+ return true+ }+ if len(ff.words) > 0 {+ for _, word := range splitWords(h.Title) {+ if ff.toLower {+ word = strings.ToLower(word)+ }+ if ff.words.Contains(word) {+ return true+ }+ }+ }+ return false }) } }@@ -618,6 +707,31 @@ type Keyword interface { String() string } +// splitWords splits s into normalized words, stripping HTML entities and+// leading/trailing punctuation from each word.+func splitWords(title string) []string {+ // Fast path: no whitespace or HTML entities means at most one token.+ if strings.IndexFunc(title, unicode.IsSpace) == -1 && strings.IndexByte(title, '&') == -1 {+ title = strings.TrimFunc(html.UnescapeString(title), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if title == "" {+ return nil+ }+ return []string{title}+ }+ raw := strings.Fields(html.UnescapeString(title))+ words := raw[:0]+ for _, w := range raw {+ w = strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+ if w != "" {+ words = append(words, w)+ }+ }+ if len(words) == 0 {+ return nil+ }+ return words+}+ // StringsToKeywords converts the given slice of strings to a slice of Keyword. func (cfg IndexConfig) StringsToKeywords(s ...string) []Keyword { kw := make([]Keyword, len(s))related/inverted_index_test.goindex b33d42955..6bb561112 100644@@ -309,6 +309,38 @@ func TestToKeywordsAnySlice(t *testing.T) { }) } +func TestSplitWords(t *testing.T) {+ c := qt.New(t)++ tests := []struct {+ in string+ want []string+ }{+ {"", nil},+ {" ", nil},+ {"hello world", []string{"hello", "world"}},+ // leading/trailing punctuation stripped+ {"...hello...", []string{"hello"}},+ {",hello, world!", []string{"hello", "world"}},+ // digits preserved+ {"go1.24", []string{"go1.24"}},+ // HTML entities decoded+ {"foo &bar", []string{"foo", "bar"}},+ // fast path: single token, no whitespace+ {"&foo&", []string{"foo"}},+ // and similar entities that decode to whitespace must not use the fast path+ {"Apples and oranges!", []string{"Apples", "and", "oranges"}},+ // punctuation-only tokens dropped+ {"hello --- world", []string{"hello", "world"}},+ // mixed spaces+ {" foo bar ", []string{"foo", "bar"}},+ }++ for _, tt := range tests {+ c.Assert(splitWords(tt.in), qt.DeepEquals, tt.want, qt.Commentf("input: %q", tt.in))+ }+}+ func BenchmarkRelatedNewIndex(b *testing.B) { pages := make([]*testDoc, 100) numkeywords := 30related/related_integration_test.goindex e1387d3c5..038d955cc 100644@@ -40,7 +40,7 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] weight = 90 [[related.indices]] name = 'keywords'- weight = 80 + weight = 80 -- content/p1.md -- --- title: p1@@ -112,13 +112,13 @@ Related 2: {{ template "list-related" $related2 }} -- layouts/single.html -- Content: {{ .Content }} - + ` b := hugolib.Test(t, files) expect := `-P1 Fragments: [p1-title] +P1 Fragments: [p1-title] P2 Fragments: [p2-title-1 p2-title-2 ref1] Len Related 1: 1 Related 2: 2@@ -128,12 +128,288 @@ Related 2: 2 b.AssertFileContent("public/"+p+"/index.html", expect) } - b.AssertFileContent("public/p1/index.html",+ b.AssertFileContent(+ "public/p1/index.html", "Related 1: 0: p2: h0: First title|ref1|::END", "Related 2: 0: p5: h0: Common p3, p4, p5|common-p3-p4-p5|::END 1: p4: h0: Common p3, p4, p5|common-p3-p4-p5|::END", ) } +// See issue 15199.+func TestRelatedFragmentsTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo+-- content/p3.md --+---+title: p3+---+## JavaScript and CSS+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeHTMLEntities(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+baseURL = "http://example.com/"+disableKinds = ["home", "section", "taxonomy", "term", "rss", "sitemap"]+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Types of apples+-- content/p2.md --+---+title: p2+---+## Something about "apples"+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2")+ b.AssertFileContent("public/p2/index.html", "Related: p1")+}++// See issue 15199.+func TestRelatedBasicTokenize(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: Hugo Templates Overview+---+-- content/p2.md --+---+title: Templates in Hugo+---+-- content/p3.md --+---+title: JavaScript and CSS+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: Templates in Hugo")+ b.AssertFileContent("public/p2/index.html", "Related: Hugo Templates Overview")+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// See issue 15199.+func TestRelatedFragmentsTokenizeApplyFilter(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ applyFilter = true+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates Overview+-- content/p2.md --+---+title: p2+---+## Templates in Hugo++## Unrelated Topic+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }}:{{ with .HeadingsFiltered }}{{ range . }}{{ .Title }}|{{ end }}{{ end }}::END {{ end }}+`+ b := hugolib.Test(t, files)++ b.AssertFileContent("public/p1/index.html", "Related: p2:Templates in Hugo|::END")+ b.AssertFileContent("public/p2/index.html", "Related: p1:Hugo Templates Overview|::END")+}++// See issue 15199.+func TestRelatedTokenizeCaseSensitive(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = false+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+-- content/p1.md --+---+title: p1+---+## Hugo Templates+-- content/p2.md --+---+title: p2+---+## hugo templates+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // toLower=false: "Hugo" != "hugo", so no match.+ b.AssertFileContent("public/p1/index.html", "Related: ")+ b.AssertFileContent("public/p2/index.html", "Related: ")+}++// See issue 15238.+func TestRelatedTokenizeMinTokenLength(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ name = 'title'+ weight = 100+ tokenize = true+ minTokenLength = 4+-- content/p1.md --+---+title: The long journey home+---+-- content/p2.md --+---+title: Planning your next big journey+---+-- content/p3.md --+---+title: Go to the top+---+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p2 share "journey" (>= minTokenLength=4); short words are filtered.+ b.AssertFileContent("public/p1/index.html", "Related: Planning your next big journey")+ b.AssertFileContent("public/p2/index.html", "Related: The long journey home")+ // All tokens in p3's title ("go", "to", "the", "top") are < minTokenLength=4.+ b.AssertFileContent("public/p3/index.html", "Related: ")+}++// Repetitive headings must not inflate a document's rank above a document+// that matches on an additional index.+func TestRelatedFragmentsTokenizeDeduplication(t *testing.T) {+ t.Parallel()++ files := `+-- hugo.toml --+disableKinds = ['home', 'section', 'taxonomy', 'term', 'rss', 'sitemap']+[related]+ includeNewer = true+ threshold = 80+ toLower = true+[[related.indices]]+ type = 'fragments'+ weight = 100+ tokenize = true+[[related.indices]]+ name = 'keywords'+ weight = 100+-- content/p1.md --+---+title: p1+keywords: ['fruit']+---+## I love apples, apples and more apples+-- content/p2.md --+---+title: p2+keywords: ['fruit']+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- content/p3.md --+---+title: p3+---+## I love apples, apples and more apples+## I love apples, apples and more apples+## I love apples, apples and more apples+-- layouts/page.html --+Related: {{ range site.RegularPages.Related . }}{{ .Title }} {{ end }}+`+ b := hugolib.Test(t, files)++ // p1 and p3 share the same unique fragment tokens; p1 also matches on+ // keywords ("fruit"). Without deduplication, p3's repeated headings+ // inflate its weight above p1's combined fragment+keywords weight.+ b.AssertFileContent("public/p2/index.html", "Related: p1 p3")+}+ func BenchmarkRelatedSite(b *testing.B) { var files strings.Builder files.WriteString(`@@ -146,11 +422,11 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "robotsTXT"] toLower = false [[related.indices]] name = 'keywords'- weight = 70 + weight = 70 [[related.indices]] name = 'pagerefs' type = 'fragments'- weight = 30 + weight = 30 -- layouts/single.html -- Len related: {{ site.RegularPages.Related . | len }} `)