Backtestshugorun-20261002-200313#15301
Improve/document security.http.urls override behavior
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
The issue asks for better visibility and documentation of how the security.http.urls override disables the dial-time resolved-address check. Patch B targets that ask directly. It adds a dedicated docs section explaining the behavior, its caveats (permissive allowlists, deny-only lists, repeating defaults plus additions), and a narrow-allowlist recommendation. It also factors the condition into IsResolvedHTTPAddressCheckEnabled with a clear doc comment and adds a hint to AccessDeniedError, so users who hit the denial learn why and how to opt out. Its tests cover the enabled and disabled cases across several custom policies, including the default patterns plus an addition, and assert the new error message. Those tests would fail before the change, since the method and hint don't exist yet. Patch A adds a short doc paragraph about the override, which partly answers the issue. Most of the patch, though, is an unrelated new feature: proxyFromEnvironment, with a custom proxyTransport, context-carried proxy URLs, a reimplemented canonicalAddr, and a new golang.org/x/net dependency. It also changes default behavior by setting base.Proxy = nil, so environment proxies, which were honored before via the cloned DefaultTransport, are now ignored unless the user opts in. That is a regression risk for existing users behind proxies. The added complexity (address matching to bypass the check for the proxy dial) adds risk and obscures the documentation fix the issue asked for. B is focused, lower-risk, better tested for the issue's concern, and improves user-facing diagnostics.
The judge saw only the two patches, labelled A and B in random order, with issue links and version numbers removed. The reasoning above is shown verbatim.
Solve
57m 40s
Wall-clock time in the container
Full test suite
No new failures
3,754 passed · 100 skipped · 3,856 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
104vs 167 merged
Changed lines, ours vs merged
The issue
View on GitHubIn d6e6f9e500eebdeae8e28de830fff2e3bfc7d534 (unreleased) we added a dial-time URL validation to catch URLs that's valid according to the whitelist but that resolves to a loopback, private or link-local IP addresses. That check is skipped it if the user has a custom security.http.urls setup, as it would not make sense to block local development servers.
See CheckAllowedHTTPAddress.
This has been reported by @Reload3d as a potential security issue, and while I think the current behaviour is the pragmatic correct approach, I'm creating this issue for visibility and to see if this can be improved further (e.g. better documentation).
The two patches
config/security/securityConfig.goindex f7cc157b8..6ca0801d9 100644@@ -126,6 +126,9 @@ type Funcs struct { type HTTP struct { // URLs to allow in remote HTTP (resources.Get, getJSON, getCSV).+ // Under the default policy, Hugo additionally refuses to connect to hosts+ // that resolve to a loopback, private, link-local or otherwise non-public+ // address. Any custom value turns that resolved-address check off. URLs Whitelist `json:"urls"` // HTTP methods to allow.@@ -224,12 +227,11 @@ func (c Config) CheckAllowedHTTPURL(u string) error { // link-local (including the cloud metadata endpoint) address would otherwise // satisfy the policy and let resources.GetRemote reach an internal endpoint. // We deny any non–global-unicast or private address here to close that gap.+//+// The check is only enforced under the default security.http.urls policy, see+// IsResolvedHTTPAddressCheckEnabled. func (c Config) CheckAllowedHTTPAddress(network, address string) error {- // Only enforced under the default hardened allowlist. If the user has- // customized security.http.urls they have opted into whatever hosts they- // listed, including internal ones (e.g. a local dev server), so we do not- // second-guess the resolved address.- if !slices.Equal(c.HTTP.URLs.patternsStrings, DefaultConfig.HTTP.URLs.patternsStrings) {+ if !c.IsResolvedHTTPAddressCheckEnabled() { return nil } deny := func(name string) error {@@ -237,6 +239,7 @@ func (c Config) CheckAllowedHTTPAddress(network, address string) error { name: name, path: "security.http.urls", policies: c.ToTOML(),+ hint: fmt.Sprintf(resolvedAddressDeniedHint, name), } } host, _, err := net.SplitHostPort(address)@@ -255,6 +258,24 @@ func (c Config) CheckAllowedHTTPAddress(network, address string) error { return nil } +const resolvedAddressDeniedHint = "the remote host resolved to %q, which is a loopback, private, link-local or otherwise non-public address. " ++ "Hugo refuses to connect to such addresses under the default security.http.urls policy. " ++ "To fetch from an internal host (e.g. a local development server), set security.http.urls explicitly; " ++ "note that any custom security.http.urls policy turns this resolved-address check off, so keep it as narrow as possible"++// IsResolvedHTTPAddressCheckEnabled reports whether CheckAllowedHTTPAddress+// enforces its resolved-address restrictions.+//+// It is only enforced under the default hardened security.http.urls policy.+// If the user has customized security.http.urls they have opted into whatever+// hosts they listed, including internal ones (e.g. a local dev server), so we+// do not second-guess the resolved address. This means that a permissive custom+// policy (e.g. ['.*']) also allows hostnames that resolve to internal+// addresses; this is documented in the security configuration docs.+func (c Config) IsResolvedHTTPAddressCheckEnabled() bool {+ return slices.Equal(c.HTTP.URLs.patternsStrings, DefaultConfig.HTTP.URLs.patternsStrings)+}+ // Special-purpose ranges that Go classifies as global unicast and // non-private, but that are never reachable on the public Internet. var nonPublicPrefixes = []netip.Prefix{@@ -463,9 +484,15 @@ type AccessDeniedError struct { path string name string policies string++ // hint is an optional explanation of why access was denied.+ hint string } func (e *AccessDeniedError) Error() string {+ if e.hint != "" {+ return fmt.Sprintf("access denied: %q is not whitelisted in policy %q; %s; the current security configuration is:\n\n%s\n\n", e.name, e.path, e.hint, e.policies)+ } return fmt.Sprintf("access denied: %q is not whitelisted in policy %q; the current security configuration is:\n\n%s\n\n", e.name, e.path, e.policies) } config/security/securityConfig_test.goindex 4ac059a67..56e289f73 100644@@ -296,6 +296,51 @@ func TestCheckAllowedHTTPAddress(t *testing.T) { } } +// The resolved-address check is only enforced under the default+// security.http.urls policy; any custom policy turns it off. A denial must+// explain why the resolved address was refused and how to opt out.+func TestCheckAllowedHTTPAddressOverride(t *testing.T) {+ t.Parallel()+ c := qt.New(t)++ decode := func(tomlConfig string) Config {+ cfg, err := config.FromConfigString(tomlConfig, "toml")+ c.Assert(err, qt.IsNil)+ pc, err := DecodeConfig(cfg)+ c.Assert(err, qt.IsNil)+ return pc+ }++ pc, err := DecodeConfig(config.New())+ c.Assert(err, qt.IsNil)+ c.Assert(pc.IsResolvedHTTPAddressCheckEnabled(), qt.IsTrue)+ err = pc.CheckAllowedHTTPAddress("tcp", "169.254.169.254:80")+ c.Assert(err, qt.IsNotNil)+ c.Assert(IsAccessDenied(err), qt.IsTrue)+ c.Assert(err, qt.ErrorMatches, `(?s).*"169\.254\.169\.254" is not whitelisted in policy "security\.http\.urls"; the remote host resolved to "169\.254\.169\.254", which is a loopback, private, link-local or otherwise non-public address.*set security\.http\.urls explicitly.*`)++ // Explicitly setting the default patterns keeps the check.+ pc = decode(`+[security.http]+urls = ['(?i)^https?://[a-z0-9]', '! (?i)^https?://\d+\.', '! (?i)localhost', '! (?i)^https?://[^/?#]*@']+`)+ c.Assert(pc.IsResolvedHTTPAddressCheckEnabled(), qt.IsTrue)+ c.Assert(pc.CheckAllowedHTTPAddress("tcp", "127.0.0.1:80"), qt.IsNotNil)++ for _, urls := range []string{+ `['.*']`,+ `['.*', '! ^https?://evil\.example\.com']`,+ `['! ^https?://evil\.example\.com']`,+ `['^http://localhost:1313/']`,+ `['(?i)^https?://[a-z0-9]', '! (?i)^https?://\d+\.', '! (?i)localhost', '! (?i)^https?://[^/?#]*@', '^http://localhost:1313/']`,+ `'none'`,+ } {+ pc := decode("[security.http]\nurls = " + urls)+ c.Assert(pc.IsResolvedHTTPAddressCheckEnabled(), qt.IsFalse, qt.Commentf(urls))+ c.Assert(pc.CheckAllowedHTTPAddress("tcp", "127.0.0.1:1313"), qt.IsNil, qt.Commentf(urls))+ }+}+ func TestCheckAllowedHTTPURLAtInPathIssue14825(t *testing.T) { t.Parallel() c := qt.New(t)docs/content/en/configuration/security.mdindex 9efbc0148..4fc4f6179 100644@@ -35,7 +35,7 @@ This is the default security configuration: : (`[]string`) Applicable to the `resources.GetRemote` function, a slice of [regular expressions](g) matching the `Content-Type` in HTTP responses that Hugo trusts, bypassing file content analysis for media type detection. `http.urls`-: (`[]string`) A slice of [regular expressions](g) matching the URLs that the `resources.GetRemote` function is allowed to access.+: (`[]string`) A slice of [regular expressions](g) matching the URLs that the `resources.GetRemote` function is allowed to access. See [remote URLs and internal addresses](#remote-urls-and-internal-addresses) for how a custom value affects the resolved-address check. `node.permissions.disable` : {{< new-in 0.161.0 />}}@@ -76,6 +76,25 @@ urls = ['.*', '! ^https?://evil\.example\.com'] Setting an allowlist to the string `none` will completely disable the associated feature. +## Remote URLs and internal addresses++The `http.urls` allowlist only inspects the text of the URL. A hostname that looks harmless can still resolve to an internal address, for example a loopback address (`127.0.0.1`), a private network address (`10.0.0.1`), or the link-local cloud metadata endpoint (`169.254.169.254`).++With the default `http.urls` allowlist, Hugo therefore also validates the address that each hostname resolves to when connecting, and refuses to connect to loopback, private, link-local, and other non-public addresses. The build fails with an `access denied` error naming the resolved address.++If you set `http.urls` to anything other than the default, Hugo skips this resolved-address check, so that you can fetch from internal hosts you trust, such as a local development server or a service on your private network. Keep in mind that:++- The resolved-address check is disabled for every URL your allowlist accepts, not only for the hosts you intended to add.+- A permissive allowlist such as `['.*']`, or one composed entirely of deny rules such as the [negation example](#negation-rules) above, lets a hostname that resolves to an internal address through.+- Repeating the default patterns and adding your own still counts as a custom allowlist.++If you build untrusted content, or build on a machine with access to sensitive internal services (for example, a cloud CI runner with a metadata endpoint), keep the default allowlist or make your custom allowlist as narrow as possible, preferably listing only the exact hosts you need:++```toml+[security.http]+urls = ['^https://api\.example\.com/', '^http://localhost:1313/']+```+ ## Environment variables You can also override your project configuration with environment variables. For example, to block `resources.GetRemote` from accessing any URL:resources/resource_factories/create/remote_test.goindex f9830c842..d16233044 100644@@ -161,6 +161,7 @@ func TestSecureBaseTransportBlocksInternalAddress(t *testing.T) { _, err = newSecureBaseTransport(security.DefaultConfig).RoundTrip(req) c.Assert(err, qt.IsNotNil) c.Assert(security.IsAccessDenied(err), qt.IsTrue)+ c.Assert(err, qt.ErrorMatches, `(?s).*"127\.0\.0\.1" is not whitelisted in policy "security\.http\.urls"; the remote host resolved to "127\.0\.0\.1", which is a loopback, private, link-local or otherwise non-public address.*`) // Customized policy: the user has opted into their own hosts, so the dial // check stands down and the fetch succeeds.
config/security/securityConfig.goindex f7cc157b8..a2708288a 100644@@ -128,6 +128,12 @@ type HTTP struct { // URLs to allow in remote HTTP (resources.Get, getJSON, getCSV). URLs Whitelist `json:"urls"` + // Whether to honor the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment+ // variables for remote HTTP. Off by default, as a proxy hides the+ // resolved destination address from the security policy.+ // <docsmeta>{ "newIn": "0.166.0" }</docsmeta>+ ProxyFromEnvironment bool `json:"proxyFromEnvironment"`+ // HTTP methods to allow. Methods Whitelist `json:"methods"` config/security/securityConfig_test.goindex 4ac059a67..e6f8c68e3 100644@@ -135,7 +135,7 @@ func TestToTOML(t *testing.T) { got := DefaultConfig.ToTOML() c.Assert(got, qt.Equals,- "[security]\n allowContent = ['! ^text/html$', '! ^text/org$']\n enableInlineShortcodes = false\n\n [security.exec]\n allow = ['^(dart-)?sass$', '^go$', '^git$', '^node$', '^postcss$']\n osEnv = ['(?i)^((HTTPS?|NO)_PROXY|PATH(EXT)?|APPDATA|TE?MP|TERM|GO\\w+|(XDG_CONFIG_)?HOME|USERPROFILE|SSH_AUTH_SOCK|DISPLAY|LANG|SYSTEMDRIVE|PROGRAMDATA)$']\n\n [security.funcs]\n getenv = ['^HUGO_', '^CI$']\n\n [security.http]\n methods = ['(?i)GET|POST']\n urls = ['(?i)^https?://[a-z0-9]', '! (?i)^https?://\\d+\\.', '! (?i)localhost', '! (?i)^https?://[^/?#]*@']\n\n [security.node]\n [security.node.permissions]\n allowAddons = ['tailwindcss']\n allowChildProcess = ['tailwindcss']\n allowRead = ['.']\n allowWorker = ['tailwindcss']\n allowWrite = []\n disable = false",+ "[security]\n allowContent = ['! ^text/html$', '! ^text/org$']\n enableInlineShortcodes = false\n\n [security.exec]\n allow = ['^(dart-)?sass$', '^go$', '^git$', '^node$', '^postcss$']\n osEnv = ['(?i)^((HTTPS?|NO)_PROXY|PATH(EXT)?|APPDATA|TE?MP|TERM|GO\\w+|(XDG_CONFIG_)?HOME|USERPROFILE|SSH_AUTH_SOCK|DISPLAY|LANG|SYSTEMDRIVE|PROGRAMDATA)$']\n\n [security.funcs]\n getenv = ['^HUGO_', '^CI$']\n\n [security.http]\n methods = ['(?i)GET|POST']\n proxyFromEnvironment = false\n urls = ['(?i)^https?://[a-z0-9]', '! (?i)^https?://\\d+\\.', '! (?i)localhost', '! (?i)^https?://[^/?#]*@']\n\n [security.node]\n [security.node.permissions]\n allowAddons = ['tailwindcss']\n allowChildProcess = ['tailwindcss']\n allowRead = ['.']\n allowWorker = ['tailwindcss']\n allowWrite = []\n disable = false", ) } @@ -156,6 +156,7 @@ func TestDecodeConfigDefault(t *testing.T) { c.Assert(pc.HTTP.Methods.Accept("get"), qt.IsTrue) c.Assert(pc.HTTP.Methods.Accept("DELETE"), qt.IsFalse) c.Assert(pc.HTTP.MediaTypes.Accept("application/msword"), qt.IsFalse)+ c.Assert(pc.HTTP.ProxyFromEnvironment, qt.IsFalse) c.Assert(pc.Exec.OsEnv.Accept("PATH"), qt.IsTrue) c.Assert(pc.Exec.OsEnv.Accept("GOROOT"), qt.IsTrue)docs/content/en/configuration/security.mdindex 9efbc0148..54ebb4521 100644@@ -34,9 +34,15 @@ This is the default security configuration: `http.mediaTypes` : (`[]string`) Applicable to the `resources.GetRemote` function, a slice of [regular expressions](g) matching the `Content-Type` in HTTP responses that Hugo trusts, bypassing file content analysis for media type detection. +`http.proxyFromEnvironment`+: {{< new-in 0.166.0 />}}+: (`bool`) Whether the `resources.GetRemote` function honors the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables. Default is `false`. When a proxy is used, Hugo connects to the proxy rather than the destination, so the resolved address validation described under `http.urls` does not apply.+ `http.urls` : (`[]string`) A slice of [regular expressions](g) matching the URLs that the `resources.GetRemote` function is allowed to access. + The default allowlist denies URLs with an IP address or `localhost` as the host name. In addition, with the default allowlist, Hugo validates the resolved address when connecting and rejects connections to loopback, private, link-local, and other non-public addresses, e.g. a host name that resolves to a cloud metadata endpoint. This validation is disabled if you override `http.urls`, as the override may intentionally allow access to hosts on your local network, such as a development server.+ `node.permissions.disable` : {{< new-in 0.161.0 />}} : (`bool`) Whether to disable the Node.js [permission model][]. When `false`, Hugo runs Node.js tools with the `--permission` flag, restricting their file system and resource access to what is explicitly allowed below. Default is `false`.docs/data/docs.yamlindex b392c6a71..bf63fb6b0 100644@@ -2240,6 +2240,7 @@ config: mediaTypes: null methods: - (?i)GET|POST+ proxyFromEnvironment: false urls: - (?i)^https?://[a-z0-9] - '! (?i)^https?://\d+\.'resources/resource_factories/create/create.goindex a34f9fc7b..1570c2648 100644@@ -16,9 +16,11 @@ package create import (+ "context" "errors" "net" "net/http"+ "net/url" "os" "path" "path/filepath"@@ -26,6 +28,8 @@ import ( "syscall" "time" + "golang.org/x/net/http/httpproxy"+ "github.com/bep/helpers/contexthelpers" "github.com/bep/logg"@@ -65,6 +69,7 @@ type contextKey uint8 const ( contextKeyResourceID contextKey = iota+ contextKeyProxyURL ) // New creates a new Client with the given specification.@@ -154,17 +159,80 @@ func New(rs *resources.Spec) *Client { // hook that validates the resolved destination address against the security // policy, so a hostname resolving to an internal address cannot be used to // reach internal endpoints. See CheckAllowedHTTPAddress.+//+// Proxies are only used when security.http.proxyFromEnvironment is enabled.+// A proxied connection is dialed to the proxy, not the destination, so the+// address check is skipped for that dial; the user has opted into trusting+// the proxy with the destination. The proxy decision is carried per request+// in its context, so a direct dial is always validated, even to the proxy's+// own address. func newSecureBaseTransport(sec security.Config) http.RoundTripper { base := http.DefaultTransport.(*http.Transport).Clone()- d := &net.Dialer{+ secureDialer := &net.Dialer{ Timeout: 30 * time.Second, KeepAlive: 30 * time.Second, Control: func(network, address string, _ syscall.RawConn) error { return sec.CheckAllowedHTTPAddress(network, address) }, }- base.DialContext = d.DialContext- return base+ base.Proxy = nil+ base.DialContext = secureDialer.DialContext++ if !sec.HTTP.ProxyFromEnvironment {+ return base+ }++ proxyFromCtx := func(ctx context.Context) *url.URL {+ u, _ := ctx.Value(contextKeyProxyURL).(*url.URL)+ return u+ }+ base.Proxy = func(req *http.Request) (*url.URL, error) {+ return proxyFromCtx(req.Context()), nil+ }+ proxyDialer := &net.Dialer{Timeout: secureDialer.Timeout, KeepAlive: secureDialer.KeepAlive}+ base.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {+ if u := proxyFromCtx(ctx); u != nil && canonicalAddr(u) == addr {+ return proxyDialer.DialContext(ctx, network, addr)+ }+ return secureDialer.DialContext(ctx, network, addr)+ }+ return &proxyTransport{base: base, proxyFunc: httpproxy.FromEnvironment().ProxyFunc()}+}++// proxyTransport resolves the proxy for each request up front and stores it+// in the request context, where both the Proxy and DialContext hooks of base+// can see it.+type proxyTransport struct {+ base *http.Transport+ proxyFunc func(*url.URL) (*url.URL, error)+}++func (t *proxyTransport) RoundTrip(req *http.Request) (*http.Response, error) {+ u, err := t.proxyFunc(req.URL)+ if err != nil {+ return nil, err+ }+ if u != nil {+ req = req.WithContext(context.WithValue(req.Context(), contextKeyProxyURL, u))+ }+ return t.base.RoundTrip(req)+}++// canonicalAddr mirrors net/http's canonicalAddr: the host:port the transport+// dials for u, with the port defaulted from the scheme.+func canonicalAddr(u *url.URL) string {+ port := u.Port()+ if port == "" {+ switch u.Scheme {+ case "http":+ port = "80"+ case "https":+ port = "443"+ case "socks5", "socks5h":+ port = "1080"+ }+ }+ return net.JoinHostPort(u.Hostname(), port) } // Copy copies r to the new targetPath.resources/resource_factories/create/remote_test.goindex f9830c842..d7bb98361 100644@@ -14,8 +14,10 @@ package create import (+ "io" "net/http" "net/http/httptest"+ "sync/atomic" "testing" qt "github.com/frankban/quicktest"@@ -173,3 +175,78 @@ urls = ['.*'] c.Assert(err, qt.IsNil) resp.Body.Close() }++// Proxies from the environment are ignored unless+// security.http.proxyFromEnvironment is enabled. When enabled, the dial-time+// address check does not apply to the proxy itself. See issue 15302.+func TestSecureBaseTransportProxyFromEnvironment(t *testing.T) {+ c := qt.New(t)++ var proxyHits, targetHits atomic.Int32+ proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ proxyHits.Add(1)+ w.Write([]byte("via proxy"))+ }))+ t.Cleanup(proxy.Close)+ target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ targetHits.Add(1)+ w.Write([]byte("direct"))+ }))+ t.Cleanup(target.Close)++ t.Setenv("HTTP_PROXY", proxy.URL)+ t.Setenv("NO_PROXY", "")+ t.Setenv("no_proxy", "")++ decode := func(toml string) security.Config {+ sec, err := security.DecodeConfig(config.FromTOMLConfigString(toml))+ c.Assert(err, qt.IsNil)+ return sec+ }+ fetch := func(sec security.Config, u string) (string, error) {+ req, err := http.NewRequest("GET", u, nil)+ c.Assert(err, qt.IsNil)+ resp, err := newSecureBaseTransport(sec).RoundTrip(req)+ if err != nil {+ return "", err+ }+ defer resp.Body.Close()+ b, err := io.ReadAll(resp.Body)+ c.Assert(err, qt.IsNil)+ return string(b), nil+ }++ // Default: proxy ignored, fetched directly.+ body, err := fetch(decode(`+[security.http]+urls = ['.*']+`), target.URL)+ c.Assert(err, qt.IsNil)+ c.Assert(body, qt.Equals, "direct")+ c.Assert(proxyHits.Load(), qt.Equals, int32(0))+ c.Assert(targetHits.Load(), qt.Equals, int32(1))++ // Enabled: the loopback proxy is dialed even under the default hardened+ // policy, and the destination is never dialed directly.+ body, err = fetch(decode(`+[security.http]+proxyFromEnvironment = true+`), "http://example.com/")+ c.Assert(err, qt.IsNil)+ c.Assert(body, qt.Equals, "via proxy")+ c.Assert(proxyHits.Load(), qt.Equals, int32(1))+ c.Assert(targetHits.Load(), qt.Equals, int32(1))++ // Enabled, but loopback destinations are never proxied: the address+ // check still applies to the direct dial, also when the destination is+ // the proxy's own address.+ for _, u := range []string{target.URL, proxy.URL} {+ _, err = fetch(decode(`+[security.http]+proxyFromEnvironment = true+`), u)+ c.Assert(security.IsAccessDenied(err), qt.IsTrue, qt.Commentf(u))+ }+ c.Assert(proxyHits.Load(), qt.Equals, int32(1))+ c.Assert(targetHits.Load(), qt.Equals, int32(1))+}