Rule backend panics

[pahaeanx]

Hi,

I was trying the rule backend for some advanced routing but only got panics on any query. I thought I might be doing something wrong and decided to try the example from the rules page but that unfortunately also panics.

Seems there's currently some common issue to the rule backend?

Output:

time=2025-07-04T12:27:01.93910506Z app=trickster level=info event="registering route paths" backendName=example-reader-cluster backendProvider=rpc upstreamHost=reader-cluster.example.com
time=2025-07-04T12:27:01.93913968Z app=trickster level=info event="registering route paths" backendName=example backendProvider=rule upstreamHost=
time=2025-07-04T12:27:01.939109798Z app=trickster level=info event="application loaded from configuration" buildTime=2025-06-30T13:29:55+0000 commitID=48f761e8244a0eeac0658cca4bdf6c6ae75d63b6 config=/trickster.conf goArch=amd64 goOS=linux goVersion=go1.24.4 logLevel=debug name=trickster pid=1 version=
time=2025-07-04T12:27:01.939167327Z app=trickster level=debug event="registering health handler path" backendName=example-writer-cluster path=/trickster/health/example-writer-cluster upstreamPath=/ upstreamVerb=GET
time=2025-07-04T12:27:01.939156071Z app=trickster level=debug event="registering backend handler path" backendHost= backendName=example frontendHosts= handledPath=/example/ handlerName=rule matchType=prefix path=/GET-HEAD
time=2025-07-04T12:27:01.939161526Z app=trickster level=debug event="registering backend handler path" backendHost=writer-cluster.example.com backendName=example-writer-cluster frontendHosts= handledPath=/example-writer-cluster/ handlerName=proxycache matchType=prefix path=/-GET-HEAD
time=2025-07-04T12:27:01.939181349Z app=trickster level=debug event="registering backend handler path" backendHost=reader-cluster.example.com backendName=example-reader-cluster frontendHosts= handledPath=/example-reader-cluster/ handlerName=proxy matchType=prefix path=/-POST-PUT-DELETE-CONNECT-OPTIONS-TRACE-PATCH-PURGE
time=2025-07-04T12:27:01.939179208Z app=trickster level=debug event="registering backend handler path" backendHost=writer-cluster.example.com backendName=example-writer-cluster frontendHosts= handledPath=/example-writer-cluster/ handlerName=proxy matchType=prefix path=/-POST-PUT-DELETE-CONNECT-OPTIONS-TRACE-PATCH-PURGE
time=2025-07-04T12:27:01.939189929Z app=trickster level=debug event="registering backend handler path" backendHost=reader-cluster.example.com backendName=example-reader-cluster frontendHosts= handledPath=/example-reader-cluster/ handlerName=proxycache matchType=prefix path=/-GET-HEAD
time=2025-07-04T12:27:01.939192243Z app=trickster level=info event="registering pprof /debug routes" routerName=metrics
time=2025-07-04T12:27:01.939186577Z app=trickster level=debug event="registering health handler path" backendName=example-reader-cluster path=/trickster/health/example-reader-cluster upstreamPath=/ upstreamVerb=GET
time=2025-07-04T12:27:01.939201538Z app=trickster level=info event="registering route paths" backendName=example-writer-cluster backendProvider=rpc upstreamHost=writer-cluster.example.com
time=2025-07-04T12:27:01.939205931Z app=trickster level=info event="registering pprof /debug routes" routerName=mgmt
time=2025-07-04T12:27:01.939464237Z app=trickster level=info event="http listener starting" address= listenerName=metricsListener port=8481
time=2025-07-04T12:27:01.939461434Z app=trickster level=debug event="starting proxy listener" address= connectionsLimit=0 port=8481 scheme=http
time=2025-07-04T12:27:01.939518678Z app=trickster level=info event="http listener starting" address= listenerName=httpListener port=8480
time=2025-07-04T12:27:01.939540373Z app=trickster level=debug event="starting proxy listener" address= connectionsLimit=0 port=8480 scheme=http
time=2025-07-04T12:27:01.939573986Z app=trickster level=info event="http listener starting" address=127.0.0.1 listenerName=mgmtListener port=8484
time=2025-07-04T12:27:01.939580887Z app=trickster level=debug event="starting proxy listener" address=127.0.0.1 connectionsLimit=0 port=8484 scheme=http
2025/07/04 12:27:05 http: panic serving 127.0.0.1:34302: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
net/http.(*conn).serve.func1()
   /usr/local/go/src/net/http/server.go:1947 +0xbe
panic({0x1060680?, 0x1d7c840?})
   /usr/local/go/src/runtime/panic.go:792 +0x132
github.com/trickstercache/trickster/v2/pkg/backends/rule.(*Client).Handler(0x114dbe8?, {0x7fd80d9c2338, 0xc00011c780}, 0xf66e92?)
   /go/src/github.com/trickstercache/trickster/pkg/backends/rule/handler.go:27 +0x1c
net/http.HandlerFunc.ServeHTTP(0x7fd80d9c2338?, {0x7fd80d9c2338?, 0xc00011c780?}, 0xc0001651f0?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/routing.RegisterPathRoutes.func1.Handler.1({0x7fd80d9c2338, 0xc00011c780}, 0xc0001651f0?)
   /go/src/github.com/trickstercache/trickster/pkg/util/middleware/bodyfilter/bodyfilter.go:32 +0x87
net/http.HandlerFunc.ServeHTTP(0x1d80d50?, {0x7fd80d9c2338?, 0xc00011c780?}, 0x11b4b40?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/routing.RegisterPathRoutes.func1.HandleCompression.5({0x147f6e0, 0xc00014cb10}, 0xc0001528c0)
   /go/src/github.com/trickstercache/trickster/pkg/encoding/handler/handler.go:60 +0x1dc
net/http.HandlerFunc.ServeHTTP(0xc000152780?, {0x147f6e0?, 0xc00014cb10?}, 0xc0004c0380?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/routing.RegisterPathRoutes.func1.WithResourcesContext.6({0x147f6e0, 0xc00014cb10}, 0xc000152780)
   /go/src/github.com/trickstercache/trickster/pkg/util/middleware/config_context.go:50 +0x24f
net/http.HandlerFunc.ServeHTTP(0x18?, {0x147f6e0?, 0xc00014cb10?}, 0xc000217ad8?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/routing.RegisterPathRoutes.func1.Decorate.9({0x147f860, 0xc0004c02a0}, 0xc000152780)
   /go/src/github.com/trickstercache/trickster/pkg/util/middleware/metrics.go:38 +0x116
net/http.HandlerFunc.ServeHTTP(0xc000540680?, {0x147f860?, 0xc0004c02a0?}, 0xc0005054b0?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/util/middleware.StripPathPrefix.func1({0x147f860?, 0xc0004c02a0?}, 0xc000152780?)
   /go/src/github.com/trickstercache/trickster/pkg/util/middleware/path.go:44 +0xf8
net/http.HandlerFunc.ServeHTTP(0xc0005054b0?, {0x147f860?, 0xc0004c02a0?}, 0xc0004dfb38?)
   /usr/local/go/src/net/http/server.go:2294 +0x29
github.com/trickstercache/trickster/v2/pkg/proxy/router/lm.(*lmRouter).ServeHTTP(0xc0004dfb88?, {0x147f860, 0xc0004c02a0}, 0xc000152780)
   /go/src/github.com/trickstercache/trickster/pkg/proxy/router/lm/lm.go:57 +0x142
github.com/trickstercache/trickster/v2/pkg/proxy/handlers/trickster/switcher.(*SwitchHandler).ServeHTTP(0x475eb9?, {0x147f860?, 0xc0004c02a0?}, 0xc0004dfb70?)
   /go/src/github.com/trickstercache/trickster/pkg/proxy/handlers/trickster/switcher/switch.go:43 +0x43
net/http.serverHandler.ServeHTTP({0xc00014c9c0?}, {0x147f860?, 0xc0004c02a0?}, 0x1?)
   /usr/local/go/src/net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc0004d4120, {0x14816e8, 0xc00014c750})
   /usr/local/go/src/net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 108
   /usr/local/go/src/net/http/server.go:3454 +0x485

All that's necessary to reproduce should be to start the most recent (i think?) image: trickstercache/trickster@sha256:9fdaadf37cf09941417111d33a1276f116b8a3976c17a3d765def35a3bfac54f with the config from the example-user-router rule example and then try a query.

In my case I had the initial test routes based around prometheus/victoriametrics and tried how it reacts to a curl 127.0.0.1:8480/example/api/v1/query_range but it seems to crash on any path, and from what I can tell, any rules.

[jranson]

Thanks so much for the detailed report. I"ll take a look at this and get it fixed up for you.

[crandles]

I'm able to reproduce this bug, fyi. Added a simple test case here: 375a660

[jranson]

The root cause is a refactor between Beta 1 and Beta 2 resulting in the population of backend clients in the clients variable occurring after the rules options are compiled instead of before. The fixes in #851 should also fix this one up when ready.

[jranson]

@pahaeanx we have this fixed up in #864 and should have it merged in over the next 24 hours.