Skip to content

Debugging

Jump to bottom
niksis02 edited this page Sep 4, 2026 · 2 revisions

Debug

Debug logging is controlled by the --log-level option:

   --log-level value    debug logger verbosity: "silent" (default, no debug output), "debug" (full request/response logging with secrets and tokens masked), or "unsafe" (full logging with NO masking -- prints access keys, secrets, session tokens, and signatures in the clear; only use for local troubleshooting, never in production) (default: "silent") [$VGW_LOG_LEVEL]
   --debug              enable debug output (deprecated: use --log-level=debug for finer-grained control) (default: false) [$VGW_DEBUG]

--log-level is a global option, so on the command line it must appear before the subcommand; placing it after fails with Incorrect Usage: flag provided but not defined: -log-level followed by the subcommand's help text. VGW_LOG_LEVEL works regardless of argument position.

./versitygw --log-level debug --access user --secret pass iam --dir /var/lib/vgw-iam

The value is case-insensitive and surrounding whitespace is trimmed; an unrecognized value is fatal at startup with invalid log level "<value>": must be one of 'silent', 'debug', 'unsafe'. The level is fixed at startup — there is no SIGHUP or runtime toggle — and when it is silent the debug middleware is never installed, so it cannot be enabled later either.

--debug / VGW_DEBUG still works but is deprecated in favour of --log-level. Whenever --debug is set, WARNING: --debug is deprecated; use --log-level=debug for finer-grained control over debug logging is printed to stderr without a timestamp, even if --log-level is also set. It only raises the level while --log-level is still silent, so --log-level=unsafe --debug stays unsafe.

Log levels

Level Meaning
silent Default. No debug output at all.
debug Full request/response logging, with secrets and tokens masked.
unsafe Full request/response logging with no masking, plus the SigV4 canonical request and string-to-sign.

Warning

A log file produced at --log-level=unsafe is itself a credential: anyone who can read it can replay or reuse the keys, tokens and signatures it contains. Local troubleshooting only.

What each level logs

silent installs no debug middleware. The following are printed regardless of level, because they are not gated by the debug logger:

  • [PANIC]: <err> — in red on stderr, when a panic was recovered.
  • [INTERNAL ERROR]: <err> — in red on stderr, for any internal failure that is not a normal API error. This is the one line that still appears in a fully silent deployment, and it is the first thing to look for.
  • The per-request access log line and the startup banner, unless --quiet.

debug additionally prints, to stdout, a boxed 120-column rendering of every request and response:

  • [URL]: <scheme>://<host><path>?<query> — the query string is redacted, so credentials passed as query parameters are not printed.
  • REQUEST HEADERS, REQUEST BODY, RESPONSE HEADERS and RESPONSE BODY blocks, with header values redacted and bodies masked as described below.
  • Each query argument is also written to stderr through the standard library logger, so those lines carry a timestamp: 2026/08/28 15:22:14 Action: ListUsers.
  • Internal [DEBUG]: diagnostics, in yellow — including the IAM policy-evaluation and identity-resolution messages (IAM API Reference, Policy Conditions).

unsafe does everything debug does, with redaction disabled, and adds material that is not printed at debug at all: the SigV4 canonical request and string-to-sign, as ---[ CANONICAL STRING ]--- and ---[ STRING TO SIGN ]--- sections under [DEBUG]: Request Signature:. For presigned (query-string) authentication a third section, ---[ SIGNED QUERY ]---, carries the complete, replayable signed query string. Troubleshooting shows this output with real values and how to read it against a SignatureDoesNotMatch.

Debug output goes to stdout, except the per-query-argument lines and the [INTERNAL ERROR]/[PANIC] lines, which go to stderr. It is written unbuffered from the request goroutine with no rate limiting or sampling, and it includes the full body of every request and response.

Masking

At debug the values below are replaced with ****; unsafe disables the replacement entirely.

Header, query-parameter and form-field names are matched case-insensitively against a fixed list:

authorization, x-amz-security-token, webidentitytoken, x-amz-signature, x-amz-credential, x-amz-server-side-encryption-customer-key, x-amz-copy-source-server-side-encryption-customer-key.

The paired …-key-md5 headers are deliberately not masked — they are only a checksum of the key, and they are useful for correlating requests.

Query-string redaction in the access log is always on and is independent of --log-level: a credential that appears in a presigned URL is redacted in the access log even at silent, and unsafe does not turn that off. The opt-out applies only to the debug logger.

XML bodies are parsed into a tree and re-rendered with the sensitive values replaced, rather than being dumped raw. Matching is on the element's local name, case-sensitively, at any nesting depth, and it applies to attribute values as well as element text.

Element / attribute name Treatment
SecretAccessKey Replaced entirely with ****
SessionToken Replaced entirely with ****
WebIdentityToken Replaced entirely with ****
AccessKeyId First 4 characters kept, remainder replaced: AKIAabcdefghijklmnop becomes AKIA****. A value of 4 characters or fewer is masked in full, so ASIA becomes ****.

A body that is empty, does not start with <, or is not well-formed XML is printed unchanged. The request body is still skipped wholesale for PutObject, UploadPart and POST-object uploads, and the response body for GetObject; everything else is masked property by property.

A form-encoded body — which is every AWS Query protocol POST, including all IAM and STS calls — is logged as one redacted key=value line per field rather than as a URL-encoded blob, so an AssumeRoleWithWebIdentity POST logs WebIdentityToken=****.

Debug logging for the standalone IAM service

--pprof applies to versitygw iam as well, starting the profiling server described below and logging pprof: listening on <addr>. A few further specifics:

  • The health endpoint configured by --health is registered ahead of the debug middleware, so requests to it produce no debug output at all. A health-checked service therefore does not flood the log.
  • The private endpoint (the mTLS/unix-socket control channel the S3 gateway uses) runs as a separate application with no debug middleware. Private calls are visible only through its access log.
  • At unsafe, the S3 gateway also prints the canonical request and string-to-sign for the private requests it signs, since both processes share one SigV4 implementation.
  • --iam-debug has no effect on versitygw iam, nor on the standalone IAM backend in the gateway. It only adds request/response tracing to the gateway's LDAP and S3 IAM backends, and produces nothing at all unless --log-level is non-silent — see Multi-Tenant.

Profiling

Profiling can be enabled while running in production, but it is recommended to only listen on localhost or private networks and not expose this endpoint to external networks.

Enabling profiling:

   --pprof value                           enable pprof debug on specified port [$VGW_PPROF]

The pprof option enables the pprof HTTP server for profiling the gateway while the process is running. See the following for more information:
https://pkg.go.dev/net/http/pprof
To enable, set the pprof option to the listening address for the pprof service. For example, to listen on localhost port 6060, set the option to "localhost:6060".

Assuming the prrof service is running on localhost:6060, here are some useful client commands for profiling.

Get stack traces for all goroutines:

curl 'http://localhost:6060/debug/pprof/goroutine?debug=1'

or

curl 'http://localhost:6060/debug/pprof/goroutine?debug=2'

Get detailed memory heap stats:

curl 'http://localhost:6060/debug/pprof/heap?debug=1'

Format for heap allocations is in https://go.dev/src/runtime/pprof/pprof.go

    	fmt.Fprintf(w, "heap profile: %d: %d [%d: %d] @ heap/%d\n",
    		total.InUseObjects(), inUseBytes,
    		total.AllocObjects, allocBytes,
    		rate)
    
    	for i := range p {
    		r := &p[i]
    		fmt.Fprintf(w, "%d: %d [%d: %d] @",
    			r.InUseObjects(), r.InUseBytes(),
    			r.AllocObjects, r.AllocBytes)
    		for _, pc := range r.Stack() {
    			fmt.Fprintf(w, " %#x", pc)
    		}
    		fmt.Fprintf(w, "\n")
    		printStackRecord(w, r.Stack(), false)
    	}

Grab and display a 30 second CPU profile on a local webserver on port 9999:

go tool pprof -http :9999 'http://localhost:6060/debug/pprof/profile?seconds=30'

Execution traces

curl -o trace.out 'http://localhost:6060/debug/pprof/trace?debug=1'
go tool trace trace.out

See this link for a good writeup on the execution tracer use: https://sourcegraph.com/blog/go/an-introduction-to-go-tool-trace-rhys-hiltner

Interactive Browser

Point a browser to http://localhost:6060/debug/pprof to see links for the options:

Profile Descriptions:

allocs:       A sampling of all past memory allocations
block:        Stack traces that led to blocking on synchronization primitives
cmdline:      The command line invocation of the current program
goroutine:    Stack traces of all current goroutines.
heap:         A sampling of memory allocations of live objects.
mutex:        Stack traces of holders of contended mutexes
profile:      CPU profile. Use the go tool pprof command to investigate the profile
threadcreate: Stack traces that led to the creation of new OS threads
trace:        A trace of execution of the current program. Use the go tool trace command to investigate the trace.

See Also

Versity Software logo image.

@versitysoftware

linkedin   twitter   facebook   instagram  

Clone this wiki locally