I Self-Hosted SigNoz to Debug My Trading Bot, and One Feature Ruined Every Other Logging Setup For Me

I'll be honest, when I signed up for this sprint I thought "observability" was just a fancier word for "logs, but with a dashboard." I've spent enough night’s print()-debugging my side projects at 2 AM to think I had this whole monitoring thing figured out. Turns out I didn't even know what I was missing until I self-hosted SigNoz and it slapped me across the face with a feature I now can't stop thinking about.

Let me back up.

Why SigNoz, and why now?

I've been building a small algorithmic/paper-trading bot on the side - nothing that's going to make me rich, just something to test strategies against live market data without risking actual money. It's got a handful of moving parts: a market data listener (websocket feed), a strategy engine that crunches signals, an order execution service that talks to a broker's paper-trading API, and a tiny FastAPI layer that serves a dashboard so I can watch it think.

The problem is that "a handful of moving parts" is exactly the kind of setup where things quietly break in ways that are miserable to trace. An order would occasionally just... not go through. No crash, no obvious error, just silence. And my only tool to investigate was tail -f across three different terminal tabs, trying to eyeball which log lines happened around the same second in each service. It worked, technically, the same way squinting at a blurry photo "works."

So when this sprint asked us to self-host SigNoz and actually explore it instead of just installing it and forgetting about it, I figured - perfect excuse to fix my own mess. SigNoz is open-source, built natively on OpenTelemetry (so no vendor-specific agents baked into my code), and it throws traces, metrics, and logs into one tool instead of making you duct-tape Prometheus + Loki + Jaeger together like some kind of observability Frankenstein. Also, not gonna lie, "open-source Datadog alternative" that I can run on my own machine for free as a broke student was doing a lot of the convincing.

Getting it running

I went with the Docker Compose route since I just wanted it on my own VPS without overengineering anything. A few honest notes from actually doing it, not just reading the docs:

• RAM matters more than you'd think. ClickHouse (the database SigNoz stores everything in) wants at least 4GB just to breathe. I initially tried this on a tiny 2GB box out of stubbornness and watched containers restart-loop into oblivion. Bumped it up and everything calmed down immediately.
• Bring up the stack, wait for ClickHouse to finish its schema migration (this takes a few minutes and feels like it's stuck even when it isn't), then hit port 8080 for the UI. First login makes you set up an admin account, and you're in.
• Ports 4317 (gRPC) and 4318 (HTTP) are where the OpenTelemetry Collector listens for incoming data. Those are the ones you actually need open to your apps.

Wiring my bot in was less painful than I expected. Since SigNoz speaks OpenTelemetry natively, I didn't need any SigNoz-specific SDK - just the standard OTel Python packages, pointed at my collector endpoint via an environment variable:

OTEL_EXPORTER_OTLP_ENDPOINT=http://<my-signoz-host>:4317

OTEL_SERVICE_NAME=order-execution-service

Auto-instrumentation picked up my FastAPI routes and outgoing HTTP calls without me writing a single tracing line by hand. For logs, I hooked up Python's logging module through the OTel logging instrumentation so every log line got tagged with trace and span IDs automatically. That one small setup step is the reason the rest of this blog post exists, so keep it in mind.

Poking around: what each tab actually gave me

Once data started flowing, I did what I assume everyone does - clicked every tab like a kid in a candy store.

Traces showed me the full waterfall of a request: market tick comes in, strategy engine evaluates it, order gets placed, broker API responds. Actual millisecond-level breakdown of where time went. I found out my "instant" order execution was quietly spending 400ms just waiting on a DNS lookup I never noticed because it wasn't hitting my terminal logs.

Metrics gave me the boring-but-essential stuff - request rates, error rates, latency percentiles - without me having to hand-roll a single Prometheus query. There's a query builder that's actually usable for someone who has only a passing relationship with PromQL.

Logs were searchable and filterable in a way that made my old grep-across-tabs method look prehistoric. Filter by severity, by service, by any attribute I'd bothered to log.

Dashboards let me build a little "mission control" panel for my bot: orders placed per minute, error rate on the broker API, p99 latency on the strategy engine. Took maybe twenty minutes to get something genuinely useful looking.

Alerts were where I set up a Slack webhook so that if my error rate on the order execution service spiked, I'd get pinged instead of finding out three hours later that my bot had been silently failing since lunch. Small thing, but it's the difference between "monitoring" and "actually being on-call for your own side project."

All of this was solid. Genuinely solid, better than I expected from something I could self-host for free. But none of it is the thing I want to talk about.

The feature that actually made me sit up: jumping from a trace straight into the exact log line that caused it

Here's the scenario that sold me. My order execution service started throwing intermittent failures -maybe 1 in 20 orders - with no pattern I could see just by staring at metrics. A latency spike here, a failure there, nothing that screamed "here's your bug."

In my old setup, this is the part where I'd lose an evening. I'd note the rough timestamp of a failed order from one log file, then go hunting through a different log file from the strategy engine, trying to match timestamps by eye, hoping my system clocks across services actually agreed with each other (narrator: they did not, always). By the time I'd stitched together what happened across three services, I'd usually forgotten what I was even looking for.

In SigNoz, I opened the trace for one of the failed orders. Waterfall view, all the spans laid out - market data fetch, strategy evaluation, order placement, broker response. I could see exactly which span had the error flag lit up red. And instead of then going to hunt for the relevant log line manually, I just clicked into that span's details panel, hit the Logs tab, and there it was - the exact log line that fired during that exact span, no searching, no timestamp math, no cross-referencing. It's connected automatically because the trace ID and span ID get injected straight into every log line emitted while that span is active, so SigNoz can literally draw a line between "this thing happened" and "here's the evidence for why."

That log line told me everything: a ConnectionResetError on the broker's paper-trading endpoint, happening specifically when two orders landed within the same 100ms window - a tiny race condition in how I was reusing a single HTTP session across concurrent requests. I would not have found that by squinting at metrics dashboards. I would have found it eventually by grepping, sure, but "eventually" was doing a lot of heavy lifting in my old workflow, and this took about ninety seconds.

And it goes both ways - I can also start from a log line and jump straight to the full trace it belongs to, which came in handy when I was scanning error logs first and wanted the bigger picture of what request caused them. Later I even used the metrics-to-logs correlation on a dashboard panel, right-clicking a latency spike and pulling up the exact logs from that time window without rebuilding a single query.

What gets me about this feature isn't that it's flashy. It's that it quietly kills the single most annoying part of debugging distributed systems: the manual stitching. Every observability tool I'd used before - or cobbled together - treated traces, logs, and metrics as three separate universes that happened to be looking at the same app. You'd open one tool, squint at a timestamp, alt-tab to another tool, squint again, and hope you were even looking at the right five-second window. SigNoz treats them as three views of the same event, because under the hood they're literally stored and linked together instead of living in three different databases with three different query languages. That's not a UI convenience, that's a fundamentally different (and better) way of thinking about what "one platform" should mean.

For a solo student project, this matters even more than it would for some giant engineering team with a dedicated SRE. I don't have the bandwidth to be a full-time detective every time something in my bot goes sideways. I have homework. I have a life, occasionally. This feature gave me back the hours I would've burned manually correlating logs, and honestly, that's the entire pitch for self-hosted observability tooling when you're a student instead of a company: your time is the resource you can least afford to waste, not your wallet.

Final thoughts

Would I have found SigNoz's other features useful on their own? Sure - the dashboards are genuinely nice to look at, the alerting saved me from an embarrassing silent failure, and having everything under one Apache-2.0-licensed, self-hostable roof means I'm not locked into anyone's pricing page as my side project (hypothetically, someday) grows. But if you asked me to pick one single reason I'd recommend SigNoz to another student debugging their own multi-service project at midnight, it's this: the moment something breaks, you are never more than one click away from the exact evidence of why. Not "roughly when." Not "somewhere around here." The actual line.

That race condition is fixed now, by the way. My orders go through cleanly. And somewhere in my bot's trace history there's a little green span that used to be red, which is a weirdly satisfying thing to look at for something I built in my dorm room.

If you're doing any kind of side project with more than one moving part - trading bot, Discord bot with a backend, whatever - I'd genuinely say wire up SigNoz before you need it, not after your third 2 AM debugging session convinces you that you do.