Structured JSON Logging: A Small SaaS Guide

Structured JSON logging turns your app’s plain-text log lines into machine-readable key-value pairs, making it far easier to search, filter, and analyze logs in modern tools like ELK, Grafana, or Datadog. This guide walks you through adding it to a small SaaS app, covering the setup, common pitfalls, and troubleshooting.

What You’ll Need

• A small SaaS app (any language/framework) with existing logging.
• Access to your app’s codebase and deployment environment.
• A log aggregation service (or at least a way to view raw logs, e.g., Docker logs, Heroku logs, or a simple file).
• Basic familiarity with your language’s logging library.

Step-by-Step

1. Choose a JSON logging library. Most ecosystems have one: for Node.js use pino or winston with a JSON formatter; for Python use python-json-logger; for Go use logrus or zap with JSON encoding; for Ruby use lograge or Ougai. Pick one that’s well-maintained and supports contextual fields.
2. Configure the logger to output JSON. Replace your default logger with the JSON-capable one. For example, with pino: const logger = require('pino')(); – it outputs JSON by default. With Python’s logging, set a JSON formatter on your handlers. Ensure the output goes to stdout (or the same destination as before) so your existing log pipeline still works.
3. Add standard fields to every log. Include at minimum: timestamp, level, message, and a service name. Many libraries add these automatically. If not, create a wrapper that injects them.
4. Incorporate request context. For web requests, add a request ID (generated per request) and include it in all logs within that request. Also log method, path, status code, and duration. This is crucial for tracing issues across multiple services.
5. Add business-specific fields. For a SaaS app, log user ID (if authenticated), tenant/account ID, feature flags, and any relevant domain data. Avoid logging sensitive data like passwords or tokens.
6. Use structured logging calls. Instead of string interpolation, pass fields as a separate object. For example, with pino: logger.info({ userId, action: 'login' }, 'user logged in'). This keeps the message clean and the fields searchable.
7. Handle errors properly. Log errors with a stack trace and error code. Most JSON loggers support an err field that serializes the error object. Ensure you log the error object itself, not just the message.
8. Update your log viewer/aggregator. If you use a service like ELK or Grafana Loki, ensure it parses JSON logs automatically. If not, configure the parser to read JSON. For simple setups, you can use jq to pretty-print JSON logs from the command line.
9. Test locally. Run your app and verify logs are valid JSON. Use a tool like jq to parse and pretty-print them. Check that all expected fields appear.
10. Deploy and monitor. After deploying, check that logs are flowing and searchable. Set up a simple dashboard or alert for error rates to see the benefit immediately.

Common Mistakes

• Logging non-serializable objects (e.g., circular references) – this can crash your logger or produce invalid JSON. Always sanitize or use a replacer function.
• Mixing JSON and plain text logs – if some logs are still plain text, your aggregator may fail to parse them. Ensure all log output goes through the JSON logger.
• Forgetting to include a timestamp – many loggers add it, but if you use a custom formatter, you might miss it. Timestamps are essential for sorting and correlating.
• Logging sensitive data – user emails, tokens, or payment info in logs can cause security and compliance issues. Always redact or omit.
• Over-logging – logging every variable at debug level can bloat storage and make it hard to find relevant lines. Use appropriate levels (debug, info, warn, error).

Tips

• Use a unique request ID for each incoming request. Generate it at the entry point and pass it through all function calls (or use async context).
• Log in context – include the user ID and tenant ID in every log that involves user actions. This makes multi-tenant debugging much easier.
• Keep messages short and static. The dynamic parts should be in fields, not in the message string. This allows better aggregation and searching.
• Use log levels consistently. Reserve error for exceptions and critical failures, warn for potential issues, info for normal operations, and debug for detailed troubleshooting.
• Consider adding a correlation ID that spans across multiple services (e.g., an external API call). This is more advanced but invaluable for microservices.

Troubleshooting

• Logs are not appearing at all. Check that your logger is configured to output to stdout/stderr and that your logging level isn’t set too high (e.g., only errors). Verify the app is actually running the new code.
• Logs are malformed JSON. Look for stray characters like commas or quotes. This often happens when you mix string concatenation with JSON objects. Use a JSON validator to find the issue.
• Logs are truncated. Some systems truncate long lines. If you have very long messages or stack traces, consider splitting them or increasing the line limit in your log shipper.
• Aggregator shows logs as plain text. Ensure your log shipper (e.g., Filebeat, Fluentd) is configured to parse JSON. Sometimes you need to set a specific parser or format.
• Performance degradation. JSON logging can be slower than plain text. If you notice a hit, consider using a faster logger (like pino) or reducing log volume. Also, avoid logging in tight loops.
• Missing fields. If you expect a field but it’s not there, check that you’re passing it correctly. For example, in pino, if you pass undefined as a value, it may be omitted. Use null instead.

Frequently Asked Questions

Do I need to change my log storage?

Not necessarily. Most log storage systems (like Elasticsearch, Loki, or even plain files) can handle JSON. You may need to adjust the parsing configuration to extract fields automatically.

Can I keep my existing log viewer?

Yes, if it supports JSON. Many tools like Kibana, Grafana, and even simple grep can work with JSON logs, though you’ll get more benefit from a tool that indexes fields.

How do I log a stack trace in JSON?

Use the error object directly. Most JSON loggers have a built-in serializer for errors that includes the stack trace. For example, in pino, just pass the error object as a field.

What should I do with logs that are not from my app (e.g., nginx)?

You can leave them as plain text, but for consistency, you could configure nginx to output JSON as well. Many log shippers can handle mixed formats, but it’s easier to have everything JSON.

Is it worth the effort for a small app?

Yes, especially if you’re already using a log aggregator. Even with a small app, structured logs save time when debugging production issues because you can filter by user, request, or error type instantly.

How do I redact sensitive data?

Most loggers allow you to define redaction patterns. For example, pino has a redact option where you can specify paths to redact. Alternatively, you can sanitize data before logging.

Related

• Diversify Beyond the S&P 500 with European ETFs
• Run Control Panel as Administrator in Windows
• Disable Voice Message Recording in WhatsApp on iPhone