Noveo developer Andrei talks about TypeLog, his custom-made solution for structuring logs and keeping them clean, context-rich and useful.
Intro
The majority of modern log management and analytics systems can parse JSON data right out of the box. Therefore, it’s important to define a log style, or structure, that could be easily ‘json-able’, and as a result, parsable without any issue.
As an alternative, one could use regular expressions for each new entry and parse their logs that way – but in reality, it’s a very limited option (and honestly, quite an unfit and un-fun one at that). It also cannot be considered as a best practice.
So, log structuring it is. There are a lot of tools that can do it for you, but they don’t always work as consistently as one would like, and their output isn’t 100% type-safe. In the end, using these solutions may turn out to be more harmful than useful.
When it comes to Golang and Python (the languages I’m working with), structured logging is now part of the official documentation for the latter, as well as part of the standard library for the former. Golang managed to become quite a ‘comfortable’ language to use out of the box, but it’s still lacking in certain areas… such as type safety – for example, one can easily insert Any objects into their logging messages. In this sense, Python is in an even rougher state, and it’s quite difficult to do structured logging in a type-safe way there.
Solution
I’ve designed TypeLog to combine some of the best parts of the already-existing structured logging solutions, and improve upon them by getting an additional benefit of type safety. For instance, there’s no need to worry about having Any objects anymore – I made sure to remove as many occurrences of the Any type as possible from the public interface parts.
Go version
As you can see below, if you try to insert invalid data into logs, you will get an error before run-time:
package types
type TaskID string
type WorkerID int
package typedlogs
import (
"log/slog"
"github.com/darklab8/go-typelog/examples/types"
"github.com/darklab8/go-typelog/typelog"
)
func TaskID(value types.TaskID) typelog.LogType { return typelog.String("task_id", string(value)) }
func WorkerID(value types.WorkerID) typelog.LogType {
return typelog.Int("worker_id", int(value))
}
package examples
var logger *typelog.Logger = typelog.NewLogger("worker")
func TestTypedLogs(t *testing.T) {
worker_id := types.WorkerID(5)
logger.Info("Worker was started", typedlogs.WorkerID(worker_id))
logger.Info("Worker was started", 123123, "asdasd") // TYPING ERROR
logger := logger.Log.WithFields(typedlogs.WorkerID(worker_id), typedlogs.TaskID("abc"))
logger.Info("Worker started task")
logger.Info("Worker finished task")
}
Python version
In order to use TypeLog to its full potential in Python, you will need to turn on Mypy (or Pyright), preferably in strict mode. Py-TypeLog comes with Mypy typelog-stubs as part of its package.
# types.py
from dataclasses import dataclass
from typing import NewType
TaskID = NewType("TaskID", int)
@dataclass(frozen=True)
class Task:
smth: str
b: int
# logtypes.py
from typing import Any, Dict
from typelog import LogType
from . import types
def TaskID(value: types.TaskID) -> LogType:
def wrapper(params: Dict[str, Any]) -> None:
params["task_id"] = str(value)
return wrapper
def Task(value: types.Task) -> LogType:
def wrapper(params: Dict[str, Any]) -> None:
params.update(value.__dict__)
return wrapper
import logging
import unittest
import typelog
from typelog import LogConfig, Loggers, get_logger
from typelog.types import LibName, LogLevel, RootLogLevel
from . import logtypes, types
logger = get_logger(__name__)
class TestExamples(unittest.TestCase):
def setUp(self) -> None:
Loggers(
RootLogLevel(logging.DEBUG),
LogConfig(LibName("examples"), LogLevel(logging.DEBUG)),
add_time=True,
).configure()
def test_basic(self) -> None:
logger.warn("Writing something", logtypes.TaskID(types.TaskID(123)))
def test_another_one(self) -> None:
task = types.Task(smth="abc", b=4)
logger.warn("Writing something", logtypes.Task(task))
logger.warn("Writing something", 123123, "324234") # TYPING ERROR
What are the advantages of using TypeLog?
- Your work is type-safe and protected from logging mistakes.
- Log record parsing is decoupled from the code you’re working on.
- Consistency: keys for variables are kept the same throughout the project.
- It’s easier to introduce various changes to your logs if needed: refactor or rename logs, modify keys and parsing rules across the whole application – in one click or with just a few lines of code.
- It’s easier to refactor the code in general, improving the ubiquitous language defined across your application.
- You will be able to parse your JSON logs using any log management and analytics solution of your choice. And on top of that, your logs will be much clearer and more useful, since all of the necessary identifiers are now consistently present in your code.
Final remarks
When you tap into the power of context-rich logging, you obtain quite an effective tool for performing program debugging at run-time. It may not seem as much, but it can actually prove to be really useful since some of the issues can only be detected at run-time, in production. Having clear, meaningful logs in such cases equals having a reliable source of data that can help to investigate the problem more fully. Plus, it can also come in handy during the process of unit test debugging – especially if you turn on warnings for your third-party libraries and switch your logging level to ‘Debug’ during test runs.
