Python Logging Handler for logging24

Installation

The logging24 python logging interface has three components:

  1. The logging24 python library.
  2. The qlg-fwd python application, to continuously forward logs from files and journals
  3. The qlg-importer python application, to ingest existing historic log files.

If you are logging from a python framework, or want to use custom python scripts to forward data to logging24, you only need to install the logging24 library. If you want to tail a log file or stream to the logging24 ingester, you need to install both, the library as well as the qlg-fwd forwarder.

Download one, or both wheels (from the next section) and use pip to install into your python environment of choice: 

python -m pip install logging24-*-py3-none-any.whl qlg_fwd-*-py3-any.whl

As stated before, for direct use of the logging24 python log hander handler only the first package is required: 

python -m pip install logging24-*-py3-none-any.whl

Download Links

As wheel installer (preferred):

As source distributions:

Basic Usage

qlg-fwd, the multi-arch forwarder

To forward data to the logging24 backend for you to start analysing your logs, you can use the python based forwarder qlg-fwd.

In order to forward /var/log/syslog with the write token ABCDEF12345 for example, you would: 

qlg-fwd tail -f /var/log/syslog ABCDEF12345

To load an access token from a file use the file:// syntax. 

qlg-fwd tail -f /var/log/syslog file:///path/to/token

Similarly to load an access token from an environment variable: 

qlg-fwd tail -f /var/log/syslog env://TOKEN_SYSLOG

When running on a systemd based system and with python-systemd available the following can be used to forward all nginx.service related logs. 

qlg-fwd journal -f "UNIT=nginx.service" file:///etc/logging/nginx.token

If the journal subcommand of qlg-fwd does not seem to be available. Please make sure that the python-systemd library is available in the runtime environment of qlg-fwd.

logging24, the python logging library

Here is a quick example on how to use the QuuxLogingHandler class, again using ABCDEF12345 as an example write token: 

import logging24 as qlg

# Create your QuuxLoggingHandler
qlg_handler = qlg.QuuxLoggingHandler("ingest.logging24.com",
                    "ABCDEF12345", level=logging.DEBUG)

# configure/create your logger as usual. e.g. use basicConfig()
logging.basicConfig()
log = logging.getLogger("demo")

# attach your handler
log.handlers.append(qlg_handler)

# now log everything to logging24
log.info("This line is logged with the write token specified above")

Write Token Handling

There are two ways to log messages to different logging contexts/write tokens.

Firstly, by creating a separate handler for each write token: 

# Create your QuuxLoggingHandler
qlg_handler1 = qlg.QuuxLoggingHandler("ingest.logging24.com",
                    "your write token", level=logging.DEBUG)
qlg_handler2 = qlg.QuuxLoggingHandler("ingest.logging24.com",
                    "another write token", level=logging.DEBUG)

# configure/create your logger as usual. e.g. use basicConfig()
logging.basicConfig()
log_demo = logging.getLogger("demo")
log_app = logging.getLogger("app")

# attach your handler to the logger
log_demo.handlers.append(qlg_handler1)
log_app.handlers.append(qlg_handler2)

# now log to logging24
log_demo.info("This line is logged with the first write token specified above.")
log_app.info("This is logged to the second logstream.")

Secondly, it is possible to decide on a per log record basis: 

# Create your QuuxLoggingHandler
qlg_handler1 = qlg.QuuxLoggingHandler("ingest.logging24.com",
                    "your write token", level=logging.DEBUG)

# configure/create your logger as usual. e.g. use basicConfig()
logging.basicConfig
log_demo = logging.getLogger("demo")

# attach your handler to the logger
log_demo.handlers.append(qlg_handler1)

# now log to logging24
log_demo.info("This line is logged with the first write token specified above.")
log_demo.info("This is logged to a second logstream.",
              extra={"qlg_token": "another log token"})

Note that the key for the token can be configured in the QuuxLoggingHandler constructor, by specifying the "token_key" keyword argument: 

qlg_handler1 = qlg.QuuxLoggingHandler("ingest.logging24.com",
            "your write token", level=logging.INFO, token_key="custom_token_key")

Log Timestamps

If nothing else is specified, the time of the logging call is used as the timestamp of the log line. In case it is necessary to ingest data that has different timestamps attached, or use timestamps from a custom source, this can easily be accomplished by specifying a custom timestamp in the log call.

The default key for timestamps is "qlg_timestamp", a logging call specifying this can look like: 

from datetime import datetime
my_time = datetime(2024, 1, 10, 12, 34)
log_demo.info("This line is logged at 12:34 on 2024-01-10.",
              extra={"qlg_timestamp": my_time})

The value of the qlg_timestamp key can be either:

Note that the key for the timestamp can be configured in the QuuxLoggingHandler constructor, by specifying the "timestamp_key" keyword argument: 

qlg_handler = qlg.QuuxLoggingHandler("ingest.logging24.com",
              "your write token", level=logging.INFO, timestamp_key="custom_timestamp_key")

A log using this key would look like this: 

log_demo.info("This line is logged at 12:34 on 2024-01-10.",
              extra={"custom_timestamp_key": my_time})