logging24
log everything · query anything
To authenticate a log forwarding agent, you must create a "write token". While doing so, you are prompted for prefix0-3 of the resulting log stream. For security reasons we never store tokens (only the their hashes). So we cannot show you the actual secret a second time. Please make sure to copy it somewhere safe (or create a new token if you lose it).
We offer several methods of forwarding logs to us. To achieve easy and versatile forwarding, we have
an implementations as a Linux x86-64 bit binary for high performance requirements,
a multi-architecture python logging handler and forwarder,
a network port for receiving logs from syslog+tls, and
a network port for receiving logs via the loki protocol.
This is the recommended method for high-performance forwarding of standard syslog files.
Find a place to install to (here we assume /opt/logging24 as destination) and download
the Linux x86-64bit forwarder binary.
To accomplish this from the command line, e.g. via ssh, you may want to use these commands:
% mkdir -p /opt/logging24 % cd /opt/logging24 % wget -O forwarder 'https://logging24.com/forwarder/forwarder.debian-stable' % chmod +x forwarder
NOTE: Any line in this document prefixed with a % is intended to be executed on a linux command line terminal.
Use our web interface to create some write tokens.
Call the forwarder on the command line and specify files and write tokens.
% cd /opt/logging24
% ./forwarder
Usage: forwarder <subcommand> [help | ...]
where subcommand can be any of:
help - show help
version - show version
forward - forward logs to target endpoint
example:
forwarder forward tail /var/log/syslog <write_token_1> \
tail /var/log/daemon.log <write_token_2> \
to ingest.logging24.com:9000
forwarder forward stream some-test-logs.txt <write_token_1> \
timestamps first-iso \
prepend 'file:some-test-logs.txt ' \
to ingest.logging24.com:9000
Typically you'll want to use tail with a filename to correctly detect
log rotation and seamlessly resume forwarding from the new file. To do this correctly,
the forwarder needs to monitor various file properties which might not be available
on special files like pipes, named fifos or sockets. For such cases, the forwarder also
provides a stream subcommand.
If you have existing logs you want to test with, these can be uploaded and timed correctly, provided the lines contain ISO-8601 timestamps (i.e. in YYYY-MM-DD HH:MM:SS format). Fractional seconds are also fine. If your logs have such timestamps, a quick upload can be done via
% ./forwarder stream some-test-logs.txt <write_token> \
timestamps first-iso \
prepend 'file:some-test-logs.txt ' \
to ingest.logging24.com:9000
We suggest using a separate log context (and thus write token) for this upload to make the test logs easy to expire later.
To have all your logs reliably forwarded into logging24 even after reboots and power cycles, it is
recommended to setup the forwarder as a system service.
The continuous operation setup depends on the exact Linux distribution, for systemd-based
ones we recommend creating a run script to execute the forwarder with your desired
tails and write tokens
% cat > /opt/logging24/run #!/bin/sh exec ./forwarder forward \ tail /var/log/syslog <write_token> \ to ingest.logging24.com:9000
Then create a corresponding logging24.service file, to make
the script known to systemd.
% cat > /etc/systemd/system/logging24.service [Unit] Description=logging24 Log Forwarding Requires=network.target [Service] Type=exec WorkingDirectory=/opt/logging24 ExecStart=/opt/logging24/run TimeoutStopSec=60 Restart=always RestartSec=10 StandardOutput=syslog StandardError=syslog SyslogIdentifier=logging24 User=root CPUAccounting=on CPUQuota=50% MemoryAccounting=on MemoryMin=64M MemoryHigh=128M MemoryMax=196M IOAccounting=on IOWeight=1 [Install] WantedBy=multi-user.target
The last step remaining it to reload the systemd scripts, enable and start the service.
% systemctl daemon-reload % systemctl enable logging24 % systemctl start logging24
In case the default binary does not run on your system,
we also provide a .tgz of the forwarder bundled with all required libraries.
The following commands will download and extract it into the recommended location:
% mkdir -p /opt/logging24 % cd /opt % wget -O logging24/forwarder.tgz 'https://logging24.com/forwarder/forwarder.library-bundle.tgz' % tar -xzvf logging24/forwarder.tgz
Note that this binary will only run from the current directory, as the compiled library lookup paths are relative.
For forwarding on other architectures, we recommend to download the Python-based log forwarder. It offers the same feature set as the binary forwarder, but platform independent. Additionally it supports forwarding logs from systemd journals using the python-systemd library.
Technically the python forwarder script itself is implemented using the logging24 python logging handler, which can be use in any native python logger infrastructure. (more on the python language specific programming interface).
System installation can proceed similar to the way outlined above, more specific
instructions are included in the README.md file.
For forwarding directly from within fluentd,
we prepared a plugin for download.
Instructions are included in the README.md.
Currently our syslog receiver is set up to work seemlessly with heroku syslog drains.
To send heroku log drains to logging24 proceed as follows: First add syslog+tls drain to your heroku app
% heroku drains:add -a YOUR_APP_NAME syslog+tls://ingest.logging24.com:9100
Next obtain the token heroku assigned to your log drain.
% heroku drains --json
Example output:
[
{
"addon": null,
"created_at": "2021-11-27T23:04:56Z",
"id": "041234ad-dffa-a511-769a-aff13656789e",
"token": "d.ef012345-6789-0abc-defg-01234567890a",
"updated_at": "2021-11-27T23:04:56Z",
"url": "syslog+tls://ingest.logging24.com:9100"
},
]
To associate the heroku log drain token with a logging24 write token,
create a new write token using the logging24 web interface. Choose "heroku"
as External ID provider and assign the heroku token as External ID.
For now the syslog+tls receiver expects the specific formatting as sent by heroku. If you have other requirements, we should be able to accomodate them on short notice.
To setup custom rsyslog forwarding, create a random identifier of the form: x.xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxxxx. We highly suggest machine generating these identifiers. For example using the following script line:
% od -vAn -N16 -tx1 < /dev/urandom | \
awk '{ gsub (" ", "", $0); print}' | \
cut -c 1-8,9-13,14-17,18-21,22-33 --output-delimiter "-" | \
awk '$0="d."$0'
Now associate the generated identifier with a logging24 write token. Create a new write token using the logging24 web interface. Choose "heroku" as External ID provider and assign the generated token as External ID.
Next you need to configure rsyslog to send the logs. Lets assume you want to forward all syslog messages (*.*) to one forward stream.
Create a config file /etc/rsyslog.d/99-qlg.conf containing:
template(name="qlgStream0" type="string"
string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% x.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx %APP-NAME% %PROCID% %MSGID% - %msg%\n")
*.* action(type="omfwd" protocol="tcp" tcp_framing="octet-counted" StreamDriver="gtls" StreamDriverMode="1" StreamAuthMode="anon" target="ingest.logging24.com" port="9100" template="qlgStream0")
Replace x.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with your generated id. Restart the rsyslogd and watch your logs appear in logging24.
% systemctl restart rsyslogd
Logs can be forwarded via the Loki HTTPS protocol by sending log batches to
https://ingest.logging24.com:9200/ with
an Authorization: Bearer <write_token> header.
For promtail, the relevant part of the config looks like this:
clients:
- url: "https://ingest.logging24.com:9200/"
batchwait: 1s
batchsize: 1048576
follow_redirects: false
enable_http2: false
backoff_config:
min_period: 500ms
max_period: 5m0s
max_retries: 10
timeout: 10s
tenant_id: ""
drop_rate_limited_batches: false
stream_lag_labels: ""
tls_config:
insecure_skip_verify: false
bearer_token: <write_token>