logging24
log everything · query anything
All search and query capabilities of logging24 are available via a simple HTTP API. The logging24 search web UI itself uses this public API to interact with our backend. The relevant code can be found in api.js.
The API is available at https://logging24.com/api/search/v1. We plan to keep all our version-numbered resources (such as this one) backward-compatible in the sense that all future parameters will be optional and all returned keys will keep their semantics. Specifically, we do not consider adding optional parameters a breaking change and will not change the version number for added optional parameters and returned keys. Therefore your code should be prepared to ignore any unknown keys in our API responses to keep working when the endpoints change in the future.
Request:
curl 'https://logging24.com/api/search/v1' \
-X POST \
-H 'Authorization: Token <Your read token>' \
--data-urlencode 'customer=<Your customer identifier>' \
--data-urlencode 'regex=.*' \
--data-urlencode 'type=BACKWARD_RESULTS' \
--data-urlencode 'beginTime=2022-08-25 21:41:00.000' \
--data-urlencode 'endTime=2022-08-25 21:57:00.000'
Response:
{
"fullHash": "7246963916FD189A297F77A970B78E0B144E0B2AAE49BD64C56CF9E161767DF6",
"events": [
{
"time":"1661464551383000000",
"message":"Aug 25 23:55:51 example-host apache[1254]: Startup completed."
}
],
"counts": [],
"totalBlocks": 10044,
"relevantBlocks": 15,
"scannedBlocks": 6,
"failedBlocks": 0,
"scannedBytes": 24327668,
"complete": false,
"stopped": false
}
Authentication to the search API happens via read tokens. For testing purposes, the token stored in the read_token cookie can be used. It is automatically created after logging in to the web UI but expires after a day. For permanent usage, you should instead create a dedicated read token.
The value can be passed either via the Authorization header (as shown in the example) or via the read_token cookie.
| fullHash | A hash built from the query parameters which is used to re-identify identical queries. This is returned for debugging purposes and allows us to track queries across our backend. | ||
| events | A list of log events, or log lines, returned by the query. Note that not all query types return events. | ||
| events[*].time | Each returned event has a timestamp in nanoseconds since the UNIX epoch. | ||
| events[*].context | An index into the contexts array, denoting the log context this event was found in. | ||
| events[*].message | The log message. This is byte-for-byte identical with the original log line (as received by our ingest). Please keep in mind that the events.message value itself is part of the result JSON and therefore JSON encoding is applied as necessary. | ||
| events[*].key | If the query type supports it, the key captured from the event content. | ||
| contexts | A list of log contexts. For many query types, this is used to communicate the event source contexts in deduplicated form. | ||
| contexts[*].customer | The customer identifier of the log context. | ||
| contexts[*].prefix{0,1,2,3} | The four log prefixes defining the context. | ||
| contexts[*].count | For some query types, this field returns counted elements per log context, e.g. total bytes stored within a context. | ||
| counts | An array of counters, typically how many matches have been counted. The array layout and semantics depends on the query type chosen. Note that not all query types return counts. | ||
| totalBlocks | Log events are grouped into blocks inside the logging24 engine. This value gives an estimate of the total number of blocks in the system. | ||
| relevantBlocks | The number of blocks which might contribute to answering the query because they fall into the queried time range and match the specified prefix0 to prefix3. | ||
| scannedBlocks | The number of blocks actually scanned to answer the query. This number will increase as scanning progresses. | ||
| failedBlocks | The number of blocks which we failed to scan due to technical problems. This should be zero, obviously. | ||
| scannedBytes | The number of bytes scanned to answer the query. This number will increase as scanning progresses. Query billing is based on this number. | ||
| scanProgress | An integer number approximating the total scanning progress. Scanning is complete once it reaches 4294967296 (i.e. 2^32). | ||
| complete | Becomes true once the result is completely computed. | ||
| stopped | Becomes true once the backend has completely stopped scanning blocks. As we scan blocks on multiple nodes concurrently, it can happen that scanning continues for a while even though the result is already complete. | ||
| error | This key is only present (and most others missing) if the request could not be processed. It contains an English text describing the error condition. | ||
| errorCode |
This key is present for some common error conditions to enable the frontend
to deal with them in helpful ways. So far there is one such error code defined:
|
Scanning large amounts of data can take some time. Users would typically like to see progress happening meanwhile. For ease of implementation, our API returns partial answers while the scanning continues asynchronously. The API client is expected to re-issue a query repeatedly (approximately once a second), until the result is complete. If no further requests for a result are issued, the asynchronous scanning will stop after a few seconds of inactivity. Except for the inactivity timeout, the scan speed does not depend on the frequency of queries.
To estimate the progress a scan has made so far, check scanProgress. Some query types can terminate earlier than this estimate would suggest, e.g. a query for the last 100 matching lines will scan the newest blocks first and conclude the scan once 100 lines have found, regardless of how many older blocks could have contributed (but ultimately didn't because they can only contain older events). In any case, a complete result can be shown as final to the user.
Query parameters are passed via the common application/x-www-form-urlencoded encoding in the request body.
| customer |
No default. The technical customer identifier. Assigned by us when we created your account. The exact value (in particular upper/lower case) can be found in the details of the read token list. |
| prefix0 |
Default: empty Only query logs where the prefix0 dimensions starts with this prefix. |
| prefix1 | Same as prefix0, but for the prefix1 dimension. |
| prefix2 | Same as prefix0, but for the prefix2 dimension. |
| prefix3 | Same as prefix0, but for the prefix3 dimension. |
| type |
Default: UNSORTED_RESULTS The type of query to run. A list of available query types and their semantics can be found below. |
| limit |
Default: 100 An integer specifying how many elements (typically events) to return. Must be from 1 to 10000 (inclusive). To implement scrolling through potentially longer event lists, the best strategy is to issue new queries for time-ordered events and update beginTime and endTime as needed to see more results. |
| beginTime |
Default: 1970-01-01 01:00:00 The earliest time an event may have to be included in the result. |
| endTime |
Default: 2254-07-22 01:34:33 Events at or after this time are excluded from the result. |
| timeBins |
Default: 1 Into how many bins the interval starting at beginTime and ending before endTime should be split. Must be from 1 to 4096 (inclusive). |
| regex |
No default. The regular expression which events must match to be included in the result. |
| xSplits |
Default: empty A comma-separated, ordered list of the thresholds to use to divide events into histogram buckets along the x dimension. The format of the individual elements depends on the format in the regex used to capture the value. |
| ySplits | Same as xSplits, but for the y dimension. |
The API accepts timestamps in ISO 8601 format. Both YYYY-MM-DDTHH:MM:SS and YYYY-MM-DD HH:MM:SS work. Fractional seconds are supported. Timezone offsets are supported. If no timezone offset is given, times are assumed to be UTC.
Returns all matching events. The events are not ordered nor preferably selected from the end of the matching time range.
Use case: Checking for absence of error messages during a time-interval.
Returns all matching events in reverse chronological order (i.e. the latest events go first in the result).
Use case: Manual log exploration. This is the query type most equivalent to tail some.log | grep regex.
Counts all matching events per time bin. The first element of counts contains the number of events in the earliest time bin, the last element the number of events in the latest time bin. The limit parameters is not applied.
Use case: Show how often a certain log pattern appears throughout time.
Use case: With regex=.*, this shows the total number of log events over time.
Counts all matching events per time bin and x- and y-dimensions bin as indicated by xSplits resp. ySplits. The limit parameters is not applied. The regex must contain (?<x>...) or (?<y>...) captures to define an event's x- and y-position.
A matching event in time bin index t, x-dimension bin index x and y-dimension bin index y increases the count array at index i := t * xBins * yBins + x * yBins + y.
As an example, in a query with 2 time bins, 2 x splits and 1 y split, the result will be split into 2 time bins (t=0, t=1), 3 x-dimension bins (x=0, x=1, x=2), and 2 y-dimension bins (y=0, y=1). The elements of count will be sorted in this order:
"counts": [
(events within t=0,x=0,y=0), (events within t=0,x=0,y=1),
(events within t=0,x=1,y=0), (events within t=0,x=1,y=1),
(events within t=0,x=2,y=0), (events within t=0,x=2,y=1),
(events within t=1,x=0,y=0), (events within t=1,x=0,y=1),
(events within t=1,x=1,y=0), (events within t=1,x=1,y=1),
(events within t=1,x=2,y=0), (events within t=1,x=2,y=1)
]
Use case: With only timeBins and ySplits, plots the progression of the captured metric over time.
Use case: With only xSplits and ySplits, returns a scatter-plot of the correlation between two captured metrics.
Finds the last event per captured key (up to 31 characters long), up to a total of limit unique keys. The regex must contain a (?<k>...) capture to define the key.
Use case: Find the most recent metric set logged for each host, pod or instance.
Use case: Find the set of useful values to show in log filter dialogs.