You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CONTRIBUTING.md
-9Lines changed: 0 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,6 @@ Before you contribute to the Fluent Bit docs, review these critical tips:
15
15
16
16
-[Sign off](#sign-off-your-git-commits) your Git commits.
17
17
- Use [soft line wraps](#line-wraps) in Markdown files.
18
-
- To link between pages, use [absolute file paths](#links).
19
18
- Review the results of [linters](#linters) for style and formatting guidance.
20
19
21
20
## Review process
@@ -69,14 +68,6 @@ The active [linters](#linters) in this repository flag certain style errors and,
69
68
70
69
The Fluent Bit docs library is built and hosted through [GitBook](https://docs.gitbook.com/). Unfortunately, GitBook doesn't support local previews for contributors, but a Fluent Bit maintainer with a dedicated GitBook account can verify that things are formatted correctly after you open a new pull request.
71
70
72
-
### Links
73
-
74
-
When cross-linking between in this repository, use a full absolute path whenever possible. For example:
75
-
76
-
```text
77
-
[LTSV](../pipeline/parsers/ltsv.md) and [Logfmt](../pipeline/parsers/logfmt.md)
78
-
```
79
-
80
71
### Line wraps
81
72
82
73
When GitBook renders pages, it treats all newlines literally, which means hard line wraps in Markdown files create awkward line breaks in the Fluent Bit docs site. Due to this, docs contributions must use soft line wraps.
-`includes`: Specifies additional YAML configuration files to [include as part of a parent file](../administration/configuring-fluent-bit/yaml/includes-section.md).
13
-
-`service`: Configures global properties of the Fluent Bit [service](../administration/configuring-fluent-bit/yaml/service-section.md).
14
-
-`pipeline`: Configures active [`inputs`, `filters`, and `outputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md).
-`plugins`: Defines paths for [custom plugins](./yaml/plugins-section.md).
18
+
-`upstream_servers`: Defines [nodes](./yaml/upstream-servers-section.md) for output plugins.
19
19
20
20
{% hint style="info" %}
21
21
YAML configuration is used in the smoke tests for containers. An always-correct up-to-date example is here: <https://github.com/fluent/fluent-bit/blob/master/packaging/testing/smoke/container/fluent-bit.yaml>.
Copy file name to clipboardExpand all lines: administration/configuring-fluent-bit/yaml/multiline-parsers-section.md
+16-4Lines changed: 16 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,19 @@
1
1
# Multiline parsers
2
2
3
-
Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information.
3
+
You can define custom [multiline parsers](../../pipeline/parsers/multiline-parsing.md) in the `multiline_parsers` section of YAML configuration files.
4
4
5
-
In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.
5
+
{% hint style="info" %}
6
6
7
-
The following example demonstrates how to define a multiline parser directly in the main configuration file, and how to include additional definitions from external files:
7
+
To define standard custom parsers, use [the `parsers` section](./parsers-section.md) of YAML configuration files.
8
+
9
+
{% endhint %}
10
+
11
+
## Syntax
12
+
13
+
To define custom parsers in the `multiline_parsers` section of a YAML configuration file, use the following syntax:
14
+
15
+
{% tabs %}
16
+
{% tab title="fluent-bit.yaml" %}
8
17
9
18
```yaml
10
19
multiline_parsers:
@@ -20,6 +29,9 @@ multiline_parsers:
20
29
next_state: cont
21
30
```
22
31
32
+
{% endtab %}
33
+
{% endtabs %}
34
+
23
35
This example defines a multiline parser named `multiline-regex-test` that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from `start_state` to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.
24
36
25
-
For more detailed information on configuring multiline parsers, including advanced options and use cases, refer to the Configuring Multiline Parsers documentation.
37
+
For information about supported configuration options for custom multiline parsers, see [configuring multiline parsers](../../pipeline/parsers/multiline-parsing.md#configuring-multiline-parsers).
Copy file name to clipboardExpand all lines: administration/configuring-fluent-bit/yaml/parsers-section.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,13 +4,13 @@ You can define customer [parsers](../pipeline/parsers.md) in the `parsers` secti
4
4
5
5
{% hint style="info" %}
6
6
7
-
To define custom multiline parsers, use [the `multiline_parsers` section](../administration/configuring-fluent-bit/yaml/multiline-parsers-section.md) of YAML configuration files.
7
+
To define custom multiline parsers, use [the `multiline_parsers` section](./multiline-parsers-section.md) of YAML configuration files.
8
8
9
9
{% endhint %}
10
10
11
11
## Syntax
12
12
13
-
To define customers parsers in the `parsers` section of a YAML configuration file, use the following syntax.
13
+
To define custom parsers in the `parsers` section of a YAML configuration file, use the following syntax.
14
14
15
15
{% tabs %}
16
16
{% tab title="fluent-bit.yaml" %}
@@ -35,7 +35,7 @@ parsers:
35
35
{% endtab %}
36
36
{% endtabs %}
37
37
38
-
For information about supported configuration options for custom parsers, see [configuring parsers](../pipeline/parsers/configuring-parser.md).
38
+
For information about supported configuration options for custom parsers, see [configuring parsers](../../pipeline/parsers/configuring-parser.md).
Copy file name to clipboardExpand all lines: pipeline/filters/multiline-stacktrace.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ Along with multiline filters, you can enable one of the following built-in Fluen
15
15
16
16
When using this filter:
17
17
18
-
- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
18
+
- The usage of this filter depends on a previous configuration of a [multiline parser](../pipeline/parsers/multiline-parsing.md) definition.
19
19
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) in combination with the `docker` or `cri` parser. To concatenate application logs like stacktraces on top of that, you can use this multiline filter.
20
20
21
21
{% hint style="warning" %}
@@ -40,7 +40,7 @@ The plugin supports the following configuration parameters:
40
40
41
41
| Property | Description |
42
42
| -------- | ----------- |
43
-
|`multiline.parser`| Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
43
+
|`multiline.parser`| Specify one or multiple [Multiline Parser definitions](../pipeline/parsers/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
44
44
|`multiline.key_content`| Key name that holds the content to process. A multiline parser definition can specify the `key_content` This option allows for overwriting that value for the purpose of the filter. |
45
45
|`mode`| Mode can be `parser` for regular expression concatenation, or `partial_message` to concatenate split Docker logs. |
46
46
|`buffer`| Enable buffered mode. In buffered mode, the filter can concatenate multiple lines from inputs that ingest records one by one (like Forward), rather than in chunks, re-emitting them into the beginning of the pipeline (with the same tag) using the `in_emitter` instance. With buffer off, this filter won't work with most inputs, except Tail. |
@@ -68,7 +68,7 @@ service:
68
68
flush: 1
69
69
log_level: info
70
70
parsers_file: parsers_multiline.yaml
71
-
71
+
72
72
pipeline:
73
73
inputs:
74
74
- name: tail
@@ -157,10 +157,10 @@ This file defines a multiline parser for the example. A second multiline parser
157
157
# Regex rules for multiline parsing
158
158
# ---------------------------------
159
159
#
160
-
# configuration hints:
160
+
# configuration hints:
161
161
#
162
162
# - first state always has the name: start_state
163
-
# - every field in the rule must be inside double quotes
163
+
# - every field in the rule must be inside double quotes
@@ -314,7 +314,7 @@ Lines that don't match a pattern aren't considered as part of the multiline mess
314
314
315
315
## Docker partial message use case
316
316
317
-
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16 KB.
317
+
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16 KB.
318
318
If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
Copy file name to clipboardExpand all lines: pipeline/inputs/tail.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -158,9 +158,9 @@ Multiline core is exposed by the following configuration:
158
158
159
159
| Key | Description |
160
160
|:-------------------|:---------------|
161
-
|`multiline.parser`| Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. |
161
+
|`multiline.parser`| Specify one or multiple [Multiline Parser definitions](../parsers/multiline-parsing.md) to apply to the content. |
162
162
163
-
[Multiline Parser](../../administration/configuring-fluent-bit/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
163
+
[Multiline Parser](../parsers/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
Copy file name to clipboardExpand all lines: pipeline/parsers.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,19 +39,19 @@ For example, a parser can turn an unstructured log entry like this:
39
39
40
40
## How parsers work
41
41
42
-
Parsers modify the data ingested by input plugins. This modification happens before Fluent Bit applies any [filters](../pipeline/filters.md) or [processors](..pipeline/processors.md) to that data.
42
+
Parsers modify the data ingested by input plugins. This modification happens before Fluent Bit applies any [filters](../pipeline/filters.md) or [processors](../pipeline/processors.md) to that data.
43
43
44
44
Each input plugin can have one active parser. Multiple plugins within the same Fluent Bit configuration file can use the same parser or use different parsers from each other.
45
45
46
46
### Default parsers and custom parsers
47
47
48
-
Fluent Bit includes a variety of [default parsers](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for parsing common data formats, like Apache and Docker logs. You can also [define custom parsers](../configuring-fluent-bit/yaml/parsers-section.md).
48
+
Fluent Bit includes a variety of [default parsers](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for parsing common data formats, like Apache and Docker logs. You can also [define custom parsers](../administration/configuring-fluent-bit/yaml/parsers-section.md).
49
49
50
50
## Add a parser to an input plugin
51
51
52
52
To add a parser to an input plugin, follow these steps.
53
53
54
-
1. Either identify the name of the [default parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) you want to use, or [define a custom parser](../configuring-fluent-bit/yaml/parsers-section.md) with your desired [configuration settings](../pipeline/parsers/configuring-parser.md).
54
+
1. Either identify the name of the [default parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) you want to use, or [define a custom parser](../administration/configuring-fluent-bit/yaml/parsers-section.md) with your desired [configuration settings](../pipeline/parsers/configuring-parser.md).
55
55
56
56
1. Add a `parsers` key to the plugin's settings in the [`inputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md#inputs) section of your YAML configuration file.
Copy file name to clipboardExpand all lines: pipeline/parsers/configuring-parser.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
# Configuring custom parsers
2
2
3
-
Use the information on this page to configure custom [parsers](../pipeline/parsers).
3
+
Use the information on this page to configure custom [parsers](../parsers.md).
4
4
5
5
{% hint style="info" %}
6
6
7
-
To define a custom parser, add an entry to the [`parsers` section](../configuring-fluent-bit/yaml/parsers-section.md) of your YAML configuration file, or create a [standalone parser file](../administration/configuring-fluent-bit/yaml/parsers-section.md#standalone-parsers-files).
7
+
To define a custom parser, add an entry to the [`parsers` section](../administration/configuring-fluent-bit/yaml/parsers-section.md) of your YAML configuration file, or create a [standalone parser file](../administration/configuring-fluent-bit/yaml/parsers-section.md#standalone-parsers-files).
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?v=4&size=40){kind=avatar}
0 commit comments