From f22a8d747e23de3cb663db382291fc86ae2d4975 Mon Sep 17 00:00:00 2001 From: "deivid.garcia.garcia" Date: Thu, 20 Nov 2025 15:17:53 +0100 Subject: [PATCH 1/5] tail: New keep_file_handle and fstat_interval_nsec options for tail input #11151 Signed-off-by: David Garcia Signed-off-by: deivid.garcia.garcia --- pipeline/inputs/tail.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 7d1ac7fa1..29596ee1b 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -37,6 +37,8 @@ The plugin supports the following configuration parameters: | `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`. | _none_ | | `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | | `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | +| `keep_file_handle` | Keep file handles open while tailing files. When set to `false`, file handles are closed after reading, which is useful for SMB/SAMBA storage backends that prevent file deletion when handles are open. **Note:** When set to `false`, log rotation detection does not work as it requires an active handle to determine where the original file was rotated to. Rotation will be detected as a truncation, so some logs might be lost. | `true` | +| `fstat_interval_nsec` | Set the interval for checking file status (`fstat`). This controls how often file metadata is checked, which is important for cloud storage backends that are sensitive to IOPS (Input/Output Operations Per Second). The value accepts time units (for example, `1s`, `500ms`, `1000000000ns`) and will be converted to nanoseconds internally. | `250ms` | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data isn't aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` | | `Generic.Encoding` | Set the non-Unicode encoding of the file data. Supported values: `ShiftJIS`, `UHC`, `GBK`, `GB18030`, `Big5`, `Win866`, `Win874`, `Win1250`, `Win1251`, `Win1252`, `Win2513`, `Win1254`, `Win1255`, and `Win1256`. | `none` | @@ -474,6 +476,38 @@ While file rotation is handled, there are risks of potential log loss when using {% endhint %} +## Tailing files from SMB/SAMBA + +When tailing files from SMB/SAMBA shares on POSIX systems or Kubernetes CSI volume mounts, file handles can prevent file deletion. Although the `FILE_SHARE_DELETE` flag is supported on Windows, it's not available on POSIX systems or when using intermediate abstractions such as Kubernetes CSI volume mounts. Set `keep_file_handle` to `false` to allow file deletion while files are being tailed. Note that this disables log rotation detection. + +Example configuration: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: tail + path: /mnt/samba/logs/*.log + keep_file_handle: false + fstat_interval_nsec: 1s +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name tail + Path /mnt/samba/logs/*.log + Keep_File_Handle false + Fstat_Interval_Nsec 1s +``` + +{% endtab %} +{% endtabs %} + ## Character encoding conversion This feature allows Fluent Bit to convert logs from various character encodings into the standard UTF-8 format. From b1dfc96e99459b9164a2ed490bec15d8a164c793 Mon Sep 17 00:00:00 2001 From: David Date: Sat, 22 Nov 2025 09:26:08 +0100 Subject: [PATCH 2/5] Update pipeline/inputs/tail.md Co-authored-by: Alexa Kreizinger Signed-off-by: David --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 29596ee1b..5e193d91f 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -37,7 +37,7 @@ The plugin supports the following configuration parameters: | `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`. | _none_ | | `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | | `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | -| `keep_file_handle` | Keep file handles open while tailing files. When set to `false`, file handles are closed after reading, which is useful for SMB/SAMBA storage backends that prevent file deletion when handles are open. **Note:** When set to `false`, log rotation detection does not work as it requires an active handle to determine where the original file was rotated to. Rotation will be detected as a truncation, so some logs might be lost. | `true` | +| `keep_file_handle` | Keep file handles open while tailing files. If `false`, file handles are closed after reading, which is useful for SMB/Samba storage backends that prevent file deletion when handles are open. However, if `false`, log rotation does not work because that requires an active handle to determine where the original file was rotated to. Rotation will be detected as a truncation, so some logs might be lost. | `true` | | `fstat_interval_nsec` | Set the interval for checking file status (`fstat`). This controls how often file metadata is checked, which is important for cloud storage backends that are sensitive to IOPS (Input/Output Operations Per Second). The value accepts time units (for example, `1s`, `500ms`, `1000000000ns`) and will be converted to nanoseconds internally. | `250ms` | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data isn't aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` | From 3240bcab243ccd33a7982abb0c8002a460a12478 Mon Sep 17 00:00:00 2001 From: David Date: Sat, 22 Nov 2025 09:26:25 +0100 Subject: [PATCH 3/5] Update pipeline/inputs/tail.md Co-authored-by: Alexa Kreizinger Signed-off-by: David --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 5e193d91f..132ac9bb7 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -476,7 +476,7 @@ While file rotation is handled, there are risks of potential log loss when using {% endhint %} -## Tailing files from SMB/SAMBA +## Tailing files from SMB/Samba When tailing files from SMB/SAMBA shares on POSIX systems or Kubernetes CSI volume mounts, file handles can prevent file deletion. Although the `FILE_SHARE_DELETE` flag is supported on Windows, it's not available on POSIX systems or when using intermediate abstractions such as Kubernetes CSI volume mounts. Set `keep_file_handle` to `false` to allow file deletion while files are being tailed. Note that this disables log rotation detection. From 96b09280b6a62f557d61694d092456822fa99be0 Mon Sep 17 00:00:00 2001 From: David Date: Sat, 22 Nov 2025 09:26:53 +0100 Subject: [PATCH 4/5] Update pipeline/inputs/tail.md Co-authored-by: Alexa Kreizinger Signed-off-by: David --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 132ac9bb7..4615811f6 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -478,7 +478,7 @@ While file rotation is handled, there are risks of potential log loss when using ## Tailing files from SMB/Samba -When tailing files from SMB/SAMBA shares on POSIX systems or Kubernetes CSI volume mounts, file handles can prevent file deletion. Although the `FILE_SHARE_DELETE` flag is supported on Windows, it's not available on POSIX systems or when using intermediate abstractions such as Kubernetes CSI volume mounts. Set `keep_file_handle` to `false` to allow file deletion while files are being tailed. Note that this disables log rotation detection. +When tailing files from SMB/Samba shares on POSIX systems or Kubernetes CSI volume mounts, file handles can prevent file deletion. Although the `FILE_SHARE_DELETE` flag is supported on Windows, it's not available on POSIX systems or when using intermediate abstractions such as Kubernetes CSI volume mounts. Set `keep_file_handle` to `false` to allow file deletion while files are being tailed. However, keep in mind that this disables log rotation detection. Example configuration: From ae0b9d15c6487126ca48f321720c8e653adcca5d Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 24 Nov 2025 11:21:11 -0800 Subject: [PATCH 5/5] Update pipeline/inputs/tail.md Signed-off-by: Alexa Kreizinger --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 4615811f6..82f2920e8 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -37,7 +37,7 @@ The plugin supports the following configuration parameters: | `tag_regex` | Set a regular expression to extract fields from the filename. For example: `(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`. | _none_ | | `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | | `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | -| `keep_file_handle` | Keep file handles open while tailing files. If `false`, file handles are closed after reading, which is useful for SMB/Samba storage backends that prevent file deletion when handles are open. However, if `false`, log rotation does not work because that requires an active handle to determine where the original file was rotated to. Rotation will be detected as a truncation, so some logs might be lost. | `true` | +| `keep_file_handle` | Keep file handles open while tailing files. If `false`, file handles are closed after reading, which is useful for SMB/Samba storage backends that prevent file deletion when handles are open. However, if `false`, log rotation does not work because that requires an active handle to determine where the original file was rotated to. Rotation will be detected as a truncation, so some logs might be lost. | `true` | | `fstat_interval_nsec` | Set the interval for checking file status (`fstat`). This controls how often file metadata is checked, which is important for cloud storage backends that are sensitive to IOPS (Input/Output Operations Per Second). The value accepts time units (for example, `1s`, `500ms`, `1000000000ns`) and will be converted to nanoseconds internally. | `250ms` | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data isn't aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` |