You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: how-to-guides/free-alternative-to-splunk-by-fluentd.md
+55-33Lines changed: 55 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Free Alternative To Splunk
2
2
3
-
[Splunk](http://www.splunk.com/) is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open-source alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
3
+
[Splunk](https://www.splunk.com/) is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open-source alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
In this guide, we will go over the installation, setup, and basic use of this combined log search solution. This article was tested on Ubuntu 16.04 and CentOS 7.4. **If you're not familiar with Fluentd**, please learn more about Fluentd first.
13
+
In this guide, we will go over the installation, setup, and basic use of this combined log search solution. This article was tested on Ubuntu 24.04. **If you're not familiar with Fluentd**, please learn more about Fluentd first.
$ tar -xf elasticsearch-8.17.1-linux-x86_64.tar.gz
45
+
$ cd elasticsearch-8.17.1
36
46
```
37
47
38
48
Once the installation is complete, start Elasticsearch:
@@ -41,16 +51,20 @@ Once the installation is complete, start Elasticsearch:
41
51
$ ./bin/elasticsearch
42
52
```
43
53
44
-
Note: You can also install Elasticsearch \(and Kibana\) using RPM/DEB packages. For details, please refer to [the official instructions](https://www.elastic.co/downloads).
54
+
{% hint style='info' %}
55
+
* You can also install Elasticsearch \(and Kibana\) using RPM/DEB packages. For details, please refer to [the official instructions](https://www.elastic.co/downloads).
56
+
* You can create enrollment token for kinaba. Use `./bin/elasticsearch-create-enrollment-token -s kibana`.
57
+
* You can reset default password for `elastic`, Use `./bin/elasticsearch-reset-password -u elastic`.
58
+
{% endhint %}
45
59
46
-
## Set Up Kibana
60
+
###Set Up Kibana
47
61
48
62
To install Kibana, download it from the official website and extract it. Kibana is an HTML/CSS/JavaScript application \([download](https://www.elastic.co/downloads/kibana)\). Use the binary for 64-bit Linux systems.
See [Plugin Management](..//installation/post-installation-guide#plugin-management) section how to install fluent-plugin-elasticsearch on your environment.
75
87
76
-
We'll configure td-agent\(Fluentd\) to interface properly with Elasticsearch. Please modify `/etc/td-agent/td-agent.conf` as shown below:
88
+
We'll configure fluent-package\(Fluentd\) to interface properly with Elasticsearch. Please modify `/etc/fluent/fluentd.conf` as shown below:
77
89
78
90
```text
79
91
# get logs from syslog
@@ -90,27 +102,34 @@ We'll configure td-agent \(Fluentd\) to interface properly with Elasticsearch. P
90
102
91
103
<match syslog.**>
92
104
@type elasticsearch
105
+
host localhost
106
+
user elastic
107
+
password (ELASTIC_USER_PASSWORD_HERE)
93
108
logstash_format true
109
+
scheme https
110
+
ssl_verify false
111
+
include_timestamp true
94
112
<buffer>
95
113
flush_interval 10s # for testing
96
114
</buffer>
97
115
</match>
98
116
```
99
117
118
+
{% hint style='warning' %}
119
+
In this article, it disables verification of TLS explicitly for elasticsearch because of demonstration. Do not disable on production.
120
+
{% endhint %}
121
+
100
122
`fluent-plugin-elasticsearch` comes with a `logstash_format` option that allows Kibana to search through the stored event logs in Elasticsearch.
101
123
102
-
Once everything has been set up and configured, start `td-agent`:
124
+
Once everything has been set up and configured, start `fluentd`:
103
125
104
126
```text
105
-
# init
106
-
$ sudo /etc/init.d/td-agent start
107
-
# or systemd
108
-
$ sudo systemctl start td-agent.service
127
+
$ sudo systemctl start fluentd
109
128
```
110
129
111
130
## Set Up `rsyslogd`
112
131
113
-
The final step is to forward the logs from your `rsyslogd` to `fluentd`. Please add the following line to `/etc/rsyslog.conf`, and restart `rsyslog`. This will forward the local syslogs to Fluentd, and Fluentd in turn will forward the logs to Elasticsearch.
132
+
The final step is to forward the logs from your `rsyslogd` to `fluentd`. Please create the file with following line to `/etc/rsyslog.d/90-fluentd.conf`, and restart `rsyslog`. This will forward the local syslogs to Fluentd, and Fluentd in turn will forward the logs to Elasticsearch.
114
133
115
134
```text
116
135
*.* @127.0.0.1:42185
@@ -119,9 +138,6 @@ The final step is to forward the logs from your `rsyslogd` to `fluentd`. Please
119
138
Please restart the `rsyslog` service once the modification is complete:
120
139
121
140
```text
122
-
# init
123
-
$ sudo /etc/init.d/rsyslog restart
124
-
# or systemd
125
141
$ sudo systemctl restart rsyslog
126
142
```
127
143
@@ -147,7 +163,7 @@ To manually send logs to Elasticsearch, please use the `logger` command:
147
163
$ logger -t test foobar
148
164
```
149
165
150
-
When debugging your `td-agent` configuration, using [`filter_stdout`](../filter/stdout.md) will be useful. All the logs including errors can be found at `/etc/td-agent/td-agent.log`.
166
+
When debugging your `fluentd` configuration, using [`filter_stdout`](../filter/stdout.md) will be useful. All the logs including errors can be found at `/etc/fluent/fluentd.log`.
151
167
152
168
```text
153
169
<filter syslog.**>
@@ -156,7 +172,13 @@ When debugging your `td-agent` configuration, using [`filter_stdout`](../filter/
0 commit comments