Skip to content

Simplify the README and remove the testing badge #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gordonmurray merged 1 commit into from
Nov 2, 2026
Merged

Simplify the README and remove the testing badge #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 108 additions & 104 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,20 @@
# Trying out Apache Doris
# Apache Doris with Docker Compose

![Static Badge](https://img.shields.io/badge/Just_testing-Not_production_ready-red)
Run Apache Doris locally and stream data from Kafka into a table you can query
with SQL. The default setup uses one frontend, one backend and one Kafka broker.
An optional configuration provides three frontends and three backends.

A small Docker Compose project for trying Apache Doris and streaming data from
Kafka. Originally put together in 2023 after reading
[Empowering cyber security by enabling seven times faster log analysis](https://doris.apache.org/blog/empowering-cyber-security-by-enabling-seven-times-faster-log-analysis).

Updated to **Doris 4.0.8**, the release labelled Stable on the
[Apache Doris download page](https://doris.apache.org/download/) when this refresh
was made. The images are pinned to that version, rather than a moving `latest` tag.
Kafka uses the official `apache/kafka:4.1.2` image in KRaft mode, without ZooKeeper.
Images are pinned to **Doris 4.0.8** and **Kafka 4.1.2**. Kafka runs in KRaft mode,
without ZooKeeper.

## Requirements

- Docker Engine or Docker Desktop with **Docker Compose v2.24.4 or newer**.
- Bash. A local MySQL client is optional; the scripts use the one in the FE image.
- For the default demo, allow at least **8 GiB of memory for Docker**, preferably
more, and enough disk space for the images and data. The configured container
memory limits total 7 GiB. The larger cluster's limits total 19 GiB.
- The official Doris images provide AMD64 and ARM64 builds. An AMD64 CPU needs
AVX2 support. Use a native image for your machine.
- Git and Bash. The frontend image includes a MySQL client.
- At least **8 GiB of memory available to Docker** for the default setup,
preferably more, plus space for the images and data. Container memory limits
total 7 GiB for the default setup and 19 GiB for the larger cluster.
- AMD64 with AVX2 support, or ARM64. Use the native image for your machine.

On a Linux Docker host, raise the memory mapping limit before starting:

Expand All @@ -34,64 +29,69 @@ printf 'vm.max_map_count=2000000\n' | sudo tee /etc/sysctl.d/99-doris.conf
sudo sysctl --system
```

With Docker Desktop, the setting applies inside its Linux VM. If startup reports
that the limit is too low, configure it there rather than on the macOS or Windows
host. Consult the [Doris deployment guide](https://doris.apache.org/docs/4.x/getting-started/quick-start/).
With Docker Desktop, this setting applies inside its Linux VM. If startup reports
a low limit, configure it there. See the
[Doris deployment guide](https://doris.apache.org/docs/4.x/getting-started/quick-start/).

## Quick start

```bash
git clone https://github.com/gordonmurray/docker_compose_apache_doris.git
cd docker_compose_apache_doris
```

## Run the demo
To change ports or the network subnet, copy `.env.example` to `.env` and edit it
before running the demo. See [Configuration](#configuration).

```bash
./scripts/demo.sh
```

This starts one frontend, one backend and Kafka, waits for readiness, creates
`kafka_topic` and `products.kafka_table`, starts a Routine Load job, and sends
three sample messages. It waits until the corresponding prices can be queried:
The script waits for healthy services, creates `kafka_topic` and
`products.kafka_table`, starts the `products.kafka_fruit` Routine Load job, and
sends three sample messages. It then queries their prices:

```text
apples 0.60
bananas 0.30
oranges 0.45
```

The table uses a unique key on `fruit`, so repeating the demo updates the sample
prices rather than adding duplicate rows. Prices use `DECIMAL(10,2)`.
The job reads from the beginning of the Kafka topic on its first creation, so
messages produced before the consumer starts are included.

Running the demo also resumes its paused `products.kafka_fruit` job once all
services are ready. Doris can pause the job while containers restart. Resuming
retains its saved Kafka offsets; it does not recreate the job or reset the topic.
The script makes one resume attempt, so persistent ingestion errors still need
to be investigated. To leave a job paused, start only the services instead.
Rerunning the demo updates the sample rows using a unique key on `fruit`; prices
use `DECIMAL(10,2)`. A new ingestion job reads the topic from the beginning.
An existing paused job is resumed once services are ready, retaining its saved
Kafka offsets. If it pauses again, inspect the cause before retrying.

To start just the services:
To start only the services, leaving any paused ingestion job paused:

```bash
docker compose up -d --wait --wait-timeout 300
```

## Connect and explore
## Connect and query

| Interface | Address |
| Interface | Default address |
| --- | --- |
| Doris Web UI | <http://localhost:8030> |
| MySQL protocol | `127.0.0.1:9030` |
| Kafka, within the Compose network | `kafka:9092` |
| Kafka, inside the Compose network | `kafka:9092` |

This local setup uses Doris user `root` with an empty password. Published ports
bind to `127.0.0.1`; Kafka and backend ports remain inside the Docker network.

Doris uses `root` with an empty password for this local demo. Published ports bind
to `127.0.0.1`. Kafka and BE ports are available only inside the Docker network.
This configuration is intended for local experimentation.
Open a SQL session using the bundled client:

```bash
# Use the client already included in the frontend image.
docker compose exec fe mysql -h127.0.0.1 -P9030 -uroot
```

Or connect with a MySQL client installed on your host:

# Or use an installed MySQL client on the host.
```bash
mysql -h127.0.0.1 -P9030 -uroot
```

Try these queries:
Run these queries in either SQL session:

```sql
SHOW FRONTENDS;
Expand All @@ -100,48 +100,61 @@ SELECT * FROM products.kafka_table ORDER BY fruit;
SHOW ROUTINE LOAD FOR products.kafka_fruit;
```

Send another message, then query again after a few seconds:
From another terminal, send a message and query the table again after a few seconds:

```bash
printf 'pears,0.75\n' | docker compose exec -T kafka \
/opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server kafka:9092 --topic kafka_topic
```

The broker advertises `kafka:9092`, which is resolvable by the Doris containers.
Use the bundled Kafka commands through `docker compose exec`; a host producer
would need a separate external Kafka listener.
Use Kafka commands through `docker compose exec`. The broker advertises
`kafka:9092`, which is resolvable inside Docker; host producers would need a
separate external listener.

## Verify the pipeline

```bash
./scripts/smoke-test.sh
```

The smoke test runs the demo, checks FE and BE membership, produces a uniquely
named message, waits for its exact price in Doris, and removes the test row.
Polling is bounded and failures print cluster and Routine Load diagnostics.
The smoke test runs the demo, checks frontend and backend membership, sends a
unique message, verifies its price in Doris, and removes the test row.

GitHub Actions validates both Compose configurations and shell scripts, then
tests ingestion and data persistence after recreating the default containers.
The larger cluster is configuration-checked in CI; run its smoke test on a
machine with enough memory.

## Stop, restart or reset

Named volumes store Doris metadata, backend storage, logs and Kafka data.
Stop and restart without deleting data:

```bash
docker compose down
./scripts/demo.sh
```

To **delete all data in the default setup**:

GitHub Actions validates both Compose configurations, checks the Bash scripts,
and runs the default smoke test. It also recreates the containers without
removing volumes, checks that the three sample rows survived, then tests ingestion
again. The larger cluster is configuration-checked in CI; run its smoke test on
a machine with enough memory.
```bash
docker compose down --volumes
```

## Optional three-frontend, three-backend cluster
## Optional larger cluster

Stop the default project first to release its ports and subnet. Its volumes are
retained. The larger configuration uses a separate project and separate volumes:
The larger configuration uses three frontends, three backends and three table
replicas. It has a separate Compose project and volumes. Stop the default setup
first to release the shared ports and subnet:

```bash
docker compose down
./scripts/demo.sh --cluster
./scripts/smoke-test.sh --cluster
```

The larger demo creates the table with **three replicas** instead of one. This is
still a single-machine learning environment. Run further commands with the same
project name and both files:
Use the same project name and both files for subsequent cluster commands:

```bash
docker compose -p trying-doris-cluster \
Expand All @@ -152,58 +165,49 @@ docker compose -p trying-doris-cluster \
exec fe mysql -h127.0.0.1 -P9030 -uroot
```

To return to the default demo, stop the larger project with the same arguments
and `down`, then run `./scripts/demo.sh`. Do not remove followers from a running
cluster by merely switching Compose files: its persisted FE membership needs its
quorum. The separate project names keep these two experiments independent.

## Configuration and data

Optionally copy `.env.example` to `.env` to change the image versions, host ports
or network prefix. Choose an unused private `/24` subnet before the first start.
The Doris entrypoints register nodes by IP, so changing that subnet with existing
metadata is not supported by this demo; use fresh volumes for a different subnet.

The mounted files in `fe/conf` and `be/conf` contain small configuration overrides.
`scripts/start-doris.sh` applies these to the image's writable configuration and
reduces JVM heaps while retaining the release's other Java options. The official
entrypoints handle node registration and derive the matching network settings.
Restart the services after changing configuration files.

Doris metadata, backend storage, logs and Kafka data use named Docker volumes.
Stopping and recreating containers preserves them:
To stop the larger cluster and return to the default setup:

```bash
docker compose down
docker compose -p trying-doris-cluster \
-f docker-compose.yml -f docker-compose.cluster.yml down
./scripts/demo.sh
```

For an intentional **reset that deletes all data in the default demo**:
Add `--volumes` to the larger cluster's `down` command only to delete its data.
Keep the projects separate: removing frontends from a persisted cluster by
switching Compose files can prevent it from reaching quorum.

```bash
docker compose down --volumes
```
## Configuration

To delete the larger demo's data, use its explicit project and both Compose files:
[`.env.example`](.env.example) contains the image versions, host ports and network
prefix. Copy it to `.env` to override the defaults. Choose an unused private `/24`
subnet before the first start. Doris registers nodes by IP, so changing the
subnet requires fresh volumes in this demo.

```bash
docker compose -p trying-doris-cluster \
-f docker-compose.yml -f docker-compose.cluster.yml down --volumes
```
[`fe/conf/fe.conf`](fe/conf/fe.conf) and [`be/conf/be.conf`](be/conf/be.conf)
contain configuration overrides. The startup script applies them to the image's
configuration while preserving its Java options and setting smaller JVM heaps.
Restart the services after editing these files.

These commands do not touch the old 2023 setup's `/data/...` bind-mounted files.
This refresh starts a new demo; it does not migrate a 2.0-alpha database. Preserve
any old data separately and follow the
[official upgrade guidance](https://doris.apache.org/docs/4.x/admin-manual/cluster-management/upgrade/)
if it needs to be retained.
The original 2023 setup used `/data/...` bind mounts and Doris 2.0-alpha.
This setup uses fresh named volumes and does not migrate or delete those old
files. If you need to retain an old database, follow the
[official upgrade guidance](https://doris.apache.org/docs/4.x/admin-manual/cluster-management/upgrade/).

## Troubleshooting

- Inspect `docker compose ps --all` and `docker compose logs --tail=100` if startup
fails. Check the host's `vm.max_map_count`, available memory and AVX2 support.
- FE and BE logs are also inside their containers under
`/opt/apache-doris/fe/log` and `/opt/apache-doris/be/log`.
- If Routine Load pauses, inspect `SHOW ROUTINE LOAD FOR products.kafka_fruit;` for its
reason, fix the cause, then use `RESUME ROUTINE LOAD FOR products.kafka_fruit;`.
- A timeout can be rerun after fixing the underlying problem. The scripts leave
containers and data in place for inspection; they never reset volumes for you.
```bash
docker compose ps --all
docker compose logs --tail=100
```

- For startup failures, check memory, `vm.max_map_count` and AVX2 support on AMD64.
- Doris logs are in `/opt/apache-doris/fe/log` and `/opt/apache-doris/be/log`
inside the corresponding containers.
- For a paused ingestion job, inspect `SHOW ROUTINE LOAD FOR products.kafka_fruit;`,
fix the reported cause, then run `RESUME ROUTINE LOAD FOR products.kafka_fruit;`.
- A timeout leaves containers and data available for inspection. Fix the cause
and rerun the script; it never resets volumes automatically.

Originally inspired by
[Empowering cyber security by enabling seven times faster log analysis](https://doris.apache.org/blog/empowering-cyber-security-by-enabling-seven-times-faster-log-analysis).
Loading