RANGER-5520:Audit Server refactoring to segregate audit ingestion and…

[rameeshm]

… dispatching functionality

What changes were proposed in this pull request?

audit-ingestor

• This would have the functionality of audit log inbounding from various plugin and committing into Kafka
• This will be refactored audit-server microservice application in the current audit-server.

audit-dispatcher.

• This should be micro-services for subscribing the audits from kafka ranger audit topic and send it into Solr / hdfs / opensearch and other destination.
• This should be configurable to take the form of Solr Audit dispatcher, HDFS audit dispatcher etc based on the destination type.
• Each dispatcher for destination type should be a separate service dedicated to commit the audits to respective destination

How was this patch tested?

Tested in Docker setup

//  Ranger Docker

docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml up -d

// Hadoop + Hive + Kafka

docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-hive.yml -f docker-compose.ranger-kafka.yml up -d

// start ranger audit server - Single one

docker compose  -f docker-compose.ranger.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-kafka.yml -f docker-compose.ranger-audit-server.yml  up -d

Commands run:

# HDFS load
docker exec -it ranger-hadoop bash -c '
export KRB5CCNAME=/tmp/krb5cc_hdfs_custom
kinit -kt /etc/keytabs/testuser1.keytab testuser1/[ranger-hadoop.rangernw@EXAMPLE.COM](mailto:ranger-hadoop.rangernw@EXAMPLE.COM)
for i in {1..80}; do
  echo "Iteration $i"
  hdfs dfs -ls /user/
  sleep 1
done'

[Copilot]

Pull request overview

This PR refactors the Ranger audit-server into two separable deployables: an audit-ingestor (plugin → REST → Kafka producer) and an audit-dispatcher (Kafka consumer(s) → Solr/HDFS), and updates distro assembly + Docker tooling accordingly.

Changes:

• Replaces the legacy audit-server module packaging with a new ranger-audit-ingestor WAR and introduces a new ranger-audit-dispatcher distribution with per-destination consumer WARs.
• Updates distro assembly descriptors and distro/pom.xml wiring to build/ship the new tarballs and WAR names/paths.
• Updates docker-compose and Dockerfiles/scripts to run the ingestor and unified dispatcher containers.

Reviewed changes

Copilot reviewed 45 out of 88 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
distro/src/main/assembly/audit-server.xml	Points audit-server distro assembly to the refactored audit-ingestor paths and WAR output.
distro/src/main/assembly/audit-dispatcher.xml	New assembly descriptor to package dispatcher scripts/configs and Solr/HDFS consumer WARs.
distro/src/main/assembly/audit-consumer-solr.xml	Updates legacy Solr consumer assembly paths to new audit-dispatcher module layout.
distro/src/main/assembly/audit-consumer-hdfs.xml	Updates legacy HDFS consumer assembly paths to new audit-dispatcher module layout.
distro/pom.xml	Updates distro dependencies and assembly descriptors to produce audit-server + audit-dispatcher artifacts.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-server.sh	Adapts startup script for audit-ingestor naming + backward-compatible env vars.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-solr.sh	Updates Solr consumer script to new extracted webapp dir naming and webapp-dir system prop.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-solr-site.xml	Refactors docker Solr consumer config to support unified startup script discovery keys.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-hdfs.sh	Updates HDFS consumer script to new extracted webapp dir naming and webapp-dir system prop.
dev-support/ranger-docker/scripts/audit-server/ranger-audit-consumer-hdfs-site.xml	Refactors docker HDFS consumer config to support unified startup script discovery keys.
dev-support/ranger-docker/docker-compose.ranger-audit-server.yml	Switches to audit-ingestor + audit-dispatcher containers; updates ports, commands, volumes, healthchecks.
dev-support/ranger-docker/Dockerfile.ranger-audit-ingestor	Builds a container for the audit-ingestor distribution and updated log/spool paths.
dev-support/ranger-docker/Dockerfile.ranger-audit-dispatcher	New unified dispatcher container that runs destination-specific consumer via start-audit-consumer.sh.
dev-support/ranger-docker/Dockerfile.ranger-audit-consumer-solr	Adjusts legacy Solr consumer Dockerfile comments for refactored origin.
dev-support/ranger-docker/Dockerfile.ranger-audit-consumer-hdfs	Adjusts legacy HDFS consumer Dockerfile comments for refactored origin.
dev-support/ranger-docker/.dockerignore	Updates included distro tarball set to include audit-dispatcher tarball.
audit-server/scripts/stop-all-services.sh	Minor output string change.
audit-server/scripts/start-all-services.sh	Minor output string change.
audit-server/pom.xml	Replaces child modules with audit-common, audit-dispatcher, audit-ingestor and adjusts shared deps.
audit-server/consumer-solr/scripts/stop-consumer-solr.sh	Removes legacy standalone Solr consumer stop script (superseded by dispatcher).
audit-server/consumer-solr/scripts/start-consumer-solr.sh	Removes legacy standalone Solr consumer start script (superseded by dispatcher).
audit-server/consumer-hdfs/scripts/stop-consumer-hdfs.sh	Removes legacy standalone HDFS consumer stop script (superseded by dispatcher).
audit-server/consumer-hdfs/scripts/start-consumer-hdfs.sh	Removes legacy standalone HDFS consumer start script (superseded by dispatcher).
audit-server/audit-ingestor/src/main/webapp/WEB-INF/web.xml	Adds ingestor web.xml for REST + Spring Security filter mapping.
audit-server/audit-ingestor/src/main/webapp/WEB-INF/security-applicationContext.xml	Adds Spring Security config for ingestor REST endpoints (JWT + delegation token filters).
audit-server/audit-ingestor/src/main/webapp/WEB-INF/applicationContext.xml	Adds Spring context wiring for ingestor component scan + scopes.
audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-server-site.xml	Updates ingestor webapp dir and renames kerberos properties under ranger.audit.server.*.
audit-server/audit-ingestor/src/main/resources/conf/logback.xml	Adds ingestor logback configuration.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/server/AuditServerConfig.java	Adds ingestor-specific config loader for ranger-audit-server-site.xml.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/server/AuditServerApplication.java	Updates app name to audit-ingestor.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/NullServletContext.java	Introduces a ServletContext stub used by ingestor security filters.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/FilterChainWrapper.java	Adds filter-chain wrapper to populate Spring Security context from auth cookie/remoteUser.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditJwtAuthFilter.java	Adds JWT auth filter integration using Ranger JWT handler configuration.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditDelegationTokenFilter.java	Updates delegation token filter config prefix to ranger.audit.server.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/security/AuditAuthEntryPoint.java	Adds entrypoint that returns 401 instead of redirecting.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/rest/RangerJsonProvider.java	Adds Jackson provider wiring using Ranger’s shared ObjectMapper.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditRecoveryManager.java	Adds recovery manager to coordinate writer/retry threads for Kafka outage spooling.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditProducer.java	Adds producer wrapper supporting idempotent config + batch send + selective retry.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditPartitioner.java	Adds plugin-aware partitioner for distributing audit events across topic partitions.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/AuditDestinationMgr.java	Adds ingestor component to initialize Kafka destination and log audit batches.
audit-server/audit-ingestor/src/main/java/javax/ws/rs/core/NoContentException.java	Adds a JAX-RS NoContentException shim to avoid jersey/jackson provider classloading failure.
audit-server/audit-ingestor/scripts/stop-audit-server.sh	Minor output string change.
audit-server/audit-ingestor/scripts/start-audit-server.sh	Minor output string change.
audit-server/audit-ingestor/pom.xml	Renames module/artifact to ranger-audit-ingestor, updates deps/plugins and final WAR name.
audit-server/audit-dispatcher/scripts/start-audit-consumer.sh	Adds unified consumer startup script selecting WAR + main class from type-specific config.
audit-server/audit-dispatcher/pom.xml	Adds dispatcher parent POM aggregating consumer-common, consumer-hdfs, consumer-solr.
audit-server/audit-dispatcher/consumer-solr/src/main/webapp/WEB-INF/web.xml	Adds Solr consumer web.xml for health endpoint via Jersey/Spring.
audit-server/audit-dispatcher/consumer-solr/src/main/webapp/WEB-INF/applicationContext.xml	Switches Solr consumer component scan to org.apache.ranger.audit.
audit-server/audit-dispatcher/consumer-solr/src/main/resources/conf/ranger-audit-consumer-solr-site.xml	Adds Solr consumer config including dispatcher startup metadata + Kafka/Solr destination settings.
audit-server/audit-dispatcher/consumer-solr/src/main/resources/conf/logback.xml	Adds Solr consumer logback configuration.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/server/SolrConsumerConfig.java	Loads common + Solr-specific config resources.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java	Adds Solr consumer health endpoint implementation.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/consumer/SolrConsumerManager.java	Adds Spring-managed lifecycle to create/start/stop Solr consumer threads.
audit-server/audit-dispatcher/consumer-solr/src/main/java/org/apache/ranger/audit/consumer/SolrConsumerApplication.java	Updates app name/config prefix usage for dispatcher-style execution.
audit-server/audit-dispatcher/consumer-solr/pom.xml	Updates parent pathing, Jersey deps/exclusions, internal deps, adds PMD and final WAR name.
audit-server/audit-dispatcher/consumer-hdfs/src/main/webapp/WEB-INF/web.xml	Adds HDFS consumer web.xml for health endpoint via Jersey/Spring.
audit-server/audit-dispatcher/consumer-hdfs/src/main/webapp/WEB-INF/applicationContext.xml	Switches HDFS consumer component scan to org.apache.ranger.audit.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/ranger-audit-consumer-hdfs-site.xml	Adds HDFS consumer config including dispatcher startup metadata + Kafka/HDFS destination settings.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/logback.xml	Adds HDFS consumer logback configuration.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/hdfs-site.xml	Adds HDFS client kerberos-related config for HDFS consumer deployments.
audit-server/audit-dispatcher/consumer-hdfs/src/main/resources/conf/core-site.xml	Adds core Hadoop security/auth_to_local config for HDFS consumer deployments.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/server/HdfsConsumerConfig.java	Loads common + HDFS-specific + core-site/hdfs-site configuration resources.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java	Adds HDFS consumer health endpoint implementation.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/consumer/HdfsConsumerManager.java	Adds Spring-managed lifecycle to create/start/stop HDFS consumer threads.
audit-server/audit-dispatcher/consumer-hdfs/src/main/java/org/apache/ranger/audit/consumer/HdfsConsumerApplication.java	Updates app name/config prefix usage for dispatcher-style execution.
audit-server/audit-dispatcher/consumer-hdfs/pom.xml	Updates parent pathing, Jersey deps/exclusions, internal deps, adds PMD and final WAR name.
audit-server/audit-dispatcher/consumer-common/src/main/resources/conf/ranger-audit-consumer-site.xml	Adds shared consumer service config (host/port/context + shared kafka/kerberos placeholders).
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerRegistry.java	Adds registry to manage destination factories and create consumers based on enabled destinations.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerRebalanceListener.java	Adds reusable rebalance listener to commit offsets and log assignment/revocation details.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerFactory.java	Adds functional interface for creating consumer instances.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumerBase.java	Adds base class for Kafka consumer configuration and shared client setup.
audit-server/audit-dispatcher/consumer-common/src/main/java/org/apache/ranger/audit/consumer/kafka/AuditConsumer.java	Adds common consumer interface contract.
audit-server/audit-dispatcher/consumer-common/pom.xml	Renames and defines the shared consumer-common artifact and dependencies/finalName.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/utils/AuditServerUtils.java	Introduces shared audit-server utility logic for destination config manipulation and topic readiness checks.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/utils/AuditServerLogFormatter.java	Adds structured log helper/builder for consistent startup and status logging.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/server/AuditServerConstants.java	Adds shared constants for server/consumer/producer configuration keys and defaults.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/server/AuditConfig.java	Adds shared configuration base class extending RangerConfiguration with resource loading helpers.
audit-server/audit-common/pom.xml	Adds ranger-audit-server-common shared jar module.

Comments suppressed due to low confidence (3)

dev-support/ranger-docker/scripts/audit-server/ranger-audit-server.sh:68

• The ingestor startup script still sets -Daudit.server.log.file=ranger-audit-server.log. Since this container now runs the audit-ingestor, consider updating the default log filename to something like ranger-audit-ingestor.log to avoid confusing log locations when both ingestor and legacy server naming exist.
  audit-server/audit-dispatcher/consumer-solr/pom.xml:260
• ranger-audit-consumer-common is declared a second time here (it already appears a few lines above with exclusions). This is redundant and can make dependency exclusions unclear; keep a single dependency entry and apply exclusions there if needed.
  audit-server/audit-dispatcher/consumer-hdfs/pom.xml:273
• ranger-audit-consumer-common is declared a second time here (it already appears a few lines above with exclusions). This is redundant and can make dependency exclusions unclear; keep a single dependency entry and apply exclusions there if needed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 88 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mneethiraj]

Is dependency on org.apache.ranger:ranger-audit-server-common necessary in distro module?

[mneethiraj]

Consider renaming audit-server.xml to audit-ingestor.xml.

[mneethiraj]

Please update audit-server to audit-ingestor in this file, and all other references - like war filename, docker container name, etc.

[mneethiraj]

Is volatile necessary on server and webappContext - given they are assigned only in start() method which is called from main()? Also, they are directly referenced only from start() and gracefulShutdown() methods.