Merge pull request #22 from SOFTNETWORK-APP/feature/select

- update client apis so as to use generic elastic result
- add elastic client companion
- add and implement scroll api : 
  - ✅ Retry with exponential backoff
  - ✅ Automatic tie-breaker for search_after
  - ✅ PIT (Point In Time) support for consistent snapshots
  - ✅ Configurable batch size
  - ✅ Shard failure detection
  - ✅ Resource cleanup
  - ✅ Detailed error messages
- add and implement metrics and monitoring
- add and implement SPI architecture for elastic client api 
- add specific module for persistence
- fix deprecated scala.collection.JavaConverters
- choose scroll strategy based on query type and underlying Elasticsearch version
- add documentation for all apis

Author: fupelaqu

.github/workflows/build.yml

@@ -44,6 +44,8 @@ jobs:
       uses: sbt/setup-sbt@v1
     - name: Set vm.max_map_count
       run: sudo sysctl -w vm.max_map_count=262144
+    - name: Cross Compile
+      run: SBT_OPTS="-Xss4M -Xms1g -Xmx4g -Dfile.encoding=UTF-8" sbt + compile
     - name: Run tests
       run: SBT_OPTS="-Xss4M -Xms1g -Xmx4g -Dfile.encoding=UTF-8" sbt compile test
 
@@ -65,5 +67,5 @@ jobs:
 #          cache: 'sbt'
       - name: Setup sbt launcher
         uses: sbt/setup-sbt@v1
-      - name: Formatting
-        run: sbt scalafmtSbtCheck scalafmtCheck test:scalafmtCheck
+      - name: Checks
+        run: sbt headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck

README.md

@@ -11,29 +11,34 @@
 ## Key Features
 
 **Unified Elasticsearch API**  
-This project provides a trait-based interface (`ElasticClientApi`) that aggregates the core functionalities of Elasticsearch: indexing, searching, updating, deleting, mapping, aliases, refreshing, and more. This design abstracts the underlying client implementation and ensures compatibility across different Elasticsearch versions.
+This project provides a trait-based interface (`ElasticClientApi`) that aggregates the core functionalities of Elasticsearch: [indexing](documentation/client/index.md), [updating](documentation/client/update.md), [deleting](documentation/client/delete.md), [bulk](documentation/client/bulk.md), [searching](documentation/client/search.md), [scrolling](documentation/client/scroll.md), [mapping](documentation/client/mappings.md), [aliases](documentation/client/aliases.md), [refreshing](documentation/client/refresh.md), and [more](documentation/client/README.md).  
+This design abstracts the underlying client implementation and ensures compatibility across different Elasticsearch versions.
 
-- `JestClientApi`: For Elasticsearch 6 using the open-source [Jest client](https://github.com/searchbox-io/Jest).
+- `JavaClientApi`: For Elasticsearch 8 and 9 using the official Java client.
 - `RestHighLevelClientApi`: For Elasticsearch 6 and 7 using the official high-level REST client.
-- `ElasticsearchClientApi`: For Elasticsearch 8 and 9 using the official Java client.
+- `JestClientApi`: For Elasticsearch 6 using the open-source [Jest client](https://github.com/searchbox-io/Jest).
 
 By relying on these concrete implementations, developers can switch between versions with minimal changes to their business logic.
 
 **SQL to Elasticsearch Query Translation**  
-Elastic Client includes a parser capable of translating SQL `SELECT` queries into Elasticsearch queries. The parser produces an intermediate representation, which is then converted into [Elastic4s](https://github.com/sksamuel/elastic4s) DSL queries and ultimately into native Elasticsearch queries. This allows data engineers and analysts to express queries in familiar [SQL](documentation/README.md) syntax.
+Elastic Client includes a parser capable of translating SQL `SELECT` queries into Elasticsearch queries. The parser produces an intermediate representation, which is then converted into [Elastic4s](https://github.com/sksamuel/elastic4s) DSL queries and ultimately into native Elasticsearch queries. This allows data engineers and analysts to express queries in familiar [SQL](documentation/sql/README.md) syntax.
 
 **Dynamic Mapping Migration**  
 Elastic Client provides tools to analyze and compare existing mappings with new ones. If differences are detected, it can automatically perform safe migrations. This includes creating temporary indices, reindexing, and renaming — all while preserving data integrity. This eliminates the need for manual mapping migrations and reduces downtime.
 
 **High-Performance Bulk API with Akka Streams**  
 Bulk operations leverage the power of Akka Streams to efficiently process and index large volumes of data. This stream-based approach improves performance, resilience, and backpressure handling, especially for real-time or high-throughput indexing scenarios.
 
+**Scroll API with automatic Scroll Strategy detection**  
+The Scroll API is also integrated with Akka Streams, enabling efficient retrieval of large datasets in a streaming fashion. This allows applications to process search results incrementally, reducing memory consumption and improving responsiveness.
+It automatically selects the optimal scrolling strategy (PIT + search_after, search_after, or classic scroll) based on your query and Elasticsearch version.
+
 **Akka Persistence Integration**  
 The project offers seamless integration with Akka Persistence. This enables Elasticsearch indices to be updated reactively based on persistent events, offering a robust pattern for event-sourced systems.
 
 ## Roadmap
 
-Future enhancements include expanding the SQL parser to support additional operations such as `INSERT`, `UPDATE`, and `DELETE`. The long-term vision is to deliver a fully functional, open-source **JDBC connector for Elasticsearch**, empowering users to interact with their data using standard SQL tooling.
+Future enhancements include expanding the SQL parser to support additional operations such as `CREATE`, `ALTER`, `INSERT`, `UPDATE`, and `DELETE`. The long-term vision is to deliver a fully functional, open-source **JDBC connector for Elasticsearch**, empowering users to interact with their data using standard SQL tooling.
 
 ## License
 

build.sbt

@@ -19,7 +19,7 @@ ThisBuild / organization := "app.softnetwork"
 
 name := "softclient4es"
 
-ThisBuild / version := "0.9.3"
+ThisBuild / version := "0.10.0"
 
 ThisBuild / scalaVersion := scala213
 
@@ -54,6 +54,22 @@ lazy val moduleSettings = Seq(
 
 ThisBuild / javacOptions ++= Seq("-source", "1.8", "-target", "1.8")
 
+ThisBuild / javaOptions ++= Seq(
+  "--add-opens=java.base/java.util=ALL-UNNAMED",
+  "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+  "--add-opens=java.base/java.lang=ALL-UNNAMED",
+  "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+  "--add-opens=java.base/java.math=ALL-UNNAMED",
+  "--add-opens=java.base/java.io=ALL-UNNAMED",
+  "--add-opens=java.base/java.net=ALL-UNNAMED",
+  "--add-opens=java.base/java.nio=ALL-UNNAMED",
+  "--add-opens=java.base/java.text=ALL-UNNAMED",
+  "--add-opens=java.base/java.time=ALL-UNNAMED",
+  "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
+)
+
+Test / javaOptions ++= (javaOptions.value)
+
 ThisBuild / resolvers ++= Seq(
   "Softnetwork Server" at "https://softnetwork.jfrog.io/artifactory/releases/",
   "Softnetwork Snapshots" at "https://softnetwork.jfrog.io/artifactory/snapshots/",
@@ -75,6 +91,7 @@ val json4s = Seq(
 ).map(_.excludeAll(jacksonExclusions: _*))
 
 ThisBuild / libraryDependencies ++= Seq(
+  "org.scala-lang.modules" %% "scala-collection-compat" % "2.11.0",
   "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
 ) // ++ configDependencies ++ json4s ++ logging
 
@@ -101,6 +118,17 @@ lazy val core = project
     sql % "compile->compile;test->test;it->it"
   )
 
+lazy val persistence = project
+  .in(file("persistence"))
+  .configs(IntegrationTest)
+  .settings(
+    Defaults.itSettings,
+    moduleSettings
+  )
+  .dependsOn(
+    core % "compile->compile;test->test;it->it"
+  )
+
 def copyTestkit(esVersion: String): Def.Initialize[Task[Unit]] = Def.task {
   val src = file("core/testkit")
   val target = baseDirectory.value
@@ -136,7 +164,7 @@ def testkitProject(esVersion: String, ss: Def.SettingsDefinition*): Project = {
     .settings(ss: _*)
     .enablePlugins(BuildInfoPlugin)
     .dependsOn(
-      core % "compile->compile;test->test;it->it"
+      persistence % "compile->compile;test->test;it->it"
     )
 }
 
@@ -383,6 +411,7 @@ lazy val root = project
   .aggregate(
     sql,
     core,
+    persistence,
     es6,
     es7,
     es8,

core/build.sbt

@@ -4,15 +4,32 @@ organization := "app.softnetwork.elastic"
 
 name := "softclient4es-core"
 
-val configDependencies = Seq(
-  "com.typesafe" % "config" % Versions.typesafeConfig
+val akka = Seq(
+  "com.typesafe.akka" %% "akka-actor" % Versions.akka,
+  "com.typesafe.akka" %% "akka-cluster-sharding-typed" % Versions.akka,
+  "com.typesafe.akka" %% "akka-slf4j" % Versions.akka,
+  "com.typesafe.akka" %% "akka-discovery" % Versions.akka,
+  "com.typesafe.akka" %% "akka-stream" % Versions.akka
+)
+
+val typesafeConfig = Seq(
+  "com.typesafe" % "config" % Versions.typesafeConfig,
+  "com.github.kxbmap" %% "configs" % Versions.kxbmap
+)
+
+val http = Seq(
+  "org.apache.httpcomponents" % "httpcore" % "4.4.12" % "provided"
 )
 
 val json4s = Seq(
   "org.json4s" %% "json4s-jackson" % Versions.json4s,
   "org.json4s" %% "json4s-ext" % Versions.json4s
 ).map(_.excludeAll(jacksonExclusions *))
 
-libraryDependencies ++= configDependencies ++
-json4s :+ "com.google.code.gson" % "gson" % Versions.gson :+
-("app.softnetwork.persistence" %% "persistence-core" % Versions.genericPersistence excludeAll (jacksonExclusions *))
+val mockito = Seq(
+  "org.mockito" %% "mockito-scala" % "1.17.12" % Test
+)
+
+libraryDependencies ++= akka ++ typesafeConfig ++ http ++
+json4s ++ mockito :+ "com.google.code.gson" % "gson" % Versions.gson :+
+  "com.typesafe.scala-logging" %% "scala-logging" % Versions.scalaLogging

core/src/main/resources/softnetwork-elastic.conf

@@ -1,11 +1,14 @@
 elastic {
-  ip = "localhost"
-  ip = ${?ELASTIC_IP}
+  # Connection settings
+  host = "localhost"
+  host = ${?ELASTIC_HOST}
+  host = ${?ELASTIC_IP} # Alternative environment variable for host IP for backward compatibility
   port = 9200
   port = ${?ELASTIC_PORT}
 
+  # Authentication
   credentials {
-    url      = "http://"${elastic.ip}":"${elastic.port}
+    url      = "http://"${elastic.host}":"${elastic.port}
     username = ""
     password = ""
 
@@ -15,7 +18,25 @@ elastic {
 
   }
 
+  # Performance
   multithreaded     = true
-  discovery-enabled = false
+  connection-timeout = 5s
+  socket-timeout     = 30s
 
+  # Cluster discovery
+  discovery {
+    enabled = false
+    frequency = 5m
+  }
+
+  # Metrics and Monitoring
+  metrics {
+    enabled = true
+    monitoring {
+      enabled = true
+      interval = 30s
+      failure-rate-threshold = 10.0  # Alert if > 10% failures
+      latency-threshold = 1000.0     # Alert if average latency > 1000ms
+    }
+  }
 }

core/src/main/scala-2.12/app/softnetwork/elastic/client/ElasticConfig.scala

@@ -1,13 +1,49 @@
+/*
+ * Copyright 2025 SOFTNETWORK
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
 package app.softnetwork.elastic.client
 
 import com.typesafe.config.{Config, ConfigFactory}
 import com.typesafe.scalalogging.StrictLogging
 import configs.Configs
 
+import java.time.Duration
+
+/** Complete Elasticsearch client configuration.
+ *
+ * @param credentials
+ *   Connection credentials (url, username, password)
+ * @param multithreaded
+ *   Enables multi-threaded mode for parallel operations
+ * @param discovery
+ *   Automatic cluster node discovery configuration
+ * @param connectionTimeout
+ *   Connection timeout to the cluster
+ * @param socketTimeout
+ *   Socket operation timeout
+ * @param metrics
+ *   Metrics and monitoring configuration
+ */
 case class ElasticConfig(
   credentials: ElasticCredentials = ElasticCredentials(),
   multithreaded: Boolean = true,
-  discoveryEnabled: Boolean = false
+  discovery: DiscoveryConfig,
+  connectionTimeout: Duration,
+  socketTimeout: Duration,
+  metrics: MetricsConfig
 )
 
 object ElasticConfig extends StrictLogging {

core/src/main/scala-2.13/app/softnetwork/elastic/client/ElasticConfig.scala

@@ -16,14 +16,35 @@
 
 package app.softnetwork.elastic.client
 
+import app.softnetwork.elastic.client.metrics.MetricsConfig
 import com.typesafe.config.{Config, ConfigFactory}
 import com.typesafe.scalalogging.StrictLogging
 import configs.ConfigReader
 
+import java.time.Duration
+
+/** Complete Elasticsearch client configuration.
+  *
+  * @param credentials
+  *   Connection credentials (url, username, password)
+  * @param multithreaded
+  *   Enables multi-threaded mode for parallel operations
+  * @param discovery
+  *   Automatic cluster node discovery configuration
+  * @param connectionTimeout
+  *   Connection timeout to the cluster
+  * @param socketTimeout
+  *   Socket operation timeout
+  * @param metrics
+  *   Metrics and monitoring configuration
+  */
 case class ElasticConfig(
   credentials: ElasticCredentials = ElasticCredentials(),
   multithreaded: Boolean = true,
-  discoveryEnabled: Boolean = false
+  discovery: DiscoveryConfig,
+  connectionTimeout: Duration,
+  socketTimeout: Duration,
+  metrics: MetricsConfig
 )
 
 object ElasticConfig extends StrictLogging {