[SPARK-53998][TESTS] Add addition E2E tests for RTM #52870
Conversation
|
Thank you, @jerrypeng . |
| import org.apache.spark.sql.test.TestSparkSession | ||
| import org.apache.spark.sql.types.{IntegerType, StringType, StructType} | ||
|
|
||
| class StreamRealTimeModeE2ESuite extends StreamRealTimeModeE2ESuiteBase { |
There was a problem hiding this comment.
Could you check the CI failure?
2025-11-04T07:01:24.2425213Z �[0m[�[0m�[31merror�[0m] �[0m�[0mFailed tests:�[0m
2025-11-04T07:01:24.2426468Z �[0m[�[0m�[31merror�[0m] �[0m�[0m org.apache.spark.sql.streaming.StreamRealTimeModeSuite�[0m
| override protected def createSparkSession = | ||
| new TestSparkSession( | ||
| new SparkContext( | ||
| "local[15]", |
There was a problem hiding this comment.
Can we use more smaller values like other tests? According to the commit logs, RTM test suites seem to use this kind of high values. I'm wondering if this is required for some reasons.
There was a problem hiding this comment.
@dongjoon-hyun when a streaming query runs in RTM, all stages of the query run concurrently. Which means the cluster needs to have the number of available slots equal to the total number of tasks across all stages.
|
Gentle ping, @jerrypeng . |
|
Gentle ping once more, @jerrypeng . |
|
Hi, @jerrypeng . Could you share us your thoughts about this PR? |
dongjoon-hyun
changed the title
|
Gentle ping, @jerrypeng . |
1 similar comment
|
Gentle ping, @jerrypeng . |
|
@dongjoon-hyun sorry for the delay. Let me take a look at the failure. |
jerrypeng
force-pushed
the
SPARK-53998-2
branch
from
dd64bea to
3a54608
Compare
|
Thank you, @jerrypeng . |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM.
Merged to master/4.1 for Apache Spark 4.1.0 RC3.
### What changes were proposed in this pull request? Add some additional end to end tests for RTM ### Why are the changes needed? To have better test coverage for RTM functionality ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A. Only tests are added ### Was this patch authored or co-authored using generative AI tooling? no Closes #52870 from jerrypeng/SPARK-53998-2. Authored-by: Jerry Peng <jerry.peng@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 7df7dad) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
@dongjoon-hyun thank you! |
|
Thank YOU! |
2 participants
What changes were proposed in this pull request?
Add some additional end to end tests for RTM
Why are the changes needed?
To have better test coverage for RTM functionality
Does this PR introduce any user-facing change?
no
How was this patch tested?
N/A. Only tests are added
Was this patch authored or co-authored using generative AI tooling?
no