Skip to content

Commit c8cb3e7

Browse files
vinodkcHyukjinKwon
authored andcommitted
[SPARK-54049][BUILD] Shade com.google.thirdparty package to fix Guava class conflicts in spark 4.0
We upgraded Guava from 14.0.1 to 30+ in  spark 4.0 . Guava 33.4.0 used in Spark 4 consists of two main packages: - `com.google.common` - `com.google.thirdparty` Prior to this PR, only the `com.google.common` package was shaded into the spark-network-common jar, while classes under `com.google.thirdparty` remained unshaded in the spark-network-common jar. This partial shading causes classloading conflicts and runtime errors when a downstream project depends on both Spark and its own version of Guava. Eg: calls to guava class `com.google.common.net.InternetDomainName` fails with the following error: ``` Caused by: java.lang.NoSuchFieldError: EXACT at com.google.common.net.InternetDomainName.findSuffixOfType(InternetDomainName.java:226) at com.google.common.net.InternetDomainName.publicSuffixIndex(InternetDomainName.java:185) at com.google.common.net.InternetDomainName.hasPublicSuffix(InternetDomainName.java:400) at com.eadx.Domain$.printDomainInfo(Domain.scala:16) at com.eadx.TestApp$.main(TestApp.scala:16) ``` **Root Cause**: `com.google.common.net.InternetDomainName` uses classes from `com.google.thirdparty.publicsuffix`. The classloader resolves `com.google.common.net.InternetDomainName` from the downstream Guava jar, while `com.google.thirdparty.publicsuffix.PublicSuffixPatterns` is loaded from Spark 4.x Guava classes, leading to binary incompatibility. Example diagnostic: ``` InternetDomainName → guava-32.0.0-jre.jar (target/.../guava-32.0.0-jre.jar) PublicSuffixPatterns → spark-network-common_2.13-4.0.0.jar (target/.../spark-network-common_2.13-4.0.0.jar) ``` ### What changes were proposed in this pull request? This PR ensures package `com.google.thirdparty` is also shaded and isolated under the sparkproject namespace in Spark, preventing downstream class conflicts and runtime errors. ### Why are the changes needed? These changes are necessary to prevent runtime errors and class conflicts for downstream projects that depend on both Spark and Guava by restoring proper isolation of shaded Guava classes in spark ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No new test cases added; used existing UT and IT. ### Was this patch authored or co-authored using generative AI tooling? No Closes #52767 from vinodkc/br_shade_guava_thirdparty. Authored-by: vinodkc <vinod.kc.in@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent c8d8cf2 commit c8cb3e7

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

pom.xml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3148,6 +3148,10 @@
31483148
<pattern>com.google.common</pattern>
31493149
<shadedPattern>${spark.shade.packageName}.guava</shadedPattern>
31503150
</relocation>
3151+
<relocation>
3152+
<pattern>com.google.thirdparty</pattern>
3153+
<shadedPattern>${spark.shade.packageName}.guava.thirdparty</shadedPattern>
3154+
</relocation>
31513155
<relocation>
31523156
<pattern>org.dmg.pmml</pattern>
31533157
<shadedPattern>${spark.shade.packageName}.dmg.pmml</shadedPattern>

0 commit comments

Comments
 (0)