Skip to content

Commit a1f8ea4

Browse files
pan3793LuciferYang
authored andcommitted
[SPARK-54636][BUILD][YARN] Correctly relocate Netty native libs for YARN ESS
### What changes were proposed in this pull request? Spark upgrade to Netty 4.2 since SPARK-53849, which brings several additional native libs, YARN ESS should correctly relocate them as it did for other existing Netty native libs. ### Why are the changes needed? Fix potential classpath conflicts issues for YARN ESS. ### Does this PR introduce _any_ user-facing change? No, Spark upgrades to Netty 4.2 in 4.1.0, which has not been released yet. ### How was this patch tested? Spark 4.0.1 ``` $ jar tf spark-4.0.1-yarn-shuffle.jar | grep META-INF/native/ | grep netty META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib META-INF/native/netty_tcnative_windows_x86_64.dll ``` master branch ``` $ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ | grep netty META-INF/native/libnetty_quiche42_linux_aarch_64.so META-INF/native/libnetty_quiche42_linux_x86_64.so META-INF/native/libnetty_quiche42_osx_aarch_64.jnilib META-INF/native/libnetty_quiche42_osx_x86_64.jnilib META-INF/native/libnetty_transport_native_io_uring42_aarch_64.so META-INF/native/libnetty_transport_native_io_uring42_riscv64.so META-INF/native/libnetty_transport_native_io_uring42_x86_64.so META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib META-INF/native/netty_quiche42_windows_x86_64.dll META-INF/native/netty_tcnative_windows_x86_64.dll ``` this PR ``` $ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ | grep netty META-INF/native/liborg_sparkproject_netty_quiche42_linux_aarch_64.so META-INF/native/liborg_sparkproject_netty_quiche42_linux_x86_64.so META-INF/native/liborg_sparkproject_netty_quiche42_osx_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_quiche42_osx_x86_64.jnilib META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_aarch_64.so META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_riscv64.so META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_x86_64.so META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib META-INF/native/netty_quiche42_windows_x86_64.dll META-INF/native/netty_tcnative_windows_x86_64.dll ``` Manually tested with Hadoop YARN (v3.4.2), with ESS enabled. <img width="1585" height="983" alt="image" src="https://github.com/user-attachments/assets/e7cfc0c9-05e5-4480-bd35-29946d1c17e9" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53382 from pan3793/SPARK-54636. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: yangjie01 <yangjie01@baidu.com> (cherry picked from commit a48e2e8) Signed-off-by: yangjie01 <yangjie01@baidu.com>
1 parent 46fcb05 commit a1f8ea4

File tree

1 file changed

+15
-0
lines changed

1 file changed

+15
-0
lines changed

common/network-yarn/pom.xml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,18 +171,25 @@
171171
<phase>package</phase>
172172
<configuration>
173173
<target>
174+
<!-- Follow https://github.com/netty/netty/blob/netty-4.2.7.Final/testsuite-shading/pom.xml to skip processing MS Windows DLLs -->
174175
<echo message="Shade netty native libraries to ${spark.shade.native.packageName}" />
175176
<unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/" />
176177
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so"
177178
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" />
179+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_x86_64.so"
180+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_x86_64.so" />
178181
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"
179182
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib" />
180183
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so"
181184
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" />
185+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_aarch_64.so"
186+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_aarch_64.so" />
182187
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
183188
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib" />
184189
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_riscv64.so"
185190
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_riscv64.so" />
191+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_riscv64.so"
192+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_riscv64.so" />
186193
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_x86_64.so"
187194
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_linux_x86_64.so" />
188195
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_aarch_64.so"
@@ -191,6 +198,14 @@
191198
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_x86_64.jnilib" />
192199
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_osx_aarch_64.jnilib"
193200
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_aarch_64.jnilib" />
201+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_linux_x86_64.so"
202+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_linux_x86_64.so" />
203+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_linux_aarch_64.so"
204+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_linux_aarch_64.so" />
205+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_osx_x86_64.jnilib"
206+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_osx_x86_64.jnilib" />
207+
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_osx_aarch_64.jnilib"
208+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_osx_aarch_64.jnilib" />
194209
<jar destfile="${shuffle.jar}" basedir="${project.build.directory}/exploded" />
195210
</target>
196211
</configuration>

0 commit comments

Comments
 (0)