Commit a1f8ea4
[SPARK-54636][BUILD][YARN] Correctly relocate Netty native libs for YARN ESS
### What changes were proposed in this pull request?
Spark upgrade to Netty 4.2 since SPARK-53849, which brings several additional native libs, YARN ESS should correctly relocate them as it did for other existing Netty native libs.
### Why are the changes needed?
Fix potential classpath conflicts issues for YARN ESS.
### Does this PR introduce _any_ user-facing change?
No, Spark upgrades to Netty 4.2 in 4.1.0, which has not been released yet.
### How was this patch tested?
Spark 4.0.1
```
$ jar tf spark-4.0.1-yarn-shuffle.jar | grep META-INF/native/ | grep netty
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_tcnative_windows_x86_64.dll
```
master branch
```
$ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ | grep netty
META-INF/native/libnetty_quiche42_linux_aarch_64.so
META-INF/native/libnetty_quiche42_linux_x86_64.so
META-INF/native/libnetty_quiche42_osx_aarch_64.jnilib
META-INF/native/libnetty_quiche42_osx_x86_64.jnilib
META-INF/native/libnetty_transport_native_io_uring42_aarch_64.so
META-INF/native/libnetty_transport_native_io_uring42_riscv64.so
META-INF/native/libnetty_transport_native_io_uring42_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_quiche42_windows_x86_64.dll
META-INF/native/netty_tcnative_windows_x86_64.dll
```
this PR
```
$ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ | grep netty
META-INF/native/liborg_sparkproject_netty_quiche42_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_quiche42_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_quiche42_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_quiche42_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_quiche42_windows_x86_64.dll
META-INF/native/netty_tcnative_windows_x86_64.dll
```
Manually tested with Hadoop YARN (v3.4.2), with ESS enabled.
<img width="1585" height="983" alt="image" src="https://github.com/user-attachments/assets/e7cfc0c9-05e5-4480-bd35-29946d1c17e9" />
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53382 from pan3793/SPARK-54636.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
(cherry picked from commit a48e2e8)
Signed-off-by: yangjie01 <yangjie01@baidu.com>1 parent 46fcb05 commit a1f8ea4
1 file changed
+15
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
171 | 171 | | |
172 | 172 | | |
173 | 173 | | |
| 174 | + | |
174 | 175 | | |
175 | 176 | | |
176 | 177 | | |
177 | 178 | | |
| 179 | + | |
| 180 | + | |
178 | 181 | | |
179 | 182 | | |
180 | 183 | | |
181 | 184 | | |
| 185 | + | |
| 186 | + | |
182 | 187 | | |
183 | 188 | | |
184 | 189 | | |
185 | 190 | | |
| 191 | + | |
| 192 | + | |
186 | 193 | | |
187 | 194 | | |
188 | 195 | | |
| |||
191 | 198 | | |
192 | 199 | | |
193 | 200 | | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
194 | 209 | | |
195 | 210 | | |
196 | 211 | | |
| |||
0 commit comments