Commit cd86533
committed
Try to use jgo for classpath instead of shaded fat jar
Needs
- mvn on classpath
- jgo from [resolve-only](apposed/jgo#37) pull request (Python >= 3.7)
Currently fails with:
```
Unable to convert into Paintera dataset: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 736, 10.36.110.15, executor 69): java.lang.NoSuchMethodError: org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream.<init>(Ljava/io/Ou
at org.janelia.saalfeldlab.n5.GzipCompression.getOutputStream(GzipCompression.java:89)
at org.janelia.saalfeldlab.n5.DefaultBlockWriter.write(DefaultBlockWriter.java:49)
at org.janelia.saalfeldlab.n5.DefaultBlockWriter.writeBlock(DefaultBlockWriter.java:83)
at org.janelia.saalfeldlab.n5.N5FSWriter.writeBlock(N5FSWriter.java:133)
at org.janelia.saalfeldlab.n5.imglib2.N5LabelMultisets.saveLabelMultisetBlock(N5LabelMultisets.java:308)
at org.janelia.saalfeldlab.n5.imglib2.N5LabelMultisets.saveLabelMultisetBlock(N5LabelMultisets.java:337)
at org.janelia.saalfeldlab.label.spark.convert.ConvertToLabelMultisetType.lambda$convertToLabelMultisetType$dce05d2$1(ConvertToLabelMultisetType.java:226)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:185)
at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:1015)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:1013)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
Probably dependency issue between spark and N5 (or other libraries) that is not an issue with shaded fat jar.1 parent 4718964 commit cd86533
1 file changed
+62
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
0 commit comments