spark 报错

] ERROR org.apache.spark.executor.Executor[91] - Exception in task 1.3 in stage 1686049.0 (TID 5731454)
java.io.FileNotFoundException: /hadoop1/hadoop/yarn/local/usercache/hdfs/appcache/application_1499847020960_0173/blockmgr-5bff41f0-76e3-4acc-bda5-1cef1bc3b57f/28/shuffle_596533_1_0.index.4a9bcf6e-ce3d-4a5d-b44a-b795dc6c6fa5 (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:143)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:127)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

2017-08-08 11:12:37 [block-manager-slave-async-thread-pool-1074] INFO org.apache.spark.storage.BlockManager[54] - Removing RDD 2984024
2017-08-08 11:12:37 [dispatcher-event-loop-25] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726286
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 0.0 in stage 1312496.0 (TID 4726286)
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931309
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931309_piece0 stored as bytes in memory (estimated size 2.5 KB, free 1007.5 MB)
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931309 took 241 ms
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931309 stored as values in memory (estimated size 4.1 KB, free 1007.5 MB)
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.streaming.kafka010.KafkaRDD[54] - Computing topic caiwudb.fso_yao_db.result, partition 0 offsets 152725 -> 152726
2017-08-08 11:12:37 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 0.0 in stage 1312496.0 (TID 4726286). 1906 bytes result sent to driver
2017-08-08 11:12:38 [dispatcher-event-loop-11] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726288
2017-08-08 11:12:38 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 1.0 in stage 1312497.0 (TID 4726288)
2017-08-08 11:12:38 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Updating epoch to 430497 and clearing cache
2017-08-08 11:12:38 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931310
[Full GC (Allocation Failure) [CMS[CMS-concurrent-preclean: 3.041/3.202 secs] [Times: user=7.55 sys=0.89, real=3.21 secs]
(concurrent mode failure): 1398144K->1398143K(1398144K), 15.5453208 secs] 2027264K->1846473K(2027264K), [Metaspace: 61392K->61392K(1105920K)], 15.5545389 secs] [Times: user=15.53 sys=0.02, real=15.56 secs]
2017-08-08 11:12:54 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931310_piece0 stored as bytes in memory (estimated size 2.5 KB, free 1007.5 MB)
2017-08-08 11:12:54 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931310 took 16423 ms
2017-08-08 11:12:54 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931310 stored as values in memory (estimated size 4.8 KB, free 1007.5 MB)
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 12.0319500 secs] 2027263K->1901436K(2027264K), [Metaspace: 61392K->61392K(1105920K)], 12.0385956 secs] [Times: user=12.00 sys=0.04, real=12.04 secs]
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Don't have map outputs for shuffle 430407, fetching them
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@10.120.65.17:45941)
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Got the output locations
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Getting 1 non-empty blocks out of 2 blocks
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Started 0 remote fetches in 36 ms
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block rdd_2987221_1 stored as values in memory (estimated size 10.0 KB, free 1007.5 MB)
2017-08-08 11:13:07 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 1.0 in stage 1312497.0 (TID 4726288). 2708 bytes result sent to driver
[GC (CMS Initial Mark) [1 CMS-initial-mark: 1398143K(1398144K)] 1958900K(2027264K), 1.1663336 secs] [Times: user=1.55 sys=0.00, real=1.17 secs]
[CMS-concurrent-mark-start]
2017-08-08 11:13:08 [dispatcher-event-loop-12] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726298
2017-08-08 11:13:08 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 4.0 in stage 1312498.0 (TID 4726298)
2017-08-08 11:13:08 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Updating epoch to 430498 and clearing cache
2017-08-08 11:13:08 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931311
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931311_piece0 stored as bytes in memory (estimated size 1974.0 B, free 1007.5 MB)
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931311 took 297 ms
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931311 stored as values in memory (estimated size 3.5 KB, free 1007.4 MB)
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Don't have map outputs for shuffle 430408, fetching them
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@10.120.65.17:45941)
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Got the output locations
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Getting 0 non-empty blocks out of 6 blocks
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Started 0 remote fetches in 0 ms
2017-08-08 11:13:09 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 4.0 in stage 1312498.0 (TID 4726298). 2364 bytes result sent to driver
[CMS-concurrent-mark: 0.947/0.947 secs] [Times: user=8.75 sys=0.31, real=0.95 secs]
[CMS-concurrent-preclean-start]
[Full GC (Allocation Failure) [CMS[CMS-concurrent-preclean: 2.397/2.549 secs] [Times: user=3.64 sys=0.10, real=2.55 secs]
(concurrent mode failure): 1398144K->1398143K(1398144K), 14.3311618 secs] 2027263K->1924237K(2027264K), [Metaspace: 61392K->61392K(1105920K)], 14.3384912 secs] [Times: user=14.33 sys=0.01, real=14.33 secs]
2017-08-08 11:13:10 [dispatcher-event-loop-2] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726302
2017-08-08 11:13:24 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 1.0 in stage 1312500.0 (TID 4726302)
2017-08-08 11:13:25 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931312
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 12.7133963 secs] 2027263K->1937836K(2027264K), [Metaspace: 61392K->61392K(1105920K)], 12.7203709 secs] [Times: user=12.73 sys=0.01, real=12.72 secs]
[GC (CMS Initial Mark) [1 CMS-initial-mark: 1398143K(1398144K)] 1958079K(2027264K), 1.0068300 secs] [Times: user=1.22 sys=0.04, real=1.01 secs]
[CMS-concurrent-mark-start]
2017-08-08 11:13:39 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931312_piece0 stored as bytes in memory (estimated size 2.5 KB, free 1007.4 MB)
2017-08-08 11:13:40 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931312 took 14203 ms
2017-08-08 11:13:40 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931312 stored as values in memory (estimated size 4.8 KB, free 1007.4 MB)
[CMS-concurrent-mark: 1.014/1.014 secs] [Times: user=9.20 sys=0.41, real=1.02 secs]
[CMS-concurrent-preclean-start]
2017-08-08 11:13:41 [Executor task launch worker-3] INFO org.apache.spark.storage.BlockManager[54] - Found block rdd_2987221_1 locally
2017-08-08 11:13:41 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block rdd_2987233_1 stored as values in memory (estimated size 9.4 KB, free 1007.4 MB)
[Full GC (Allocation Failure) [CMS[CMS-concurrent-preclean: 3.047/3.255 secs] [Times: user=5.63 sys=0.33, real=3.26 secs]
(concurrent mode failure): 1398144K->1398143K(1398144K), 15.4392635 secs] 2027263K->1905416K(2027264K), [Metaspace: 61392K->61392K(1105920K)], 15.4510082 secs] [Times: user=15.44 sys=0.05, real=15.45 secs]
2017-08-08 11:13:56 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 1.0 in stage 1312500.0 (TID 4726302). 2498 bytes result sent to driver
2017-08-08 11:13:57 [dispatcher-event-loop-10] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726306
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 5.0 in stage 1312501.0 (TID 4726306)
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Updating epoch to 430499 and clearing cache
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931313
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931313_piece0 stored as bytes in memory (estimated size 1974.0 B, free 1007.4 MB)
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931313 took 243 ms
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931313 stored as values in memory (estimated size 3.5 KB, free 1007.4 MB)
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Don't have map outputs for shuffle 430409, fetching them
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@10.120.65.17:45941)
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.MapOutputTrackerWorker[54] - Got the output locations
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Getting 1 non-empty blocks out of 6 blocks
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.storage.ShuffleBlockFetcherIterator[54] - Started 0 remote fetches in 0 ms
2017-08-08 11:13:57 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 5.0 in stage 1312501.0 (TID 4726306). 7229 bytes result sent to driver
[Full GC (Allocation Failure) [CMS: 1398144K->1398144K(1398144K), 12.7199865 secs] 2027263K->1883155K(2027264K), [Metaspace: 61370K->61370K(1105920K)], 12.7270290 secs] [Times: user=12.72 sys=0.02, real=12.72 secs]
2017-08-08 11:13:58 [dispatcher-event-loop-26] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726312
2017-08-08 11:14:11 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 1.0 in stage 1312503.0 (TID 4726312)
2017-08-08 11:14:11 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931314
[GC (CMS Initial Mark) [1 CMS-initial-mark: 1398144K(1398144K)] 1957495K(2027264K), 1.0782571 secs] [Times: user=1.64 sys=0.00, real=1.08 secs]
[CMS-concurrent-mark-start]
2017-08-08 11:14:12 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931314_piece0 stored as bytes in memory (estimated size 2.4 KB, free 1007.4 MB)
2017-08-08 11:14:12 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931314 took 1549 ms
2017-08-08 11:14:12 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931314 stored as values in memory (estimated size 4.5 KB, free 1007.4 MB)
2017-08-08 11:14:12 [Executor task launch worker-3] INFO org.apache.spark.storage.BlockManager[54] - Found block rdd_2987233_1 locally
2017-08-08 11:14:12 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Finished task 1.0 in stage 1312503.0 (TID 4726312). 1983 bytes result sent to driver
[CMS-concurrent-mark: 1.177/1.227 secs] [Times: user=9.81 sys=0.58, real=1.23 secs]
[CMS-concurrent-preclean-start]
[Full GC (Allocation Failure) [CMS[CMS-concurrent-preclean: 3.131/3.384 secs] [Times: user=3.58 sys=0.08, real=3.38 secs]
(concurrent mode failure): 1398144K->1398143K(1398144K), 16.1807502 secs] 2027263K->1927225K(2027264K), [Metaspace: 61370K->61370K(1105920K)], 16.1943176 secs] [Times: user=16.18 sys=0.01, real=16.20 secs]
2017-08-08 11:14:13 [dispatcher-event-loop-13] INFO org.apache.spark.executor.CoarseGrainedExecutorBackend[54] - Got assigned task 4726318
2017-08-08 11:14:30 [Executor task launch worker-3] INFO org.apache.spark.executor.Executor[54] - Running task 1.0 in stage 1312505.0 (TID 4726318)
2017-08-08 11:14:31 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Started reading broadcast variable 931315
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 12.5547268 secs] 2027263K->1941909K(2027264K), [Metaspace: 61370K->61370K(1105920K)], 12.5627255 secs] [Times: user=12.57 sys=0.02, real=12.56 secs]
[GC (CMS Initial Mark) [1 CMS-initial-mark: 1398143K(1398144K)] 1961591K(2027264K), 1.3199670 secs] [Times: user=1.47 sys=0.00, real=1.33 secs]
[CMS-concurrent-mark-start]
[CMS-concurrent-mark: 0.998/0.998 secs] [Times: user=9.46 sys=0.53, real=1.00 secs]
[CMS-concurrent-preclean-start]
2017-08-08 11:14:46 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931315_piece0 stored as bytes in memory (estimated size 14.5 KB, free 1007.4 MB)
[Full GC (Allocation Failure) [CMS[CMS-concurrent-preclean: 2.587/2.707 secs] [Times: user=3.67 sys=0.11, real=2.71 secs]
(concurrent mode failure): 1398143K->1398144K(1398144K), 14.9466043 secs] 2027263K->1981147K(2027264K), [Metaspace: 61370K->61370K(1105920K)], 14.9532691 secs] [Times: user=14.95 sys=0.01, real=14.95 secs]
2017-08-08 11:15:02 [Executor task launch worker-3] INFO org.apache.spark.broadcast.TorrentBroadcast[54] - Reading broadcast variable 931315 took 30519 ms
[Full GC (Allocation Failure) [CMS: 1398144K->1398144K(1398144K), 12.6370073 secs] 2027263K->2014671K(2027264K), [Metaspace: 61370K->61370K(1105920K)], 12.6445066 secs] [Times: user=12.65 sys=0.00, real=12.64 secs]
[GC (CMS Initial Mark) [1 CMS-initial-mark: 1398144K(1398144K)] 2016682K(2027264K), 1.0483755 secs] [Times: user=1.32 sys=0.01, real=1.05 secs]
[CMS-concurrent-mark-start]
2017-08-08 11:15:16 [Executor task launch worker-3] INFO org.apache.spark.storage.memory.MemoryStore[54] - Block broadcast_931315 stored as values in memory (estimated size 37.8 KB, free 1007.4 MB)
[Full GC (Allocation Failure) [CMS[CMS-concurrent-mark: 0.774/0.895 secs] [Times: user=5.59 sys=0.13, real=0.90 secs]
(concurrent mode failure): 1398144K->1398144K(1398144K), 11.8773907 secs] 2027263K->2022652K(2027264K), [Metaspace: 60885K->60885K(1105920K)], 11.8840551 secs] [Times: user=14.54 sys=0.03, real=11.88 secs]
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 12.6779874 secs] 2027263K->2025250K(2027264K), [Metaspace: 60877K->60877K(1105920K)], 12.6863345 secs] [Times: user=12.69 sys=0.01, real=12.69 secs]
[Full GC (Allocation Failure) [CMS: 1398144K->1398144K(1398144K), 14.6889951 secs] 2027263K->2026904K(2027264K), [Metaspace: 60625K->60625K(1105920K)], 14.6975841 secs] [Times: user=14.65 sys=0.00, real=14.69 secs]
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 13.7280953 secs] 2027229K->2027054K(2027264K), [Metaspace: 60483K->60483K(1105920K)], 13.7586430 secs] [Times: user=13.78 sys=0.01, real=13.76 secs]
[Full GC (Allocation Failure) [CMS: 1398143K->1398143K(1398144K), 12.8305680 secs] 2027228K->2027106K(2027264K), [Metaspace: 60483K->60483K(1105920K)], 12.8374358 secs] [Times: user=12.84 sys=0.01, real=12.84 secs]
[Full GC (Allocation Failure) [CMS: 1398143K->1398143K(1398144K), 13.1167582 secs] 2027263K->2027231K(2027264K), [Metaspace: 60431K->60431K(1105920K)], 13.1215934 secs] [Times: user=13.13 sys=0.00, real=13.12 secs]
[Full GC (Allocation Failure) [CMS: 1398143K->1398143K(1398144K), 13.2881426 secs] 2027241K->2027232K(2027264K), [Metaspace: 60431K->60431K(1105920K)], 13.2938949 secs] [Times: user=13.30 sys=0.01, real=13.29 secs]
[Full GC (Allocation Failure) [CMS: 1398143K->1398143K(1398144K), 13.8317961 secs] 2027232K->2027144K(2027264K), [Metaspace: 60431K->60431K(1105920K)], 13.8371026 secs] [Times: user=13.85 sys=0.01, real=13.84 secs]
[Full GC (Allocation Failure) [CMS: 1398144K->1398143K(1398144K), 12.9987488 secs] 2027264K->2027246K(2027264K), [Metaspace: 60422K->60422K(1105920K)], 13.0061373 secs] [Times: user=13.02 sys=0.02, real=13.00 secs]
[Full GC (Allocation Failure) [CMS