一个很简单的分享
2015-12-29 本文已影响1674人
Troubleshooting
首先说一下分享起因，前几天下载 openjdk 源码时，官方提倡是这样：
$ hg clone http://hg.openjdk.java.net/jdk9/jdk9
$ cd jdk9
$ bash get_sources.sh
因为网络原因，这问题首先是出在第一条命令上，反复几次都中途报错下不完。
就在这种等待时间不能确定，而且运行结果又未知的情况下
我把它写成如下脚本，运行之后我去睡觉，第二天一看已下完！
#!/usr/bin/bash
hg clone http://hg.openjdk.java.net/jdk9/jdk9
while [ $? -ne 0 ]
do
hg clone http://hg.openjdk.java.net/jdk9/jdk9
done
当然这只是开始，真正的源码还没开始下载，接下来对下载下来的 jdk9/get_sources.sh 还要继续做类似改动，改动部分在文件底部，改成这样之后即可，还是一样在下边加上 while 循环：
# Get clones of all absent nested repositories (harmless if already exist)
sh ./common/bin/hgforest.sh clone "$@"
while [ $? -ne 0 ]
do
sh ./common/bin/hgforest.sh clone "$@"
done
# Update all existing repositories to the latest sources
sh ./common/bin/hgforest.sh pull -u
while [ $? -ne 0 ]
do
sh ./common/bin/hgforest.sh pull -u
done
注意观察上边脚本，其实我第一次尝试改动的是这个文件jdk9/common/bin/hgforest.sh，只不过这个不好改，而且像这种用版本控制工具来下载代码，如果一个目录已经成功下载完成，它是会预先检查出来而不会重新下载的，所以上边的改动方法就是最简单的，起因讲述完毕。
接下来的分享回到我们熟悉的 git 上边来，还是众所周知的网络原因，在github上下载代码也时常会有上述情况发生，如果时间不是问题，例如说一宿时间下完就行，那么上述方法依然可行。
接下来继续引申一下，如果想从github上几乎同时下载多个项目源码，或者再退一步，第一个还没下完，我就想下第二个、第三个的话。到底是多开窗口，多开选项卡，还是后台运行。如果此时把之前的网络原因考虑在内，DRY断定前述种种皆不可行。可能已经被猜到了，应该写一个“多线程”的bash脚本来完成这件事。
至于“多线程”的bash脚本……，但是我想到了一个更好的方法，借助 Makefile来做这件事异常简单！
首先写一个稍微通用一点的上述简单脚本，可以叫做 get_any_sources.sh：
#!/usr/bin/bash
git clone https://github.com/$1.git
while [ $? -ne 0 ]
do
git clone https://github.com/$1.git
done
再写一个仅有两行的 Makefile即可：
%:
bash get_any_sources.sh $@
接下来随便在 github 上找一些项目来下载，像下面这样来用：
# nice make -j4 docker/docker sunspot/sunspot apple/swift facebook/react
注意 -j 4 要加才不枉费上边的两行 Makefile，不然就没有同时下载的效果了，至于 -j 几 至少是同时下载的项目数，另外 nice 并不是必需的。
运行结果如下，发现没有 docker 第一次下载时失败了。
bash get_sources.sh docker/docker
bash get_sources.sh sunspot/sunspot
bash get_sources.sh apple/swift
bash get_sources.sh facebook/react
Cloning into 'docker'...
Cloning into 'sunspot'...
Cloning into 'react'...
Cloning into 'swift'...
remote: Counting objects: 314457, done.
remote: Compressing objects: 100% (7/7), done.
remote: Counting objects: 17273, done.
remote: Counting objects: 65503, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 314457 (delta 0), reused 0 (delta 0), pack-reused 314450
Receiving objects: 100% (314457/314457), 64.27 MiB | 1.95 MiB/s, done.
Resolving deltas: 100% (257725/257725), done.B | 43.00 KiB/s
Checking connectivity... done.
fatal: unable to access 'https://github.com/docker/docker.git/': SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Cloning into 'docker'...
remote: Counting objects: 126929, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 126929 (delta 6), reused 2 (delta 2), pack-reused 126917
Receiving objects: 100% (126929/126929), 76.08 MiB | 1.08 MiB/s, done.
Resolving deltas: 100% (84561/84561), done. MiB | 73.00 KiB/s
Checking connectivity... done.
remote: Total 65503 (delta 0), reused 0 (delta 0), pack-reused 65490
Receiving objects: 100% (65503/65503), 73.17 MiB | 119.00 KiB/s, done.
Resolving deltas: 100% (41363/41363), done. MiB | 68.00 KiB/s
Checking connectivity... done.
remote: Total 17273 (delta 0), reused 0 (delta 0), pack-reused 17273
Receiving objects: 100% (17273/17273), 148.55 MiB | 197.00 KiB/s, done.
Resolving deltas: 100% (11316/11316), done.
Checking connectivity... done.
最后用事实说明一下这个如此简单的分享的意义所在，你们看下我本地下载 jdk9 源码时多少次下载失败！每一段 WARNING *** 之后的 # Repositories: ... 就是一次中途失败之后的重新开始。
It works @ jdk9 $ bash get_source.sh
# Repositories: corba jaxp jaxws langtools jdk hotspot nashorn
corba: hg clone http://hg.openjdk.java.net/jdk9/jdk9/corba corba
jaxp: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jaxp jaxp
corba: requesting all changes
jaxp: requesting all changes
corba: adding changesets
jaxp: adding changesets
corba: adding manifests
corba: adding file changes
jaxp: adding manifests
jaxp: adding file changes
corba: added 735 changesets with 5227 changes to 2580 files
corba: updating to branch default
corba: 1186 files updated, 0 files merged, 0 files removed, 0 files unresolved
jaxws: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jaxws jaxws
jaxws: requesting all changes
jaxws: adding changesets
jaxws: adding manifests
jaxws: adding file changes
jaxp: added 859 changesets with 11287 changes to 8203 files
jaxp: updating to branch default
jaxp: 3174 files updated, 0 files merged, 0 files removed, 0 files unresolved
langtools: hg clone http://hg.openjdk.java.net/jdk9/jdk9/langtools langtools
langtools: requesting all changes
langtools: adding changesets
langtools: adding manifests
langtools: adding file changes
langtools: added 3182 changesets with 29716 changes to 9483 files
jaxws: transaction abort!
jaxws: rollback completed
langtools: updating to branch default
jaxws: abort: stream ended unexpectedly (got 16588 bytes, expected 32956)
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
langtools: 7504 files updated, 0 files merged, 0 files removed, 0 files unresolved
hotspot: hg clone http://hg.openjdk.java.net/jdk9/jdk9/hotspot hotspot
jdk: requesting all changes
jdk: adding changesets
hotspot: requesting all changes
hotspot: adding changesets
hotspot: adding manifests
jdk: adding manifests
hotspot: adding file changes
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 13753 bytes, expected 16914)
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 6117 bytes, expected 6563)
hotspot: transaction abort!
hotspot: rollback completed
hotspot: abort: stream ended unexpectedly (got 1111 bytes, expected 2764)
WARNING: hotspot exited abnormally (255)
WARNING: jaxws exited abnormally (255)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jaxws jdk hotspot nashorn
jaxws: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jaxws jaxws
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
jdk: requesting all changes
jaxws: requesting all changes
jaxws: adding changesets
jdk: adding changesets
jaxws: adding manifests
jaxws: transaction abort!
jaxws: rollback completed
jaxws: abort: connection ended unexpectedly
hotspot: hg clone http://hg.openjdk.java.net/jdk9/jdk9/hotspot hotspot
hotspot: requesting all changes
hotspot: adding changesets
hotspot: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 322 bytes, expected 327)
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
hotspot: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 3043939 bytes, expected 5631663)
hotspot: added 9699 changesets with 55379 changes to 7785 files
hotspot: updating to branch default
hotspot: 5911 files updated, 0 files merged, 0 files removed, 0 files unresolved
WARNING: jaxws exited abnormally (255)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jaxws jdk nashorn
jaxws: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jaxws jaxws
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
jdk: requesting all changes
jdk: adding changesets
jaxws: requesting all changes
jaxws: adding changesets
jaxws: adding manifests
jaxws: adding file changes
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 39 bytes, expected 82)
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jaxws: transaction abort!
jaxws: rollback completed
jaxws: abort: stream ended unexpectedly (got 1998 bytes, expected 10818)
WARNING: jaxws exited abnormally (255)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jaxws jdk nashorn
jaxws: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jaxws jaxws
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
jaxws: requesting all changes
jaxws: adding changesets
jaxws: adding manifests
jdk: requesting all changes
jdk: adding changesets
jaxws: adding file changes
jaxws: added 654 changesets with 21107 changes to 10761 files
jaxws: updating to branch default
jaxws: 3717 files updated, 0 files merged, 0 files removed, 0 files unresolved
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 9281 bytes, expected 12428)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 322 bytes, expected 327)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
nashorn: requesting all changes
nashorn: adding changesets
jdk: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 2 bytes, expected 80)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
jdk: requesting all changes
jdk: adding changesets
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 32 bytes, expected 8245)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
nashorn: requesting all changes
jdk: adding changesets
nashorn: adding changesets
nashorn: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 103 bytes, expected 245)
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: connection ended unexpectedly
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
jdk: adding changesets
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 208 bytes, expected 304)
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 9154 bytes, expected 13169)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
jdk: requesting all changes
jdk: adding changesets
nashorn: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: connection ended unexpectedly
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: Connection reset by peer
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
jdk: requesting all changes
jdk: adding changesets
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 19253 bytes, expected 22530)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 322 bytes, expected 327)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
jdk: adding changesets
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
jdk: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 100683 bytes, expected 134527)
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 133822 bytes, expected 192443)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
jdk: adding changesets
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 322 bytes, expected 327)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
nashorn: requesting all changes
nashorn: adding changesets
jdk: adding changesets
nashorn: adding manifests
nashorn: adding file changes
jdk: adding manifests
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 28837 bytes, expected 73222)
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 44518 bytes, expected 68365)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
nashorn: requesting all changes
jdk: adding changesets
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 103 bytes, expected 245)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
jdk: adding changesets
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
jdk: transaction abort!
jdk: rollback completed
jdk: abort: stream ended unexpectedly (got 128 bytes, expected 542)
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 6 bytes, expected 80)
WARNING: jdk exited abnormally (255)
WARNING: nashorn exited abnormally (255)
# Repositories: jdk nashorn
jdk: hg clone http://hg.openjdk.java.net/jdk9/jdk9/jdk jdk
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
jdk: requesting all changes
jdk: adding changesets
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
jdk: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 5259 bytes, expected 5633)
jdk: adding file changes
jdk: added 13344 changesets with 128118 changes to 44542 files
jdk: updating to branch default
jdk: 24310 files updated, 0 files merged, 0 files removed, 0 files unresolved
WARNING: nashorn exited abnormally (255)
# Repositories: nashorn
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: transaction abort!
nashorn: rollback completed
nashorn: abort: stream ended unexpectedly (got 3763 bytes, expected 12310)
WARNING: nashorn exited abnormally (255)
# Repositories: nashorn
nashorn: hg clone http://hg.openjdk.java.net/jdk9/jdk9/nashorn nashorn
nashorn: requesting all changes
nashorn: adding changesets
nashorn: adding manifests
nashorn: adding file changes
nashorn: added 1581 changesets with 13304 changes to 3961 files
nashorn: updating to branch default
nashorn: 3093 files updated, 0 files merged, 0 files removed, 0 files unresolved
# Repositories: . corba jaxp jaxws langtools jdk hotspot nashorn
.: cd . && hg pull -u
corba: cd corba && hg pull -u
jaxp: cd jaxp && hg pull -u
jaxws: cd jaxws && hg pull -u
.: pulling from http://hg.openjdk.java.net/jdk9/jdk9
langtools: cd langtools && hg pull -u
corba: pulling from http://hg.openjdk.java.net/jdk9/jdk9/corba
jdk: cd jdk && hg pull -u
jaxp: pulling from http://hg.openjdk.java.net/jdk9/jdk9/jaxp
hotspot: cd hotspot && hg pull -u
nashorn: cd nashorn && hg pull -u
jaxws: pulling from http://hg.openjdk.java.net/jdk9/jdk9/jaxws
jdk: pulling from http://hg.openjdk.java.net/jdk9/jdk9/jdk
nashorn: pulling from http://hg.openjdk.java.net/jdk9/jdk9/nashorn
langtools: pulling from http://hg.openjdk.java.net/jdk9/jdk9/langtools
hotspot: pulling from http://hg.openjdk.java.net/jdk9/jdk9/hotspot
.: searching for changes
.: no changes found
jaxp: searching for changes
jaxp: no changes found
jaxws: searching for changes
jaxws: no changes found
langtools: searching for changes
langtools: no changes found
nashorn: searching for changes
nashorn: no changes found
jdk: searching for changes
jdk: no changes found
hotspot: searching for changes
hotspot: no changes found
corba: searching for changes
corba: no changes found
