2019-04-18
1. data 的下载，data的完整性，data的打包
2. 下载数据 wget curl
3. wget --accept "*.gtf" --no-directories --recursive --no-parent \
http://genomics.someuniversity.edu/labsite/annotation.html
从一个页面下载自己想要的格式的数据 gtf
4.Rcurl pycurl
5. rsync 比较传递大文件
6. scp 快速拷贝一个文件
7.数据完整性检验 checksum algorithms MD5 SHA-1
8. echo "Bioinformatics is fun" | shasum
echo "Bioinformatic is fun" | shasum
9. shasum data/*fastq > fastq_checksums.sha
然后
shasum -c fastq_checksums.sha
10 md5sum
11. diff 计算文件的不同点在哪
先用checksum比较出来文件不同，然后用diff比较不同点在哪
diff -u gene-1.bed gene-2.bed
12 patch
13 cat less grep
14. 文件压缩 gzip是比较常用的，bzip一般在生物信息中用于归档
15. trimmer in.fastq.gz | gzip > out.fastq.gz
trimer可以识别gz,但是输入的是非压缩的加入gzip作为压缩
16.gzip 使用gzip后会替代原来的文件
gunzip 会解压然后替代原来的gz文件
gzip -c in.fastq > in,fastq.gz
gunzip -c infastq.gz > duplicate_in.fastq
不替换原来的
17. $ ls
in.fastq.gz in2.fastq
$ gzip -c in2.fastq >> in.fastq.gz
18.cat in.fastq in2.fastq | gzip > in.fastq.gz
cat 读入两个文件然后压缩到一个文件
19. $ zgrep --color -i -n "AGATAGAT" Csyrichta_TAGGACT_L008_R1_001.fastq.gz
20. 每次下完数据检测数据的完整性，然后将shasum的值保存到 README
21. R's knitr Sweave
22.
tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q
23. cut awk
24. head 1.bed
head -n 3 1.bed
tail -n 2 2.bed
25.
26. 查看前两行和后两行的数据
27.
28 less
q 可以推出h
h可以获得帮助
b 向上翻页
j 向下翻页
k 向上翻页
/ 输入想寻找的序列
29 wc
by default wc output the number of worlds lines characters
30
wc -l number of lines
31 ls -l 查看文件大小
ls -lh
31 awk -F "\t" '{print NF; exit}' Mus_musculus.GRCm38.75_chr1.bed
awk
32
33. cut -f 2 1.bed | head -n 3
cut -f 第二列
34. grep -v "^#" Mus_musculus.GRCm38.75_chr1.gtf | cut -f1,4,5 | head -n 3
grep -v "^#" Mus_musculus.GRCm38.75_chr1.gtf | cut -f1,4,5 > test.txt
35. grep -v "^#" Mus_musculus.GRCm38.75_chr1.gtf | cut -f 1-8 | column -t
| head -n 3
column-t 比较容易看
36. column -s"," -t Mus_musculus.GRCm38.75_chr1_bed.csv | head -n 3
37. grep
grep "Olfr" gene.txt | head -n 5
“” 不是必须的但是为了避免错误最好是带上
--color 参数
-v 排除某个 “” 只要含有就排除
但是如果想保留含有+其他的-w
grep -B1 "AGATCGG" contam.fastq | head -n 6
B1 与序列匹配的前一行
grep -A2 "AGATCGG" contam.fastq | head -n 6
A2 与序列匹配的后两行
38 identifiers for both “Olfr1413” and “Olfr1411
$ grep "Olfr141[13]" Mus_musculus.GRCm38.75_chr1_genes.txt
39 $ grep -E "(Olfr1413|Olfr1411)" Mus_musculus.GRCm38.75_chr1_genes.txt
-E参数或者egrep 来匹配我们需要的
40 grep has an option to count how many lines match a pattern: -c. For example, suppose
we wanted a quick look at how many genes start with “Olfr”:
$ grep -c "\tOlfr" Mus_musculus.GRCm38.75_chr1_genes.txt
40 假设我们想知道在Mus_musculus.GRCm38.75_chr1.gtf中有多少小RNA gene_biotype 中的snRNA
$ grep -c 'gene_biotype "snRNA"' Mus_musculus.GRCm38.75_chr1.gtf
41. grep -o to extract only the matching part of the pattern
$ grep -o "Olfr.*" Mus_musculus.GRCm38.75_chr1_genes.txt | head -n 3
42. $ grep -E -o 'gene_id "\w+"' Mus_musculus.GRCm38.75_chr1.gtf | head -n 5
suppose we wanted to extract all values of the “gene_id” field from the last column
of our Mus_musculus.GRCm38.75_chr1.gtf file. This is easy with -o:
43.
$ grep -E -o 'gene_id "(\w+)"' Mus_musculus.GRCm38.75_chr1.gtf | \
cut -f2 -d" " | \
sed 's/"//g' | \
sort | \
uniq > mm_gene_id.txt
44 hexdump
编码情况
ASCII
UTF-8
file + 文件名查看编码情况
45. sort | uniq
$ sort -k1,1 -k2,2n example.bed
46. uniq
$ sort letters.txt | uniq
uniq -c 标记出有多少个重复的
46. $ grep -v "^#" Mus_musculus.GRCm38.75_chr1.gtf | cut -f3 | sort | uniq -c | \
sort -rn
47 join
-a 参数
48 awk
49 bioawk
50 sed
sed 's/chrom/chr/' chroms.txt | head -n 3
he syntax of sed’s substitute is s/pattern/replacement/.
s/pattern/replacement/g
s/pattern/replacement/i
51 POSIX Basic Regular Expressions (BRE).
52 mkfifo
53.program --in1 in1.txt --in2 in2.txt \
--out1 >(gzip > out1.txt.gz) --out2 >(gzip > out2.txt.gz)
