NCBI数据、获取使用
一、NCBI各类数据储存库
1、Sequence Read Archive (SRA):raw sequence data and alignment information from
high throughput sequencing platforms including 454, illumina, SOLiD and PacBio.
此外, European Nucleotide Archive (ENA) 和 DNAnexus SRA 均有SRA数据。2、Gene Expression Omnibus (GEO):RNA-Seq, ChIP-seq, RIP-seq, HiC-seq, methyl-seq, expression data such as microarray, SAGE or mass spectrometry data sets.
3、Database of Short Genetic Variations (dbSNP):variation data, such as single nucleotide variations, microsatellites, and small-scale insertions and deletions.
4、Database of Genomic Structural Variations (dbVar):genomic structural variation data, such as large insertions, deletions, translocations etc.
5、Database of Expressed Sequence Tags (dbEST)
6、Transcriptome Shotgun Assembly Sequence Database (TSA):transcriptome assemblies
7、Whole Genome Shotgun Submissions (WGS):incomplete genome assemblies.
二、Entrez
如何使用Entrez?NCBI提供了一个API网络接口Entrez E-utilsand 和一个命令行工具Entrez Direct。
Entrez E-utils
query URL :https://service.nih.gov?param1=value1¶m2=value2
在shell中,&需要使用转义符\,或者将URL添加引号‘’
curl -s https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi? id=AF086833.2\&db=nuccore\&rettype=fasta|head
curl -s 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?id=AF086833.2&db=nuccore&rettype=fasta'|head
具体命令参考文档:https://www.ncbi.nlm.nih.gov/books/NBK25500/
Entrez Direct
具体命令参考文档:https://www.ncbi.nlm.nih.gov/books/NBK179288/
einfo
抓取描述信息,einfo -dbs 查看所有数据库类型
esearch
esearch -help 查看帮助,esearch 查找的记录可以传递给efetch下载
esearch -db .. -query .. | efetch -formate ..
esearch -db sra -query PRJNA257197 | efetch -format runinfo > info.csv (runinfo — 所有搜索的信息)
efetch
It can even produce the sequence from reverse strands(reverse complement):
三、sratoolkit
1、prefetch 可以从远程站点下载文件
prefetch --option-file ids.list
2、fast-dump : downloads data in FASTQ format, 初始sra data 保存在 ~/ncbi/public/sra/
fastq-dump SRR1553607 --split-files( separate paired end reads)
下载project里多个SRA数据:
在上一步得到的info.csv文件中,第一列为这个project里所有的SRA文件,取第一列run编号,用xargs批量下载
cat info.csv|cut-f 1 -d','|grepSRR|head>ids ; cat ids |xargs-n 1 fastq-dump -X 10000 --split-files $1
3、sra-stat
sra-stat --xml --quick SRR1553610 生成xml报告
四、 SeqKit
seqkit stat *.gz 查看全部fasta/q文件的简要信息
seqkit fx2tab --name --only-id --gc *.fna 显示GC含量
seqkit fx2tab -H -n -i -B a -B c -B ac *.fna 显示A、C、AC含量
随机取出0.001比例的id号,并根据id从总体中提取序列:seqkit sample --proportion 0.001 duplicated-reads.fq.gz|seqkitseq --name --only-id>id.txt ;seqkit grep --pattern-file id.txt duplicated-reads.fq.gz > duplicated-reads.subset.fq.gz
seqkit grep --pattern-file id2.txt --invert-match viral.1.1.genomic.fna.gz > clean.fa 取出list上的序列
seqkit grep --pattern-file id2.txt viral.1.1.genomic.fna.gz|seqkitlocate--ignore-case --only-positive-strand --pattern K+ --pattern N+ 显示含K、N的数据
seqkit rmdup --by-seq --ignore-case duplicated-reads.fq.gz 去除重复
seqkit locate --degenerate --ignore-case --pattern-file enzymes.fa *.fna 匹配短序列如酶结合位点
seqkit sort--by-length/full header (--by-name) or sequence content (--by-seq)
seqkit split --by-id --id-regexp"\[(.+)\]" 根据ID, number of parts, size of each part, or sequence region分离
csvtk join-H -t<(seqkit fx2tab 1.fa)<(seqkit fx2tab 2.fa)|sed's/\t\t//'|seqkit tab2fx 合并两个文件
五、other tool
get subreads from a single ZMW : bamtools filter -in subreads.bam -out zmw.bam -tag 'zm..'
get the consensus sequence from the subreads : # ccs [options] INPUT OUTPUT