TCGA数据库下载后文件合并与转换

1、从TCGA上下载癌症相关数据到rawdata，下载一般用TCGA_gdc-client。以原始的Seq-count为例。

gdc-client download -m gdc_manifest_20200923_032916.txt -d path/to/gz_file

2、加入cart之后下载manifest.json文件。（PS：这个文件真的无比讨厌，处理起来很费经）
3、将数据解压缩之后生成看起来像乱码的文件夹。此时利用perl脚本将这每个文件夹中的文件放入另外一个file文件夹当中，然后将文件名改成data_in_one便于识别。

###将所有压缩文件移到一个名为files的文件里面
###用法为cmd perl 1.move_gzFileToFiles.pl
use strict;
use warnings;
use File::Copy;
my $newDir="files";
unless(-d $newDir)
{
mkdir $newDir or die $!;
}
my @allFiles=glob("*");
foreach my $subDir(@allFiles)
{
if((-d $subDir) && ($subDir ne $newDir))
{
opendir(SUB,"./$subDir") or die $!;
while(my $file=readdir(SUB))
{
if($file=~/\.gz$/)
{
#`cp ./$subDir/$file ./$newDir`;
copy("$subDir/$file","$newDir") or die "Copy failed: $!";
}
}
close(SUB);
}
}
```
4、将刚才的file文件修改为raw_data。利用R脚本将data_in_one文件夹当中的文件进行逐一合并，形成一个seq-count矩阵。这一步骤时间有点久，耐心。
```
library(R.utils)
library(hash)
library(data.table)
library(jsonlite)
setwd("E:/3.FPKM")
dir.create("data_in_one")
for (i in list.files("raw_data/")){
    b <- paste("raw_data/",i,sep="")
    pathname <- paste(b,dir(b),sep="/")
    file.copy(pathname,"data_in_one/")
}  
dz <- fromJSON("metadata.cart.2020-09-28.json")
temp_tcgaid <- as.character(lapply(dz$associated_entities,function(x){a <- x$entity_submitter_id;return(a)}))
temp_filename <- as.character(dz$file_name)
h <- hash(temp_filename,temp_tcgaid)
cishu <- 0
for (i in 1:length(dz$file_name)){
    if (cishu==0){
        test <- fread(paste("data_in_one",dz$file_name[i],sep="/"))
        test <- test[-((nrow(test)-5):nrow(test)),]
        exp <- matrix(NA,nrow(test),length(dz$file_name))
        rownames(exp) <- as.data.frame(test)[,1]
        tcgaid <- c()
        for (j in keys(h)){
            tcgaid_temp <- h[[j]]
            tcgaid <- paste(tcgaid,tcgaid_temp,sep=",")
        }
        ttt <- lapply(strsplit(tcgaid,",")[[1]],function(x){if(x != ""){return (x)}})
        ttt <- as.character(ttt)[-1]
        colnames(exp) <- ttt
        cishu <- cishu + 1
    }
    if (cishu > 0){
        test <- fread(paste("data_in_one",dz$file_name[i],sep="/"))
        test <- test[-((nrow(test)-5):nrow(test)),]
        new_h <- hash(test$V1,test$V2)
        for (j in rownames(exp)){
            file_name_new <- dz$file_name[i]
            exp[j,h[[file_name_new]]] <- new_h[[j]]
        }
    }
}
normalSample <- c()
tumorSample <- c()
for ( i in colnames(exp)){
    sample <- unlist(strsplit(i,"-"))[4]
    if(grepl("^1",sample)){
        normalSample <- paste(normalSample,i,sep=",")
    }else{
        tumorSample <- paste(tumorSample,i,sep=",")
    }
}
if ("normalSample" %in% ls()){
    normal_name <- strsplit(normalSample,",")[[1]][-1]
    tumor_name <- strsplit(tumorSample,",")[[1]][-1]
    if (length(normal_name) == 1){
        temp_normal <- as.data.frame(exp[,normal_name])
        colnames(temp_normal) <- normal_name
        normal_data <- temp_normal
    }else{
        normal_data <- exp[,normal_name]
    }
    tumor_data <- exp[,tumor_name]
    total_sort_sample <- merge(normal_data,tumor_data,by="row.names",all=T)
}else{
    total_sort_sample <- tumor_data
}
zanshi <- c("id")
for (i in colnames(total_sort_sample)[-1]){zanshi <- paste(zanshi,i,sep=",")}
colnames(total_sort_sample) <- unlist(strsplit(zanshi,","))
write.table(total_sort_sample,file="RNAmatrix.txt",sep="\t",row.names=F,quote=F)

5、因为从TCGA上下载的文件的基因名字是ensemble上的名字，类似于ENSG00000150093，所以先下载Homo_sapiens.GRCh38.98.chr.gtf文件，然后利用这个文件进行转换。也可以不转换，因为有的时候转换完了有些基因的名字是重复的，后期在利用DESeq2进行差异表达分析的时候会报错。

##用法perl 3.GeneName2Symble.pl
use strict;
use warnings;
my $gtfFile="Homo_sapiens.GRCh38.98.chr.gtf";
my $expFile="DEG_treat_vs_control.txt";
my $outFile="DEG_treat_vs_control_symbol.txt";
my %hash=();
open(RF,"$gtfFile") or die $!;
while(my $line=<RF>)
{
chomp($line);
if($line=~/gene_id \"(.+?)\"\;.+gene_name "(.+?)"\;.+gene_biotype \"(.+?)\"\;/)
{
$hash{$1}=$2;
}
}
close(RF);
open(RF,"$expFile") or die $!;
open(WF,">$outFile") or die $!;
while(my $line=<RF>)
{
if($.==1)
{
print WF $line;
next;
}
chomp($line);
my @arr=split(/\t/,$line);
$arr[0]=~s/(.+)\..+/$1/g;
if(exists $hash{$arr[0]})
{
$arr[0]=$hash{$arr[0]};
print WF join("\t",@arr) . "\n";
}
}
close(WF);
close(RF)
```