学习资料：想免费用谷歌资源训练神经网络？Colab 详细使用教程

作者 Jinkey（微信公众号 jinkey-love，官网 https://jinkey.ai

!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# 授权登录，仅第一次的时候会鉴权
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

遍历目录

其中 id 是获取文件的唯一标识。根据 mimeType 可以知道 Colab 文件类型：

# 列出根目录的所有文件
# "q" 查询条件教程详见：https://developers.google.com/drive/v2/web/search-parameters
file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file1 in file_list:
  print('title: %s, id: %s, mimeType: %s' % (file1['title'], file1['id'], file1["mimeType"]))

title: Anaconda3-5.1.0-Linux-x86_64.sh, id: 18mZ7tzwtTP48Rao9_WYwa-CHpr-E7u3Y, mimeType: text/x-sh
title: “Snippets: Importing libraries”, id: 1RK-dXQCOZB3-2sXdUMaLaXFj7tl1se-g, mimeType: application/vnd.google.colab
title: “TensorFlow with GPU”, id: 1eXw6z5xsAcTZN2mZwPKvfJhM7QREJZzk, mimeType: application/vnd.google.colab
title: “Getting started with BigQuery”, id: 1AskA0vNFXWUPZXwRs0f6w58UOxoNz9Sl, mimeType: application/vnd.google.colab
title: events.out.tfevents.1516971784.LAPTOP-970LLVBL, id: 1TD455CEuiwhvp_UqnhmGcuVrFl7AbBQk, mimeType: application/octet-stream
title: “demo_breastcancer_neural_network.ipynb”, id: 1BIodOhedtdh926pDvcK7CXg1O-LD7q7e, mimeType: application/vnd.google.colab
title: Colab Notebooks, id: 1dHhYv3OJoova841R--Or8wAgMZ_fhdhK, mimeType: application/vnd.google-apps.folder

如果想查询 Colab 文件夹下的文件，查询条件可以这么写：

# '目录 id' in parents
file_list = drive.ListFile({'q': "'1dHhYv3OJoova841R--Or8wAgMZ_fhdhK' in parents and trashed=false"}).GetList()

读取文件内容

可以直接读取格式为 .txt（mimeType: text/plain）的文件，读取代码：

file = drive.CreateFile({'id': "替换成你的 .txt 文件 id"}) 
file.GetContentString()

# 安装 PyDrive 操作库，该操作每个 notebook 只需要执行一次
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

def login_google_drive():
  # 授权登录，仅第一次的时候会鉴权
  auth.authenticate_user()
  gauth = GoogleAuth()
  gauth.credentials = GoogleCredentials.get_application_default()
  drive = GoogleDrive(gauth)
  return drive

def list_file(drive):
  file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
  for file1 in file_list:
    print('title: %s, id: %s, mimeType: %s' % (file1['title'], file1['id'], file1["mimeType"]))
    

drive = login_google_drive()
list_file(drive)

title: Anaconda3-5.1.0-Linux-x86_64.sh, id: 18mZ7tzwtTP48Rao9_WYwa-CHpr-E7u3Y, mimeType: text/x-sh
title: “Snippets: Importing libraries”, id: 1RK-dXQCOZB3-2sXdUMaLaXFj7tl1se-g, mimeType: application/vnd.google.colab
title: “TensorFlow with GPU”, id: 1eXw6z5xsAcTZN2mZwPKvfJhM7QREJZzk, mimeType: application/vnd.google.colab
title: “Getting started with BigQuery”, id: 1AskA0vNFXWUPZXwRs0f6w58UOxoNz9Sl, mimeType: application/vnd.google.colab
title: events.out.tfevents.1516971784.LAPTOP-970LLVBL, id: 1TD455CEuiwhvp_UqnhmGcuVrFl7AbBQk, mimeType: application/octet-stream
title: “demo_breastcancer_neural_network.ipynb”, id: 1BIodOhedtdh926pDvcK7CXg1O-LD7q7e, mimeType: application/vnd.google.colab
title: Colab Notebooks, id: 1dHhYv3OJoova841R--Or8wAgMZ_fhdhK, mimeType: application/vnd.google-apps.folder

Google 在线深度学习神器 Colab

!df -lh               # 查看虚拟机硬盘容量

Filesystem      Size  Used Avail Use% Mounted on
overlay          40G  4.8G   33G  13% /
tmpfs           6.4G     0  6.4G   0% /dev
tmpfs           6.4G     0  6.4G   0% /sys/fs/cgroup
/dev/sda1        46G  5.5G   40G  13% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           6.4G     0  6.4G   0% /sys/firmware

!cat /proc/cpuinfo | grep model\ name # 查看cpu配置

model name  : Intel(R) Xeon(R) CPU @ 2.20GHz
model name  : Intel(R) Xeon(R) CPU @ 2.20GHz

!cat /proc/meminfo | grep MemTotal  # 查看内存容量

MemTotal:       13341856 kB

# 安装requests, 爬虫必备
!pip install requests
# 安装 lxml, 解析xpath语法
!pip install lxml

Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests)
Collecting lxml
  Downloading lxml-4.1.1-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)
�[K    100% |████████████████████████████████| 5.6MB 201kB/s 
�[?25hInstalling collected packages: lxml
Successfully installed lxml-4.1.1

# 将获取的数据同步到github仓库
!apt install git

Reading package lists... Done
Building dependency tree       
Reading state information... Done
git is already the newest version (1:2.14.1-1ubuntu4).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

!pip3 install -U pyspark
!pip3 install -U mxnet-cu90
!pip3 install -U tensorflow-gpu
!pip3 install -U keras
!pip3 install -U mlk

Collecting pyspark
Collecting py4j==0.10.6 (from pyspark)
  Using cached py4j-0.10.6-py2.py3-none-any.whl
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.6 pyspark-2.3.0
Collecting mxnet-cu90
  Using cached mxnet_cu90-1.1.0-py2.py3-none-manylinux1_x86_64.whl
tcmalloc: large alloc 1073750016 bytes == 0x546ba000 @  0x7f0b051a31c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8
Requirement already up-to-date: requests==2.18.4 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu90)
Collecting graphviz==0.8.1 (from mxnet-cu90)
  Using cached graphviz-0.8.1-py2.py3-none-any.whl
Collecting numpy<=1.13.3 (from mxnet-cu90)
  Downloading numpy-1.13.3-cp36-cp36m-manylinux1_x86_64.whl (17.0MB)
�[K    100% |████████████████████████████████| 17.0MB 81kB/s 
�[?25hRequirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests==2.18.4->mxnet-cu90)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests==2.18.4->mxnet-cu90)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests==2.18.4->mxnet-cu90)
Requirement already up-to-date: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests==2.18.4->mxnet-cu90)
Installing collected packages: graphviz, numpy, mxnet-cu90
  Found existing installation: numpy 1.14.0
    Uninstalling numpy-1.14.0:
      Successfully uninstalled numpy-1.14.0
Successfully installed graphviz-0.8.1 mxnet-cu90-1.1.0 numpy-1.13.3
Collecting tensorflow-gpu
  Downloading tensorflow_gpu-1.6.0-cp36-cp36m-manylinux1_x86_64.whl (209.2MB)
�[K    38% |████████████▎                   | 80.3MB 43.1MB/s eta 0:00:03�[K    99% |████████████████████████████████| 209.2MB 36.4MB/s eta 0:00:01