AWS EC2 服务建立云端Deep Learning开发环境

1. AWS EC2 的建立

• AMI 选择

Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-48db9d28

• GPU Instance 选择

目前只有g2.2xlarge是最廉价的方案，里面的硬盘空间最大为60g

• 因此需要添加 EBS 硬盘来扩充空间

Root - /dev/sda1 60GB ebs - /dev/sdb 200GB

2. Access EC2 through ssh

• 使用ssh连接系统

• weiwei_0903.pem 是下载到本地一个目录的key

• 然后执行下面语句

ssh -i "weiwei_0903.pem" ubuntu@ec2-52-53-235-35.us-west-1.compute.amazonaws.com

• 让 know_host 记住这个IP地址即可

3. 加载 EBS 到刚才建立的 GPU Instance

• 查看EBS是不是存在

ubuntu@ip-*-*-*-*:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk `-xvda1 202:1 0 8G 0 part / xvdb 202:16 0 100G 0 disk /home/ubuntu/workspace

• 其中 xvdb 是我单独添加的 EBS 硬盘。在最初，MOUNTPOINT下的/home/ubuntu/workspace应该是没有的，可以通过下面的步骤完成。

• 查询EBS是否已经有 File System

[ec2-user ~]$ sudo file -s /dev/xvdb /dev/xvdb: data

• 返回值是data意味着这个device目前没有文件系统，需要进一步格式化

[ec2-user ~]$ sudo mkfs -t ext4 /dev/xvdb

• 再次查看

ec2-user ~]$ sudo file -s /dev/xvdb /dev/xvdb: Linux rev 1.0 ext4 filesystem data, UUID=1701d228-e1bd-4094-a14c-8c64d6819362 (needs journal recovery) (extents) (large files) (huge files)

• 挂载格式化好的device到当前目录

ubuntu@ip-*-*-*-*:~$ sudo mount /dev/xvdb workspace ubuntu@ip-*-*-*-*:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk `-xvda1 202:1 0 8G 0 part / xvdb 202:16 0 100G 0 disk /home/ubuntu/workspace

• 此时就可以看到xvdb这个硬盘的挂载点。

• 添加权限

$ sudo chmod go+rw workspace

• 关于其他Storage

/dev/sda = /dev/xvda in the instance 8Gb "EBS persistent storage" /dev/sdb = /dev/xvdb in the instance 400Gb "Non persistent storage"

• 查看 Storage

df -h

4. 安装相关软件

• 基本依赖库

sudo apt-get update 
sudo apt-get upgrade 
sudo apt-get install -y build-essential git python-pip libfreetype6-dev libxft-dev libncurses-dev libopenblas-dev gfortran python-matplotlib libblas-dev liblapack-dev libatlas-base-dev python-dev python-pydot linux-headers-generic linux-image-extra-virtual unzip python-numpy swig python-pandas python-sklearn unzip wget pkg-config zip g++ zlib1g-dev 
sudo pip install -U pip

• Install Python2.7, Anconda, CUDA 7.5.178, CUDNN 7.0, Tensorflow 0.10.0

git clone https://gist.github.com/weiweikong/374e93d9ccb88ea45341268a06897259 aws-tensorflow-python2.7-setup

• 注意给bash文件权限

• Set a folder to /mnt

# stop on error 
set -e 
############################################
 # install into /mnt/bin 
sudo mkdir -p /mnt/bin 
sudo chown ubuntu:ubuntu /mnt/bin

• Install Anaconda

wget https://repo.continuum.io/archive/Anaconda2-4.1.1-Linux-x86_64.sh 
bash Anaconda2-4.1.1-Linux-x86_64.sh -b -p /mnt/bin/anaconda2 
rm Anaconda2-4.1.1-Linux-x86_64.sh 
echo 'export PATH="/mnt/bin/anaconda2/bin:$PATH"' >> ~/.bashrc

• Install Required Packages

# install the required packages 
sudo apt-get update && sudo apt-get -y upgrade 
sudo apt-get -y install linux-headers-$(uname -r) linux-image-extra-`uname -r`

• Install CUDA 7.5

# install cuda wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.5-18_amd64.deb 
sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
rm cuda-repo-ubuntu1404_7.5-18_amd64.deb 
sudo apt-get update 
sudo apt-get install -y cuda

• Manually download CUDNN 7.5 and upload

scp -i your_pem_file.pem cudnn-7.5-linux-x64-v5.0-ga.tgz ubuntu@ec2-54-67-18-98.us-west-1.compute.amazonaws.com:~/.

• Install cuDNN 7.5.1

# get cudnn 
tar xvzf cudnn-7.5-linux-x64-v5.1.tgz 
cd cuda 
sudo cp lib64/* /usr/local/cuda/lib64/ 
sudo cp include/* /usr/local/cuda/include/ 
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn* 

echo 'export CUDA_HOME=/usr/local/cuda 
export CUDA_ROOT=/usr/local/cuda 
export PATH=$PATH:$CUDA_ROOT/bin:$HOME/bin 
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_ROOT/lib64 
' >> ~/.bashrc

• Install Tensorflow with only cuDNN 7.5.1 and Python 2.7

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0-cp27-none-linux_x86_64.whl /mnt/bin/anaconda2/bin/pip install $TF_BINARY_URL

• Install Caffe 依赖库

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler 
sudo apt-get install --no-install-recommends libboost-all-dev 
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

• 配置 Caffe 文件并编译

cp Makefile.config.example Makefile.config
# Adjust Makefile.config (for example, if using Anaconda Python, or if cuDNN is desired)

mkdir build 
cd build 
cmake ..
make all 
make test 
make runtest

• Test Caffe Install

sh data/mnist/get_mnist.sh 
sh examples/mnist/create_mnist.sh 
sh examples/mnist/train_lenet.sh

• Monitor Code

# install monitoring programs 
sudo wget https://git.io/gpustat.py -O /usr/local/bin/gpustat 
sudo chmod +x /usr/local/bin/gpustat 
sudo nvidia-smi daemon 
sudo apt-get -y install htop

• Ref: How to install CUDA Toolkit and cuDNN for deep learning - PyImageSearch

4.1 Trouble Shooting

• Caffe 遇到 locale::facet::_S_create_c_locale name not valid

  • add export LC_ALL="en_US.UTF-8" to bashrc

  • Ref:https://gist.github.com/wangruohui/679b05fcd1466bb0937f

• Tensorflow 遇到 AttributeError: 'GFile' object has no attribute 'size'

sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

• libdc1394 error: Failed to initialize libdc1394

sudo ln /dev/null /dev/raw1394

• Cafee 遇到 Aborted at 1458527401 (unix time) try "date -d @1458527401" if you are using GNU date

• Check Nvidia GPUs List

$ nvidia-smi 

• Set Specific GPU Visible

CUDA_VISIBLE_DEVICES=1 Only device 1 will be seen 
CUDA_VISIBLE_DEVICES=0,1 Devices 0 and 1 will be visible 
CUDA_VISIBLE_DEVICES=”0,1” Same as above, quotation marks are optional
CUDA_VISIBLE_DEVICES=0,2,3 Devices 0, 2, 3 will be visible; device 1 is masked

• 目前选择两个GPU工作不会报错

• Ref: https://github.com/BVLC/caffe/issues/1993

4.2 Access to EC2 using FileZilla

• Download FileZilla and setup.

• Add .pem key file

  • Edit -> Settings -> Connection -> SFTP

  • Select the .pem file and add it to the list.

• Add EC2 connection

  • File -> Site Manager

  • Host: EC2 Public IP (Could be check under EC2 Console)

  • Protocol: SFTP

  • Login Type: Normal

  • User: ubuntu

  • Password: /

• Press Connect