2020-12-13 docker build for deep

2020-12-13  本文已影响0人  罗志鹏_6145

docker build for deepLearing env mxnet-cu10

base image

nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

pull image

docker pull [OPTIONS] NAME[:TAG|@DIGEST]
docker pull nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
Untitled.png

run container

sudo nvidia-docker run -it --name base-10.1-cudnn7 -v /home/qy/Documents/df/:/df  05e3d30bd56a bash

update source.list

将 /home/qy/Documents/df/ 目录里source.list 拷贝到容器的 /etc/apt/sources.list
执行： apt update

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse

# 预发布软件源，不建议启用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

然后生成新的镜像 nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 (原来的镜像名一样）

Untitled 1.png 
sudo docker commit -m "add vim" -a "qyzhizi" xxxxx(容器id) xxxxx:xx(待生成的镜像名：tag)

all build

为了使用Dockerfile来构建，为了构建的方便，每次构建一点，下一次的构建在上一次的基础上进行。第一次构建从在新nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 镜像上，从1.0版本迭代到3.25版本。然后发现在中间过程中cuda的版本安装错了。不过没关系，在原来2.5版本上，从新构建得到2.5-cu10版本，然后继续构建，由于有之前的Dockerfile ,在2.5cu10版本上直接得到最终版本3.0-cu10.
除此之外，从1.0版本开始，所有版本都以 registry.cnshanghai.aliyuncs.com/qyzhizi/dl_base
为镜像名。另外1.0版的镜像提交到了阿里云的镜像库。

Untitled 2.png Untitled 3.png

1.0

FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

# for chinese supports
ENV LANG C.UTF-8
RUN rm -rf /var/lib/apt/lists/* \
           /etc/apt/sources.list.d/cuda.list \
           /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-get update && apt-get install -y --no-install-recommends \
    curl \
    vim \
    wget \
    tmux \
    ca-certificates \
    git \
 && rm -rf /var/lib/apt/lists/*

2.2

FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

# for chinese supports
ENV LANG C.UTF-8
RUN rm -rf /var/lib/apt/lists/* \
           /etc/apt/sources.list.d/cuda.list \
           /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-get update && apt-get install -y --no-install-recommends \
    curl \
    vim \
    wget \
    tmux \
    ca-certificates \
    git \
 && rm -rf /var/lib/apt/lists/*

## Install Miniconda and Python 3.8
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV HOME=/root
ENV PATH=$PATH:$HOME/miniconda/bin
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& bash ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda clean -ya \
&& echo ". $HOME/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc

CMD ["bash"]

2.4(2.3)

FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

# for chinese supports
ENV LANG C.UTF-8
RUN rm -rf /var/lib/apt/lists/* \
           /etc/apt/sources.list.d/cuda.list \
           /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-get update && apt-get install -y --no-install-recommends \
    curl \
    vim \
    wget \
    tmux \
    ca-certificates \
    git \
 && rm -rf /var/lib/apt/lists/*

## Install Miniconda and Python 3.8
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV HOME=/root
ENV PATH=$PATH:$HOME/miniconda/bin
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& bash ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda clean -ya \
&& echo ". $HOME/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc

COPY . $HOME
RUN PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" \ 
&& conda config --set show_channel_urls yes \
&& conda env create -f ~/environment.yml
CMD ["bash"]

environment.yml

name: gluon1
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
dependencies:
- python=3.6

2.5-cu10 (2.5)

FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

# for chinese supports
ENV LANG C.UTF-8
RUN rm -rf /var/lib/apt/lists/* \
           /etc/apt/sources.list.d/cuda.list \
           /etc/apt/sources.list.d/nvidia-ml.list && \
    apt-get update && apt-get install -y --no-install-recommends \
    curl \
    vim \
    wget \
    tmux \
    ca-certificates \
    git \
 && rm -rf /var/lib/apt/lists/*

## Install Miniconda and Python 3.8
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV HOME=/root
ENV PATH=$PATH:$HOME/miniconda/bin
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& bash ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh \
&& conda clean -ya \
&& echo ". $HOME/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc

COPY . $HOME
RUN PIP_INSTALL="python -m pip --no-cache-dir install --upgrade" \ 
&& conda config --set show_channel_urls yes \
&& conda env create -f ~/environment.yml

RUN source ~/miniconda/etc/profile.d/conda.sh \
&& conda activate gluon1 \
&& pip install -r ~/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/ \
&& conda clean -ya \
&& conda deactivate 

CMD ["bash"]

requirement.txt

区别：
2.5-cu10: mxnet-cu101
2.5: mxnet-cu102

mxnet-cu101
d2lzh==1.0.0
jupyter==1.0.0
matplotlib==2.2.2
pandas==0.23.4

3.0-cu10(最终版本），在2.5-cu10的基础上

FROM registry.cn-shanghai.aliyuncs.com/qyzhizi/dl_base:2.5-cu10

## settings
RUN mkdir ~/notebook && \
    source ~/miniconda/etc/profile.d/conda.sh && \
    conda activate gluon1 && \
    pip install https://github.com/mli/notedown/tarball/master &&\
    jupyter notebook --allow-root --generate-config && \
    echo "c.NotebookApp.contents_manager_class = \
    'notedown.NotedownContentsManager'" >>~/.jupyter/jupyter_notebook_config.py

## copy notebooks
RUN  mkdir ~/gluon-tutorials-zh
ADD . /root/gluon-tutorials-zh

CMD source ~/miniconda/etc/profile.d/conda.sh && \
    conda activate gluon1 && cd ~/gluon-tutorials-zh && \
    jupyter notebook --ip=0.0.0.0 --allow-root

3.0

FROM registry.cn-shanghai.aliyuncs.com/qyzhizi/dl_base:2.5

## setup notedown
RUN source ~/miniconda/etc/profile.d/conda.sh &&\
    conda activate gluon1 && \
    pip install https://github.com/mli/notedown/tarball/master &&\
    conda deactivate
#
## copy notebooks
#RUN  mkdir /gluon-tutorials-zh
#COPY / /gluon-tutorials-zh/
#
CMD ["bash"]
#CMD source activate gluon && cd /gluon-tutorials-zh && \
#    jupyter notebook --ip=0.0.0.0 --allow-root

3.1

FROM registry.cn-shanghai.aliyuncs.com/qyzhizi/dl_base:3.0

## settings
RUN mkdir ~/notebook && \
    source ~/miniconda/etc/profile.d/conda.sh && \
    conda activate gluon1 && \
    jupyter notebook --allow-root --generate-config && \
    echo "c.NotebookApp.contents_manager_class = \
    'notedown.NotedownContentsManager'" >>~/.jupyter/jupyter_notebook_config.py
#
## copy notebooks
#RUN  mkdir /gluon-tutorials-zh
#COPY / /gluon-tutorials-zh/
#
CMD ["bash"]
#CMD source activate gluon && cd /gluon-tutorials-zh && \
#    jupyter notebook --ip=0.0.0.0 --allow-root

3.2(3.25）

FROM registry.cn-shanghai.aliyuncs.com/qyzhizi/dl_base:3.0

## settings
RUN mkdir ~/notebook && \
    source ~/miniconda/etc/profile.d/conda.sh && \
    conda activate gluon1 && \
    jupyter notebook --allow-root --generate-config && \
    echo "c.NotebookApp.contents_manager_class = \
    'notedown.NotedownContentsManager'" >>~/.jupyter/jupyter_notebook_config.py
## copy notebooks
RUN  mkdir ~/gluon-tutorials-zh
ADD . /root/gluon-tutorials-zh

CMD source ~/miniconda/etc/profile.d/conda.sh && \
    conda activate gluon1 && cd ~/gluon-tutorials-zh && \
    jupyter notebook --ip=0.0.0.0 --allow-root
上一篇下一篇

猜你喜欢

热点阅读