overlayFSInDocker 2022-03-14
overlayFS
In computer operating systems, union mounting is a way of combining multiple directories into one that appears to contain their combined contents
overlay FileSystem是union mounting的一种具体实现
overlayFS的构成
Overlay文件系统类似于堆叠的文件系统
An overlay filesystem combines two filesystems - an ‘upper’ filesystem and a ‘lower’ filesystem. An overlay sits on top of existing filesystems, and combines an upper and a lower directory tree (which can be from different filesystems), in order to present a unified representation of both directory trees. Where objects with the same name exist in both directory trees, then their treatment depends on the object type:
- File: the object in the upper directory tree appears in the overlay, whilst the object in the lower directory tree is hidden
- Directory: the contents of each directory object are merged to create a combined directory object in the overlay
more info for merge rules, see: https://windsock.io/the-overlay-filesystem/
overlayFS的读写规则——下层只读，上层读写：
- the lower directory of the filesystem is read-only, and could be an overlay itself
- the upper directory of the filesystem can be both read to and written from
- When a process reads a file, the overlayfs filesystem driver looks in the upper directory and reads the file from there if it’s present. Otherwise, it looks in the lower directory.
- When a process writes a file, overlayfs will just write it to the upper directory.
What if remove files from "merge" directory but they are truly stored in lower directory? When a file or directory that originates in the upper directory is removed from the overlay, it's also removed from the upper directory. If a file or directory that originates in the lower directory is removed from the overlay, it remains in the lower directory, but a 'whiteout' is created in the upper directory.
创建overlayFS的命令
At mount time, the two directories given as mount options “lowerdir” and “upperdir” are combined into a merged directory:
mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /merged
notice that overlayFS has a workdir option, beside two other directories lowerdir and upperdir, which needs to be an empty directory.
The "workdir" needs to be an empty directory on the same filesystem as upperdir.
The "merged" directory is used to mount the combined filesystem into
what's more, overlayfs supports having multiple lower directories, you can run:
mount -t overlay overlay -o lowerdir:/dir1:/dir2:/dir3:...:/dir25,upperdir=...
overlayFS in docker
镜像（image），是若干read-only layers，通过overlay文件系统“合并”成一个统一文件系统，隐藏了多个只读层的细节
容器（Container），和镜像一样，也是一个或多个只读层的堆叠“合并”，只不过不同的是，最上层比镜像多了一个可读写的统一文件系统层。即：容器 = 镜像 + 可读写层
docker中初识overlayFS
如果你的机器上正在运行docker，那么键入mount | grep overlay你会看到类似下面的东西
overlay on /var/lib/docker/overlay2/8d27b297a36f2d2dade214b01c4a2b3cf389/merged
type overlay
(rw,relatime,
# lowerdir
lowerdir=/var/lib/docker/overlay2/l/I5I7QY5EVPP4WUUXIKJWJMMOQZ:/var/lib/docker/overlay2/l/5VQ5GHIQ6TZUUFPRRXPK4GCT2L:/var/lib/docker/overlay2/l/R6TVL6NM3PY64QERHOZHLENYTH:/var/lib/docker/overlay2/l/VRGZ4LBO4YAAJIABKFLCCAULIU:/var/lib/docker/overlay2/l/FQIK6OUHUM3Q6UQBNHJGBRYAUL:/var/lib/docker/overlay2/l/6XCQF3KYIWO24TESEJZFXRIPOO:/var/lib/docker/overlay2/l/QN7TPWWVT5UXCVBITUO7UHIQRP:/var/lib/docker/overlay2/l/2YWOI25IUHZPKPWULORTKFUJDK:/var/lib/docker/overlay2/l/6TY66M3E655V2257ML2XTKYKEZ,
# upperdir
upperdir=/var/lib/docker/overlay2/8d27b297a36f2d2dade214b01c4a2b3cf389/diff,
# workdir
workdir=/var/lib/docker/overlay2/8d27b297a36f2d2dade214b01c4a2b3cf389/work,
index=off)
上面的信息做了一些脱敏和标识，意思是说：
/var/lib/docker/overlay2/8d27b297a36f2d2dade214b01c4a2b3cf389/merged这个overlayFS是通过这样的lowerdir、upperdir和workdir union而来的
可以看到，命令中涉及的各种目录的前缀都是/var/lib/docker/overlay2，这其实就是各种容器所使用的overlay file system
/var/lib/docker/overlay2
使用docker image inspect imageName剖析docker的getting-started image
[root@VM-165-116-centos ~]# docker image inspect docker/getting-started
[
{
...,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/a4cdd6e41364634cc299ed8aee2badf3d4345f5ab2ff5006df218714e29a596d/diff:/var/lib/docker/overlay2/dbb5d4267865fd93e102cf40c8896b48372c20301c7dedc3dcd674d517e8f142/diff:/var/lib/docker/overlay2/783678dcf22ac6422756c9de988ccc79b0908cc8718cc1b70bcff75ac090f9fe/diff:/var/lib/docker/overlay2/38c604f97b55c67343d541ab499562186fa9c0d720926fb78479cb2f2bc9d3d6/diff:/var/lib/docker/overlay2/4977a1f7e69870ca91aaee3fcbdf9a713032931fb9a780cc03b9cbb70116cbfb/diff:/var/lib/docker/overlay2/99d6524646cf6f183568bbbb986a64d0b83a7644b62c180b6e072ed68e068945/diff:/var/lib/docker/overlay2/487edf20bf3397e2a63970bbbb58ae599feb8f3158dcbdde07239a6c344dc6a9/diff",
"MergedDir": "/var/lib/docker/overlay2/f1bb521db98f9656b2aa39db62803a4fb5b79e01e0664189bb2034db3dbab170/merged",
"UpperDir": "/var/lib/docker/overlay2/f1bb521db98f9656b2aa39db62803a4fb5b79e01e0664189bb2034db3dbab170/diff",
"WorkDir": "/var/lib/docker/overlay2/f1bb521db98f9656b2aa39db62803a4fb5b79e01e0664189bb2034db3dbab170/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116",
"sha256:a42a23cd7b07a408e16c433f60eb97332b0dcd14313479a8094a69ad59da29fa",
"sha256:96131b349b16fb6244f73c696598f88b897bcbf33d59850461fd1cb356bd5f92",
"sha256:2b60f0243825842ad041e53cf9c53a05b80515f863102e02b18f9593627c6bd3",
"sha256:2f2df3ae0cadcae154d0ef616bd7b47430af04c7550e1605a9a85afacf796690",
"sha256:058eb06e0efd35550de1b6fb13702e3f33b3ebf21b9ed17b5d336e74c31c9c5f",
"sha256:43da65dcaf22955a64efed86273cfd69f7009402888a483dc33b0b34b6bc726b",
"sha256:e7572aa9a832f47d9bf9164aba78e6cde47caa631018cbc51030078d55837feb"
]
},
...
}
]
[root@VM-165-116-centos ~]#
可以从GraphDriver.Data里面看到getting-started image的lower，merged，upper和workdir目录
现在看下getting-started image的层标识目录/var/lib/docker/overlay2/f1bb521db98f9656b2aa39db62803a4fb5b79e01e0664189bb2034db3dbab170/
[root@VM-165-116-centos f1bb521db98f9656b2aa39db62803a4fb5b79e01e0664189bb2034db3dbab170]# ls
committed diff link lower work
其中，diff目录存放的是overlay之后形成的当前层的文件，link和lower的内容分别是当前层和下一层的软链接名字
see:
https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/
https://zhuanlan.zhihu.com/p/374924046
