Openshift生产环境部署规范

容器会在物联网中大放异彩

1. 磁盘目录挂载

• master
  磁盘格式：xfs
  /var/log
  /var/lib/docker
  /var/lib/etcd [ssd]
• node
  磁盘格式：xfs
  /var/log
  /var/lib/docker

2. 关闭swap

swapoff -a
cat /etc/fstab ## 注释掉swap

3. 打开seLinux enabled

touch /.autorelabel
sed -i 's/SELINUX=disabled/SELINUX=permissive/' /etc/selinux/config

4. 更改resolve.conf

$ cat /etc/resolv.conf
search cluster.local
nameserver 192.168.0.2

5. 时间同步

$ ansible all -m package -a 'name=chrony state=present'

## chronyd服务端配置
$ cat /etc/chrony.conf
server 55.15.226.193 iburst
allow 55.15.226.0/24
local stratum 10

强制同步时间

## chrony客户端配置
systemctl sources -v
systemctl stop chronyd
chronyd -q 'pool 55.15.226.193 iburst'

6.创建docker 用户组

groupadd docker

7. docker-storage设置

/etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "

8. 网卡配置

• NetworkManager, 是一个提供网络检测和配置网络的工具，在Node节点需要使用它来自动配置节点的dnsmasq作为默认的网络入口。
• 网络设备的配置中/etc/sysconfig/network-scripts/ifcfg-eth*默认NM_CONTROLLED是被设置为yes,如果它被设置为no，那么NetworkManager应用将不会去自动创建dnsmasq相关的配置，所以此时需要手动配置dnsmasq。

添加文件

$ cat /etc/dnsmasq.d/origin-upstream-dns.conf
server=192.168.0.2
$ cat /etc/origin/node/resolv.conf
nameserver 192.168.0.2

参考 install-config-network-using-firewalld

9. 双网卡

一张网卡配置为访问业务流量
另一张网卡配置为访问存储NAS流量

10.外部节点相关组件

• 时间同步服务（chronyd）
• DNS(dnsmasq)
• 镜像仓库(docker-distribution)
• 负载均衡器（Haproxy）

11. 外部镜像仓库授权

将私有镜像仓库的CA文件拷贝到镜像仓库所在服务器的/etc/pki/ca-trust/source/anchors/目录下

$ ansible all -m copy -a 'src=registry.crt dest=/etc/pki/ca-trust/source/anchors/registry.crt'

12. 内核优化（openshift安装会自动配置）

$ ansible all -m package -a 'name=tuned state=present'
$ ansible all -m service -a 'name=tuned state=started enabled=true'
$ ansible all -m shell -a 'tuned-adm profile throughput-performance'

13. ansible设置reserved

[OSEv3:vars]
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80'], 'system-reserved':['cpu=200m', 'memory=1G'], 'kube-reserved':['cpu=200m','memory=1G']}

14. ansible中设置Docker存储type及Docker与etcd额外磁盘

[OSEv3:vars]
# Docker setup for extra disks on nodes
container_runtime_docker_storage_setup_device=/dev/vdb
container_runtime_docker_storage_type=overlay2
openshift_node_local_quota_per_fsgroup=512Mi

[masters:vars]
container_runtime_extra_storage=[{'device': '/dev/vdc', 'path': '/var/lib/origin/openshift.local.volumes', 'options': 'gquota', 'filesystem': 'xfs', 'format': 'True'}, {'device': '/dev/vdd', 'path': '/var/lib/etcd', 'hosts': 'masters', 'filesystem': 'xfs', 'format': 'True'}]

[nodes:vars]
container_runtime_extra_storage=[{'device': '/dev/vdc', 'path': '/var/lib/origin/openshift.local.volumes', 'options': 'gquota', 'filesystem': 'xfs', 'format': 'True'}]

15. 设置日志自动归档

1. journal日志归档
   设置/etc/systemd/journald.conf

$ cat /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
#Seal=yes
#SplitMode=uid
SyncIntervalSec=1s
RateLimitInterval=1s
RateLimitBurst=10000
SystemMaxUse=1G
SystemKeepFree=20%
SystemMaxFileSize=10M
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
MaxRetentionSec=3days
MaxFileSec=1day
ForwardToSyslog=False
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=False
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
$ systemctl restart systemd-journald

或者部署时更新以下文件内容(openshift 3.9以上)
roles/openshift_node/defaults/main.yml

...
journald_vars_to_replace:
- { var: Storage, val: persistent }
- { var: Compress, val: yes }
- { var: SyncIntervalSec, val: 1s }
- { var: RateLimitInterval, val: 1s }
- { var: RateLimitBurst, val: 10000 }
- { var: SystemMaxUse, val: 1G }
- { var: SystemKeepFree, val: 20% }
- { var: SystemMaxFileSize, val: 10M }
- { var: MaxRetentionSec, val: 3days }
- { var: MaxFileSec, val: 1day }
- { var: ForwardToSyslog, val: no }
- { var: ForwardToWall, val: no }
...

2. message日志归档
   只收集warning以上的日志/etc/rsyslog.conf

$ cat /etc/rsyslog.conf
*.warning;mail.none;authpriv.none;cron.none  /var/log/messages

将message日志只保留最近三天的日志

$ cat /etc/logrotate.d/syslog
/var/log/cron
/var/log/messages
{
  rotate 3
  sharedscripts
  postrotate
     /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
  endscript
}

参考文章：
linux journalctl 命令
配置 logrotate 的终极指导