Ceph集群的搭建與運維
Ceph集群的搭建與運維
Ceph的存儲過程
文件->obj->pool->pg->osd->disk
- 文件被分片成對象
- 對象存放於特定的pool
- pool由多個pg組成
- pg對應多個osd
- osd直接對應disk
機器環境
- 4台centos7機器
- 默認最小集群是一個mon節點,兩個osd節點
admin 10.37.129.10
ceph-node1 10.37.129.11
ceph-node2 10.37.129.12
ceph-node3 10.37.129.13
設置ssh免密碼登陸
- 各節點修改/etc/hosts文件
admin 10.37.129.10
ceph-node1 10.37.129.11
ceph-node2 10.37.129.12
ceph-node3 10.37.129.13
- 在admin節點生成密鑰對並派發
# 在admin節點
ssh-keygen -t rsa
ssh-copy-id root@ceph-node1
ssh-copy-id root@ceph-node2
ssh-copy-id root@ceph-node3
各節點環境配置
- 各節點關閉防火牆
sudo systemctl stop firewalld.service #停止 firewall
sudo systemctl disable firewalld.service #禁止 firewall 開機啟動
- 各節點關閉selinux
vi /etc/selinux/config
SELINUX=disabled
reboot
- 各節點安裝時間同步工具
yum install chrony -y
systemctl restart chronyd
systemctl enable chronyd
chronyc source -v (查看時間是否同步,*表示同步完成)
- 各節點配置yum源
vi /etc/yum.repo.d/ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/$basearch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/noarch
enabled=1
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.163.com/ceph/rpm-kraken/el7/SRPMS
enabled=0
priority=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
在admin節點部署
- 在admin節點安裝部署工具
yum -y install ceph-deploy
- 創建部署文件目錄
mkdir -p /etc/ceph
cd /etc/ceph
- 創建一個集群(創建mon節點)
ceph-deploy new ceph-node1 ceph-node2 (也就是定義2各mon節點)
- 配置ceph.conf
vi ceph.conf
加入:
public network =10.37.129.0/24
- 開始部署
ceph-deploy install admin ceph-node1 ceph-node2 ceph-node3
- 初始化 mon節點 並收集所有的秘鑰
ceph-deploy mon create-initial
- 創建osd
ceph-deploy osd create ceph-node1:/dev/sdb ceph-node2:/dev/sdb
- 使用ceph-deploy把配置文件和admin秘鑰推送到管理節點和ceph節點
ceph-deploy admin admin ceph-node1 ceph-node2 ceph-node3
- 給每一個節點的keyring 增加 r 許可權(各節點執行)
chmod +r /ect/ceph/ceph.client.admin.keyring
- 檢查集群狀況
ceph health
新增一個osd節點和mon節點
- 在admin節點操作
1、新增一個osd ceph-node3
ceph-deploy osd create ceph-node3:/dev/sdb
2、在ceph-node1增加一個元數據的角色
ceph-deploy mds create ceph-node1
3、增加一個新的mon在ceph-node3 上面
ceph-deploy mon add ceph-node3
誤刪osd恢複流程
- 記錄osd狀態:ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.01469 root default
-2 0.00490 host ceph-node1
0 0.00490 osd.0 up 1.00000 1.00000
-3 0.00490 host ceph-node2
1 0.00490 osd.1 up 1.00000 1.00000
-4 0.00490 host ceph-node3
2 0.00490 osd.2 up 1.00000 1.00000
- 模擬刪除osd.2
ceph osd out osd.2
systemctl stop ceph-osd@2 (osd id)
ceph osd crush remove osd.2
ceph auth del osd.2
ceph osd rm osd.2
- 在刪除的osd節點上進行恢復操作
進入到其掛載的目錄 例如
cd /var/lib/ceph/osd/ceph-1
more fsid
ceph osd create uuid
ceph auth add osd.2 osd allow * mon allow rwx -i /var/lib/ceph/osd/ceph-1/keyring
ceph osd crush add 2 osd.2(osd id) 0.0049 (權重) host=ceph-node3
ceph osd in osd.2
systemctl start ceph-osd@2
推薦閱讀: