Ceph 集群部署
來自： http://blog.csdn.net/chinagissoft/article/details/50491429
Ceph的底層是RADOS，它的意思是“A reliable, autonomous, distributed object storage”。
RADOS由兩個組件組成：
? OSD： Object Storage Device，提供存儲資源。即osd
? Monitor：維護整個Ceph集群的全局狀態。即mon
RADOS具有很強的擴展性和可編程性，Ceph基于RADOS開發了Object Storage、Block Storage、FileSystem。
Ceph另外兩個組件是：
? MDS：用于保存CephFS的元數據。即mds
? RADOS Gateway：對外提供REST接口，兼容S3和Swift的API。即rgw
RADOS由兩個組件組成：
? OSD： Object Storage Device，提供存儲資源。即osd
? Monitor：維護整個Ceph集群的全局狀態。即mon
RADOS具有很強的擴展性和可編程性，Ceph基于RADOS開發了Object Storage、Block Storage、FileSystem。
Ceph另外兩個組件是：
? MDS：用于保存CephFS的元數據。即mds
? RADOS Gateway：對外提供REST接口，兼容S3和Swift的API。即rgw
環境準備
1，準備四臺ubuntu1404 server
192.168.3.106 client （部署ceph-deploy）
192.168.3.8 node1 （部署mon,mds）
192.168.3.9 node2 （部署osd)
192.168.3.10 node3 (部署osd)
2，為每臺配置hosts
root@node2:~# cat /etc/hosts 127.0.0.1 localhost 192.168.3.8 node1 192.168.3.9 node2 192.168.3.10 node3 192.168.3.106 client
3，開啟root 用戶ssh
vi /etc/ssh/sshd_config
修改PermitRootLogin yes
執行service ssh restart 重啟ssh服務
4，配置client 到各個節點的無密碼登錄
node1,node2,node3 執行
ssh-keygen
client執行：
ssh-keygen
ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3
root@client:~# ssh-copy-id node1 The authenticity of host 'node1 (192.168.3.8)' can't be established. ECDSA key fingerprint is 1b:16:32:16:e8:92:4e:f2:8b:98:1f:1a:9b:e4:27:e9. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@osd's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'node1'" and check to make sure that only the key(s) you wanted were added.測試無密碼登錄
ssh mon ,登錄成功！
root@client:~# ssh node1
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Sun Jan 10 11:09:47 CST 2016
System load: 0.01 Processes: 93
Usage of /: 19.7% of 5.51GB Users logged in: 1
Memory usage: 5% IP address for eth0: 192.168.3.8
Swap usage: 0%
Graph this data and manage this system at:
https://landscape.canonical.com/
Last login: Sun Jan 10 10:34:52 2016
root@node1:~#
安裝ceph 集群
1，安裝ceph-deploy執行：apt-get install ceph-deploy
2，配置一個mon 節點 node
執行ceph-deploy new node1 ，這里必須用主機名
root@client:~/ceph# ceph-deploy new node1 [ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy new node1 [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][DEBUG ] Resolving host node1 [ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.3.8 [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [node1][DEBUG ] connected to host: client [node1][INFO ] Running command: ssh -CT -o BatchMode=yes node1 [ceph_deploy.new][DEBUG ] Monitor initial members are ['node1'] [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.3.8'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
執行成功后目錄下回生成 如下三個文件
ceph.mon.keyring ，ceph.conf，ceph.log
編輯ceph.conf 添加 osd_pool_default_size = 2
root@client:~/ceph# vi ceph.conf [global] fsid = 3a562301-bd64-45c5-aaa0-ef57e3dfd76f mon_initial_members = node1 mon_host = 192.168.3.8 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_size = 2
3，安裝ceph 到三個節點上
執行ceph-deploy install client node1 node2 node3
root@client:~/ceph# ceph-deploy install client node1 node2 node3 [ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy install client node1 node2 node3 [ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster ceph hosts client node1 node2 node3 [ceph_deploy.install][DEBUG ] Detecting platform for host client ... [client][DEBUG ] connected to host: client [client][DEBUG ] detect platform information from remote host [client][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: Ubuntu 14.04 trusty [client][INFO ] installing ceph on client [client][INFO ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q install --assume-yes ca-certificates [client][DEBUG ] Reading package lists... [client][DEBUG ] Building dependency tree...
[node3][DEBUG ] Unpacking ceph-mds (0.80.10-0ubuntu1.14.04.3) ... [node3][DEBUG ] Processing triggers for man-db (2.6.7.1-1ubuntu1) ... [node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ... [node3][DEBUG ] ureadahead will be reprofiled on next reboot [node3][DEBUG ] Setting up ceph-fs-common (0.80.10-0ubuntu1.14.04.3) ... [node3][DEBUG ] Setting up ceph-mds (0.80.10-0ubuntu1.14.04.3) ... [node3][DEBUG ] ceph-mds-all start/running [node3][DEBUG ] Processing triggers for ureadahead (0.100.0-16) ... [node3][INFO ] Running command: ceph --version [node3][DEBUG ] ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70) Unhandled exception in thread started by sys.excepthook is missing lost sys.stderr
到各個node去執行ceph --version 檢查ceph安裝結果
root@node2:~# ceph --version ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
激活監控節點
root@client:~/ceph# ceph-deploy mon create-initial
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node1
[ceph_deploy.mon][DEBUG ] detecting platform for host node1 ...
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Ubuntu 14.04 trusty
[node1][DEBUG ] determining if provided host has same hostname in remote
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] deploying mon to node1
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] remote hostname: node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create the mon path if it does not exist
[node1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node1/done
[node1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node1/done
[node1][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create the monitor keyring file
[node1][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i node1 --keyring /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] ceph-mon: mon.noname-a 192.168.3.8:6789/0 is local, renaming to mon.node1
[node1][DEBUG ] ceph-mon: set fsid to 3a562301-bd64-45c5-aaa0-ef57e3dfd76f
[node1][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node1 for mon.node1
[node1][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-node1.mon.keyring
[node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[node1][DEBUG ] create the init path if it does not exist
[node1][DEBUG ] locating the `service` executable...
[node1][INFO ] Running command: initctl emit ceph-mon cluster=ceph id=node1
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[node1][DEBUG ] ********************************************************************************
[node1][DEBUG ] status for monitor: mon.node1
[node1][DEBUG ] {
[node1][DEBUG ] "election_epoch": 2,
[node1][DEBUG ] "extra_probe_peers": [],
[node1][DEBUG ] "monmap": {
[node1][DEBUG ] "created": "0.000000",
[node1][DEBUG ] "epoch": 1,
[node1][DEBUG ] "fsid": "3a562301-bd64-45c5-aaa0-ef57e3dfd76f",
[node1][DEBUG ] "modified": "0.000000",
[node1][DEBUG ] "mons": [
[node1][DEBUG ] {
[node1][DEBUG ] "addr": "192.168.3.8:6789/0",
[node1][DEBUG ] "name": "node1",
[node1][DEBUG ] "rank": 0
[node1][DEBUG ] }
[node1][DEBUG ] ]
[node1][DEBUG ] },
[node1][DEBUG ] "name": "node1",
[node1][DEBUG ] "outside_quorum": [],
[node1][DEBUG ] "quorum": [
[node1][DEBUG ] 0
[node1][DEBUG ] ],
[node1][DEBUG ] "rank": 0,
[node1][DEBUG ] "state": "leader",
[node1][DEBUG ] "sync_provider": []
[node1][DEBUG ] }
[node1][DEBUG ] ********************************************************************************
[node1][INFO ] monitor: mon.node1 is running
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO ] processing monitor mon.node1
[node1][DEBUG ] connected to host: node1
[node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[ceph_deploy.mon][INFO ] mon.node1 monitor has reached quorum!
[ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /etc/ceph/ceph.client.admin.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from node1.
[ceph_deploy.gatherkeys][DEBUG ] Checking node1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] fetch remote file
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from node1.執行成功后本地生成三個key 文件，
ceph.bootstrap-mds.keyring，ceph.bootstrap-osd.keyring，ceph.client.admin.keyring
root@client:~/ceph# ll total 100 drwxr-xr-x 2 root root 4096 Jan 12 14:07 ./ drwx------ 5 root root 4096 Jan 12 13:53 ../ -rw-r--r-- 1 root root 71 Jan 12 14:07 ceph.bootstrap-mds.keyring -rw-r--r-- 1 root root 71 Jan 12 14:07 ceph.bootstrap-osd.keyring -rw-r--r-- 1 root root 63 Jan 12 14:07 ceph.client.admin.keyring -rw-r--r-- 1 root root 251 Jan 12 13:53 ceph.conf -rw-r--r-- 1 root root 66652 Jan 12 14:07 ceph.log -rw-r--r-- 1 root root 73 Jan 12 13:52 ceph.mon.keyring
并且mon 節點 node1 上會有ceph_mon 進程
root@node1:~# ps aux|grep ceph root 1518 0.2 1.8 154700 19284 ? Ssl 14:48 0:00 /usr/bin/ceph-mon --cluster=ceph -i node1 -f root 1846 0.0 0.2 11748 2208 pts/0 S+ 14:49 0:00 grep --color=auto ceph
配置osd 節點
給node2,node3 分別添加一塊硬盤8G,創建分區/dev/sdb1,格式化為xfs 文件系統 ，
root@node3:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe5f14ca9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-16777215, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215): 16000000
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe5f14ca9
Device Boot Start End Blocks Id System
/dev/sdb1 2048 16000000 7998976+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@node3:~# mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=4, agsize=499936 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=1999744, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node3:~#添加osd 節點
執行ceph-deploy osd prepare node2:/dev/sdb1 node3:/dev/sdb1
root@client:~/ceph# ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy osd prepare node2:/dev/sdb1
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node2:/dev/sdb1:
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node2 disk /dev/sdb1 journal None activate False
[node2][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb1
[node2][WARNIN] Error: Partition(s) 1 on /dev/sdb1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
[node2][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4, agsize=499936 blks
[node2][DEBUG ] = sectsz=512 attr=2, projid32bit=0
[node2][DEBUG ] data = bsize=4096 blocks=1999744, imaxpct=25
[node2][DEBUG ] = sunit=0 swidth=0 blks
[node2][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[node2][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2
[node2][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1
[node2][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
[ceph_deploy.osd][DEBUG ] Host node2 is now ready for osd use.
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr激活osd 節點
執行 ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1
root@client:~/ceph# ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1 [ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy osd activate node2:/dev/sdb1 node3:/dev/sdb1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/dev/sdb1: node3:/dev/sdb1: [node2][DEBUG ] connected to host: node2 [node2][DEBUG ] detect platform information from remote host [node2][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty [ceph_deploy.osd][DEBUG ] activating host node2 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will use init type: upstart [node2][INFO ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1 [node2][WARNIN] got monmap epoch 1 [node2][WARNIN] 2016-01-12 15:26:22.781005 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway [node2][WARNIN] 2016-01-12 15:26:23.132088 7f1967a62800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway [node2][WARNIN] 2016-01-12 15:26:23.136228 7f1967a62800 -1 filestore(/var/lib/ceph/tmp/mnt.LRhdRc) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory [node2][WARNIN] 2016-01-12 15:26:23.386713 7f1967a62800 -1 created object store /var/lib/ceph/tmp/mnt.LRhdRc journal /var/lib/ceph/tmp/mnt.LRhdRc/journal for osd.0 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f [node2][WARNIN] 2016-01-12 15:26:23.386942 7f1967a62800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.LRhdRc/keyring: can't open /var/lib/ceph/tmp/mnt.LRhdRc/keyring: (2) No such file or directory [node2][WARNIN] 2016-01-12 15:26:23.387056 7f1967a62800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.LRhdRc/keyring [node2][WARNIN] added key for osd.0 [node3][DEBUG ] connected to host: node3 [node3][DEBUG ] detect platform information from remote host [node3][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty [ceph_deploy.osd][DEBUG ] activating host node3 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will use init type: upstart [node3][INFO ] Running command: ceph-disk-activate --mark-init upstart --mount /dev/sdb1 [node3][WARNIN] got monmap epoch 1 [node3][WARNIN] 2016-01-12 15:26:27.482403 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway [node3][WARNIN] 2016-01-12 15:26:28.286329 7f5efe4de800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway [node3][WARNIN] 2016-01-12 15:26:28.416139 7f5efe4de800 -1 filestore(/var/lib/ceph/tmp/mnt.l2aD7V) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory [node3][WARNIN] 2016-01-12 15:26:29.444301 7f5efe4de800 -1 created object store /var/lib/ceph/tmp/mnt.l2aD7V journal /var/lib/ceph/tmp/mnt.l2aD7V/journal for osd.1 fsid 3a562301-bd64-45c5-aaa0-ef57e3dfd76f [node3][WARNIN] 2016-01-12 15:26:29.444558 7f5efe4de800 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.l2aD7V/keyring: can't open /var/lib/ceph/tmp/mnt.l2aD7V/keyring: (2) No such file or directory [node3][WARNIN] 2016-01-12 15:26:29.444832 7f5efe4de800 -1 created new key in keyring /var/lib/ceph/tmp/mnt.l2aD7V/keyring [node3][WARNIN] added key for osd.1 Unhandled exception in thread started by sys.excepthook is missing lost sys.stderr
在osd 節點上驗證激活成功
執行 ps aux | grep ceph-osd|grep -v grep
root@node2:~# ps aux | grep ceph-osd|grep -v grep root 1752 3.9 2.9 518216 29572 ? Ssl 15:26 0:03 /usr/bin/ceph-osd --cluster=ceph -i 0 -f root@node2:~#查看/dev/sdb1 分區已經掛載成功
root@node2:~# df -TH Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/ubuntu1404--vg-root ext4 6.0G 1.4G 4.3G 24% / none tmpfs 4.1k 0 4.1k 0% /sys/fs/cgroup udev devtmpfs 510M 4.1k 510M 1% /dev tmpfs tmpfs 105M 476k 104M 1% /run none tmpfs 5.3M 0 5.3M 0% /run/lock none tmpfs 521M 0 521M 0% /run/shm none tmpfs 105M 0 105M 0% /run/user /dev/sda1 ext2 247M 40M 195M 17% /boot /dev/sdb1 xfs 8.2G 5.5G 2.8G 67% /var/lib/ceph/osd/ceph-0
分別在兩個osd 節點中，把/dev/sdb1分區加入到/etc/fstab 中開機自動掛載
添加 /dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults 0 0
root@node2:~# vi /etc/fstab # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> /dev/mapper/ubuntu1404--vg-root / ext4 errors=remount-ro 0 1 # /boot was on /dev/sda1 during installation UUID=54b04b5b-52ce-485f-95ef-70c667cfd8b3 /boot ext2 defaults 0 2 /dev/mapper/ubuntu1404--vg-swap_1 none swap sw 0 0 /dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults 0 0 ~配置mds服務
在node1 上添加 metadata server
執行ceph-deploy mds create node1
root@client:~/ceph# ceph-deploy mds create node1
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy mds create node1
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node1:node1
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.mds][DEBUG ] remote host will use upstart
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][DEBUG ] create path if it doesn't exist
[node1][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node1/keyring
[node1][INFO ] Running command: initctl emit ceph-mds cluster=ceph id=node1
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr檢查mds 進程
root@node1:~# ps aux|grep mds root 2025 0.1 1.6 165288 16280 ? Ssl 15:34 0:00 /usr/bin/ceph-mds --cluster=ceph -i node1 -f root 2046 0.0 0.2 11748 2216 pts/0 S+ 15:36 0:00 grep --color=auto mds
創建一個ceph 文件系統
root@node1:~# ceph osd pool create cephfs_data 10
pool 'cephfs_data' created
root@node1:~# ceph osd pool create cephfs_metadata 10
pool 'cephfs_metadata' created
root@node1:~# ceph fs new cephfs cephfs_metadata cephfs_data
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph fs ls
no valid command found; 10 closest matches:
fsid
Error EINVAL: invalid command
root@node1:~# ceph mds stat
e4: 1/1/1 up {0=node1=up:active}
root@node1:~#同步配置文件
執行 ceph-deploy admin node1 node2 node3
root@client:~/ceph# ceph-deploy admin node1 node2 node3
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy admin node1 node2 node3
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] get remote short hostname
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node3
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][DEBUG ] get remote short hostname
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf查看集群健康狀態
root@node1:~# ceph health HEALTH_OK
使用rbd
新建一個ceph pool
rados mkpool test
在pool中新建一個鏡像（ip 為mon節點）
rbd create test-1 --size 4096 -p test -m 172.17.106.122
把鏡像映射到pool塊設備中
rbd map test-1 -p test --name client.admin
查看rbd的映射關系
rbd showmapped
root@node1:~# rados mkpool test successfully created pool test root@node1:~# lsmod |grep rbd root@node1:~# rbd create test-1 --size 4096 -p test -m 192.168.3.8 root@node1:~# rbd map test-1 -p test --name client.admin root@node1:~# rbd showmapped id pool image snap device 0 test test-1 - /dev/rbd0
把新建的鏡像ceph塊進行格式化
root@node1:~# mkfs.ext4 -m0 /dev/rbd0
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
262144 inodes, 1048576 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done掛載
root@node1:~# mount /dev/rbd0 /mnt
root@node1:~# df -TH Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/ubuntu1404--vg-root ext4 6.0G 1.4G 4.3G 24% / none tmpfs 4.1k 0 4.1k 0% /sys/fs/cgroup udev devtmpfs 510M 4.1k 510M 1% /dev tmpfs tmpfs 105M 472k 104M 1% /run none tmpfs 5.3M 0 5.3M 0% /run/lock none tmpfs 521M 0 521M 0% /run/shm none tmpfs 105M 0 105M 0% /run/user /dev/sda1 ext2 247M 40M 195M 17% /boot /dev/rbd0 ext4 4.1G 8.4M 4.1G 1% /mnt現在可以使用/mnt 目錄了
本文由用戶 duimian97oe 自行上傳分享，僅供網友學習交流。所有權歸原作者，若您的權利被侵害，請聯系管理員。
轉載本站原創文章，請注明出處，并保留原始鏈接、圖片水印。
本站是一個以用戶分享為主的開源技術平臺，歡迎各類分享！