通過pymongo測試MongoDB的高可用性
原文出處： 肖鵬（@進擊的麥兜兜） 自薦
1 實驗環境搭建
三臺Ubuntu 14.04 64bit Server。
$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.1 LTS Release: 14.04 Codename: trusty
Mongodb version db version v2.6.7.
Pymongo version 2.7
1.1 Install mongodb
在三臺機器上安裝mongodb
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10 $ echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list $ sudo apt-get update $ sudo apt-get install -y mongodb-org
1.2 Configure mongodb
我們要配置一個三個節點的Replication，如下所示：
1.2.1 三個節點能相互通信，配置DNS。
三臺機器的hostname分別為：mongodb1 mongodb2 mongodb3
配置host：
$ more /etc/hosts 127.0.0.1 localhost 10.75.44.10 mongodb1 10.75.44.11 mongodb2 10.75.44.12 mongodb3
保證之間能相互ping通：
$ ping mongodb1 PING mongodb1 (10.75.44.10) 56(84) bytes of data. 64 bytes from mongodb1 (10.75.44.10): icmp_seq=1 ttl=64 time=0.037 ms ^C --- mongodb1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms penxiao@mongodb1:~$ ping mongodb2 PING mongodb2 (10.75.44.11) 56(84) bytes of data. 64 bytes from mongodb2 (10.75.44.11): icmp_seq=1 ttl=64 time=0.277 ms ^C --- mongodb2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.277/0.277/0.277/0.000 ms penxiao@mongodb1:~$ ping mongodb3 PING mongodb3 (10.75.44.12) 56(84) bytes of data. 64 bytes from mongodb3 (10.75.44.12): icmp_seq=1 ttl=64 time=0.193 ms ^C --- mongodb3 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.193/0.193/0.193/0.000 ms
1.2.2. 配置mongodb
盡量使用MongoDB默認端口27017. 使用bind_ip配置選項配置綁定的IP地址，能讓三個節點相互訪問。
1) 啟動三個mongodb進程
修改三臺host的mongodb1的配置： 添加一個相同的replicate set的名字:rs1。
$ more /etc/mongod.conf replSet=rs1
啟動三臺host的mongod進程
$ sudo service mongod start/restart
2) 然后進到mongodb1的mongo shell里初始化配置。
penxiao@mongodb1:~$ mongo
MongoDB shell version: 2.6.7
connecting to: test
>
>
> use adminuse admin
switched to db admin
> rs.conf()rs.conf()
null
> rs.status()
{
"startupStatus" : 3,
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "can't get local.system.replset config from self or any seed (EMPTYCONFIG)"
}
>
>
> rs.initiate()
{
"info2" : "no configuration explicitly specified -- making one",
"me" : "mongodb1:27017",
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
> rs.status()
{
"set" : "rs1",
"date" : ISODate("2015-01-16T02:45:35Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "mongodb1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 510,
"optime" : Timestamp(1421376326, 1),
"optimeDate" : ISODate("2015-01-16T02:45:26Z"),
"electionTime" : Timestamp(1421376326, 2),
"electionDate" : ISODate("2015-01-16T02:45:26Z"),
"self" : true
}
],
"ok" : 1
}
rs1:PRIMARY> 可以看到此時因為只有一個member，并且本身就是PRIMARY.
2) 添加member
把另外兩臺host添加進來。
rs1:PRIMARY> rs.add('mongodb2')
{ "ok" : 1 }
rs1:PRIMARY> rs.add('mongodb3')
{ "ok" : 1 }
rs1:PRIMARY>
rs1:PRIMARY> rs.conf()
{
"_id" : "rs1",
"version" : 3,
"members" : [
{
"_id" : 0,
"host" : "mongodb1:27017"
},
{
"_id" : 1,
"host" : "mongodb2:27017"
},
{
"_id" : 2,
"host" : "mongodb3:27017"
}
]
}
rs1:PRIMARY> rs.status()
{
"set" : "rs1",
"date" : ISODate("2015-01-16T02:48:36Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "mongodb1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 691,
"optime" : Timestamp(1421376444, 1),
"optimeDate" : ISODate("2015-01-16T02:47:24Z"),
"electionTime" : Timestamp(1421376326, 2),
"electionDate" : ISODate("2015-01-16T02:45:26Z"),
"self" : true
},
{
"_id" : 1,
"name" : "mongodb2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 99,
"optime" : Timestamp(1421376444, 1),
"optimeDate" : ISODate("2015-01-16T02:47:24Z"),
"lastHeartbeat" : ISODate("2015-01-16T02:48:35Z"),
"lastHeartbeatRecv" : ISODate("2015-01-16T02:48:35Z"),
"pingMs" : 0,
"syncingTo" : "mongodb1:27017"
},
{
"_id" : 2,
"name" : "mongodb3:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 72,
"optime" : Timestamp(1421376444, 1),
"optimeDate" : ISODate("2015-01-16T02:47:24Z"),
"lastHeartbeat" : ISODate("2015-01-16T02:48:36Z"),
"lastHeartbeatRecv" : ISODate("2015-01-16T02:48:35Z"),
"pingMs" : 0,
"syncingTo" : "mongodb1:27017"
}
],
"ok" : 1
}
rs1:PRIMARY> 結果可以看到mongodb1是PRIMARY，其它兩個host是SECONDARY。
3) 刪除member
http://docs.mongodb.org/manual/tutorial/remove-replica-set-member/
- Shut down the mongod instance for the member you wish to remove. To shut down the instance, connect using the mongo shell and the db.shutdownServer() method.
- Connect to the replica set’s current primary. To determine the current primary, use db.isMaster() while connected to any member of the replica set.
- Use rs.remove() in either of the following forms to remove the member:
rs.remove("mongod3.example.net:27017")
rs.remove("mongod3.example.net") MongoDB disconnects the shell briefly as the replica set elects a new primary. The shell then automatically reconnects. The shell displays a DBClientCursor::init call() failed error even though the command succeeds.
至此mongodb的配置完成了。
2 測試內容
2.1 Test case 1: Basic operations
測試對Replica Set的基本連接，以及基本寫操作，和Failover后的恢復操作，
使用pymongo
Connecting to a Replica Set
>>> from pymongo import MongoClient
>>> MongoClient("10.75.44.10", replicaset='rs1')
MongoClient([u'mongodb2:27017', u'mongodb1:27017', u'mongodb3:27017'])
>>> 對數據庫進行基本的寫操作：
>>> from pymongo import MongoClient
>>> db = MongoClient("10.75.44.10", replicaset='rs1').demo
>>> db
Database(MongoClient([u'mongodb2:27017', u'mongodb1:27017', u'mongodb3:27017']), u'demo')
>>> db.connection.host
'10.75.44.10'
>>> db.connection.port
27017
>>> 看到目前對于數據庫的操作是PRIMARY，也就是host mongodb1。 然后對數據庫寫入一條record：
>>> db.test.insert({'x':1})
ObjectId('54b8b1a1c77b3b3b354869a3')
>>> db.test.find_one()
{u'x': 1, u'_id': ObjectId('54b8b1a1c77b3b3b354869a3')}
>>> 此時，把mongodb1的mongod stop掉。
>>> db.test.find_one()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 713, in find_one
for result in cursor.limit(-1):
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1038, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 982, in _refresh
self.__uuid_subtype))
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 906, in __send_message
res = client._send_message_with_response(message, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 1186, in _send_message_with_response
sock_info = self.__socket(member)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 913, in __socket
"%s %s" % (host_details, str(why)))
pymongo.errors.AutoReconnect: could not connect to 10.75.44.10:27017: [Errno 111] Connection refused
>>> db.test.find_one()
{u'x': 1, u'_id': ObjectId('54b8b1a1c77b3b3b354869a3')}
>>> db.connection.host
u'mongodb3'
>>> db.connection.port
27017
>>> 發現有個pymongo.errors.AutoReconnect的異常，不過馬上恢復了，而且此時的操作數據庫變成了mongodb3，也就是現在的PRIMARY.
如果需要從Secondary讀取數據，可以設置ReadPreference.
>>> from pymongo.read_preferences import ReadPreference
>>> db.test.find_one()
{u'x': 1, u'_id': ObjectId('54b8b1a1c77b3b3b354869a3')}
>>> db.test.find_one(read_preference=ReadPreference.SECONDARY)
{u'x': 1, u'_id': ObjectId('54b8b1a1c77b3b3b354869a3')}
>>> 2.2 Test case 2: PRIMARY lost connection with all SECONDARY
假如PRIMARY和其它所有的SECONDARY失去聯系了，那么PRIMARY就無法進行讀寫操作了。
>>> db.test.insert({'x':3})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 402, in insert
gen(), check_keys, self.uuid_subtype, client)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 1125, in _send_message
raise AutoReconnect(str(e))
pymongo.errors.AutoReconnect: not master
>>>
>>>
>>>
>>>
>>> db.test.insert({'x':3})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 363, in insert
client._ensure_connected(True)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 924, in _ensure_connected
self.__ensure_member()
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 797, in __ensure_member
member, nodes = self.__find_node()
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 888, in __find_node
raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: [Errno 111] Connection refused, [Errno 111] Connection refused, mongodb3:27017 is not primary or master
>>> 直到有至少兩個Replica Set的host連接，然后選出新的PRIMARY。
2.3 Test case 3: Write Concern
根據Write Concern for Replica Sets的介紹：
>>> db
Database(MongoClient([u'mongodb2:27017', u'mongodb1:27017', u'mongodb3:27017']), u'demo')
>>> db.write_concern
{}
>>> db.write_concern = {'w':2, 'wtimeout':5000}
>>> db.write_concern
{'wtimeout': 5000, 'w': 2}
>>> db.test.insert({'y':2})
ObjectId('54b8b89dc77b3b3b354869a4') 默認的write concern是空的配置。write concern有四個參數：w,wtimeout,j, fsync。
其中比較重要的是w和wtimeout。
w: (integer or string)If this is a replica set, write operations will block until they have been replicated to the specified
number or tagged set of servers. w= always includes the replica set primary (e.g. w=3 means write to
the primary and wait until replicated to two secondaries). Setting w=0 disables write acknowledgement and
all other write concern options.
wtimeout: (integer) Used in conjunction with w. Specify a value in milliseconds to control how long to wait for write
propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised.
Reference
Three Member Replica Sets from mongodb.org
http://docs.mongodb.org/manual/replication/
How to Setup MongoDB Replication Using Replica Set and Arbiters