Hadoop子項目 ZooKeeper 3.3.4 發布
ZooKeeper是Hadoop的正式子項目，它是一個針對大型分布式系統的可靠協調系統，提供的功能包括：配置維護、名字服務、分布式同步、組服務等。ZooKeeper的目標就是封裝好復雜易出錯的關鍵服務，將簡單易用的接口和性能高效、功能穩定的系統提供給用戶。
Zookeeper是Google的Chubby一個開源的實現.是高有效和可靠的協同工作系統.Zookeeper能夠用來leader選舉,配 置信息維護等.在一個分布式的環境中,我們需要一個Master實例或存儲一些配置信息,確保文件寫入的一致性等.Zookeeper能夠保證如下3點:
- Watches are ordered with respect to other events, other watches, and
asynchronous replies. The ZooKeeper client libraries ensures that
everything is dispatched in order. - A client will see a watch event for a znode it is watching before seeing the new data that corresponds to that znode.
- The order of watch events from ZooKeeper corresponds to the order of the updates as seen by the ZooKeeper service.
在Zookeeper中,znode是一個跟Unix文件系統路徑相似的節點,可以往這個節點存儲或獲取數據.如果在創建znode時Flag設置 為EPHEMERAL,那么當這個創建這個znode的節點和Zookeeper失去連接后,這個znode將不再存在在Zookeeper 里.Zookeeper使用Watcher察覺事件信息,當客戶端接收到事件信息,比如連接超時,節點數據改變,子節點改變,可以調用相應的行為來處理數 據.Zookeeper的Wiki頁面展示了如何使用Zookeeper來處理事件通知,隊列,優先隊列,鎖,共享鎖,可撤銷的共享鎖,兩階段提交.
那么Zookeeper能幫我們作什么事情呢?簡單的例子:假設我們我們有個20個搜索引擎的服務器(每個負責總索引中的一部分的搜索任務)和一個 總服務器(負責向這20個搜索引擎的服務器發出搜索請求并合并結果集),一個備用的總服務器(負責當總服務器宕機時替換總服務器),一個web的 cgi(向總服務器發出搜索請求).搜索引擎的服務器中的15個服務器現在提供搜索服務,5個服務器正在生成索引.這20個搜索引擎的服務器經常要讓正在 提供搜索服務的服務器停止提供服務開始生成索引,或生成索引的服務器已經把索引生成完成可以搜索提供服務了.使用Zookeeper可以保證總服務器自動 感知有多少提供搜索引擎的服務器并向這些服務器發出搜索請求,備用的總服務器宕機時自動啟用備用的總服務器,web的cgi能夠自動地獲知總服務器的網絡 地址變化.這些又如何做到呢?
- 提供搜索引擎的服務器都在Zookeeper中創建znode,zk.create("/search/nodes/node1",
"hostname".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateFlags.EPHEMERAL); - 總服務器可以從Zookeeper中獲取一個znode的子節點的列表,zk.getChildren("/search/nodes", true);
- 總服務器遍歷這些子節點,并獲取子節點的數據生成提供搜索引擎的服務器列表.
- 當總服務器接收到子節點改變的事件信息,重新返回第二步.
- 總服務器在Zookeeper中創建節點,zk.create("/search/master", "hostname".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateFlags.EPHEMERAL);
- 備用的總服務器監控Zookeeper中的"/search/master"節點.當這個znode的節點數據改變時,把自己啟動變成總服務器,并把自己的網絡地址數據放進這個節點.
- web的cgi從Zookeeper中"/search/master"節點獲取總服務器的網絡地址數據并向其發送搜索請求.
- web的cgi監控Zookeeper中的"/search/master"節點,當這個znode的節點數據改變時,從這個節點獲取總服務器的網絡地址數據,并改變當前的總服務器的網絡地址.
項目地址：http://zookeeper.apache.org/
Apache ZooKeeper 3.3.4 主要改進：
| Bug | |
| [ZOOKEEPER-961] | Watch recovery after disconnection when connection string contains a prefix |
| [ZOOKEEPER-1006] | QuorumPeer "Address already in use" -- regression in 3.3.3 |
| [ZOOKEEPER-1046] | Creating a new sequential node results in a ZNODEEXISTS error |
| [ZOOKEEPER-1049] | Session expire/close flooding renders heartbeats to delay significantly |
| [ZOOKEEPER-1069] | Calling shutdown() on a QuorumPeer too quickly can lead to a corrupt log |
| [ZOOKEEPER-1087] | ForceSync VM arguement not working when set to "no" |
| [ZOOKEEPER-1097] | Quota is not correctly rehydrated on snapshot reload |
| [ZOOKEEPER-1117] | zookeeper 3.3.3 fails to build with gcc >= 4.6.1 on Debian/Ubuntu |
| [ZOOKEEPER-1154] | Data inconsistency when the node(s) with the highest zxid is not present at the time of leader election |
| [ZOOKEEPER-1156] | Log truncation truncating log too much - can cause data loss |
| [ZOOKEEPER-1174] | FD leak when network unreachable |
| [ZOOKEEPER-1189] | For an invalid snapshot file(less than 10bytes size) RandomAccessFile stream is leaking. |
| [ZOOKEEPER-1203] | Zookeeper systest is missing Junit Classes |
| [ZOOKEEPER-1206] | Sequential node creation does not use always use digits in node name given certain Locales. |
| [ZOOKEEPER-1208] | Ephemeral node not removed after the client session is long gone |
| [ZOOKEEPER-1212] | zkServer.sh stop action is not conformat with LSB para 20.2 Init Script Actions |
| [ZOOKEEPER-1264] | FollowerResyncConcurrencyTest failing intermittently |
| [ZOOKEEPER-1271] | testEarlyLeaderAbandonment failing on solaris - clients not retrying connection |
| [ZOOKEEPER-1283] | building 3.3 branch fails with Ant 1.8.2 (success with 1.7.1 though) |
| 改進 | |
| [ZOOKEEPER-1103] | In QuorumTest, use the same "for ( .. try { break } catch { } )" pattern in testFollowersStartAfterLeaders as in testSessionMove. |
| [ZOOKEEPER-1239] | add logging/stats to identify fsync stalls |
| [ZOOKEEPER-1301] | backport patches related to the zk startup script from 3.4 to 3.3 release |