大數據平臺CDH集群離線搭建

jopen 8年前發布 | 26K 次閱讀 MySQL 大數據 Cloudera 分布式/云計算/大數據

標簽： Cloudera-Manager CDH Hadoop 部署集群

摘要：管理、部署Hadoop集群需要工具，Cloudera Manager便是其一。本文先是簡要對比了當前的類似工具，而后詳細記錄了以離線方式部署CDH集群>的步驟。

[TOC]

前言

以Apache Hadoop為主導的大數據技術的出現，使得中小型公司對于大數據的存儲與處理也擁有了武器。目前Hadoop有不少發行版：華為發行版收費、Intel發行版收費、Cloudera發行版（Cloudera’s Distribution Including Apache Hadoop，簡稱 CDH ）免費、Hortonworks發行版（Hortonworks Data Platform，簡稱 HDP ）免費等，所有這些發行版均是基于Apache Hadoop社區版衍生出來的。

部署、管理擁有數十數百甚至更多節點的Hadoop集群，也需要先進武器。Hortonworks公司的Apache Ambari項目的目的就是通過軟件來配置、監控和管理Hadoop（HDP）集群，以使Hadoop的管理更加簡單。Ambari提供了一個基于它自身RESTful的api實現的直觀的、簡單易用的web界面。Cloudera公司也提供了類似的工具：Cloudera Manager（簡稱 CM ）來配置、監控和管理CDH集群。

需特別注意的是Cloudera Manager與操作系統的版本關系 el7暫不支持 ，按照官方文檔的要求來，否則安裝會有問題。

注意用戶。本文是基于操作系統CentOS 6.5, 64-bit；Cloudera Manager 5.3.6；JDK 1.7 版本進行部署的。

部署步驟

網絡配置（所有節點）

[root@cdh-server ~]# vi /etc/sysconfig/network #修改hostname:
NETWORKING=yes
HOSTNAME=cdh-server
[root@cdh-server ~]# vi /etc/hosts #修改ip與主機名的對應關系:
192.168.180.173 cdh-server
192.168.180.175 node175
[root@cdh-server ~]# service network restart #重啟網絡服務生效</pre> 
 
安裝JDK（所有節點）

#卸載OpenJDK
[root@cdh-server user1]# rpm -qa | grep java
[root@cdh-server user1]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
[root@cdh-server user1]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
[root@cdh-server user1]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
安裝JDK
[root@cdh-server user1]# chmod a+x jdk-7u79-linux-x64.rpm
[root@cdh-server user1]# rpm -ivh jdk-7u79-linux-x64.rpm
[root@cdh-server user1]# echo "JAVA_HOME=/usr/java/jdk1.7.0_79/" >></pre> 
 
安裝MySQL（主節點）

[user1@cdh-server]$ cd /home/user1
[user1@cdh-server]$ tar -zxvf mysql-5.6.26-linux-glibc2.5-x86_64.tar.gz
[user1@cdh-server]$ mv mysql-5.6.26-linux-glibc2.5-x86_64 mysql-5.6.26
[user1@cdh-server]$ cd mysql-5.6.26/
[user1@cdh-server]$ vi support-files/my.cnf #新建文件 
 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[mysqld]
character-set-server=utf8
default-storage-engine=INNODB
Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /home/user1/mysql-5.6.26/data
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /home/user1/mysql-5.6.26/data
You can set .._buffer_pool_size up to 50 - 80 %
of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</pre> 
 
初始化MySQL（主節點）

[user1@cdh-server]$ ./scripts/mysql_install_db --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data --user=user1
[user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
[user1@cdh-server]$ ./bin/mysqladmin -u root password '123456' 
 [user1@cdh-server mysql-5.6.26]$ ./bin/mysql -uroot -p'123456'
hive
mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
Activity Monitor使用
mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)
Navigator Audit Server使用
mysql> create database audit DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)
Navigator Metadata Server
mysql> create database metadata DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)
mysql> grant all privileges on . to 'root'@'localhost' identified by '123456' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on . to 'root'@'cdh-server' identified by '123456' with grant option;
Query OK, 0 rows affected (0.00 sec)
this user scm is for  cloudera manager
mysql> grant all privileges on . to 'scm'@'localhost' identified by 'scm' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on . to 'scm'@'cdh-server' identified by 'scm' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)</pre> 
 
部署/啟動CM Server（主節點）

[user1@cdh-server ~]$ tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
[user1@cdh-server ~]$ cp mysql-connector-java-5.1.33-bin.jar ./cm-5.3.6/share/cmf/lib/
[user1@cdh-server ~]$ su - root
[root@cdh-server ~]# cd /home/user1/
[root@cdh-server user1]# cp -rf cloudera /opt
[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
[root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
[root@cdh-server user1]# mv manifest.json /opt/cloudera/parcel-repo/manifest.json
[root@cdh-server user1]# ./cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost:3306 -uroot -p123456 --scm-host localhost scm scm scm
[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log</pre> 
 
關閉防火墻（所有節點）

#停止iptables
[root@cdh-server user1]# service iptables stop
通過瀏覽器訪問驗證
http://192.168.180.173:7180/</pre> 
 
部署/啟動CM Agent（從節點）

[root@cdh-server user1]# tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
[root@cdh-server user1]# vi cm-5.3.6/etc/cloudera-scm-agent/config.ini 
 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hostname of the CM server.
server_host=localhost
server_host=cdh-server
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</pre> 
 
[root@cdh-server user1]# useradd -G sys --home=/home/user1/cm-5.3.6/run/cloudera-scm-server --no-create-home  --comment "Cloudera SCM User" cloudera-scm
[root@cdh-server user1]# useradd --comment "Cloudera SCM User" cloudera-scm #若上一步執行正確，則此步省略
[root@cdh-server user1]# echo 0 > /proc/sys/vm/swappiness
[root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log</pre> 
 
配置CDH

 登陸Cloudera Manager http://192.168.180.173 :7180/，并新建集群Cluster_user1，進行各服務的配置啟動。 
#安裝配置hive出錯時，在hiveServer上:
[root@hive-server user1]# cp mysql-connector-java-5.1.33-bin.jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/hive/lib/
同理：use this jar for Navigator Audit Server and Navigator Metadata Server or Activity Server
[root@cdh-server user1]# cp mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar</pre> 
 
其他

停止集群步驟
 
   停止Cloudera Management Service和Cluster_user1
 

   從節點停止Agent
 [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent stop
 

   主節點停止Server
 [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server stop
 

 
啟動集群步驟
 
   主節點啟動MySQL
 [user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
[user1@cdh-server]$ ps -a | grep mysql
 

   從節點啟動Agent
 [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log
 

   主節點啟動Server
 [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log
 

   
    
    啟動各服務

     登陸Cloudera Manager http://192.168.180.173 :7180/，進行各服務的檢查啟動。 

   
 

 
啟動our集群步驟
node90：
 
   主節點啟動MySQL
  (1). 用 user1 賬號 啟動mysql
 cd /home/user1/mysql-5.6.26
 ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &

 (2). 驗證是否啟動成功    ps -ef | grep mysql 
 

   主節點啟動Server
 

   從節點啟動Agent
 

 
node98：
 
   從節點啟動Agent
 

 
node96：
 
   從節點啟動Agent
 

</ul>
 </div>
來自： http://segmentfault.com/a/1190000004331498

本文由用戶 jopen 自行上傳分享，僅供網友學習交流。所有權歸原作者，若您的權利被侵害，請聯系管理員。

轉載本站原創文章，請注明出處，并保留原始鏈接、圖片水印。

本站是一個以用戶分享為主的開源技術平臺，歡迎各類分享！

本文地址：http://www.baiduhome.net/lib/view/open1453201603261.html

MySQL 大數據 Cloudera 分布式/云計算/大數據

大數據平臺CDH集群離線搭建

前言

部署步驟

網絡配置（所有節點）

相關經驗

相關資訊

相關文檔

目錄