storm集群和單點安裝

來自： http://my.oschina.net/goudingcheng/blog/614236

Storm實時分析平臺的起源，Storm分布式集群實施
流式處理場景：算法交易
storm組件
1.nimbus集群中心，控制節點
2.supervisor nodes taskernode
3.the zookeeper cluster 交換信息
storm數據模型：由許多記錄組成，送入
環境準備
1.關閉防火墻
2.修改hosts
192.168.88.197 user5.hadoop.com
192.168.88.198 user6.hadoop.com
192.168.88.196 user7.hadoop.com
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
tar -zxvf zookeeper-3.3.6.tar.gz
cd zookeeper-3.3.6
mkdir data
cd conf/
cp -rp zoo_sample.cfg zoo.cfg
dataDir=/root/soft/zookeeper-3.3.6/data
前一個端口選舉，后一個端口zookeeper進行通訊
server.1=user5.hadoop.com:2888:3888
server.2=user6.hadoop.com:2888:3888
server.3=user7.hadoop.com:2888:3888
cd ~
scp -rp  zookeeper-3.3.6/ root@ip:/home/grid
[root@localhost  data]# cd /root/soft/zookeeper-3.3.6/data

創建myid
在dataDir(/home/grid/zookeeper-3.4.6/data)中創建一個myid
因為server.1=user5.hadoop.com:2888:3888指定的是1，所以在user5.hadoop.com的
機器上echo “1” >myid
其余兩臺機器配置，user6.hadoop.com myid=2 user7.hadoop.com myid=3

/root/soft/zookeeper-3.3.6/bin
[root@localhost  bin]#
[root@localhost  bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@localhost  bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: leader
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower
[root@localhost bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower

安裝python
https://www.python.org/downloads/ftp/python2.6.6
python
[root@localhost Desktop]# tar -zxf Python-2.7.2.tgz
[root@localhost soft]# cd Python-2.7.2
 ./configure --prefix=/usr/local/python27
 make
 make install
 [root@localhost Python-2.7.2]# ls /usr/local/python27/ -al
total 28
drwxr-xr-x  6 root root 4096 Feb  5 23:29 .
drwxr-xr-x 12 root root 4096 Feb  5 23:28 ..
drwxr-xr-x  2 root root 4096 Feb  5 23:29 bin
drwxr-xr-x  3 root root 4096 Feb  5 23:29 include
drwxr-xr-x  4 root root 4096 Feb  5 23:29 lib
drwxr-xr-x  3 root root 4096 Feb  5 23:29 share
[root@localhost Python-2.7.2]# mv /usr/bin/python /usr/bin/python_old
[root@localhost Python-2.7.2]# ln -s /usr/local/python27/bin/python /usr/bin/
[root@localhost Python-2.7.2]# python
Python 2.7.2 (default, Feb  5 2016, 23:28:47)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[root@localhost Python-2.7.2]# vi /usr/bin/yum
#!/usr/bin/python2.4

0.9之前的安裝步驟
同樣安裝JDK和python，zookeeper集群
2.和前面一樣，只是要多安裝storm的兩個依賴庫，zeroMQ和JZMQ
1.搭建zookeper集群
2.安裝storm依賴庫【JDK和python，zeroMQ，JZMQ】
3.下載并且解壓storm發布版本
4.修改storm.yaml文件
5.啟動storm后臺進程

[root@localhost soft]# wget http://mirror.bit.edu.cn/apache/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz
[root@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
t@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
@localhost soft]# mv apache-storm-0.9.1-incubating storm-0.9.1
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
storm.yaml
[root@localhost conf]# vi storm.yaml

  storm.zookeeper.servers:
     - "192.168.88.197"
     - "192.168.88.198"
     - "192.168.88.196"
#
 nimbus.host: "192.168.88.197"
 storm.local.dir:"/root/soft/storm-0.9.1/data"
 supervisor.slots.ports:
  -6700
  -6701
  -6702
  -6703



/root/soft/storm-0.9.1
[root@localhost soft]# scp -rp storm-0.9.1/ root@user6.hadoop.com:~/soft
[root@localhost soft]# scp -rp storm-0.9.1/ root@user7.hadoop.com:~/soft
.BASH_PROFILE
STORM_HOME=/root/soft/storm-0.9.1
PATH=$PATH:$HOME/bin:$STORM_HOME/bin
export STORM_HOME最下面

[root@localhost storm-0.9.1]# source /etc/profile
[root@localhost storm-0.9.1]# echo $PATH
/root/soft/storm-0.9.1/bin
[root@localhost ~]# jps確保三臺都能夠正常啟動
5215 QuorumPeerMain
23327 Jps

[root@localhost conf]# storm nimbus &在nimbus節點
[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@user6 ~]# storm supervisor &
[root@user6 ~]# jps
\5165 QuorumPeerMain
24473 Jps
24354 supervisor


[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@localhost conf]# storm logviewer &
[root@localhost conf]# storm ui &網址8080端口
[root@localhost conf]# jps
5536 nimbus
5841 logviewer
6013 core
5431 QuorumPeerMain
6074 Jps


[root@user6 ~]# jps
5165 QuorumPeerMain
24678 Jps
24354 supervisor


安裝git
yum install git
安裝maven
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
解壓配置環境變量
[root@localhost soft]# git clone git://github.com/nathanmarz/storm-starter.git
[root@localhost soft]# cd storm-starter
[root@localhost storm-starter]# ls
LICENSE     multilang    README.markdown  storm-starter.iml
m2-pom.xml  project.clj  src              test
[root@localhost storm-starter]# mv m2-pom.xml pom.xml
[root@localhost storm-starter]# vi pom.xml
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>0.9.1-incubating</version>
      <!-- keep storm out of the jar-with-dependencies -->
      <scope>provided</scope>
    </dependency>
    
[root@localhost storm-starter]# mvn compile exec:java -Dstorm.topology=storm.starter.wordCountTopology

mvn package
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.RollingTopwords
遠程模式
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology ExclamationTopology
storm kill ExclamationTopology
安裝ZeroMQ
安裝依賴包
yum install gcc-c++ libuuid-devel
對于libuuid-devel
先安裝ncurses-5.9.tar
在下載ftp://ftp.kernel.org/pub/linux/utils/util-linux/v2.21/util-linux-2.21.1.tar.gz  
編譯安裝
下載http://download.zeromq.org/zeromq-2.1.7.tar.gz
編譯安裝JZMQ
git clone git://github.com/nathanmarz/jzmq.git
cd jzmq
[root@localhost jzmq]# ./autogen.sh
[root@localhost jzmq]# ./configure
make
make install
下載并解壓storm第三臺
yum install unzip
wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
ocalhost soft]# unzip storm-0.8.1.zip
[root@localhost soft]# cd storm-0.8.1
cd storm-0.8.1
修改conf/storm.yaml文件在配置的每一項前加空格，冒號后也要加空格
 storm.zookeeper.servers:
     - "192.168.88.197"
     - "192.168.88.198"
     - "192.168.88.199"
#
 nimbus.host: "192.168.88.199"
storm.local.dir: "/root/soft/storm-0.8.1/data"
supervisor.slots.ports:
 - 6700
 - 6701
 - 6702
 - 6703
[root@localhost conf]# jps
5215 QuorumPeerMain
13883 Jps
修改環境變量

[root@localhost ~]# storm nimbus &
[1] 13974
[root@localhost ~]# jps
[root@localhost ~]# jps
14127 Jps
13974 nimbus
5215 QuorumPeerMain
14037 logviewer
14059 core