大數據架構-使用HBase和Solr將存儲與索引放在不同的機器上

jopen 10年前發布 | 42K 次閱讀 HBase Solr 搜索引擎

大數據架構-使用HBase和Solr將存儲與索引放在不同的機器上

摘要： HBase可以通過協處理器 Coprocessor 的 方式向Solr發出請求，Solr對于接收到的數據可以做相關的同步：增、刪、改索引的操作，這樣就可以同時使用HBase存儲量大和Solr檢索性能高 的優點了，更何況HBase和Solr都可以集群。這對海量數據存儲、檢索提供了一種方式，將存儲與索引放在不同的機器上，是大數據 架構的必須品。
關鍵詞： HBase,  Solr,  Coprocessor ,  大數據 ,  架構
 
正如我的之前的博客“ Solr與HBase架構設計 ”中所述，HBase和Solr可以通過協處理器 Coprocessor 的方式向Solr發出請求，Solr對于接收到的數據可以做相關的同步：增、刪、改索引的操作。將存儲與索引放在不同的機器上，這是大數據架構的必須品，但目前還有很多不懂得此道的同學，他們對于這種思想感到很新奇，不過，這絕對是好的方向，所以不懂得抓緊學習吧。
有個朋友給我的那篇博客留言，說CDH也可以做這樣的事情，我還沒有試過，他還問我要與此相關的代碼，于是我就稍微整理了一下，作為本篇文章的主要內容。關于CDH的事，我會盡快嘗試，有知道的同學可以給我留言。
下面我主要講述一下，我測試對HBase和Solr的性能時，使用HBase 協處理器向HBase添加數據所編寫的相關代碼，及解釋說明。
 
 
一、編寫HBase協處理器Coprocessor
 
?一旦有數據postPut，就立即對Solr里相應的Core更新。這里使用了 ConcurrentUpdateSolrServer，它是Solr速率性能的保證，使用它不要忘記在Solr里面配置autoCommit喲。
 
/* 
   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *版權：王安琪</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *描述：監視HBase，一有數據postPut就向<span style="text-decoration:underline;">Solr</span>發送，本類要作為觸發器添加到HBase</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改時間：2014-05-27</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改內容：新增</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> */</span>        </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">package<span style="color:windowtext;"> solrHbase.test;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> java.io.UnsupportedEncodingException;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> ***;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">public<span style="color:windowtext;"> </span>class<span style="color:windowtext;"> SorlIndexCoprocessorObserver </span>extends<span style="color:windowtext;"> BaseRegionObserver {</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">final</span> Logger <span style="color:#0000c0;">LOG</span> = LoggerFactory</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            .<span style="font-style:italic;">getLogger</span>(SorlIndexCoprocessorObserver.<span style="color:#7f0055;">class</span>);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">final</span> String <span style="color:#0000c0;">solrUrl</span> = <span style="color:#2a00ff;">"http://192.1.11.108:80/solr/core1"</span>;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">final</span> SolrServer <span style="color:#0000c0;">solrServer</span> = <span style="color:#7f0055;">new</span> ConcurrentUpdateSolrServer(</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#0000c0;">solrUrl</span>, 10000, 20);</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#3f5fbf;">/**</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     * 建立<span style="text-decoration:underline;">solr</span>索引</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     * </span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     * <span style="color:#7f9fbf;">@throws</span> UnsupportedEncodingException</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     */</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#646464;">@Override</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">void</span> postPut(<span style="color:#7f0055;">final</span> ObserverContext<regioncoprocessorenvironment> e,</regioncoprocessorenvironment></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">final</span> Put put, <span style="color:#7f0055;">final</span> WALEdit edit, <span style="color:#7f0055;">final</span> <span style="color:#7f0055;">boolean</span> writeToWAL)</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">throws</span> UnsupportedEncodingException {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        inputSolr(put);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    }</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">void</span> inputSolr(Put put) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">try</span> {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#0000c0;">solrServer</span>.add(TestSolrMain.<span style="font-style:italic;">getInputDoc</span>(put));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (Exception ex) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#0000c0;">LOG</span>.error(ex.getMessage());</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">}</span>        </div>

</td>

 </tr>

</tbody>

</table>

</div>

 
注意：getInputDoc是這個HBase協處理器Coprocessor的精髓所在，它可以把HBase內的Put里的內容轉化成Solr需要的值。其中 String fieldName = key.substring(key.indexOf( columnFamily ) + 3,  key.indexOf( "我在這" )).trim(); 這里有一個亂碼字符，在這里看不到，請大家注意一下。
 
public static SolrInputDocument getInputDoc(Put put) { 
   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        SolrInputDocument doc = <span style="color:#7f0055;">new</span> SolrInputDocument();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        doc.addField(<span style="color:#2a00ff;">"test_ID"</span>, Bytes.<span style="font-style:italic;">toString</span>(put.getRow()));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">for</span> (KeyValue c : put.getFamilyMap().get(Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#0000c0;">columnFamily</span>))) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            String key = Bytes.<span style="font-style:italic;">toString</span>(c.getKey());</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            String value = Bytes.<span style="font-style:italic;">toString</span>(c.getValue());</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">if</span> (value.isEmpty()) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                <span style="color:#7f0055;">continue</span>;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            String fieldName = key.substring(key.indexOf(<span style="color:#0000c0;">columnFamily</span>) + 3,</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                    key.indexOf(<span style="color:#2a00ff;">""</span>)).trim();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            doc.addField(fieldName, value);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">return</span> doc;</span>        </div>

    }</td>

 </tr>

</tbody>

</table>

</div>

 
二、編寫測試程序入口代碼main
 
?這段代碼向HBase請求建了一張表，并將模擬的數據，向HBase連續地提交數據內容，在HBase中不斷地插入數據，同時記錄時間，測試插入性能。

</div>

 
    • /* 
       <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *版權：王安琪</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *描述：測試HBaseInsert，HBase插入性能</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改時間：2014-05-27</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改內容：新增</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> */</span>        </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">package<span style="color:windowtext;"> solrHbase.test;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> hbaseInput.HbaseInsert;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> ***;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">public<span style="color:windowtext;"> </span>class<span style="color:windowtext;"> TestHBaseMain {</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> Configuration <span style="color:#0000c0;">config</span>;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> String <span style="color:#0000c0;">tableName</span> = <span style="color:#2a00ff;">"angelHbase"</span>;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> HTable <span style="color:#0000c0;">table</span> = <span style="color:#7f0055;">null</span>;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">private</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">final</span> String <span style="color:#0000c0;">columnFamily</span> = <span style="color:#2a00ff;">"wanganqi"</span>;</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#3f5fbf;">/**</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     * <span style="color:#7f9fbf;">@param</span> args</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f5fbf;font-family:'Consolas';">     */</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">void</span> main(String[] args) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#0000c0;">config</span> = HBaseConfiguration.<span style="font-style:italic;">create</span>();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#0000c0;">config</span>.set(<span style="color:#2a00ff;">"hbase.zookeeper.quorum"</span>, <span style="color:#2a00ff;">"192.103.101.104"</span>);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        HbaseInsert.<span style="font-style:italic;">createTable</span>(<span style="color:#0000c0;">config</span>, <span style="color:#0000c0;">tableName</span>, <span style="color:#0000c0;">columnFamily</span>);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">try</span> {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#0000c0;">table</span> = <span style="color:#7f0055;">new</span> HTable(<span style="color:#0000c0;">config</span>, Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#0000c0;">tableName</span>));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">for</span> (<span style="color:#7f0055;">int</span> k = 0; k < 1; k++) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                Thread t = <span style="color:#7f0055;">new</span> Thread() {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">void</span> run() {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                        <span style="color:#7f0055;">for</span> (<span style="color:#7f0055;">int</span> i = 0; i < 100000; i++) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                            HbaseInsert.<span style="font-style:italic;">inputData</span>(<span style="color:#0000c0;">table</span>,</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    PutCreater.<span style="font-style:italic;">createPuts</span>(1000, <span style="color:#0000c0;">columnFamily</span>));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                            Calendar c = Calendar.<span style="font-style:italic;">getInstance</span>();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                            String dateTime = c.get(Calendar.<span style="color:#0000c0;">YEAR</span>) + <span style="color:#2a00ff;">"-"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">MONTH</span>) + <span style="color:#2a00ff;">"-"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">DATE</span>) + <span style="color:#2a00ff;">"T"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">HOUR</span>) + <span style="color:#2a00ff;">":"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">MINUTE</span>) + <span style="color:#2a00ff;">":"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">SECOND</span>) + <span style="color:#2a00ff;">":"</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + c.get(Calendar.<span style="color:#0000c0;">MILLISECOND</span>) + <span style="color:#2a00ff;">"Z 寫入: "</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                                    + i * 1000;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                            System.<span style="color:#0000c0;">out</span>.println(dateTime);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                    }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                };</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                t.start();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (IOException e1) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            e1.printStackTrace();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    }</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">}</span>        </div>

    </td>

     </tr>

</tbody>

    </table>

    </div>

     
    ?下面的是與HBase相關的操作，把它封裝到一個類中，這里就只有建表與插入數據的相關代碼。
     
    /* 
       <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *版權：王安琪</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *描述：與HBase相關操作，建表與插入數據</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改時間：2014-05-27</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> *修改內容：新增</span>        </div>

   <div>
    <span style="font-size:10pt;color:#3f7f5f;font-family:'Consolas';"> */</span>        </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">package<span style="color:windowtext;"> hbaseInput;</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> ***;</span></span>        </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">import<span style="color:windowtext;"> org.apache.hadoop.hbase.client.Put;</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:#7f0055;font-family:'Consolas';">public<span style="color:windowtext;"> </span>class<span style="color:windowtext;"> HbaseInsert {</span></span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">void</span> createTable(Configuration config, String tableName,</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            String columnFamily) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        HBaseAdmin hBaseAdmin;</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">try</span> {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            hBaseAdmin = <span style="color:#7f0055;">new</span> HBaseAdmin(config);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">if</span> (hBaseAdmin.tableExists(tableName)) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                <span style="text-decoration:underline;color:#7f0055;">return<span style="color:windowtext;">;</span></span></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            HTableDescriptor tableDescriptor = <span style="color:#7f0055;">new</span> HTableDescriptor(tableName);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            tableDescriptor.addFamily(<span style="color:#7f0055;">new</span> HColumnDescriptor(columnFamily));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            hBaseAdmin.createTable(tableDescriptor);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            hBaseAdmin.close();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (MasterNotRunningException e) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            e.printStackTrace();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (ZooKeeperConnectionException e) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            e.printStackTrace();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (IOException e) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            e.printStackTrace();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    }</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    <span style="color:#7f0055;">public</span> <span style="color:#7f0055;">static</span> <span style="color:#7f0055;">void</span> inputData(HTable table, ArrayList<put> puts) {</put></span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">try</span> {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            table.put(puts);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            table.flushCommits();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            puts.clear();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        } <span style="color:#7f0055;">catch</span> (IOException e) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            e.printStackTrace();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">    }</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">}</span>        </div>

    </td>

     </tr>

</tbody>

    </table>

    </div>

     
    三、編寫模擬數據Put
     
    向HBase中寫入數據需要構造Put，下面是我構造模擬數據Put的方式，有字符串的生成，我是由mmseg提供的詞典 words.dic 中隨機讀取一些詞語連接起來，生成一句字符串的，下面的代碼沒有體現，不過很easy，你自己造你自己想要的數據就OK了。
     
  • public static Put createPut(String columnFamily) { 
       <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        String ss = <span style="font-style:italic;">getSentence</span>();</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        </span>         <span style="font-family:Consolas;font-size:10pt;color:#7f0055;">byte</span>         <span style="color:windowtext;font-family:Consolas;font-size:10pt;">[] family = Bytes.</span>         <span style="color:windowtext;font-family:Consolas;font-size:10pt;">toBytes</span>         <span style="color:windowtext;font-family:Consolas;font-size:10pt;">(columnFamily);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">byte</span>[] rowKey = Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#2a00ff;">""</span> + Math.<span style="font-style:italic;">abs</span>(<span style="color:#0000c0;">r</span>.nextLong()));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        Put put = <span style="color:#7f0055;">new</span> Put(rowKey);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        put.add(family, Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#2a00ff;">"DeviceID"</span>),</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">                Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#2a00ff;">""</span> + Math.<span style="font-style:italic;">abs</span>(<span style="color:#0000c0;">r</span>.nextInt())));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        ******</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        put.add(family, Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#2a00ff;">"</span></span>         <span style="color:#2a00ff;font-family:Consolas;">Company_mmsegsm</span>         <span style="font-size:10pt;color:windowtext;font-family:'Consolas';"><span style="color:#2a00ff;">"</span>), Bytes.<span style="font-style:italic;">toBytes</span>(<span style="color:#2a00ff;">"ss"</span>));</span>        </div>

   <div>
     
   </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">return</span> put;</span>        </div>

        }</td>

     </tr>

</tbody>

    </table>

    </div>

     
    當然在運行上面這個程序之前，需要先在Solr里面配置好你需要的列信息，HBase、Solr安裝與配置，它們的基礎使用方法將會在之后的文章中介紹 大數據架構-使用HBase和Solr將存儲與索引放在不同的機器上 。在這里，Solr的列配置就跟你使用createPut生成的Put搞成一樣的列名就行了，當然也可以使用動態列的形式。
     
    四、直接對Solr性能測試
     
    如果你不想對HBase與Solr的相結合進行測試，只想單獨對Solr的性能進行測試，這就更簡單了，完全可以利用上面的代碼段來測試，稍微組裝一下就可以了。
     
    private static void sendConcurrentUpdateSolrServer(final String url, 
       <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            <span style="color:#7f0055;">final</span> <span style="color:#7f0055;">int</span> count) <span style="color:#7f0055;">throws</span> SolrServerException, IOException {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        SolrServer solrServer = <span style="color:#7f0055;">new</span> ConcurrentUpdateSolrServer(url, 10000, 20);</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        <span style="color:#7f0055;">for</span> (<span style="color:#7f0055;">int</span> i = 0; i < count; i++) {</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">            solrServer.add(<span style="font-style:italic;">getInputDoc</span>(PutCreater.<span style="font-style:italic;">createPut</span>(<span style="color:#0000c0;">columnFamily</span>)));</span>        </div>

   <div>
    <span style="font-size:10pt;color:windowtext;font-family:'Consolas';">        }</span>        </div>

        }</td>

     </tr>

</tbody>

    </table>

    </div>

     
     
    希望可以幫助到你規格嚴格-功夫到家。這次的文章代碼又偏多了點，但代碼是解釋思想的最好的語言，我的提倡就是盡可能的減少代碼的注釋，盡力簡化你的代碼，使你的代碼足夠的清晰易懂，甚至于相似于偽代碼了，這也是《重構》這本書里所提倡的。


    大數據架構-使用HBase和Solr將存儲與索引放在不同的機器上

     本文由用戶 jopen 自行上傳分享，僅供網友學習交流。所有權歸原作者，若您的權利被侵害，請聯系管理員。
     轉載本站原創文章，請注明出處，并保留原始鏈接、圖片水印。
     本站是一個以用戶分享為主的開源技術平臺，歡迎各類分享！

    相關經驗

    相關資訊

    相關文檔

    • sesese色