JStorm 2.1.1 發布,分布式計算系統

jopen 8年前發布 | 10K 次閱讀

Storm 是一個類似Hadoop MapReduce的系統, 用戶按照指定的接口實現一個任務,然后將這個任務遞交給JStorm系統,Jstorm將這個任務跑起來,并且按7 * 24小時運行起來,一旦中間一個worker 發生意外故障, 調度器立即分配一個新的worker替換這個失效的worker。因此,從應用的角度,JStorm 應用是一種遵守某種編程規范的分布式應用。從系統角度, JStorm一套類似MapReduce的調度系統。 從數據的角度, 是一套基于流水線的消息處理機制。實時計算現在是大數據領域中最火爆的一個方向,因為人們對數據的要求越來越高,實時性要求也越來越快,傳統的Hadoop Map Reduce,逐漸滿足不了需求,因此在這個領域需求不斷。

新特性:

  1. 1.5~6x performance boost from worst to best scenarios compared to JStorm-2.1.0

  2. Add application-level auto-batch

  3. Add independent control channel to separate control msgs from biz msgs to guarantee high priority for control msgs

  4. Dramatic performance boost in metrics, see "Improvements" section

  5. Support jdk1.8

  6. Add Nimbus hook and topology hook

  7. Metrics system:

    1. Support disable/enable metrics on the fly

    2. Add jstorm metrics design docs, see JSTORM-METRICS.md

  8. JStorm web UI:

    1. Add zookeeper viewer in web UI, thanks to @dingjun84

    2. Add log search and deep log search, support both backward search and forward search

    3. Support log file download

  9. Support changing log level on the fly

  10. Change error structure in zk, add errorLevel, errorCode and duration.

  11. Add supervisor health check

  12. Add -Dexclude.jars option to enable filtering jars manually

Bug 修復:

  1. Fix deadlock when starting workers

  2. Fix the bug that when localstate file is empty, supervisor can't start

  3. Fix kryo serialization for HeapByteBuffer in metrics

  4. Fix total memory usage calculation

  5. Fix the bug that empty worker is assigned when configured worker number is bigger than the actual number for user defined scheduler

  6. Fix UI log home directory

  7. Fix XSS security bug in web UI

  8. Don't start TopologyMetricsRunnable thread in local mode, thanks to @L-Donne

  9. Fix JSTORM-141, JSTORM-188 that TopologyMetricsRunnable consumes too much CPU

  10. Remove MaxTenuringThreshold JVM option support jdk1.8, thanks to @249550148

  11. Fix possible NPE in MkLocalShuffer

下載地址:


來自: http://www.oschina.net//news/71306/jstorm-2-1-1

 本文由用戶 jopen 自行上傳分享,僅供網友學習交流。所有權歸原作者,若您的權利被侵害,請聯系管理員。
 轉載本站原創文章,請注明出處,并保留原始鏈接、圖片水印。
 本站是一個以用戶分享為主的開源技術平臺,歡迎各類分享!