Apache Kudu v0.9.0 發布，一個數據存儲系統

為了應對先前發現的這些趨勢，有兩種不同的方式：持續更新現有的Hadoop工具或者重新設計開發一個新的組件。其目標是：

• 對數據掃描(scan)和隨機訪問(random access)同時具有高性能，簡化用戶復雜的混合架構;
• 高CPU效率，最大化先進處理器的效能;
• 高IO性能，充分利用先進永久存儲介質;
• 支持數據的原地更新，避免額外的數據處理、數據移動

我們為了實現這些目標，首先在現有的開源項目上實現原型，但是最終我們得出結論：需要從架構層作出重大改變。而這些改變足以讓我們重新開發一個全新的數據存儲系統。經過多年的努力如今終于可以分享我們 多年來的努力成果：Kudu，一個新的數據存儲系統。

更新日志

不兼容的更改

• The KuduTableInputFormat command has changed the way in which it handles scan predicates, including how it serializes predicates to the job configuration object. The new configuration key is kudu.mapreduce.encoded.predicate. Clients using theTableInputFormatConfigurator are not affected.

• The kudu-spark sub-project has been renamed to follow naming conventions for Scala. The new name is kudu-spark_2.10.

• Default table partitioning has been removed. All tables must now be created with explicit partitioning. Existing tables are unaffected. See the schema design guide for more details.

新功能

• KUDU-1002 Added support for UPSERT operations, whereby a row is inserted if it does not already exist, but updated if it does. Support for UPSERT is included in Java, C++, and Python APIs, but not in Impala.

• KUDU-1306 Scan token API for creating partition-aware scan descriptors. This API simplifies executing parallel scans for clients and query engines.

• Gerrit 2848 Added a kudu datasource for Spark. This datasource uses the Kudu client directly instead of using the MapReduce API. Predicate pushdowns for spark-sql and Spark filters are included, as well as parallel retrieval for multiple tablets and column projections. See an example of Kudu integration with Spark.

• Gerrit 2992 Added the ability to update and insert from Spark using a Kudu datasource.

改進

• KUDU-1415 Added statistics in the Java client such as the number of bytes written and the number of operations applied.

• KUDU-1451 Improved tablet server restart time when the tablet server needs to clean up of a lot previously deleted tablets. Tablets are now cleaned up after they are deleted.

問題修復

• KUDU-678 Fixed a leak that happened during DiskRowSet compactions where tiny blocks were still written to disk even if there were no REDO records. With the default block manager, it usually resulted in block containers with thousands of tiny blocks.

• KUDU-1437 Fixed a data corruption issue that occured after compacting sequences of negative INT32 values in a column that was configured with RLE encoding.

其他值得關注的變化

All Kudu clients have longer default timeout values, as listed below.

Java

• The default operation timeout and the default admin operation timeout are now set to 30 seconds instead of 10.

• The default socket read timeout is now 10 seconds instead of 5.

C++

• The default admin timeout is now 30 seconds instead of 10.

• The default RPC timeout is now 10 seconds instead of 5.

• The default scan timeout is now 30 seconds instead of 15.

• Some default settings related to I/O behavior during flushes and compactions have been changed: The default forflush_threshold_mb has been increased from 64MB to 1000MB. The default cfile_do_on_finish has been changed fromclose to flush. Experiments using YCSB indicate that these values will provide better throughput for write-heavy applications on typical server hardware.

下載 

• Kudu 0.9.0 source tarball
• Source code (zip)
• Source code (tar.gz)