在Android中使用FlatBuffers
總覽
先來看一下 FlatBuffers 項目已經為我們提供了什么，而我們在將 FlatBuffers 用到我們的項目中時又需要做什么的整體流程。如下圖：
在使用 FlatBuffers 時，我們需要以特殊的格式定義我們的結構化數據，保存為 .fbs 文件。 FlatBuffers 項目為我們提供了編譯器，可用于將 .fbs 文件編譯為Java文件，C++文件等，以用于我們的項目。 FlatBuffers 編譯器在我們的開發機，比如Ubuntu，Mac上運行。這些源代碼文件是基于 FlatBuffers 提供的Java庫生成的，同時我們也需要利用這個Java庫的一些接口來序列化或解析數據。
我們將 FlatBuffers 編譯器生成的Java文件及 FlatBuffers 的Java庫導入我們的項目，就可以用 FlatBuffers 來對我們的結構化數據執行序列化和反序列化了。盡管每次手動執行 FlatBuffers 編譯器生成Java文件非常麻煩，但不像 Protocol Buffers 那樣，當前還沒有Google官方提供的gradle插件可用。不過，我們這邊開發了一個簡單的 FlatBuffers gradle插件，后面會簡單介紹一下，歡迎大家使用。
接下來我們更詳細地看一下上面流程中的各個部分。
下載、編譯FlatBuffers編譯器
我們可以在如下位置：
https://github.com/google/flatbuffers/releases
獲取官方發布的打包好的版本。針對Windows平臺有編譯好的可執行安裝文件，對其它平臺還是打包的源文件。我們也可以指向clone repo的代碼，進行手動編譯。這里我們從GitHub上clone代碼并手動編譯編譯器：
$ git clone https://github.com/google/flatbuffers.git
Cloning into 'flatbuffers'...
remote: Counting objects: 7340, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 7340 (delta 16), reused 0 (delta 0), pack-reused 7290
Receiving objects: 100% (7340/7340), 3.64 MiB | 115.00 KiB/s, done.
Resolving deltas: 100% (4692/4692), done.
Checking connectivity... done.
下載代碼之后，我們需要用cmake工具來為flatbuffers生成Makefile文件并編譯：
$ cd flatbuffers/
$ cmake CMakeLists.txt
-- The C compiler identification is AppleClang 7.3.0.7030031
-- The CXX compiler identification is AppleClang 7.3.0.7030031
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/netease/Projects/OpenSource/flatbuffers
$ make && make install
安裝之后執行如下命令以確認已經裝好：
$ flatc --version
flatc version 1.4.0 (Dec 7 2016)
flatc沒有為我們提供 --help 選項，不過加了錯誤的參數時這個工具會為我們展示詳細的用法：
$ flatc --help
flatc: unknown commandline argument: --help
usage: flatc [OPTION]... FILE... [-- FILE...]
--binary -b Generate wire format binaries for any data definitions.
--json -t Generate text output for any data definitions.
--cpp -c Generate C++ headers for tables/structs.
--go -g Generate Go files for tables/structs.
--java -j Generate Java classes for tables/structs.
--js -s Generate JavaScript code for tables/structs.
--csharp -n Generate C# classes for tables/structs.
--python -p Generate Python files for tables/structs.
--php Generate PHP files for tables/structs.
-o PATH Prefix PATH to all generated files.
-I PATH Search for includes in the specified path.
-M Print make rules for generated files.
--version Print the version number of flatc and exit.
--strict-json Strict JSON: field names must be / will be quoted,
no trailing commas in tables/vectors.
--allow-non-utf8 Pass non-UTF-8 input through parser and emit nonstandard
\x escapes in JSON. (Default is to raise parse error on
non-UTF-8 input.)
--defaults-json Output fields whose value is the default when
writing JSON
--unknown-json Allow fields in JSON that are not defined in the
schema. These fields will be discared when generating
binaries.
--no-prefix Don't prefix enum values with the enum type in C++.
--scoped-enums Use C++11 style scoped and strongly typed enums.
also implies --no-prefix.
--gen-includes (deprecated), this is the default behavior.
If the original behavior is required (no include
statements) use --no-includes.
--no-includes Don't generate include statements for included
schemas the generated file depends on (C++).
--gen-mutable Generate accessors that can mutate buffers in-place.
--gen-onefile Generate single output file for C#.
--gen-name-strings Generate type name functions for C++.
--escape-proto-ids Disable appending '_' in namespaces names.
--gen-object-api Generate an additional object-based API.
--cpp-ptr-type T Set object API pointer type (default std::unique_ptr)
--raw-binary Allow binaries without file_indentifier to be read.
This may crash flatc given a mismatched schema.
--proto Input is a .proto, translate to .fbs.
--schema Serialize schemas instead of JSON (use with -b)
--conform FILE Specify a schema the following schemas should be
an evolution of. Gives errors if not.
--conform-includes Include path for the schema given with --conform
PATH
FILEs may be schemas, or JSON files (conforming to preceding schema)
FILEs after the -- must be binary flatbuffer format files.
Output files are named using the base file name of the input,
and written to the current directory or the path given by -o.
example: flatc -c -b schema1.fbs schema2.fbs data.json
創建 .fbs 文件
flatc支持將為 Protocol Buffers 編寫的 .proto 文件轉換為 .fbs 文件，如：
$ ls
addressbook.proto
$ flatc --proto addressbook.proto
$ ls -l
total 16
-rw-r--r-- 1 netease staff 431 12 7 17:21 addressbook.fbs
-rw-r--r--@ 1 netease staff 486 12 1 15:18 addressbook.proto
Protocol Buffers消息文件中的一些寫法， FlatBuffers 編譯器還不能很好的支持，如option java_package，option java_outer_classname，和嵌套類。這里我們基于 FlatBuffers 編譯器轉換的 .proto 文件來獲得我們的 .fbs 文件：
// Generated from addressbook.proto
namespace com.example.tutorial;
enum PhoneType : int {
MOBILE = 0,
HOME = 1,
WORK = 2,
}
namespace com.example.tutorial;
table Person {
name:string (required);
id:int;
email:string;
phone:[com.example.tutorial._Person.PhoneNumber];
}
namespace com.example.tutorial._Person;
table PhoneNumber {
number:string (required);
type:int;
}
namespace com.example.tutorial;
table AddressBook {
person:[com.example.tutorial.Person];
}
root_type AddressBook;</code></pre>
編譯 .fbs 文件
可以通過如下命令編譯 .fbs 文件：
$ flatc --java -o out addressbook.fbs
--java用于指定編譯的目標編程語言。-o 參數則用于指定輸出文件的路徑，如過沒有提供則將當前目錄用作輸出目錄。 FlatBuffers 編譯器按照為不同的數據結構聲明的namespace生成目錄結構。對于上面的例子，會生成如下的這些文件：
$ find out
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}span.s1 {font-variant-ligatures: no-common-ligatures}
$ find out/
out/
out//com
out//com/example
out//com/example/tutorial
out//com/example/tutorial/_Person
out//com/example/tutorial/_Person/PhoneNumber.java
out//com/example/tutorial/AddressBook.java
out//com/example/tutorial/Person.java
out//com/example/tutorial/PhoneType.java</code></pre>
在Android項目中使用FlatBuffers
我們將前面由 .fbs 文件生成的Java文件拷貝到我們的項目中。我們前面提到的， FlatBuffers 的Java庫比較薄，當前并沒有發不到jcenter這樣的maven倉庫中，因而我們需要將這部分代碼也拷貝到我們的額項目中。 FlatBuffers 的Java庫在其repo倉庫的 java 目錄下。引入這些文件之后，我們的代碼結構如下：
添加訪問 FlatBuffers 的類：
package com.netease.volleydemo;
import com.example.tutorial.AddressBook;
import com.example.tutorial.Person;
import com.example.tutorial._Person.PhoneNumber;
import com.google.flatbuffers.FlatBufferBuilder;
import java.nio.ByteBuffer;
/**
- Created by hanpfei0306 on 16-12-5.
*/
public class AddressBookFlatBuffers {
public static ByteBuffer encodeTest(String[] names) {
FlatBufferBuilder builder = new FlatBufferBuilder(0);
int[] personOffsets = new int[names.length];
for (int i = 0; i < names.length; ++ i) {
int name = builder.createString(names[i]);
int email = builder.createString("zhangsan@gmail.com");
int number1 = builder.createString("0157-23443276");
int type1 = 1;
int phoneNumber1 = PhoneNumber.createPhoneNumber(builder, number1, type1);
int number2 = builder.createString("136183667387");
int type2 = 0;
int phoneNumber2 = PhoneNumber.createPhoneNumber(builder, number2, type2);
int[] phoneNubers = new int[2];
phoneNubers[0] = phoneNumber1;
phoneNubers[1] = phoneNumber2;
int phoneNumbersPos = Person.createPhoneVector(builder, phoneNubers);
int person = Person.createPerson(builder, name, 13958235, email, phoneNumbersPos);
personOffsets[i] = person;
}
int persons = AddressBook.createPersonVector(builder, personOffsets);
AddressBook.startAddressBook(builder);
AddressBook.addPerson(builder, persons);
int eab = AddressBook.endAddressBook(builder);
builder.finish(eab);
ByteBuffer buf = builder.dataBuffer();
return buf;
}
public static ByteBuffer encodeTest(String[] names, int times) {
for (int i = 0; i < times - 1; ++ i) {
encodeTest(names);
}
return encodeTest(names);
}
public static AddressBook decodeTest(ByteBuffer byteBuffer) {
AddressBook addressBook = null;
addressBook = AddressBook.getRootAsAddressBook(byteBuffer);
return addressBook;
}
public static AddressBook decodeTest(ByteBuffer byteBuffer, int times) {
AddressBook addressBook = null;
for (int i = 0; i < times; ++ i) {
addressBook = decodeTest(byteBuffer);
}
return addressBook;
}
}</code></pre>
使用 flatbuf-gradle-plugin
我們有開發一個 FlatBuffers 的gradle插件，以方便開發 。這個插件的設計有參考Google的protobuf-gradle-plugin，功能與用法也與protobuf-gradle-plugin類似。在這個項目中，我們也有為 FlatBuffers 的Java庫創建一個module。
編譯并發布protobuf-gradle-plugin
從github上下載代碼：
$ git clone https://github.com/hanpfei/flatbuffers.git
然后將代碼導入Android Studio，將看到如下的代碼結構：
app 模塊是一個demo程序，flatbuf-gradle-plugin 模塊是 FlatBuffers 的gradle插件，而flatbuffers模塊則是 FlatBuffers 的Java庫。
為了使用 flatbuf-gradle-plugin，可以將插件發布到本地文件系統。這可以通過修改flatbuf-gradle-plugin/build.gradle來完成，修改 uploadArchives task 的 repository 指向本地文件系統，如：
uploadArchives {
repositories {
mavenDeployer {
pom.groupId = 'com.netease.hearttouch'
pom.artifactId = 'ht-flatbuf-gradle-plugin'
pom.version = '0.0.1-SNAPSHOT'
repository(url: 'file:///Users/netease/Projects/CorpProjects/ht-flatbuffers/app/plugin')
}
}
}
執行uploadArchives task，編譯并發布flatbuf-gradle-plugin到本地文件系統。
應用flatbuf-gradle-plugin
修改應用程序的 build.gradle 以應用 flatbuf-gradle-plugin 。
- 為buildscript添加對 flatbuf-gradle-plugin 的依賴：
buildscript {
//目前先發布在本地，后面會通過maven進行引用
repositories {
maven {
url "file:///Users/netease/Projects/CorpProjects/ht-flatbuffers/app/plugin"
}
jcenter()
mavenCentral()
}
dependencies {
classpath 'com.netease.hearttouch:ht-flatbuf-gradle-plugin:0.0.1-SNAPSHOT'
}
}
- 在 apply plugin: 'com.android.application' 后面應用flatbuf的plugin：
apply plugin: 'com.android.application'
apply plugin: 'com.netease.flatbuf'
-
添加flatbuf塊，對flatbuf-gradle-plugin的執行做配置：
flatbuf {
flatc {
path = '/usr/local/bin/flatc'
}
generateFlatTasks {
all().each { task ->
task.builtins {
remove java
}
task.builtins {
java { }
}
}
}
}</code></pre>
flatc 塊用于配置 FlatBuffers 編譯器，這里我們指定用我們之前手動編譯的編譯器。
task.builtins 的塊必不可少，這個塊用于指定我們要為那些編程語言生成代碼，這里我們為Java生成代碼。
</li>
- 指定 .fbs 文件的路徑
sourceSets {
main {
flat {
srcDir 'src/main/flat'
}
}
}
我們將 FlatBuffers 的IDL文件放在src/main/flat目錄下。
</ol>
這樣我們就不用再那么麻煩每次手動執行protoc了。
FlatBuffers、Protobuf及JSON對比測試
FlatBuffers相對于Protobuf的表現又如何呢？這里我們用數據說話，對比一下FlatBuffers格式、JSON格式與Protobuf的表現。測試同樣用fastjson作為JSON的編碼解碼工具。
測試用的數據結構所有的數據結構，Protobuf相關的測試代碼，及JSON的測試代碼同 在Android中使用Protocol Buffers 一文所述，FlatBuffers的測試代碼如下：
package hearttouch.netease.com.myapplication;
import com.example.tutorial.AddressBook;
import com.example.tutorial.Person;
import com.example.tutorial._Person.PhoneNumber;
import com.google.flatbuffers.FlatBufferBuilder;
import java.nio.ByteBuffer;
/**
- Created by hanpfei0306 on 16-12-5.
*/
public class AddressBookFlatBuffers {
public static ByteBuffer encodeTest(String[] names) {
FlatBufferBuilder builder = new FlatBufferBuilder(0);
int[] personOffsets = new int[names.length];
for (int i = 0; i < names.length; ++ i) {
int name = builder.createString(names[i]);
int email = builder.createString("zhangsan@gmail.com");
int number1 = builder.createString("0157-23443276");
int type1 = 1;
int phoneNumber1 = PhoneNumber.createPhoneNumber(builder, number1, type1);
int number2 = builder.createString("136183667387");
int type2 = 0;
int phoneNumber2 = PhoneNumber.createPhoneNumber(builder, number2, type2);
int[] phoneNubers = new int[2];
phoneNubers[0] = phoneNumber1;
phoneNubers[1] = phoneNumber2;
int phoneNumbersPos = Person.createPhoneVector(builder, phoneNubers);
int person = Person.createPerson(builder, name, 13958235, email, phoneNumbersPos);
personOffsets[i] = person;
}
int persons = AddressBook.createPersonVector(builder, personOffsets);
AddressBook.startAddressBook(builder);
AddressBook.addPerson(builder, persons);
int eab = AddressBook.endAddressBook(builder);
builder.finish(eab);
ByteBuffer buf = builder.dataBuffer();
return buf;
}
public static ByteBuffer encodeTest(String[] names, int times) {
for (int i = 0; i < times - 1; ++ i) {
encodeTest(names);
}
return encodeTest(names);
}
public static AddressBook decodeTest(ByteBuffer byteBuffer) {
AddressBook addressBook = null;
addressBook = AddressBook.getRootAsAddressBook(byteBuffer);
return addressBook;
}
public static AddressBook decodeTest(ByteBuffer byteBuffer, int times) {
AddressBook addressBook = null;
for (int i = 0; i < times; ++ i) {
addressBook = decodeTest(byteBuffer);
}
return addressBook;
}
}</code></pre>
通過如下的這段代碼來執行測試：
private class ProtoTestTask extends AsyncTask<Void, Void, Void> {
private static final int BUFFER_LEN = 8192;
private void compress(InputStream is, OutputStream os)
throws Exception {
GZIPOutputStream gos = new GZIPOutputStream(os);
int count;
byte data[] = new byte[BUFFER_LEN];
while ((count = is.read(data, 0, BUFFER_LEN)) != -1) {
gos.write(data, 0, count);
}
gos.finish();
gos.close();
}
private int getCompressedDataLength(byte[] data) {
ByteArrayInputStream bais =new ByteArrayInputStream(data);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
compress(bais, baos);
} catch (Exception e) {
}
return baos.toByteArray().length;
}
private void dumpDataLengthInfo(byte[] protobufData, String jsonData, ByteBuffer flatbufData) {
int compressedProtobufLength = getCompressedDataLength(protobufData);
int compressedJSONLength = getCompressedDataLength(jsonData.getBytes());
int compressedFlatbufLength = getCompressedDataLength(flatbufData.array());
Log.i(TAG, String.format("%-120s", "Data length"));
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s%-20s%-20s", "Protobuf", "Protobuf (GZIP)",
"JSON", "JSON (GZIP)", "Flatbuf", "Flatbuf (GZIP)"));
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s%-20s%-20s",
String.valueOf(protobufData.length), compressedProtobufLength,
String.valueOf(jsonData.getBytes().length), compressedJSONLength,
String.valueOf(flatbufData.array().length), compressedFlatbufLength));
}
private void doEncodeTest(String[] names, int times) {
long startTime = System.nanoTime();
byte[] protobufData = AddressBookProtobuf.encodeTest(names, times);
long protobufTime = System.nanoTime();
protobufTime = protobufTime - startTime;
startTime = System.nanoTime();
String jsonData = AddressBookJson.encodeTest(names, times);
long jsonTime = System.nanoTime();
jsonTime = jsonTime - startTime;
startTime = System.nanoTime();
ByteBuffer flatbufData = AddressBookFlatBuffers.encodeTest(names, times);
long flatbufTime = System.nanoTime();
flatbufTime = flatbufTime - startTime;
dumpDataLengthInfo(protobufData, jsonData, flatbufData);
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s", "Encode Times", String.valueOf(times),
"Names Length", String.valueOf(names.length)));
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s%-20s%-20s",
"ProtobufTime", String.valueOf(protobufTime),
"JsonTime", String.valueOf(jsonTime),
"FlatbufTime", String.valueOf(flatbufTime)));
}
private void doEncodeTest10(int times) {
doEncodeTest(TestUtils.sTestNames10, times);
}
private void doEncodeTest50(int times) {
doEncodeTest(TestUtils.sTestNames50, times);
}
private void doEncodeTest100(int times) {
doEncodeTest(TestUtils.sTestNames100, times);
}
private void doEncodeTest(int times) {
doEncodeTest10(times);
doEncodeTest50(times);
doEncodeTest100(times);
}
private void doDecodeTest(String[] names, int times) {
byte[] protobufBytes = AddressBookProtobuf.encodeTest(names);
ByteArrayInputStream bais = new ByteArrayInputStream(protobufBytes);
long startTime = System.nanoTime();
AddressBookProtobuf.decodeTest(bais, times);
long protobufTime = System.nanoTime();
protobufTime = protobufTime - startTime;
String jsonStr = AddressBookJson.encodeTest(names);
startTime = System.nanoTime();
AddressBookJson.decodeTest(jsonStr, times);
long jsonTime = System.nanoTime();
jsonTime = jsonTime - startTime;
ByteBuffer flatbufData = AddressBookFlatBuffers.encodeTest(names);
startTime = System.nanoTime();
AddressBookFlatBuffers.decodeTest(flatbufData, times);
long flatbufTime = System.nanoTime();
flatbufTime = flatbufTime - startTime;
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s", "Decode Times", String.valueOf(times),
"Names Length", String.valueOf(names.length)));
Log.i(TAG, String.format("%-20s%-20s%-20s%-20s%-20s%-20s",
"ProtobufTime", String.valueOf(protobufTime),
"JsonTime", String.valueOf(jsonTime),
"FlatbufTime", String.valueOf(flatbufTime)));
}
private void doDecodeTest10(int times) {
doDecodeTest(TestUtils.sTestNames10, times);
}
private void doDecodeTest50(int times) {
doDecodeTest(TestUtils.sTestNames50, times);
}
private void doDecodeTest100(int times) {
doDecodeTest(TestUtils.sTestNames100, times);
}
private void doDecodeTest(int times) {
doDecodeTest10(times);
doDecodeTest50(times);
doDecodeTest100(times);
}
@Override
protected Void doInBackground(Void... params) {
TestUtils.initTest();
doEncodeTest(5000);
doDecodeTest(5000);
return null;
}
@Override
protected void onPostExecute(Void aVoid) {
super.onPostExecute(aVoid);
}
}</code></pre>
這里我們執行3組編碼測試及3組解碼測試。對于編碼測試，第一組的單個數據中包含10個Person，第二組的包含50個，第三組的包含100個，然后對每個數據分別執行5000次的編碼操作。
對于解碼測試，三組中單個數據同樣包含10個Person、50個及100個，然后對每個數據分別執行5000次的解碼碼操作。
在Galaxy Nexus的Android 4.4.4 CM平臺上執行上述測試，最終得到如下結果：
編碼后數據長度對比 (Bytes)
Person個數
Protobuf
Protobuf（GZIP）
JSON
JSON（GZIP）
Flatbuf
Flatbuf（GZIP)
10
860
290
1703
343
2048
521
50
4300
978
8463
1043
8192
1822
100
8600
1825
16913
1902
16384
3417
相同的數據，經過編碼，在壓縮前FlatBuffers的數據長度和JSON的數據長度接近，而Protobuf的數據長度則只有前兩者的大概一半。而在用GZIP壓縮后，Protobuf的數據長度與JSON的接近，而FlatBuffers的數據長度則接近兩者的兩倍。
編碼性能對比 (S)
Person個數
Protobuf
JSON
FlatBuffers
10
6.000
8.952
12.464
50
26.847
45.782
56.752
100
50.602
73.688
108.426
編碼性能Protobuf相對于JSON有較大幅度的提高，而FlatBuffers則有較大幅度的降低。
解碼性能對比 (S)
Person個數
Protobuf
JSON
FlatBuffers
10
0.255
10.766
0.014
50
0.245
51.134
0.014
100
0.323
101.070
0.006
解碼性能方面，Protobuf相對于JSON，有著驚人的提升。Protobuf的解碼時間幾乎不隨著數據長度的增長而有太大的增長，而JSON則隨著數據長度的增加，解碼所需要的時間也越來越長。而FlatBuffers則由于無需解碼，在性能方面相對于前兩者更有著非常大的提升。
來自：http://www.jianshu.com/p/3bac6bc80db7