使用 Twisted Python 和 Treq 進行 HTTP 壓力測試
從事API相關的工作很有挑戰性，在高峰期保持系統的穩定及健壯性就是其中之一，這也是我們在Mailgun做很多壓力測試的原因。
這么久以來，我們已經嘗試了很多種方法，從簡單的ApacheBench到復雜些的自定義測試套。但是本貼講述的，是一種使用python進行“快速粗糙”卻非常靈活的壓力測試的方法。
使用python寫HTTP客戶端的時候，我們都很喜歡用
Requests library 。這也是我們向我們的API用戶們推薦的。Requests 很強大，但有一個缺點，它是一個模塊化的每線程一個調用的東西，很難或者說不可能用它來快速的產生成千上萬級別的請求。
Treq on Twisted簡介
為解決這個問題我們引入了Treq ( Github庫 )。Treq是一個HTTP客戶端庫，受Requests影響，但是它運行在Twisted上，具有Twisted典型的強大能力：處理網絡I/O時它是異步且高度并發的方式。
Treq并不僅僅限于壓力測試：它是寫高并發HTTP客戶端的好工具，比如網頁抓取。Treq很優雅、易于使用且強大。這是一個例子：
>>> from treq import get
>>> def done(response):
... print response.code
... reactor.stop()
>>> get("http://www.github.com").addCallback(done)
>>> from twisted.internet import reactor
>>> reactor.run()
200>>> fromtreqimportget
>>> defdone(response):
... printresponse.code
... reactor.stop()
>>> get("http://www.github.com").addCallback(done)
>>> fromtwisted.internetimportreactor
>>> reactor.run()
200簡單的測試腳本
如下是一個使用Treq的簡單腳本，用最大可能量的請求來對單一URL進行轟炸。
#!/usr/bin/env python
from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor, task
from twisted.web.client import HTTPConnectionPool
import treq
import random
from datetime import datetime
req_generated = 0
req_made = 0
req_done = 0
cooperator = task.Cooperator()
pool = HTTPConnectionPool(reactor)
def counter():
'''This function gets called once a second and prints the progress at one
second intervals.
'''
print("Requests: {} generated; {} made; {} done".format(
req_generated, req_made, req_done))
# reset the counters and reschedule ourselves
req_generated = req_made = req_done = 0
reactor.callLater(1, counter)
def body_received(body):
global req_done
req_done += 1
def request_done(response):
global req_made
deferred = treq.json_content(response)
req_made += 1
deferred.addCallback(body_received)
deferred.addErrback(lambda x: None) # ignore errors
return deferred
def request():
deferred = treq.post('http://api.host/v2/loadtest/messages',
auth=('api', 'api-key'),
data={'from': 'Loadtest <test@example.com>',
'to': 'to@example.org',
'subject': "test"},
pool=pool)
deferred.addCallback(request_done)
return deferred
def requests_generator():
global req_generated
while True:
deferred = request()
req_generated += 1
# do not yield deferred here so cooperator won't pause until
# response is received
yield None
if __name__ == '__main__':
# make cooperator work on spawning requests
cooperator.cooperate(requests_generator())
# run the counter that will be reporting sending speed once a second
reactor.callLater(1, counter)
# run the reactor
reactor.run()#!/usr/bin/env python
fromtwisted.internetimportepollreactor
epollreactor.install()
fromtwisted.internetimportreactor, task
fromtwisted.web.clientimportHTTPConnectionPool
importtreq
importrandom
fromdatetimeimportdatetime
req_generated = 0
req_made = 0
req_done = 0
cooperator = task.Cooperator()
pool = HTTPConnectionPool(reactor)
defcounter():
'''This function gets called once a second and prints the progress at one
second intervals.
'''
print("Requests: {} generated; {} made; {} done".format(
req_generated, req_made, req_done))
# reset the counters and reschedule ourselves
req_generated = req_made = req_done = 0
reactor.callLater(1, counter)
defbody_received(body):
global req_done
req_done += 1
defrequest_done(response):
global req_made
deferred = treq.json_content(response)
req_made += 1
deferred.addCallback(body_received)
deferred.addErrback(lambda x: None) # ignore errors
return deferred
defrequest():
deferred = treq.post('http://api.host/v2/loadtest/messages',
auth=('api', 'api-key'),
data={'from': 'Loadtest <test@example.com>',
'to': 'to@example.org',
'subject': "test"},
pool=pool)
deferred.addCallback(request_done)
return deferred
defrequests_generator():
global req_generated
while True:
deferred = request()
req_generated += 1
# do not yield deferred here so cooperator won't pause until
# response is received
yieldNone
if __name__ == '__main__':
# make cooperator work on spawning requests
cooperator.cooperate(requests_generator())
# run the counter that will be reporting sending speed once a second
reactor.callLater(1, counter)
# run the reactor
reactor.run()輸出結果：
2013-04-25 09:30 Requests: 327 generated; 153 sent; 153 received 2013-04-25 09:30 Requests: 306 generated; 156 sent; 156 received 2013-04-25 09:30 Requests: 318 generated; 184 sent; 154 received
2013-04-25 09:30 Requests: 327 generated; 153 sent; 153 received 2013-04-25 09:30 Requests: 306 generated; 156 sent; 156 received 2013-04-25 09:30 Requests: 318 generated; 184 sent; 154 received
“Generated”類的數字代表被Twisted反應器準備好但是還沒有發送的請求。這個腳本為了簡潔性忽略了所有錯誤處理。為它添加超時狀態的信息就留給讀者作為一個練習。
這個腳本可以當做是一個起始點，你可以通過拓展改進它來自定義特定應用下的處理邏輯。建議你在改進的時候用 collections.Counter 來替代丑陋的全局變量。這個腳本運行在單線程上，想通過一臺機器壓榨出最大量的請求的話，你可以用類似 mulitprocessing 的技術手段。
愿你樂在壓力測試！
本文由用戶 jopen 自行上傳分享，僅供網友學習交流。所有權歸原作者，若您的權利被侵害，請聯系管理員。
轉載本站原創文章，請注明出處，并保留原始鏈接、圖片水印。
本站是一個以用戶分享為主的開源技術平臺，歡迎各類分享！