• <menu id="gyiem"><menu id="gyiem"></menu></menu>
  • <menu id="gyiem"><code id="gyiem"></code></menu>

    Kafka設計解析(五)- Kafka性能測試方法及Benchmark報告

    原創文章,轉載請務必將下面這段話置于文章開頭處。(已授權InfoQ中文站發布
    本文轉發自技術世界原文鏈接 http://www.luozeyang.com/2015/12/31/KafkaColumn5_kafka_benchmark

    摘要

      本文主要介紹了如何利用Kafka自帶的性能測試腳本及Kafka Manager測試Kafka的性能,以及如何使用Kafka Manager監控Kafka的工作狀態,最后給出了Kafka的性能測試報告。

    性能測試及集群監控工具

      Kafka提供了非常多有用的工具,如Kafka設計解析(三)- Kafka High Availability (下)中提到的運維類工具——Partition Reassign Tool,Preferred Replica Leader Election Tool,Replica Verification Tool,State Change Log Merge Tool。本文將介紹Kafka提供的性能測試工具,Metrics報告工具及Yahoo開源的Kafka Manager。

    Kafka性能測試腳本

    • $KAFKA_HOME/bin/kafka-producer-perf-test.sh 該腳本被設計用于測試Kafka Producer的性能,主要輸出4項指標,總共發送消息量(以MB為單位),每秒發送消息量(MB/second),發送消息總數,每秒發送消息數(records/second)。除了將測試結果輸出到標準輸出外,該腳本還提供CSV Reporter,即將結果以CSV文件的形式存儲,便于在其它分析工具中使用該測試結果
    • $KAFKA_HOME/bin/kafka-consumer-perf-test.sh 該腳本用于測試Kafka Consumer的性能,測試指標與Producer性能測試腳本一樣

    Kafka Metrics

      Kafka使用Yammer Metrics來報告服務端和客戶端的Metric信息。Yammer Metrics 3.1.0提供6種形式的Metrics收集——Meters,Gauges,Counters,Histograms,Timers,Health Checks。與此同時,Yammer Metrics將Metric的收集與報告(或者說發布)分離,可以根據需要自由組合。目前它支持的Reporter有Console Reporter,JMX Reporter,HTTP Reporter,CSV Reporter,SLF4J Reporter,Ganglia Reporter,Graphite Reporter。因此,Kafka也支持通過以上幾種Reporter輸出其Metrics信息。

    使用JConsole查看單服務器Metrics

      使用JConsole通過JMX,是在不安裝其它工具(既然已經安裝了Kafka,就肯定安裝了Java,而JConsole是Java自帶的工具)的情況下查看Kafka服務器Metrics的最簡單最方便的方法之一。
      首先必須通過為環境變量JMX_PORT設置有效值來啟用Kafka的JMX Reporter。如export JMX_PORT=19797。然后即可使用JConsole通過上面設置的端口來訪問某一臺Kafka服務器來查看其Metrics信息,如下圖所示。

    JConsole Kafka JMX
      使用JConsole的一個好處是不用安裝額外的工具,缺點很明顯,數據展示不夠直觀,數據組織形式不友好,更重要的是不能同時監控整個集群的Metrics。在上圖中,在kafka.cluster->Partition->UnderReplicated->topic4下,只有2和5兩個節點,這并非因為topic4只有這兩個Partition的數據是處于復制狀態的。事實上,topic4在該Broker上只有這2個Partition,其它Partition在其它Broker上,所以通過該服務器的JMX Reporter只看到了這兩個Partition。

    通過Kafka Manager查看整個集群的Metrics

      Kafka Manager是Yahoo開源的Kafka管理工具。它支持如下功能

    • 管理多個集群
    • 方便查看集群狀態
    • 執行preferred replica election
    • 批量為多個Topic生成并執行Partition分配方案
    • 創建Topic
    • 刪除Topic(只支持0.8.2及以上版本,同時要求在Broker中將delete.topic.enable設置為true)
    • 為已有Topic添加Partition
    • 更新Topic配置
    • 在Broker JMX Reporter開啟的前提下,輪詢Broker級別和Topic級別的Metrics
    • 監控Consumer Group及其消費狀態
    • 支持添加和查看LogKafka

      安裝好Kafka Manager后,添加Cluster非常方便,只需指明該Cluster所使用的Zookeeper列表并指明Kafka版本即可,如下圖所示。

    Add Cluster

      這里要注意,此處添加Cluster是指添加一個已有的Kafka集群進入監控列表,而非通過Kafka Manager部署一個新的Kafka Cluster,這一點與Cloudera Manager不同。

    Kafka Benchmark

      Kafka的一個核心特性是高吞吐率,因此本文的測試重點是Kafka的吞吐率。
      本文的測試共使用6臺安裝Red Hat 6.6的虛擬機,3臺作為Broker,另外3臺作為Producer或者Consumer。每臺虛擬機配置如下

    • CPU:8 vCPU, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz,2 Sockets,4 Cores per socket,1 Thread per core
    • 內存:16 GB
    • 磁盤:500 GB

      開啟Kafka JMX Reporter并使用19797端口,利用Kafka-Manager的JMX polling功能監控性能測試過程中的吞吐率。

      本文主要測試如下四種場景,測試的指標主要是每秒多少兆字節數據,每秒多少條消息。

    Producer Only

      這組測試不使用任何Consumer,只啟動Broker和Producer。

    Producer Number VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:分別測試1,2,3個Producer時的吞吐量
      測試目標:如Kafka設計解析(一)- Kafka背景及架構介紹所介紹,多個Producer可同時向同一個Topic發送數據,在Broker負載飽和前,理論上Producer數量越多,集群每秒收到的消息量越大,并且呈線性增漲。本實驗主要驗證該特性。同時作為性能測試,本實驗還將監控測試過程中單個Broker的CPU和內存使用情況
      測試結果:使用不同個數Producer時的總吞吐率如下圖所示
    Producer Number VS. Throughput

      由上圖可看出,單個Producer每秒可成功發送約128萬條Payload為100字節的消息,并且隨著Producer個數的提升,每秒總共發送的消息量線性提升,符合之前的分析。

      性能測試過程中,Broker的CPU和內存使用情況如下圖所示。
    Broker CPU Usage

      由上圖可知,在每秒接收約117萬條消息(3個Producer總共每秒發送350萬條消息,平均每個Broker每秒接收約117萬條)的情況下,一個Broker的CPU使用量約為248%,內存使用量為601 MB。

    Message Size VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,3個Producer
      測試項目:分別測試消息長度為10,20,40,60,80,100,150,200,400,800,1000,2000,5000,10000字節時的集群總吞吐量
      測試結果:不同消息長度時的集群總吞吐率如下圖所示
    Message Size VS. Throughput

      由上圖可知,消息越長,每秒所能發送的消息數越少,而每秒所能發送的消息的量(MB)越大。另外,每條消息除了Payload外,還包含其它Metadata,所以每秒所發送的消息量比每秒發送的消息數乘以100字節大,而Payload越大,這些Metadata占比越小,同時發送時的批量發送的消息體積越大,越容易得到更高的每秒消息量(MB/s)。其它測試中使用的Payload為100字節,之所以使用這種短消息(相對短)只是為了測試相對比較差的情況下的Kafka吞吐率。

    Partition Number VS. Throughput

      實驗條件:3個Broker,1個Topic,無Replication,異步模式,3個Producer,消息Payload為100字節
      測試項目:分別測試1到9個Partition時的吞吐量
      測試結果:不同Partition數量時的集群總吞吐率如下圖所示
    Partition Number VS. Throughput

      由上圖可知,當Partition數量小于Broker個數(3個)時,Partition數量越大,吞吐率越高,且呈線性提升。本文所有實驗中,只啟動3個Broker,而一個Partition只能存在于1個Broker上(不考慮Replication。即使有Replication,也只有其Leader接受讀寫請求),故當某個Topic只包含1個Partition時,實際只有1個Broker在為該Topic工作。如之前文章所講,Kafka會將所有Partition均勻分布到所有Broker上,所以當只有2個Partition時,會有2個Broker為該Topic服務。3個Partition時同理會有3個Broker為該Topic服務。換言之,Partition數量小于等于3個時,越多的Partition代表越多的Broker為該Topic服務。如前幾篇文章所述,不同Broker上的數據并行插入,這就解釋了當Partition數量小于等于3個時,吞吐率隨Partition數量的增加線性提升。
      當Partition數量多于Broker個數時,總吞吐量并未有所提升,甚至還有所下降。可能的原因是,當Partition數量為4和5時,不同Broker上的Partition數量不同,而Producer會將數據均勻發送到各Partition上,這就造成各Broker的負載不同,不能最大化集群吞吐量。而上圖中當Partition數量為Broker數量整數倍時吞吐量明顯比其它情況高,也證實了這一點。

    Replica Number VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,異步模式,3個Producer,消息Payload為100字節
      測試項目:分別測試1到3個Replica時的吞吐率
      測試結果:如下圖所示
    Replica Number VS. Throughput

      由上圖可知,隨著Replica數量的增加,吞吐率隨之下降。但吞吐率的下降并非線性下降,因為多個Follower的數據復制是并行進行的,而非串行進行。

      

    Consumer Only

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:分別測試1到3個Consumer時的集群總吞吐率
      測試結果:在集群中已有大量消息的情況下,使用1到3個Consumer時的集群總吞吐量如下圖所示

    Consumer Number VS. Throughput

      由上圖可知,單個Consumer每秒可消費306萬條消息,該數量遠大于單個Producer每秒可消費的消息數量,這保證了在合理的配置下,消息可被及時處理。并且隨著Consumer數量的增加,集群總吞吐量線性增加。
      根據Kafka設計解析(四)- Kafka Consumer設計解析所述,多Consumer消費消息時以Partition為分配單位,當只有1個Consumer時,該Consumer需要同時從6個Partition拉取消息,該Consumer所在機器的I/O成為整個消費過程的瓶頸,而當Consumer個數增加至2個至3個時,多個Consumer同時從集群拉取消息,充分利用了集群的吞吐率。

    Producer Consumer pair

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:測試1個Producer和1個Consumer同時工作時Consumer所能消費到的消息量
      測試結果:1,215,613 records/second

    Kafka系列文章

    郭俊 Jason wechat
    歡迎關注作者微信公眾號【大數據架構】
    您的贊賞將支持作者繼續原創分享
    速赢彩app www.77802s.com | www.997263.com | s4212.com | www.pj99111.com | www.hc779.com | 7003k.com | www.5446e.com | mmm5682.com | www.79095l.com | 07322222.com | www.00797a.com | 77442007.com | www.62500.cc | www.55268yy.com | www.630565.com | www.333133p.com | c00351.com | www.hy985.com | 预览68963555.com | www.1111284.com | 3616p.com | www.bccp877.com | 4270qq.com | www.3066ii.com | 0638.com | www.3890v.com | 4433mmmm.com | www.71233o.com | www.ylg20000.com | 444000tt.com | www.089728.com | zz811.com | 1517726.com | www.234677.com | 6363dd.com | www.998855e.com | 皇冠娱乐场.com | www.1754b.com | 3656xx.com | www.427786.com | 69990e.com | www.629293.com | www.005004.com | 18438h.com | www.blr2244.com | bet36500600.com | www.15czj.com | www.6020200.com | 3822f.com | www.54968.com | 83378e.com | www.66073.cc | www.778849.com | 57bet.cc | www.66ffl.com | www.69111k.com | 18438b.com | www.06820w.com | 2998j.com | www.818897.com | www.9966145.com | 178120.com | www.109115.com | 3121ee.com | www.743552.com | www.23111yte.com | 78112233.com | www.7714f.com | www.zr609.com | sha145.com | www.8473c.com | 9679z.com | www.810609.com | www.c31d.vip | bali3377.com | www.493600.com | www.lpj2222.com | 1813dd.com | www.970358.com | www.709msc.com | u82365.com | www.369062.com | www.83568t.com | 2455m.com | 80850p.com | www.6880vv.com | 4433mmmm.com | 444144.com | www.00840z.com | www.797077.com | 67890www.com | www.78700e.com | www.44118.net | 2767b.com | 9649z.com | www.4331c.com | www.6491w.com | 8003xl.com | www.50064g.com | www.v6060.com | 6150m.com | www.373553.com | www.q2894.com | 2618h.com | 1389w.com | www.876879.com | www.77731z.com | 30006hd.com | mobile.baidu.com | www.60108d.com | www.399555.com | 3169e.com | 44077x.com | www.95999w.com | www.24xjw.com | 99332007.com | 129880.net | www.pj55713.com | www.1389m.com | 92266J.com | www.553398.com | www.b35qq.com | dh4546.com | 22225002.com | www.898364.com | www.351532.com | www.hg8786.tv | b11988.com | www.956568.com | www.cr677.net | 3616x.com | 11989z.com | www.771645.com | www.780sun.com | www.6372008.com | 500000991.com | a82365.com | www.794918.com | www.js57666.com | www.7t68.com | 22556b.com | 8036zz.com | www.99clf.com | www.o69096.com | www.36530088.com | 3676vip.com | www.43131r.com | www.yh6688.bet | www.7045d.com | 33ww8331.com | 4195cc.com | www.8667j.com | www.33707.com | www.2222pj.com | 97987-3.com | www.131873.com | www.6678699.com | www.875808.com | www.tbb001.com | 474.com | www.055520.cc | www.2418n.com | www.c7.cc | www.2846o.com | 500000516.com | 83378o.com | 77005002.com | www.54400i.com | www.255ak.com | www.a83377.com | www.bst62.com | www.5554dhy.com | 6261ii.com | 28835533.com | www.170569.com | www.5522h.cc | www.065915.com | www.sscb99.cc | www.655666a.com | 23800z.com | x4255.com | 3678q.com | www.771746.com | www.52072z.com | www.ba502.com | www.ggu.99233u.com | www.10050813.com | www.hg226.com | 08159i.com | 2595o.com | 1331yy.com | www.51515v.com | www.599380.com | www.3978h.com | www.12136f.com | www.38001.com | www.22266638.com | 8036mm.com | 647768.com | aaa8996.com | 80850ff.com | www.244343.com | www.64059.com | www.6678699.com | www.d063801.com | www.38345b.com | www.66632.com | www.xpj130.com | 500000516.com | swtyxxx.com | 43330.com | 4995r.com | 4647f.com | hg999333t.com | www.572899.com | www.922069.com | www.5091z.com | www.97655g.com | www.89777b.com | www.7920x.com | www.678js222.com | www.678js444.com | www.88325h.com | www.00773l.com | www.85857z.com | www.729666.com | www.ylzz6662.com | www.653505.com | www.bobifa.net | www.1869h.com | 2613j.com | 11ll8332.com | 3301885.com | t58955.com | 7200199.com | tyc99998.com | 6766ee.com | 00uu8331.com | ff555z.com | 500000351.com | a77304.com | uuu5682.com | 11rr8332.com | 83377y.com | 2355n.com | 22556w.com | 66876z.com | 99663885.com | www.ag355.com | www.bet2828365.com | www.87680d.com | www.0636d.com | www.b3657.com | www.56520n.com | www.4625g.com | www.suncity88.net | www.vns9983.com | www.7830i.com | www.hg66997.cc | www.qqqqkkkk.com | www.20209.com | www.238288.com | www.56655l.com | www.dzj0004.com | www.19019e.com | www.9737rr.me | www.45598n.com | www.1764z.com | www.yh70123.com | www.33997g.com | www.26299c.com | www.hm2999.com | www.65yb.com | www.496689.com | www.66376o.com | 5589r.com | 27365c.com | 61789m.com | w69441155.com | 500000518.com | 5295l.com | www.655666a.com | www.706sunbet.com | www.159kj.com | www.yh8878a.com | www.40288l.com | www.pujing95.com | www.1368n.cc | www.ck5578.com | www.805357.com | www.80065o.com | ff555i.com | 18438s.com | www.622230.com | ra9993.com | w2649.com | 34n35.com | 7806.com | aa2267.com | www.xpjvip07.com | www.2302321.com | www.jy879.com | www.js89c.vip | www.60123c.com | www.485689.com | 6095hb.com | 888992255.com | 60114.net | www.yinhecc77.com | www.ks1385.com | www.280.cc | www.50024j.com | www.378016.com | ll5144.com | blr8855.com | 3169f.com | www.ty7000.com | www.671451.com | www.165888.com | www.890886.com | amjs9944.com | 27878vv.com | www.546005.com | www.5956fff.com | www.x999789.com | www.826927.com | 3122x.com | 22gg8332.com | www.r27229.com | www.6606488.com | www.525011.com | www.328229.com | 876878t.com | 6261n.com | www.8814e7.com | www.10999j.com | www.7782d.com | hkmh33.com | 86611z.com | www.yh6099.com | www.6776dd.com | www.66376b.com | www.543977.com | 2934g.com | nn500o.com | www.9374l.com | www.d5669.com | www.50074j.com | 3991100.com | 6594gg.com | www.hg58.pw | www.js89v.vip | www.617069.com | 3788q.com | 76678.com | www.mhcp8.com | www.89777d.com | www.343918.com | cp088t.com | 7811mm.com | www.r3065.com | www.072wy.com | 2381nn.com | 8827fff.com | www.ym7171.com | www.9989575.com | www.011225.com | 0805q.com | www.7658800.com | www.111msc.com | www.41518m.com | 00773n.com | zhcp15.com | www.1168c.com | www.799666aa.com | 00048l.com | 1079.cc | www.glc555.com | www.22fzc.com | yy4255.com | www.pj9087.com | www.06387777.com | www.332052.com | 67890jj.com | www.741199b.com | www.hyi682.com | 7874.tt | bst979.com | www.788jyh.com | www.959zf.com | 9002y.com | www.900y.com | www.2418p.com | 6betbet365.com | feicai020.com | www.28000r.com | www.680955.com | 6261zz.com | www.wn869.com | www.6608598.com | www.338052.com | 35442999.com | www.7111e.com | www.yi636.com | 30175500.com | www.v30226.com | www.6939h.com | 3405e.com | www.hg0377.com | www.90935.com | 73999n.com | www.ra6060.com | www.s27229.com | www.48080c.com | www.33qxc.com | 30171166.com | www.pj5718.com | www.8618aa.com | 62222w.com | www.3814b.com | www.927320.com | 4165.com | www.808888j.com | www.374511.com | u86811.com | www.bc858.cc | www.549077.com | 83138y.com | www.h7764.com | 840pj.com | www.3aobo.com | www.5856877.com | qqtheme.com | www.58777t.com | www.49956t.com | 162970.com | www.05899.com | www.653750.com | 61328822.com | www.535083.com | 3066aa.com | www.537msc.com | www.flff5.com | 496ss.com | www.610567.com | www.620316.com | d35151.com | www.742988.com | 9649.com | www.28003.com | www.673519.com | 7945oo.com | www.586806.com | amjs9944.com | www.tqdc555.com | www.ch8234.com | 42422.com | www.78680n.com | ylzz4447.com | www.888365q.com | www.90305a.com | www.00048js.com | www.7920h.com | 90307o.com | www.246hao.com | www.648689.com | 04666u.com | www.99638n.com | rrr67890.com | www.64566e.com | bwinwww.com | www.pj2348.com | www.78949u.com | www.00755.com | www.flcb3.com | 1077gg.com | www.7375.com | 18438z.com | www.689797.com | 316c.cc | www.38361.com | 0088cpw.com | www.8867e.com | www.196081.com | 1705z.com | www.64040.com | gggg19927.com | www.2418z.com | s86226.com | www.2807777.com | 025939.com | www.3116j.com | feicai0873.com | www.c44qq.com | 1018xpj.com | www.32208d.com | 83929.com | www.899866.com | 22mm8331.com | www.9996pp.com | www.477575.com | www.5557888.com | www.602913.com | www.63877w.com | www.808399.com | www.vns67999.com | www.795535.com | www.pjyunnan.com | www.602450.com | www.bet365601.com | www.627297.com | www.323006.com | www.717885.com | www.ylg05.com | www.411515.com | www.8344567.com | www.099323.com | www.44477222.com | 3178rr.com | www.bet365zh.com | 838388a.com | www.bwinyz42.com | 33395757.com | www.1115.com | 00992007.com | www.3890w.com | www.66664400.com | www.627791.com | www.25288b.com | www.618477.com | www.58777l.com | b5923.com | www.bai3322.com | bet99331.com | www.509261.com | js7.ch | www.33588z.com | www.033033m.com | www.582062.com | www.8sands.com | ee2205.com | www.848777u.com | sjg003.com | www.1199.am | pj666677.com | www.493356.com | www.45778.com | youfa111.vip | www.535087.com | vip99550.com | www.hx1181.com | www.yl00444.com | wd6667.com | www.673888k.com | 4488kkk.com | www.35155h.com | www.8494r.com | qq365t.com | www.373351.com | 9646f.com | www.71233s.com | www.912399.com | 1483jj.com | www.4212c.com | 7555i.com | www.9478d.com | www.o168a.com | www.89894d.com | www.wd817.com | 896288.com | www.5086g.com | www.09569c.com | hg999333q.com | www.78680e.com | d3144.com | www.277474.com | www.938d.cc | 55967d.com | www.mcw6.com | www.700733.com | 1705g.com | www.50080m.com | www.9996ff.com | 1v939.com | www.37377i.com | www.hg5455.com | 3544u.com | www.eee2848.com | 2306.cm | 00pp8331.com | www.32031y.com | xpj677l.com | 53166o.com | www.38775hh.com | 0311v.com | 3379qq.com | www.99553.com | zg163.net | 44488r.com | www.hg1777.com | www.wugou135.com | 8890832.com | www.3a005.com | www.vns1209.com | y2523.com | www.7239p.com | www.330099w.com | rbbet22.com | www.4975.am | www.77537a.com | 222v22.com | www.8816g.com | www.652005.com | 4368.com | www.550414.com | www.9737cc.com | 2418p.com | 3189rr.com | www.80865.com | www.40818g.com | P35cc.com | www.008ac.com | www.597567b.com | rf033.com | www.88266i.com | www.pjc88.vip | www.60656.com | 915907854.com | www.7782f.com | www.56774.com | df-bet.net | 18775g.com | www.99552oo.com | www.pjbeijing.com | 5856k.com | www.hjin8.com | www.91500999.com | 31780000.com | hg999333y.com | www.32123g.com | www.12706.com | 063971.com | bet888777.com | www.81233x.com | www.35789.com | 8547u.com | 23231381.com | www.50052z.com | www.hg8907.com | 0747w.com | 4869822.com | www.5091u.com | www.808888p.com | 7334x.com | 77889193.com | www.859505.com | www.47506s.com | www.8694h.com | 3559t.com | www.65707i.com | www.94383.com | www.hgw0088.com | 112e.net | 3308b.com | www.2373b.com | www.5446vv.com | www.333222p.com | 80878h.com | www.61233e.com | www.yh7774.com | www.www-58944.com | blh13579.com | 62222c.com | www.424988.com | www.3775n.com | www.22076.com | p533.cc | 6601vip.us | www.506752.com | www.9996ee.com | www.333639.com | 060785.cn | 338050.com | www.833314.com | www.63800.com | www.55690.com | 00665004.com | 7893w34.com | 69111q.com | www.03882.cc | www.45598i.com | www.yh9333.com | 1315559.com | 569pj.com | www.584311.com | www.js69z.cc | www.jz30280.com | www.l27229.com | zz00558.com | 41866.com | www.175902.com | www.25688n.com | www.377666o.com | www.hg7945.com | lehu556.com | pj00pp.com | 2127mm.com | www.641577.com | www.jsbet008.com | www.623667.com | www.mmtx789.com | 308827.com | 550111a.com | 7240k.com | www.550919.com | www.86267y.com | www.9570115.com | www.hg7725.com | www.00amjs.com | x55.am | 55818g.com | 2709f.com | www.587833.com | www.50732x.com | www.490804.com | www.5049v.com | www.0747700.com | huc288.com | vns7711.com | 4995f.com | shsxiwl.cn | www.525059.com | www.5854z.cc | www.ykylc01.com | www.bwinyz05.com | www.vs88.com | www.55268ww.com | 38989a.com | 52599s.com | t1458.com | k1458.com | www.50064u.com | www.918795.com | www.66332y.com | www.9187s.com | www.4136b.com | www.ke00852.com | www.25288v.com | www.887.ag | 1077vv.com | 2490j.com | lll4165.com | ly88888885.com | 1654xx.com | 28288ii.com | www.226105.com | www.901068.com | www.6364p.com | www.181442.com | www.773570.com | www.91779m.com | www.9980.la | www.358vns.com | www.yh8426.com | www.551701.com | www.yh8006.com | 6830q.com | vip3144.co | 55855.com | xhtd12345.com | hg008.am | 341a.com | 3559w.com | 7720u.com | 1407r.com | aobo88888.com | 8520n.com | www.106295.com | www.507670.com | www.737198.com | www.998855a.com | www.008ac.com | www.8667x.com | www.53900n.com | www.b387387.com | www.954321z.com | www.9996yy.com | www.798345.com | www.blr6644.com | www.78680p.com | www.835993.com | www.12136i.com | www.jsdc9555.com | www.839282.com | www.wns66.me | www.52062w.com | www.52062n.com | www.36yl.com | www.33522s.cc | www.291693.com | www.46630.info | www.g83377.com | www.86730.com | www.56011j.com | www.h36555.com | www.99094u.com | www.h6650.com | www.954321a.com | www.3978u.com | www.60108t.com | www.8039s.com | www.58ipo.com | www.985579.com | www.761996.com | www.536210.com | www.223805.com | 9958922.com | 3196.com | 80567n.com | 23800.com | 87965ii.com | 82365u.com | zz38648.com | r2306.com | 0311v.com | www.pjshandong.com | www.4938e.com | www.836666e.com | www.118sjz.cc | www.c526688.com | www.1559504.com | www.68365j.com | www.aobo191.com | www.75850.cc | www.857981.com | www.226105.com | jsdas8.com | 33318s.com | 38749.com | 2142299.com | 2019q.cc | www.bet3650714.com | www.xhtd05.com | www.kj306.com | www.4996gx.com | www.zzyl63.com | www.36166b.com | www.876433.com | www.80075a.com | bet89226.com | 7380jj.com | q8q9.com | bm1394.com | www.2222k0.com | www.2492.cc | www.245885.com | www.36584444.com | www.4dwj.com | www.235671.com | bet88w.com | 2077.am | 8381c.com | www.bjd333.com | www.xhtd1985.com | www.175702.com | www.0088hgee.com | www.35918k.com | www.627792.com | jj2649.com | 70118g.com | f5858.com | www.0377365.com | www.810msc.com | www.hgdc700.com | www.7239q.com | www.763236.com | 1483u.com | 37775145.com | 4442546.com | www.l22365.com | www.1389nn.com | www.r999992.com | www.3933g.cc | www.55228a.com | mgm3242.co | 411yh.com | www.558809.com | www.9377msc.com | www.0033sun.com | www.2394.com | 15856i.com | 333pj8.cc | dazhuangjia.com | www.ljw036.com | www.4136c.com | www.83033c.com | www.464689.com | 80850xx.com | 112m.net | www.9068aa.com | www.99113w.com | www.879528.com | www.370833.com | beiyong.bet | lh66h.com | www.pj3390.com | www.77658zz.com | www.44607.com | 31325i.com | 97340055.com | www.gh0031.com | www.qazjw.com | www.5856858.com | www.560706.com | 1591ooo.com | s47479.com | www.js7669.com | www.60886z.com | www.c1396.com | 3467u.cc | jbb93.cc | www.7036ee.com | www.663558.com | www.hm8222.com | 500000530.com | 28288vv.com | www.10050856.com | www.880317.com | www.801896.com | 3659699.com | 403144.com | www.48282222.com | www.725641.com | 910088.com | 2685a.com | www.xpj6878.com | www.3032ii.com | www.810619.com | yinhe100.cc | 1479r.com | www.808888q.com | www.00840s.com | 908080f.com | 7m365.com | www.hg6069.com | www.hd8677.com | www.029079.com | 8344222.com | www.7111g.com | www.468860.com | www.447478.com | 19880v.com | www.yh5579.com | www.6900666.com | www.607315.com | bh55654.com | www.4972zz.com | www.883399b.com | www.277739.com | 4255ss.com | www.bb2229.com | www.79966.com | chengbet330.com | feicai0744.com | www.3388v.com | www.3552m.com | 11zz8331.com | www.a2a000.cc | www.17k0048.com | www.638802.com | wnsr8820.com | www.520195.com | www.221222.cc | abc777.com | i2894.com | www.cr677.com | www.820335.com | 867wn.com | www.427567.com | www.ky1001.com | 6594rr.com | www.3983112.com | www.lhg5555.com | www.202803.com | 28288q.com | www.4238.com | www.08xyc.com | 4182n.com | www.hai6666.com | www.7714n.com | 3616t.com | www.63877o.com | www.t63568.com | 228888q.com | 38989p.com | www.xj6004.com | www.39500u.com | 635230.com | www.55070l.com | www.331215.com | bopaihang.com | www.56766.com | www.609716.com | 22aa8332.com | www.138cpc.com | www.672566.com | yabo1188.com | www.f9478.com | www.679000.com | 33cc8331.com | www.9478.com | www.508277.com | ca9033.com | www.bet91486.com | www.161172.com | 4997e.com | www.377666o.com | 983888k.com | www.cx0007.com | www.548185.com | 2851t.com | www.990333.net | www.12455g.com | 88119o.com | www.853721.com | www.663566.com | yz222999.com | www.v69096.com | 44ee8331.com | www.4123020.com | www.28891c.com | tgf8876.vip | www.11004100.com | www.qucw9.com | 4995v.com | www.00773g.com | 32424b.com | www.a3a999.cc | www.16065b.com | 5437r.com | www.5882323.com | www.676065.com | feicai0744.com | www.5886rr.com | 8520.com | www.hg0084.com | www.975367.com | 80850u.com | www.w84d.com | uu4675.com | www.5139sss.com | www.973709.com | 8547ss.com | www.777xx.cc | 88850uu.com | www.tt55666.com | www.541811.com | 6220c.com | www.959595.la | 3304y.com | www.919301.com | 61789e.com | www.hg2968.com | www.1035r.com | a87570.com | www.69567q.com | 4052g.com | www.h1765.com | www.128760.com | www.hg1850.com | www.12455n.com | 0015zz.com | www.42070011.com | 2233blg.com | www.3cq3.com | 91019.net | www.9679j.com | www.656270.com | 20772200.com | www.52072t.com |