• <menu id="gyiem"><menu id="gyiem"></menu></menu>
  • <menu id="gyiem"><code id="gyiem"></code></menu>

    Kafka設計解析(五)- Kafka性能測試方法及Benchmark報告

    原創文章,轉載請務必將下面這段話置于文章開頭處。(已授權InfoQ中文站發布
    本文轉發自技術世界原文鏈接 http://www.luozeyang.com/2015/12/31/KafkaColumn5_kafka_benchmark

    摘要

      本文主要介紹了如何利用Kafka自帶的性能測試腳本及Kafka Manager測試Kafka的性能,以及如何使用Kafka Manager監控Kafka的工作狀態,最后給出了Kafka的性能測試報告。

    性能測試及集群監控工具

      Kafka提供了非常多有用的工具,如Kafka設計解析(三)- Kafka High Availability (下)中提到的運維類工具——Partition Reassign Tool,Preferred Replica Leader Election Tool,Replica Verification Tool,State Change Log Merge Tool。本文將介紹Kafka提供的性能測試工具,Metrics報告工具及Yahoo開源的Kafka Manager。

    Kafka性能測試腳本

    • $KAFKA_HOME/bin/kafka-producer-perf-test.sh 該腳本被設計用于測試Kafka Producer的性能,主要輸出4項指標,總共發送消息量(以MB為單位),每秒發送消息量(MB/second),發送消息總數,每秒發送消息數(records/second)。除了將測試結果輸出到標準輸出外,該腳本還提供CSV Reporter,即將結果以CSV文件的形式存儲,便于在其它分析工具中使用該測試結果
    • $KAFKA_HOME/bin/kafka-consumer-perf-test.sh 該腳本用于測試Kafka Consumer的性能,測試指標與Producer性能測試腳本一樣

    Kafka Metrics

      Kafka使用Yammer Metrics來報告服務端和客戶端的Metric信息。Yammer Metrics 3.1.0提供6種形式的Metrics收集——Meters,Gauges,Counters,Histograms,Timers,Health Checks。與此同時,Yammer Metrics將Metric的收集與報告(或者說發布)分離,可以根據需要自由組合。目前它支持的Reporter有Console Reporter,JMX Reporter,HTTP Reporter,CSV Reporter,SLF4J Reporter,Ganglia Reporter,Graphite Reporter。因此,Kafka也支持通過以上幾種Reporter輸出其Metrics信息。

    使用JConsole查看單服務器Metrics

      使用JConsole通過JMX,是在不安裝其它工具(既然已經安裝了Kafka,就肯定安裝了Java,而JConsole是Java自帶的工具)的情況下查看Kafka服務器Metrics的最簡單最方便的方法之一。
      首先必須通過為環境變量JMX_PORT設置有效值來啟用Kafka的JMX Reporter。如export JMX_PORT=19797。然后即可使用JConsole通過上面設置的端口來訪問某一臺Kafka服務器來查看其Metrics信息,如下圖所示。

    JConsole Kafka JMX
      使用JConsole的一個好處是不用安裝額外的工具,缺點很明顯,數據展示不夠直觀,數據組織形式不友好,更重要的是不能同時監控整個集群的Metrics。在上圖中,在kafka.cluster->Partition->UnderReplicated->topic4下,只有2和5兩個節點,這并非因為topic4只有這兩個Partition的數據是處于復制狀態的。事實上,topic4在該Broker上只有這2個Partition,其它Partition在其它Broker上,所以通過該服務器的JMX Reporter只看到了這兩個Partition。

    通過Kafka Manager查看整個集群的Metrics

      Kafka Manager是Yahoo開源的Kafka管理工具。它支持如下功能

    • 管理多個集群
    • 方便查看集群狀態
    • 執行preferred replica election
    • 批量為多個Topic生成并執行Partition分配方案
    • 創建Topic
    • 刪除Topic(只支持0.8.2及以上版本,同時要求在Broker中將delete.topic.enable設置為true)
    • 為已有Topic添加Partition
    • 更新Topic配置
    • 在Broker JMX Reporter開啟的前提下,輪詢Broker級別和Topic級別的Metrics
    • 監控Consumer Group及其消費狀態
    • 支持添加和查看LogKafka

      安裝好Kafka Manager后,添加Cluster非常方便,只需指明該Cluster所使用的Zookeeper列表并指明Kafka版本即可,如下圖所示。

    Add Cluster

      這里要注意,此處添加Cluster是指添加一個已有的Kafka集群進入監控列表,而非通過Kafka Manager部署一個新的Kafka Cluster,這一點與Cloudera Manager不同。

    Kafka Benchmark

      Kafka的一個核心特性是高吞吐率,因此本文的測試重點是Kafka的吞吐率。
      本文的測試共使用6臺安裝Red Hat 6.6的虛擬機,3臺作為Broker,另外3臺作為Producer或者Consumer。每臺虛擬機配置如下

    • CPU:8 vCPU, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz,2 Sockets,4 Cores per socket,1 Thread per core
    • 內存:16 GB
    • 磁盤:500 GB

      開啟Kafka JMX Reporter并使用19797端口,利用Kafka-Manager的JMX polling功能監控性能測試過程中的吞吐率。

      本文主要測試如下四種場景,測試的指標主要是每秒多少兆字節數據,每秒多少條消息。

    Producer Only

      這組測試不使用任何Consumer,只啟動Broker和Producer。

    Producer Number VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:分別測試1,2,3個Producer時的吞吐量
      測試目標:如Kafka設計解析(一)- Kafka背景及架構介紹所介紹,多個Producer可同時向同一個Topic發送數據,在Broker負載飽和前,理論上Producer數量越多,集群每秒收到的消息量越大,并且呈線性增漲。本實驗主要驗證該特性。同時作為性能測試,本實驗還將監控測試過程中單個Broker的CPU和內存使用情況
      測試結果:使用不同個數Producer時的總吞吐率如下圖所示
    Producer Number VS. Throughput

      由上圖可看出,單個Producer每秒可成功發送約128萬條Payload為100字節的消息,并且隨著Producer個數的提升,每秒總共發送的消息量線性提升,符合之前的分析。

      性能測試過程中,Broker的CPU和內存使用情況如下圖所示。
    Broker CPU Usage

      由上圖可知,在每秒接收約117萬條消息(3個Producer總共每秒發送350萬條消息,平均每個Broker每秒接收約117萬條)的情況下,一個Broker的CPU使用量約為248%,內存使用量為601 MB。

    Message Size VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,3個Producer
      測試項目:分別測試消息長度為10,20,40,60,80,100,150,200,400,800,1000,2000,5000,10000字節時的集群總吞吐量
      測試結果:不同消息長度時的集群總吞吐率如下圖所示
    Message Size VS. Throughput

      由上圖可知,消息越長,每秒所能發送的消息數越少,而每秒所能發送的消息的量(MB)越大。另外,每條消息除了Payload外,還包含其它Metadata,所以每秒所發送的消息量比每秒發送的消息數乘以100字節大,而Payload越大,這些Metadata占比越小,同時發送時的批量發送的消息體積越大,越容易得到更高的每秒消息量(MB/s)。其它測試中使用的Payload為100字節,之所以使用這種短消息(相對短)只是為了測試相對比較差的情況下的Kafka吞吐率。

    Partition Number VS. Throughput

      實驗條件:3個Broker,1個Topic,無Replication,異步模式,3個Producer,消息Payload為100字節
      測試項目:分別測試1到9個Partition時的吞吐量
      測試結果:不同Partition數量時的集群總吞吐率如下圖所示
    Partition Number VS. Throughput

      由上圖可知,當Partition數量小于Broker個數(3個)時,Partition數量越大,吞吐率越高,且呈線性提升。本文所有實驗中,只啟動3個Broker,而一個Partition只能存在于1個Broker上(不考慮Replication。即使有Replication,也只有其Leader接受讀寫請求),故當某個Topic只包含1個Partition時,實際只有1個Broker在為該Topic工作。如之前文章所講,Kafka會將所有Partition均勻分布到所有Broker上,所以當只有2個Partition時,會有2個Broker為該Topic服務。3個Partition時同理會有3個Broker為該Topic服務。換言之,Partition數量小于等于3個時,越多的Partition代表越多的Broker為該Topic服務。如前幾篇文章所述,不同Broker上的數據并行插入,這就解釋了當Partition數量小于等于3個時,吞吐率隨Partition數量的增加線性提升。
      當Partition數量多于Broker個數時,總吞吐量并未有所提升,甚至還有所下降。可能的原因是,當Partition數量為4和5時,不同Broker上的Partition數量不同,而Producer會將數據均勻發送到各Partition上,這就造成各Broker的負載不同,不能最大化集群吞吐量。而上圖中當Partition數量為Broker數量整數倍時吞吐量明顯比其它情況高,也證實了這一點。

    Replica Number VS. Throughput

      實驗條件:3個Broker,1個Topic,6個Partition,異步模式,3個Producer,消息Payload為100字節
      測試項目:分別測試1到3個Replica時的吞吐率
      測試結果:如下圖所示
    Replica Number VS. Throughput

      由上圖可知,隨著Replica數量的增加,吞吐率隨之下降。但吞吐率的下降并非線性下降,因為多個Follower的數據復制是并行進行的,而非串行進行。

      

    Consumer Only

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:分別測試1到3個Consumer時的集群總吞吐率
      測試結果:在集群中已有大量消息的情況下,使用1到3個Consumer時的集群總吞吐量如下圖所示

    Consumer Number VS. Throughput

      由上圖可知,單個Consumer每秒可消費306萬條消息,該數量遠大于單個Producer每秒可消費的消息數量,這保證了在合理的配置下,消息可被及時處理。并且隨著Consumer數量的增加,集群總吞吐量線性增加。
      根據Kafka設計解析(四)- Kafka Consumer設計解析所述,多Consumer消費消息時以Partition為分配單位,當只有1個Consumer時,該Consumer需要同時從6個Partition拉取消息,該Consumer所在機器的I/O成為整個消費過程的瓶頸,而當Consumer個數增加至2個至3個時,多個Consumer同時從集群拉取消息,充分利用了集群的吞吐率。

    Producer Consumer pair

      實驗條件:3個Broker,1個Topic,6個Partition,無Replication,異步模式,消息Payload為100字節
      測試項目:測試1個Producer和1個Consumer同時工作時Consumer所能消費到的消息量
      測試結果:1,215,613 records/second

    Kafka系列文章

    郭俊 Jason wechat
    歡迎關注作者微信公眾號【大數據架構】
    您的贊賞將支持作者繼續原創分享
    速赢彩app www.79500a.com | www.ya638.com | www.677256.com | www.6868xj.com | 750.com | www.83033e.com | www.2418f.com | www.1118kj.cc | 55nn8332.com | 45637r.com | www.60123u.com | www.790964.com | www.5966uuu.com | www.xjs8880.com | x33q.vip | 518cp-6.com | www.czg2.com | www.050881.com | www.67799.com | rr1331.com | r08199.com | 82093.com | www.02czj.com | www.xj7006.com | www.3844d.com | ccc5002.com | 66887web.com | 168cp-h.com | www.c9662.com | www.flb677.com | www.377606.com | 8989770.com | 3405vvv.com | 365225a.com | www.585808.com | www.hg5013.com | www.c44uu.com | www.pjbet777.com | 3ttlcp.com | 3222y.cc | 1669b.com | www.599260.com | www.337756.com | www.89677m.com | www.a4116.com | yabo4000.com | 7m365.com | 28288pp.com | 9y88.vip | www.324770.com | www.26163j.com | www.vnsr368.com | www.z3285.com | www.39695t.com | 4028722.com | 61328899.com | 15a43.net | www.327988.com | www.089wy.com | www.1764m.com | www.99113u.com | www.hg70999.com | 4444661.com | f86811.com | 838388h.com | wnsr858.com | www.590791.com | www.26299q.com | www.3434aaa.com | www.4078n.com | www.767666.com | www.58777t.com | 1213ppp.com | yz6388.com | g82365.com | 1705c.com | bwin980.co | www.xinyc3.com | www.982082.com | www.26163c.com | www.x999789.com | www.35252g.com | www.85857u.com | www.suncity553.com | www.hg9812.com | 20772244.com | 88948999.com | 2381hh.com | yhanhui.vip | 3846xxx.com | pj520.com | 6647v.com | ly8004.com | 2373w.com | 23233u.com | www.131763.com | www.424221.com | www.809387.com | www.csgc6.com | www.89911.cc | www.8667u.com | www.80767b.com | www.aa15.cc | www.hg7022.com | www.l3410.com | www.b30666.vip | www.50999a.com | www.3846j.com | www.d6969.com | www.kj8858.com | www.101698.com | www.44118s.com | www.38138g.com | www.686102a.com | www.50999n.com | www.3846j.com | www.370190.com | www.rb208.com | www.hg8376.com | www.52564.com | www.kj8858.com | www.04080.com | www.873999.com | www.7099666.com | www.23336.com | www.56748.com | www.8838c.com | www.d88118.com | www.678130.com | www.922646.com | www.27038.com | www.73518.com | www.sg8886.com | www.07.bet | www.79500v.com | www.cp505b.com | www.rgcp22.com | www.163a44.com | www.bet91487.com | www.2997770.com | www.07679z.com | www.xpj55123.com | www.926530.com | www.550021m.com | www.66653h.com | www.43818p.com | www.55558c.com | www.39391.cc | www.961205.com | www.633813.com | www.305995.com | 1592n.com | 22296yy.com | 1389o.com | 10050533.com | 4189222.com | kk38648.com | 7935u.com | www.86611m.com | www.21365nn.com | www.4052i.com | www.212323.com | www.5506488.com | www.64566f.com | www.96877.com | www.15355v.com | www.88ckb.com | www.620477.com | www.35918u.com | 3242a.com | 55899o.com | 6487aaa.com | hg22211.com | www.hg990022.com | www.hg8069.com | www.hv588.com | www.3066xx.com | www.ejylc16.com | www.5095q.com | www.606930.com | z03.com | 12742n.com | 3775622.com | 8827ii.com | www.xpj8t8.com | 36088.com | 00774aa.com | 131o.net | wx11b.com | www.56733e.com | www.xpj2778.com | www.2y932.com | www.8080999i.com | www.60123s.com | www.745557.com | 15856e.com | 14977.com | 333790a.com | 2373l.com | www.0005156.com | www.27363i.com | www.h36588.com | www.22csj.com | www.91233v.com | 4123rr.com | mg4377.com | www.12007.com | www.5994.com | www.39937.com | www.1035h.com | www.26299u.com | 99589.net | www.43468.com | www.4449.cc | www.0062737.com | www.618msc.com | www.60886a.com | www.3933s.cc | 16690022.com | 2324hhh.com | www.514234.com | www.777888yh.com | www.8722.am | www.987580.com | cc76669.com | 3559pp.com | www.3aobo.com | www.508345.com | www.618146.com | www.681048.com | ag88.com | 8036r.com | www.k2229k.com | www.2021x.com | www.bxcp3333.com | 5443mm.com | 7508c.com | www.5454i.com | www.9895g.com | www.lczg7.com | aipin888.vip | lp06.com | www.567878.com | www.184088.com | www.822138.com | 3473p.com | 37570d.com | www.zr55555.com | www.9989581.com | www.332907.com | wfcp.com | www.js507710.com | www.666jyh.com | www.66ffi.com | q61788.com | feicai0571.com | www.550js.com | www.scfyou.com | www.137105.com | 9506c.com | www.39695d.com | www.2000e.cc | www.394277.com | 5360ll.com | www.benz4422s.com | www.80188z.com | www.799343.com | 4809y.com | www.65533b.com | www.808888n.com | www.6832y.com | 58222aa.com | 2613r.com | www.twcp05.com | www.085wy.com | 5478y.com | www.81866z.com | www.6776hh.com | www.777337a.com | 6002r.com | www.hg2968.com | www.32031q.com | www.151376.com | vipamdc.com | www.399777.com | www.77803j.com | 5309u.com | hg07111.com | www.58118h.com | www.911283.com | b2624.com | www.55268nn.com | www.1764r.com | 0033.cc | 37570h.com | www.cp0014.com | www.979133.com | 2234.so | www.22447712.com | www.20016.cc | nn500y.com | www.567sbet.com | www.68993258.com | 55984y.com | www.pj5568.com | www.30350t.com | pu65.com | 1407v.com | www.6687z.com | www.84499u.com | 883399v.com | www.pj88h.com | www.224344.com | 617055.com | www.f92776.com | www.452815.com | 01885u.com | www.9299.cc | www.500232.com | 00ee8331.com | www.686102a.com | www.106396.com | 3225n.com | www.4058cc.com | www.60123v.com | ttt2848.com | www.469700.com | ra9993.com | uu38648.com | www.69567o.com | 9895i.com | www.88188b.cc | www.zzyl64.com | 66876k.com | www.365815j.com | www.60123m.com | 83138x.com | www.174886.com | www.716866.com | yh123456.com | www.9895y.com | 54770.com | www.799007.com | www.sdsy.com | 402d2.com | www.js253.com | www.609813.com | 2864t.com | www.88325o.com | hg77712.com | www.am0674.com | www.53900n.com | 8037dd.com | www.11557.com | www.021067.com | www.1111jsc.com | www.6939j.com | n8381.com | www.352233.com | www.370583.com | xx888i.com | www.hg09a.com | hc096.net | www.11c668.com | sjgc6.com | www.365775.com | www.5095n.com | jing26.com | www.1754v.com | 7779z.cc | www.72tk.com | www.092603.com | www.m33678.com | www.16878r.com | 80850s.com | www.y6086.com | 2306i.com | www.47343.com | www.188975.com | www.682855.com | www.12455l.com | mm33.vip | www.hbet98.com | 33318m.com | www.79500x.com | www.60yyhh.com | www.104665.com | www.50064b.com | 9659a.com | www.60886t.com | 3726s.com | www.4212h.com | 3121ww.com | www.992254.com | www.028314.com | www.56520j.com | www.910883.com | 2267v.com | www.50024m.com | x7570xx.com | www.340.com | 22098z.com | www.1813bb.com | 56988mm.com | www.jj55826.com | 8577e.cc | www.k7817.vip | xxx8827.com | www.047222.com | 142788.com | www.4788msc.com | www.41518b.com | www.mng44.com | www.196148.com | www.suncity88.net | www.581860.com | www.77vn777.cc | www.599091.com | www.223456s.com | www.805357.com | www.94911.cc | www.560905.com | www.4625p.com | www.558421.com | www.67797r.com | www.393830.com | www.4853c.com | www.196103.com | www.828685.com | qq365t.com | www.kj8888.com | 500000512.com | www.99418.com | 0885.com | www.7225b.com | 80878w.com | www.33987.com | 58532222.com | www.8039b.com | www.556zr.com | www.739915.com | www.i22365.com | www.81233v.com | www.ibm888.net | 7240k.com | www.80969y.com | 5443w.com | www.1434r.com | jj7742.com | www.87498.com | www.04222e.com | www.553028.com | www.22244066.com | 2805u.com | www.00618d.com | www.5854t.cc | jsh500.com | www.qj35.com | www.yy733.com | www.068617.com | www.zs5544.com | 077ll.com | www.0622qqq.com | 7686v.com | www.66fft.com | www.s30226.com | jxf964.com | www.sbd025.net | 3559iiii.com | www.36166t.com | www.799003.com | www.43131s.com | www.1168h.com | 22299cc.com | www.866365.Com | www.56733r.com | www.hcw887.com | www.38238y.com | 0683g.com | www.89894x.com | www.7435j.com | 1408t.com | www.ff55826.com | 3552s.com | www.2350r.com | www.66166s.com | f83377.com | www.8905u.com | spj09.com | www.372082.com | www.cfcp555.com | nn3336.com | www.8667q.com | www.4tgpd.com | 2690js.com | www.1764m.com | 3018yy.com | www.89894o.com | www.340dh.com | 2757i.com | www.811597.com | www.986msc.com | 777hg111.com | www.ya055.com | www.82508b.com | 4022xx.com | www.89338k.com | www.21365ll.com | 67877q.com | www.980442.com | www.7720v.com | 11qq8332.com | www.68993229.com | www.www-hg127.com | 068.cc | www.vn888123.cc | www.hg582.com | 4195e.com | www.flb277.com | www.pj9087.com | t8159.cc | www.bet379.com | www.333133n.com | 55991277.com | www.26878g.com | www.xpj386.com | c53.cc | www.995684.com | www.3696699.com | 3663yh.com | www.231002.com | www.7366005.com | 50128a.com | 168cp-h.com | www.68993253.com | www.vns55545.com | 8722vvvv.com | www.896950.com | www.120529.com | 537a.cc | 9895i.com | www.96756.com | www.14086622.com | 84435072.com | www.55676.com | www.hg001111.com | 76543e.com | www.bai355.com | www.2y938.com | 131r.net | 11989f.com | www.0270m.com | www.81776.com | www.50064j.com | www.2078a.com | www.7830z.com | 8381uu.com | www.3775a.com | www.77117076.com | ll3336.com | www.371935.com | www.73990t.com | www.ydb44.com | 55977.cc | www.374277.com | www.7036ll.com | www.7893w13.com | 6118m.com | www.426060.com | www.33303.com | www.87msc.com | amhg099.com | www.202909.com | www.3775n.com | www.663929.com | 5856ww.com | v997a.com | www.61233r.com | www.esb125.com | sss8827.com | 234902.com | www.899465.com | www.wap.j3app.com | www.hg8662.com | 6487eee.com | www.128368.com | www.yh920955.com | www.113470.com | 2306k.com | 7508f.com | www.216777aa.com | www.i1432.com | www.10899e.com | 21366l.com | www.109883.com | www.y94600.com | www.df5550.com | bj299.com | dlzzx.com | www.hj2111.com | www.306654.com | www.67229t.com | 822.cc | 5231007.com | www.hqcp6.com | www.7366009.com | www.2078r.com | bs78933.com | 88051.net | www.856458.com | www.0088hgcc.com | www.6y7y.net | 500599.com | 37688u.com | www.53911a.com | www.50064w.com | www.5446gg.com | www.3311sb.com | 22tt8331.com | rr4255.com | www.348577.com | www.998855aa.com | www.04567o.com | www.20909l.com | 8449ll99.com | bjzhuowen.com | www.359500.com | www.28462000.com | www.6667ylg.com | www.330099x.com | feicai0518.com | viv.qipai78.net | www.86267a.com | www.5522h.cc | www.ag3366.com | www.666549.com | www.aicaiaa.com | 29918f.com | 77885002.com | www.338071.com | www.33112g.com | www.115527g.com | www.ylg883.com | 5855tt.com | 3967v.com | 4060.com | 80850p.com | www.895465.com | www.t639.com | www.80188h.com | www.22997.com | www.6491l.com | 88851a.com | yl6944.com | 3016vvv.com | www.612212.com | www.50064y.com | www.ba506.com | www.77731g.com | www.wnsr999.com | 52688h.com | 2140011.com | p1.cc | 365225a.com | www.218251.com | www.qucw4.com | www.6482v.com | www.45598u.com | www.hg8270.com | www.h7788u.com | 7935d.com | 00774kk.com | 2767f.com | 1479.cc | 9566362.com | www.689477.com | www.50069.com | www.x88789.com | www.7225f.com | www.94099.com | www.688222.com | www.066119.com | 66876q.com | 540855.com | www.505170.com | www.38345i.com | www.37766.cc | www.619757.com | www.65533b.com | www.73gcw.com | www.0967002.com | www.ee00668.com | www.dzj.cm | www.38333.com | www.vnsr22888.com | www.365815z.com | www.ylg292.com | 7989d.com | 20160044.com | u1915u.com | yypxrht.cn | vv5443.com | vv2649.com | 5756-5856.com | www.354177.com | www.846631.com | www.022sx.cc | www.08588y.com | www.bet73g.com | www.hg860033.com | www.99553.com | www.781617.com | www.4058a.com | www.58665w.com | www.pj8333.com | www.jm40.com | www.666suncity.com | www.hg0462.com | www.2846i.com | www.192022.com | www.vns994.cc | www.hg8004.com | 2618h.com | 8185v.com | 33115g.com | 2019xx.cc | x8877.com | 046967.com | 61325599.com | heji271.com | ii38648.com | 55797h.com | e99345.am | a99345.am | 55797i.com | 2096f.com | 588ylw.com | 2jsqqq.com | 44332007.com | 37570i.com | 8827qq.com | 06064688.com | www.bmcp1344.com | www.80a88.com | www.766848.com | www.3459j.com | www.900652.com | www.yvnsr.com | www.1600yy.com | www.tm012.com | www.896suncity.com | www.fh990.com | www.1064a.com | www.4058ii.com | www.7768js.com | www.56011g.com | www.1239999.cc | www.50021i.com | www.9818t.cc | www.csgc6.com | www.725660.com | www.349677.com | ly88888883.com | bct6688.com | dh074.com | yg7777.com | 66671t.com | 2127app.com | zhcp60.com | 9995144.com | www.yh57788.com | www.656022.com | www.5555566666.com | www.77731x.com | www.bwin980.co | www.68365v.com | www.97714.com | www.8667a.com | www.910977.com | www.327477.com | 3136vv.com | 08778f.com | 797720.com | 3522w.vip | 69990v.com | www.333133u.com | www.84786.com | www.28000r.com | www.w84b.com | www.0270n.com | www.fh6116.com | www.292269.com | 8742gg.com | 4288hh.com | 2757a.com | 500000349.com | www.3459e.com | www.304999.com | www.bet91487.com | www.826018.com | www.27775.cc | www.504133.com | 131ggg.net | 2767dcom | 5309f.com | www.sjs07.com | www.hg4748.com | www.l5503.com | www.78680s.com | www.26163k.com | www.507633.com | x45638.com | 33382bb.com | ss3189.com | www.sb9909.com | www.899111.com | www.j3410.com | www.50026f.com | www.458689.com | hg0505.com | 3559yyyy.com | hgw168k.com | www.jing6663.com | www.07bet8.com | www.7158.net | www.811672.com | 66876s.com | 9646k.com | 23800a.com | www.255251.com | www.899741.com | www.lb5555.com | www.976516.com | hg2015.cc | 35ii.vip | 1144011.com | www.30370000.com | www.10999e.com | www.61233g.com | www.193161.com | 4022rr.com | 0025i.com | www.tyc600.com | www.8850w3.com | www.16878u.com | www.286967.com | 00765.com | 0033u.cc | www.89880033.com | www.377666g.com | www.9356c.com | x886.com | 4109.com | www.hg9952.com | www.597567f.com | www.67797a.com | www.55526i.com | www.954321v.com | www.520691.com | 8547.com | 131ccc.net | www.yz889.com | www.109115.com | www.ck9494.com | 66667375.com | 159666t.com | www.33302a6.com | www.627464.com | www.950633.com | 818555.vip | 922630.com | www.h7733.com | www.blr4444.com | www.599487.com | dd63777.com | 47749u.com | www.hg2286.com | www.7239r.com | 4379i.com | 4022y.com | www.21365nn.com | www.4996dl.com | www.igcp1.com | 159666f.com | 3122a.com | www.wwww.6230z.com | www.13434f.com | www.038909.com | 5856sss.com | www.7830q.com | www.7225q.com | www.937016.com | 4508dd.com | 970816.com | www.668cp22.com | www.5095p.com | 3467s.cc | 308827.com | www.56655z.com | www.32123j.com | 2848.com | 3178x.com | www.1799msc.com | www.50732y.com | hy9996.com | 2190k.com | www.1389ff.com | www.25688c.com | 3405jj.com | 97799y.com | www.971055.com | www.069wy.com | 33558448.com | xpj66666.vip | www.55070r.com | www.912451.com | 1669s.com | www.39695z.com | www.xpj66123.com | www.001562.com | 3522a.vip | www.20976.com | www.87668g.com | dfmcsoft.com | 5295uu.com | www.8473b.com | www.701269.com | 3662l.com | www.dhygw46.com | www.js7591.com | ylzz6.cc | hggjtg6.com | www.6880ll.com | www.460689.com | 44009193.com | www.ca338.com | www.9155d.com | 8006009.com | www.995f.net | www.ykylc08.com | 33567i.com | 6220kk.com | www.1851137.com | www.452806.com | sha165.com | www.tyc800.com | www.hx6695.com | bh55234.com | www.hg58.pw | www.75538d.com | 89777i.com | www.cr1116.com | www.4833041.com | 1429t.com | sun7766.com | www.6687y.com | 70118k.com | 3136bb.com | www.7225g.com | www.52303i.com | 9949j.com | www.bet63e.com | www.022190.com | 2506g.com | www.6880rr.com | 59964tt.com | ybcai.com | www.178633.com | 6261hh.com | c388k.com | www.yl8553hb.com | 908080f.com | www.664848.com | www.37377d.com | 37111c.com | www.vns8489.com | www.55885c.com | 85850.com | www.736suncity.com | www.hczx6.com | 3559cp.net | www.5966eee.com | www.547577.com | 2613x.com | www.js69111.com | 3178ss.com | www.85770w.com | www.?junzi8888.com | 33uu8331.com | www.803303.com | www.757209.com | 8f06.com | www.13k0048.com | 4182g.com | www.3657k.com | www.61233h.com | 7240mcom | www.d9478.com | www.312477.com | 1077qqq.com | www.0088hgbb.com | 35503.com | www.071097.com | www.915655.com | 159666w.com | www.327969.com | c53965.com | www.hg6288.com | www.cf9905.com | x33i.vip | 98955p.com | www.5197365.com | www.65707x.com | 54140011.com | www.0590777.com | bbb7570.com | www.h7788z.com | www.2373g.com | 33rr8331.com | www.078121.top | 30170077.com | www.9997068.com | www.163cp1.cc | 54140077.com | www.07163e.com | 89777a.com | www.4694o.com | 5144s.com | www.78929.com | www.919709.com | 6220q.com | www.89777b.com | 2009666.com | www.19yh1.com | 7736d.com | www.88570.com | www.560923.com | 80892yy.com | www.71233j.com | 4465o.com | www.6386c.com | 3559ii.com | www.20199zz.com | alpk66.com | www.777cm1.com | www.601497.com | www.86611c.com | www.78700.com | 24670044.com | www.402997.com | 3121g.com | www.7415ee.com | 83086e.com | www.314455.com | c31ge25j.com | www.15960000.com | www.206991.com | www.4625p.com | www.909888.com | hga008.al | www.81gcw.com | 4023h.com | www.0270p.com | 38054913.com | www.38358.com | 04666p.com | www.38775mm.com | 0234z.com | www.055260.com | 6766ww.com | www.00772t.com | hg999333u.com | www.rgcp.vip | 9679a.com | www.cp55569.com | 2019p.cc | www.bet66622.com | 79889h.com | www.ggp.99233p.com | 55984e.com |