• <menu id="gyiem"><menu id="gyiem"></menu></menu>
  • <menu id="gyiem"><code id="gyiem"></code></menu>

    Spark SQL 性能優化再進一步 CBO 基于代價的優化

    原創文章,轉載請務必將下面這段話置于文章開頭處。
    本文轉發自技術世界原文鏈接 http://www.luozeyang.com/spark/cbo/

    本文所述內容均基于 2018年9月17日 Spark 最新 Release 2.3.1 版本。后續將持續更新

    Spark CBO 背景

    上文Spark SQL 內部原理中介紹的 Optimizer 屬于 RBO,實現簡單有效。它屬于 LogicalPlan 的優化,所有優化均基于 LogicalPlan 本身的特點,未考慮數據本身的特點,也未考慮算子本身的代價。

    本文將介紹 CBO,它充分考慮了數據本身的特點(如大小、分布)以及操作算子的特點(中間結果集的分布及大小)及代價,從而更好的選擇執行代價最小的物理執行計劃,即 SparkPlan。

    Spark CBO 原理

    CBO 原理是計算所有可能的物理計劃的代價,并挑選出代價最小的物理執行計劃。其核心在于評估一個給定的物理執行計劃的代價。

    物理執行計劃是一個樹狀結構,其代價等于每個執行節點的代價總合,如下圖所示。


    CBO 總代價

    而每個執行節點的代價,分為兩個部分

    • 該執行節點對數據集的影響,或者說該節點輸出數據集的大小與分布
    • 該執行節點操作算子的代價

    每個操作算子的代價相對固定,可用規則來描述。而執行節點輸出數據集的大小與分布,分為兩個部分:1) 初始數據集,也即原始表,其數據集的大小與分布可直接通過統計得到;2)中間節點輸出數據集的大小與分布可由其輸入數據集的信息與操作本身的特點推算。

    所以,最終主要需要解決兩個問題

    • 如何獲取原始數據集的統計信息
    • 如何根據輸入數據集估算特定算子的輸出數據集

    Statistics 收集

    通過如下 SQL 語句,可計算出整個表的記錄總數以及總大小

    1
    ANALYZE TABLE table_name COMPUTE STATISTICS;

    從如下示例中,Statistics 一行可見, customer 表數據總大小為 37026233 字節,即 35.3MB,總記錄數為 28萬,與事實相符。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    spark-sql> ANALYZE TABLE customer COMPUTE STATISTICS;
    Time taken: 12.888 seconds

    spark-sql> desc extended customer;
    c_customer_sk bigint NULL
    c_customer_id string NULL
    c_current_cdemo_sk bigint NULL
    c_current_hdemo_sk bigint NULL
    c_current_addr_sk bigint NULL
    c_first_shipto_date_sk bigint NULL
    c_first_sales_date_sk bigint NULL
    c_salutation string NULL
    c_first_name string NULL
    c_last_name string NULL
    c_preferred_cust_flag string NULL
    c_birth_day int NULL
    c_birth_month int NULL
    c_birth_year int NULL
    c_birth_country string NULL
    c_login string NULL
    c_email_address string NULL
    c_last_review_date string NULL

    # Detailed Table Information
    Database jason_tpc_ds
    Table customer
    Owner jason
    Created Time Sat Sep 15 14:00:40 CST 2018
    Last Access Thu Jan 01 08:00:00 CST 1970
    Created By Spark 2.3.2
    Type EXTERNAL
    Provider hive
    Table Properties [transient_lastDdlTime=1536997324]
    Statistics 37026233 bytes, 280000 rows
    Location hdfs://dw/tpc_ds/customer
    Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    InputFormat org.apache.hadoop.mapred.TextInputFormat
    OutputFormat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    Storage Properties [field.delim=|, serialization.format=|]
    Partition Provider Catalog
    Time taken: 1.691 seconds, Fetched 36 row(s)

    通過如下 SQL 語句,可計算出指定列的統計信息

    1
    ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS [column1] [,column2] [,column3] [,column4] ... [,columnn];

    從如下示例可見,customer 表的 c_customer_sk 列最小值為 1, 最大值為 280000,null 值個數為 0,不同值個數為 274368,平均列長度為 8,最大列長度為 8。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    spark-sql> ANALYZE TABLE customer COMPUTE STATISTICS FOR COLUMNS c_customer_sk, c_customer_id, c_current_cdemo_sk;
    Time taken: 9.139 seconds
    spark-sql> desc extended customer c_customer_sk;
    col_name c_customer_sk
    data_type bigint
    comment NULL
    min 1
    max 280000
    num_nulls 0
    distinct_count 274368
    avg_col_len 8
    max_col_len 8
    histogram NULL

    除上述示例中的統計信息外,Spark CBO 還直接等高直方圖。在上例中,histogram 為 NULL。其原因是,spark.sql.statistics.histogram.enabled 默認值為 false,也即 ANALYZE 時默認不計算及存儲 histogram。

    下例中,通過 SET spark.sql.statistics.histogram.enabled=true; 啟用 histogram 后,完整的統計信息如下。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    spark-sql> ANALYZE TABLE customer COMPUTE STATISTICS FOR COLUMNS c_customer_sk,c_customer_id,c_current_cdemo_sk,c_current_hdemo_sk,c_current_addr_sk,c_first_shipto_date_sk,c_first_sales_date_sk,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address,c_last_review_date;
    Time taken: 125.624 seconds

    spark-sql> desc extended customer c_customer_sk;
    col_name c_customer_sk
    data_type bigint
    comment NULL
    min 1
    max 280000
    num_nulls 0
    distinct_count 274368
    avg_col_len 8
    max_col_len 8
    histogram height: 1102.3622047244094, num_of_bins: 254
    bin_0 lower_bound: 1.0, upper_bound: 1090.0, distinct_count: 1089
    bin_1 lower_bound: 1090.0, upper_bound: 2206.0, distinct_count: 1161
    bin_2 lower_bound: 2206.0, upper_bound: 3286.0, distinct_count: 1124

    ...

    bin_251 lower_bound: 276665.0, upper_bound: 277768.0, distinct_count: 1041
    bin_252 lower_bound: 277768.0, upper_bound: 278870.0, distinct_count: 1098
    bin_253 lower_bound: 278870.0, upper_bound: 280000.0, distinct_count: 1106

    從上圖可見,生成的 histogram 為 equal-height histogram,且高度為 1102.36,bin 數為 254。其中 bin 個數可由 spark.sql.statistics.histogram.numBins 配置。對于每個 bin,勻記錄其最小值,最大值,以及 distinct count。

    值得注意的是,這里的 distinct count 并不是精確值,而是通過 HyperLogLog 計算出來的近似值。使用 HyperLogLog 的原因有二

    • 使用 HyperLogLog 計算 distinct count 速度快速
    • HyperLogLog 計算出的 distinct count 可以合并。例如可以直接將兩個 bin 的 HyperLogLog 值合并算出這兩個 bin 總共的 distinct count,而無須從重新計算,且合并結果的誤差可控

    算子對數據集影響估計

    對于中間算子,可以根據輸入數據集的統計信息以及算子的特性,可以估算出輸出數據集的統計結果。


    Spark SQL CBO Operator Estimation

    本節以 Filter 為例說明算子對數據集的影響。

    對于常見的 Column A < value B Filter,可通過如下方式估算輸出中間結果的統計信息

    • 若 B < A.min,則無數據被選中,輸出結果為空
    • 若 B > A.max,則全部數據被選中,輸出結果與 A 相同,且統計信息不變
    • 若 A.min < B < A.max,則被選中的數據占比為 (B.value - A.min) / (A.max - A.min),A.min 不變,A.max 更新為 B.value,A.ndv = A.ndv * (B.value - A.min) / (A.max - A.min)

    Spark SQL CBO Filter Estimation



    上述估算的前提是,字段 A 數據均勻分布。但很多時候,數據分布并不均勻,且當數據傾斜嚴重是,上述估算誤差較大。此時,可充分利用 histogram 進行更精確的估算


    Spark SQL CBO Filter Estimation with Histogram

    啟用 Historgram 后,Filter Column A < value B的估算方法為

    • 若 B < A.min,則無數據被選中,輸出結果為空
    • 若 B > A.max,則全部數據被選中,輸出結果與 A 相同,且統計信息不變
    • 若 A.min < B < A.max,則被選中的數據占比為 height(<B) / height(All),A.min 不變,A.max = B.value,A.ndv = ndv(<B)

    在上圖中,B.value = 15,A.min = 0,A.max = 32,bin 個數為 10。Filter 后 A.ndv = ndv(<B.value) = ndv(<15)。該值可根據 A < 15 的 5 個 bin 的 ndv 通過 HyperLogLog 合并而得,無須重新計算所有 A < 15 的數據。

    算子代價估計

    SQL 中常見的操作有 Selection(由 select 語句表示),Filter(由 where 語句表示)以及笛卡爾乘積(由 join 語句表示)。其中代價最高的是 join。

    Spark SQL 的 CBO 通過如下方法估算 join 的代價

    1
    2
    Cost = rows * weight + size * (1 - weight)
    Cost = CostCPU * weight + CostIO * (1 - weight)

    其中 rows 即記錄行數代表了 CPU 代價,size 代表了 IO 代價。weight 由 spark.sql.cbo.joinReorder.card.weight 決定,其默認值為 0.7。

    Build側選擇

    對于兩表Hash Join,一般選擇小表作為build size,構建哈希表,另一邊作為 probe side。未開啟 CBO 時,根據表原始數據大小選擇 t2 作為build side


    Spark SQL build side without CBO

    而開啟 CBO 后,基于估計的代價選擇 t1 作為 build side。更適合本例


    Spark SQL build side with CBO

    優化 Join 類型

    在 Spark SQL 中,Join 可分為 Shuffle based Join 和 BroadcastJoin。Shuffle based Join 需要引入 Shuffle,代價相對較高。BroadcastJoin 無須 Join,但要求至少有一張表足夠小,能通過 Spark 的 Broadcast 機制廣播到每個 Executor 中。

    在不開啟 CBO 中,Spark SQL 通過 spark.sql.autoBroadcastJoinThreshold 判斷是否啟用 BroadcastJoin。其默認值為 10485760 即 10 MB。

    并且該判斷基于參與 Join 的表的原始大小。

    在下圖示例中,Table 1 大小為 1 TB,Table 2 大小為 20 GB,因此在對二者進行 join 時,由于二者都遠大于自動 BroatcastJoin 的閾值,因此 Spark SQL 在未開啟 CBO 時選用 SortMergeJoin 對二者進行 Join。

    而開啟 CBO 后,由于 Table 1 經過 Filter 1 后結果集大小為 500 GB,Table 2 經過 Filter 2 后結果集大小為 10 MB 低于自動 BroatcastJoin 閾值,因此 Spark SQL 選用 BroadcastJoin。


    Spark SQL join type selection with CBO

    優化多表 Join 順序

    未開啟 CBO 時,Spark SQL 按 SQL 中 join 順序進行 Join。極端情況下,整個 Join 可能是 left-deep tree。在下圖所示 TPC-DS Q25 中,多路 Join 存在如下問題,因此耗時 241 秒。

    • left-deep tree,因此所有后續 Join 都依賴于前面的 Join 結果,各 Join 間無法并行進行
    • 前面的兩次 Join 輸入輸出數據量均非常大,屬于大 Join,執行時間較長


    Spark SQL multi join

    開啟 CBO 后, Spark SQL 將執行計劃優化如下


    Spark SQL multi join reorder with CBO

    優化后的 Join 有如下優勢,因此執行時間降至 71 秒

    • Join 樹不再是 left-deep tree,因此 Join 3 與 Join 4 可并行進行,Join 5 與 Join 6 可并行進行
    • 最大的 Join 5 輸出數據只有兩百萬條結果,Join 6 有 1.49 億條結果,Join 7相當于小 Join

    Spark 系列文章

    郭俊 Jason wechat
    歡迎關注作者微信公眾號【大數據架構】
    您的贊賞將支持作者繼續原創分享
    速赢彩app www.082706.com | 111000rr.com | 99151q.com | shenlaoxi.com | 33115h.com | www.pj9666.com | 3246.com | 473041.com | www.987072.com | www.7782l.com | zhcp72.com | dzhcp.com | www.09809.com | www.9g9g9g.com | hrbbdyy.com | 6150z.com | www.20288r.com | www.hg15588.com | www.34788a.com | js92359.com | www.448222.com | www.6939w.com | 15222.com | 878365mm.com | www.350677.com | www.892070.com | 80850ss.com | www.67797y.com | www.88xpj68.com | www.217302.com | 70118p.com | www.kj803.com | www.9818o.cc | 5906gg.com | www.pj88137.com | www.93gj08.com | y854.com | www.pj6522.com | www.07679v.com | 5622m.com | 56988v.com | www.x32939.com | www.590808.com | 67890qqq.com | www.dzcp4444.com | www.262072.com | 56787xx.com | www.sun181.com | www.955474.com | 187037.com | www.569389.com | www.22030.cc | 641768.com | www.t456x.com | www.748767.com | ylg123456.net | www.ii6241.com | www.647311.com | 80850ww.com | www.1168z.com | www.293255.com | 8036p.com | www.4521x.com | 9902666.com | www.hg9876.com | www.9422a.com | sxkcl.net | www.53589.com | www.53900b.com | uc.cc | www.47bm.com | www.196095.com | 30006f.com | www.511106.com | 66445003.com | www.js276.com | www.810531.com | 33ww8332.com | www.99094k.com | 5231007.com | www.yh8402.com | www.can80.com | 8037uuu.com | www.6bet005.com | 3544t.com | www.v456456.com | www.2588cai.com | 4288zz.com | www.9737yy.me | 50000.com | www.22gg940.com | www.lqc7.com | feicai0972.com | www.js520988.cn | 4775522.com | www.472702.com | 3121gg.com | www.888vip6.com | www.707619.com | feicai023.com | www.13434e.com | nn3336.com | www.1118084.com | 4066c.com | www.282555b.com | www.040415.com | www.js06123.com | www.950633.com | 776059.com | www.52072j.com | 21051199.com | www.68568c.com | hh4675.com | www.ra79.am | 69442277.com | www.7140588.com | 9679l.com | www.88325c.com | 83377r.com | www.6880rr.com | 5002sss.com | www.c555.com | 5589222.com | www.4694h.com | 1389u.com | www.4212e.com | zhcp18.com | www.88c668.com | 1168n.com | www.27363m.com | 38455.com | www.911137.com | 98345k.com | www.7415b.com | 4066zz.com | www.4078d.com | bwin8o.com | www.1559507.com | 66300vip17.com | www.0600z.cc | 9998jsc.com | www.71233k.com | 922545.com | www.5xca.com | www.hg97hg.com | www.249770.com | www.cs55222.com | 08159b.com | www.jbb08.net | 2566h9.com | www.03a88.com | 38365m.com | www.624202.com | 2851k.com | www.en06.com | www.hg68111.com | v0099.tv | www.4996xn.com | 54146655.com | www.68689h.com | 953644.com | www.361929.com | www.pj5504.com | 1366y.com | www.1851138.com | dh41866.com | www.50064t.com | 2998z.com | www.235127.com | www.569389.com | 51439522.com | www.98478.com | 15a39.net | www.210927.com | www.6809pj.com | v997e.com | www.832090.com | 6146t.com | www.292199.com | www.07773.com | 993311.com | www.624202.com | www.glc22.com | www.192722.com | www.818msc.net | 4052mm.com | www.26878p.com | www.76543z.com | s2554.com | www.792074.com | 6789801.com | www.582977.com | www.08bwin.com | 3807x.com | www.2934s.com | www.mg7088.com | 1669m.com | www.ihg5511.com | www.9599566.net | 38821199.com | www.0343h.com | www.6802pj.com | 3169.com | www.37377d.com | www.hg9765.com | 9411ggg.com | www.vns11.me | www.862365.com | v32365.com | www.1466a.com | www.0853c.com | 8742gg.com | www.amjs119.com | www.hg990077.com | 87665.com | www.88065d.com | www.yxlmwx.net | 553955.com | www.66368.com | www.38144144.com | 00773cc.com | www.375967.com | www.55070z.com | 0600i.cc | mm3189.com | www.dyloto9.com | www.33384066.com | 5219d.com | www.627212.com | www.5446dd.com | 7989h.com | 2013880.com | www.772875.com | www.77537l.com | 33599ff.com | www.890752.com | www.8313o.com | 209899.com | 3007811.com | www.7782x.com | www.222505.com | 2190d.com | 27878nn.com | www.25688e.com | www.990998.com | 61327788.com | 33tt8331.com | www.hy2811.com | www.4212s.com | www.6033b.com | 36407788.com | www.530951.com | www.792087.com | www.09569i.com | 3890u.com | js7001.com | www.16878o.com | www.444202.com | 6403n.com | 87965tt.com | www.582833.com | www.18617.com | www.w8586.com | 01885k.com | 4997t.com | www.3890z.com | www.80969a.com | www.ao899.com | bogou2222.com | www.109882.com | www.68cc2.com | www.5504h.com | www.3435999.com | bwin8f.com | www.359215.com | www.37377f.com | www.828501.com | www.68666n.com | 4461x.com | 67877e.com | www.6611d.cc | www.js58123.com | www.448868.com | www.033033p.com | 23599s.com | 9649z.com | www.99ckb.com | www.js89w.vip | www.105353.com | 5003.cc | 4488qqq.com | 56988.cc | www.202340.com | www.52072z.com | www.91779b.com | www.57798.com | 1479r.com | 36408899.com | 3650581.com | 7720776.com | www.622by.com | www.bb4625.com | www.751cp44.com | www.7025i.com | 4997c.com | uuu4165.com | 38244i.com | www.332071.com | www.33068.cc | www.fg9907.com | www.huayi388.com | www.2377msc.com | www.558077.com | 8905k.com | 3957o.com | 12772266.com | 4880s.com | www.321693.com | www.5cpb.com | www.50080q.com | www.blr9944.com | www.7415u.com | www.660077.com | www.4972jj.com | 1346.com | 2506f.com | 80850tt.com | js365dl.com | 6190js.com | 80878n.com | www.77114o.com | www.616383.com | www.0014r.com | www.20899.com | www.8w08.com | www.38138k.com | www.c44ll.com | www.688558.com | 2268jc.com | www.332807.com | www.73031.cc | www.555xin2.com | www.v0050.com | www.2945n.com | www.66710.cc | 4255zzz.com | 2015.com | www.hjdc48.com | www.50054j.com | feicai.com | y854.cc | www.ra0008.com | www.9478n.com | 1775l.com | 7935b.com | www.50999q.com | www.okw5.com | v939.com | www.bd0088.com | www.9422j.com | 230466.com | 0241.com | www.5446ww.com | www.691663.com | 463.cm | www.yh8352.com | www.10888.com | 90856y.com | 8901r.com | www.333jyh.com | www.803243.com | 2566h2.com | www.hxcp300.com | www.89894m.com | 6999004.com | www.lpj555.com | www.99552pp.com | www.50026u.com | emr9836.com | xpj99.am | www.682855.com | www.78680q.com | 61321166.com | 1859009.com | www.y6098.com | 55545z.com | 2613z.com | www.huangma22.com | www.206881.com | 56988f.vip | www.65066hh.com | www.346977.com | 375656.com | www.5099ss.com | www.341011.com | q2146.com | www.26123bb.com | www.43131w.com | 2373.com | www.7240k.com | 32212j.com | www.m32126.cc | www.8582pp.com | 013311.com | www.777307.com | www.57022.cc | 57157l.com | www.80388.com | www.638466.com | 08820088com | www.cn365u.com | 00048j.com | www.033033z.com | www.8998933.com | 1064a.com | www.xinhao77.com | www.700590.com | 4647y.com | www.351532.com | 99306e.com | www.40818e.com | www.33997d.com | 0011buyu.com | www.4963j.com | www.36787e.com | www.pj2666.com | www.6653g.com | yabo535.com | www.85857e.com | www.41518a.com | www.80852.com | www.84499q.com | 62222w.com | www.55545g.com | wlb22.com | www.o22365.com | www.5441x.com | 51200.com | www.38138r.com | e47479.com | www.kx151.com | www.655101.com | hg1228.com | www.81233hh.com | hga22200.com | www.6687l.com | 1227010.com | www.76775g.com | www.072211.com | www.hg755.com | www.igcp5.com | hgyz66.com | www.t09738.com | 509.com | www.0041331.com | 6868pp.cc | www.bet3650914.com | bwin990.co | www.88837s.com | www.701545.com | www.6033l.com | www.3416r.com | 444000vvv.com | www.088014.com | 47749p.com | www.25229.com | so55555.cc | www.178441.com | 2383.com | www.32031r.com | 00778t.com | www.1559509.com | 3559p.com | www.89599r.com | 58802b.com | www.8967c.com | 472709.com | www.099923.com | 55332007.com | www.1168f.com | js66661.com | www.67df.com | 205235.com | www.8863tt.com | ff1331.com | www.8473j.com | 363369.com | www.v88134.com | 2490s.com | www.87708c.com | bc1633.com | www.60886y.com | 3967c.com | www.x666678.com | 25288q.com | www.60108m.com | p88.cc | www.47d.cc | www.vns9999.cc | www.572539.com | www.55155h.com | www.035079.com | www.6888tm.com | 61652v.com | www.751cp22.com | 15a42.net | www.79095c.com | 4011.cc | www.yh7774.com | z2554.com | www.yw344.com | www.www-866766.com | www.317787.com | www.xpj1506.com | 131jj.net | www.ts018.com | 01234t.com | www.4331h.com | 3222i.cc | www.813230.com | www.a7892904.com | 4195d.com | www.xpj2008.co | 7686m.com | www.8bd.com | oo5144.com | www.324877.com | www.yyyy0066.com | 793929.com | www.b98478.com | 4812e.com | www.055520.cc | www.c668ln.com | 33432k.com | www.12455u.com | www.9544j.com | www.77114i.com | www.184897.com | yyy4255.com | www.73736d.com | www.518hwx.com | 17715.com | www.4136w.com | 3559oo.com | www.fcyl1.com | www.y5777.com | 9484.com | www.win1232.com | hjdc975.com | www.220892.com | www.5504j.com | 55967s.com | www.0719.cc | www.yh48826.com | 918dp.com | www.cr678.net | www.pj0567.com | 3991100.com | www.am8333.tw | z88XX.com | www.361979.com | www.1700116.com | jib666666.com | www.896950.com | www.www899000.com | 095.com | www.5086u.com | www.77777ppp.com | 61652f.com | www.50051vip.com | www.dhycp8811.com | 2805i.com | www.33112.com | www.xj4466.com | 925150.com | www.39500x.com | www.ljw032.com | 5988365.com | www.43818f.com | www.xpj70065.com | n888vv.com | www.8667g.com | www.32788a.com | 2147002.com | www.hf6880.com | www.44488807.com | s40033.com | www.645326.com | www.808888e.com | b8381.com | zhenxin360.com | www.79095t.com | www.bm1100.vip | 5906rr.com | www.95999w.com | www.606881.com | 1443f.com | www.848966.com | www.356062.com | 0747i.com | 8988444.com | www.12345604.cc | www.boma0190.com | 7249j.com | www.988456.vip | www.5504z.com | 3242.com | 90856x.com | www.01088.com | www.xpj8812.com | xpj5345.com | www.196776.com | www.44118p.com | www.365109h.com | 33382ff.com | www.c4532.com | www.28000k.com | 5004p.com | c99345.am | www.8816aa.com | www.999msc.net | 28758r.com | 1077www.com | www.21202l.com | www.drf0432.com | xpj6001.com | qijiduchang.vip | www.60108w.com | www.g78306.com | 65005h.com | 6033p.com | www.33598f.com | www.523c51.com | 3344vn77.com | 6033.com | www.820155.com | www.1559502.com | www.1347-05.com | lh66r.com | www.355284.com | www.8722.am | www.yxinct.com | jinguanylc9.com | 2595u.com | www.61411.cc | www.28000n.com | www.b3652.com | 6175ll.com | www.235035.com | www.6939v.com | www.pj79777.com | 2381k.com | 66300.org | www.361550.com | www.81608e.com | www.bsyedu.com | 56988.com | 38989j.com | www.316083.com | www.ylylc08.com | www.102149.com | 39520.net | sjg008.com | www.29277e.com | www.61655l.com | www.430578.com | www.223456p.com | 51133ooo.com | 30019f.com | www.qml4.com | www.js89a.vip | www.048888.com | 66066.com | 61999.com | 00dd8332.com | www.fh17.com | www.2299nsb.com | www.004433.com | 496677.com | 0729y.com | 44488r.com | www.cb5388.com | www.rrqp999.com | www.88100.com | www.vns82222.com | 983888q.com | 3424a.com | www.131333.com | www.33588t.com | www.91779v.com | www.656888.com | 5369m.com | 53262uu.com | 4675099.com | www.509265.com | www.29277hh.com | www.4996hh.com | www.f8867.com | www.9130.com | 62222t.com | 4036i.com | 0033l.cc | www.652623.com | www.61655n.com | www.44118n.com | www.56655b.com | www.dd7920.com | 1624hh.com | 73055t.com | 87965oo.com | 6175c.com | www.283550.com | www.007hy.cc | www.088014.com | www.516297.com | www.wz2255.com | www.1495s6.com | www.56733s.com | 2015.com | yun888.com | 365ms.net | 06006u.com | www.181599.com | www.937299.com | www.60123f.com | www.189a56.com | www.01bet8.com | www.752suncity.com | www.81849.com | www.hg1895.com | 3648991.com | 99777.com | bdg1166.com | 3334661.com | wfcp111000.com | 7196ff.com | www.530891.com | www.c5615.com | www.15355b.com | www.37377d.com | www.js89g.vip | www.38138d.com | www.1705.com | www.kj88a.com | www.hg7651.com | www.7111i.com | www.r32126.cc | www.81866w.com | 55818i.com | youfa888.vip | 73884.com | 4340s.com | 67890ggg.com | 45637r.com | 3222q.cc | 007369.com | 1479b.com | 148030.com | yabo9000.com | zunyi803.com | 3928722.com | 2381006.com | bosw8.com | 3544u.com | 115119.com | 9856bb.com | 22wb8.com | 518cp8.com | xianjinwanggg.com | 33oo8331.com | 65005.com | 2018.so | 500000972.com | 99111nn.com | 2934k.com | 060939.com | 667766z.com | 6033p.com | ylg2566.com | 1347.com | yg7777.com | 85850.com | kkkk19927.com | w7776.com | 3726z.com | 11005v.com | ylzz522.com | gao482.com | pp3189.com | 033r.net | www.hr1822.com | www.xpj1123.com | www.816801.com | www.ejylc11.com | www.1205.com | www.574119.com | www.p98478.com | www.a98478.com | www.60123w.com | www.8499d.com | www.ya350.com | www.5522q.cc | www.949410.com | www.591017.com | www.377192.com | q759.top | 50038s.com | 4123ss.com | 50067v.com | gg40033.com | 642221.com | 3121j.com | www.85770l.com | www.363788.com | www.21365kk.com | www.004439.com | www.xx3088.com | www.71071m.com | www.877579.com | www.888ag.net | www.55676g.com | www.7714v.com | www.5086e.com | www.632590.com | www.106526.com | kk3189.com | vip66657.com | 3807jj.com | 201877880.com | 7777ag2.com | www.alpk8.com | www.hg9777.com | www.622206.com | www.3116d.com | www.x88678.com | www.36166g.com | www.820158.com | www.609719.com | 7196.com | 3559uuuu.com | 3957m.com | 3405u.com | www.vns338.com | www.lefa888.com | www.boh.cc | www.79095c.com | www.88065c.com | www.824879.com | xpj345345.com | 0652v.com | 4255fff.com | 44cc8331.com | www.9068ww.com | www.5555566666.com | www.35700dd.com | www.2418002.com | www.sytg7.com | 4379t.com | bmw8099.com | c3405.com | www.81866x.com | www.k0678.com | www.28000p.com | www.93gj03.com | www.hx1162.com | v61653.com | 04666j.com | 17798888.com | www.hg748.com | www.606933.com | www.o69096.com | www.99788r.com | www.591015.com | 7779822.com | 3978.com | www.ylg4499.com | www.900168.com | www.lhg888.com | www.36788j.com | www.188737.com | 28758.com | 983888j.com | www.365775.com | www.97222a.com | www.6939k.com | www.689477.com | 159666328.com | lc99u.com | www.bet3651114.com | www.305015.me | www.86333b.com | www.ay7799.com | b3144.com | qq40033.com | www.7652.cc | www.79500v.com | www.yun870.com | www.544090.com | 1766.com | 2013.com | www.ii7966i.com | www.578888.vip | www.coz5.com | 3650755.com | wuxixhh.com | www.56520f.com | www.386464.com | www.62118b.com | www.550436.com | hh3336.com | 59889x.com | www.xpj66989.com | www.win1235.com | www.359629.com | 15a2.net | 4812222.com | www.2945j.com | www.97655q.com | 67877e.com | 8037vvv.com | www.20161166.com | www.33678dd.com | www.901307.com | 4116a.com | 3202s.com | www.767613.com | www.3978l.com | 112k.net | 6118d.com | www.a80070.com | www.blr4444.com | www.497918.com | cp014.com | www.pj0908.com | www.8080999j.com | www.911790.com | 8381o.com | www.vns998.com | www.5099tt.com | www.122888a.com | 67y88.vip | www.js72999.com | www.bwinyz01.com | www.904867.com | 3189j.com | www.ylg82.com | www.38138n.com | www.899621.com | 28839s.com | www.4107e.com | www.00778n.com | www.610659.com | 380111666.com | www.zq.hk | www.0088hgii.com | www.77803v.com | 4288z.com | www.pu382.com | www.fg500.com | www.35155e.com | 9694s.com | www.gssuu168.com | www.100v.cc | 32355dd.com | www.hg755.com | www.blr4444.com | yl-26.com | 0448099.com | www.c44ss.com | www.76wcp.com | blr365365.com | www.cai73dd.com | www.906777.com | 39992266.com | n47479.com | www.5446oo.com | www.586840.com | mi519.com | www.68228p.com | www.fl62.com | 9101902.com | www.y0001.cc | www.77802f.com | 60540022.com | www.365815z.com | www.222387.com | 3644s.com | www.yptang.com | www.75600d.com | drcp555.com | 5855qq.com | www.883399h.com | www.021037.com | 2767k.com | www.115527m.com | www.256968.com | hg5009.com | www.9170182.com | www.025938.com | 3169g.com | www.3066gg.com | 80368j.com | 2805t.com | www.x063801.com | 91019u.com | www.glc66.com | www.12136h.com | 1319996.com | www.h067.com | www.4331k.com | 80892w.com | www.hg726.com | www.6832k.com | bet577t.com | www.111492.com | www.657630.com | 365qxw.am | www.7893.net | 3089a.com | www.yh8217.com | www.7737ee.com | 25115.am | www.006074.com | www.222hy.cc | 36407755.com | www.yzcp2025.com | www.053883.com | 3379d.com | www.7036hh.com | 364493.com | www.06111.com | www.5091n.com | 3950r.com | www.3116q.com | 3399vn77.com | www.a22365.com | www.1368e.cc | 2jsyyy.com | www.xpj668.com | 817458.com | www.u27229.com | www.1368b.cc | wnsr828.com | www.biying930vip.com | 8790y.com | www.330099g.com | www.55228g.com | 4022pp.com | www.ao3399.com | 4066bb.com | www.4912f.com | www.770960.com | 3089m.com | www.hd8671.com | j669933.com | www.yfcp066.com | 33115hh.com | www.k22365.com | www.gai50.com | hh2649.com | www.hd8675.com | 4052mm.com | www.88166q.com | 55665002.com | www.byc06.com | www.517844.com | 8052.am | www.77016.net | hc3389.com | www.w84r.com | 4646755.com | www.3535h.com | spj5577.com | www.97780099.net | www.631799.com | 6218w.com | www.8499l.com | 3844rr.com | www.889580.com | 6723888.com |