• <menu id="gyiem"><menu id="gyiem"></menu></menu>
  • <menu id="gyiem"><code id="gyiem"></code></menu>

    Spark CommitCoordinator 保證數據一致性

    原創文章,轉載請務必將下面這段話置于文章開頭處。
    本文轉發自技術世界原文鏈接 http://www.luozeyang.com/spark/committer/

    本文所述內容均基于 2018年9月17日 Spark 最新 Release 2.3.1 版本,以及 hadoop-2.6.0-cdh-5.4.4

    概述

    Spark 輸出數據到 HDFS 時,需要解決如下問題:

    • 由于多個 Task 同時寫數據到 HDFS,如何保證要么所有 Task 寫的所有文件要么同時對外可見,要么同時對外不可見,即保證數據一致性
    • 同一 Task 可能因為 Speculation 而存在兩個完全相同的 Task 實例寫相同的數據到 HDFS中,如何保證只有一個 commit 成功
    • 對于大 Job(如具有幾萬甚至幾十萬 Task),如何高效管理所有文件

    commit 原理

    本文通過 Local mode 執行如下 Spark 程序詳解 commit 原理

    1
    2
    3
    sparkContext.textFile("/json/input.zstd")
    .map(_.split(","))
    .saveAsTextFile("/jason/test/tmp")

    在詳述 commit 原理前,需要說明幾個述語

    • Task,即某個 Application 的某個 Job 內的某個 Stage 的一個 Task
    • TaskAttempt,Task 每次執行都視為一個 TaskAttempt。對于同一個 Task,可能同時存在多個 TaskAttemp
    • Application Attempt,即 Application 的一次執行

    在本文中,會使用如下縮寫

    • ${output.dir.root} 即輸出目錄根路徑
    • ${appAttempt} 即 Application Attempt ID,為整型,從 0 開始
    • ${taskAttemp} 即 Task Attetmp ID,為整型,從 0 開始

    檢查 Job 輸出目錄

    在啟動 Job 之前,Driver 首先通過 FileOutputFormat 的 checkOutputSpecs 方法檢查輸出目錄是否已經存在。若已存在,則直接拋出 FileAlreadyExistsException
    Check output path

    Driver執行setupJob

    Job 開始前,由 Driver(本例使用 local mode,因此由 main 線程執行)調用 FileOuputCommitter.setupJob 創建 Application Attempt 目錄,即 ${output.dir.root}/_temporary/${appAttempt}
    Setup job

    Task執行setupTask

    由各 Task 執行 FileOutputCommitter.setupTask 方法(本例使用 local mode,因此由 task 線程執行)。該方法不做任何事情,因為 Task 臨時目錄由 Task 按需創建。
    Setup task

    按需創建 Task 目錄

    本例中,Task 寫數據需要通過 TextOutputFormatgetRecordWriter 方法創建 LineRecordWriter。而創建前需要通過 FileOutputFormat.getTaskOutputPath設置 Task 輸出路徑,即 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt}/${fileName}。該 Task Attempt 所有數據均寫在該目錄下的文件內
    Create task output file

    檢查是否需要 commit

    Task 執行數據寫完后,通過 FileOutputCommitter.needsTaskCommit 方法檢查是否需要 commit 它寫在 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 下的數據。

    檢查依據是 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 目錄是否存在
    Need commmit task

    如果需要 commit,并且開啟了 Output commit coordination,還需要通過 RPC 由 Driver 側的 OutputCommitCoordinator 判斷該 Task Attempt 是否可以 commit
    Need commmit task detail

    之所以需要由 Driver 側的 CommitCoordinator 判斷是否可以 commit,是因為可能由于 speculation 或者其它原因(如之前的 TaskAttemp 未被 Kill 成功)存在同一 Task 的多個 Attemp 同時寫數據且都申請 commit 的情況。

    CommitCoordinator

    當申請 commitTask 的 TaskAttempt 為失敗的 Attempt,則直接拒絕

    若該 TaskAttempt 成功,并且 CommitCoordinator 未允許過該 Task 的其它 Attempt 的 commit 請求,則允許該 TaskAttempt 的 commit 請求

    若 CommitCoordinator 之前已允許過該 TaskAttempt 的 commit 請求,則繼續同意該 TaskAttempt 的 commit 請求,即 CommitCoordinator 對該申請的處理是冪等的。

    若該 TaskAttempt 成功,且 CommitCoordinator 之前已允許該 Task 的其它 Attempt 的 commit 請求,則直接拒絕當前 TaskAttempt 的 commit 請求
    Coordinator handle request

    OutputCommitCoordinator 為了實現上述功能,為每個 ActiveStage 維護一個如下 StageState

    1
    2
    3
    4
    private case class StageState(numPartitions: Int) {
    val authorizedCommitters = Array.fill[TaskAttemptNumber](numPartitions)(NO_AUTHORIZED_COMMITTER)
    val failures = mutable.Map[PartitionId, mutable.Set[TaskAttemptNumber]]()
    }

    該數據結構中,保存了每個 Task 被允許 commit 的 TaskAttempt。默認值均為 NO_AUTHORIZED_COMMITTER

    同時,保存了每個 Task 的所有失敗的 Attempt

    commitTask

    當 TaskAttempt 被允許 commit 后,Task (本例由于使用 local model,因此由 task 線程執行)會通過如下方式 commitTask。

    mapreduce.fileoutputcommitter.algorithm.version 的值為 1 (默認值)時,Task 將 taskAttemptPath 即 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 重命令為 committedTaskPath 即 ${output.dir.root}/_temporary/${appAttempt}/${taskAttempt}
    Commit task v1

    mapreduce.fileoutputcommitter.algorithm.version 的值為 2,直接將taskAttemptPath 即 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 內的所有文件移動到 outputPath 即 ${output.dir.root}/
    Commit task v2

    commitJob

    當所有 Task 都執行成功后,由 Driver (本例由于使用 local model,故由 main 線程執行)執行 FileOutputCommitter.commitJob

    mapreduce.fileoutputcommitter.algorithm.version 的值為 1,則由 Driver 單線程遍歷所有 committedTaskPath 即 ${output.dir.root}/_temporary/${appAttempt}/${taskAttempt},并將其下所有文件移動到 finalOutput 即 ${output.dir.root}
    Commit job v1

    mapreduce.fileoutputcommitter.algorithm.version 的值為 2,則無須移動任何文件。因為所有 Task 的輸出文件已在 commitTask 內被移動到 finalOutput 即 ${output.dir.root}
    Commit job v2

    所有 commit 過的 Task 輸出文件移動到 finalOutput 即 ${output.dir.root} 后,Driver 通過 cleanupJob 刪除 ${output.dir.root}/_temporary/ 下所有內容
    Cleanup job

    recoverTask

    上文所述的 commitTask 與 commitJob 機制,保證了一次 Application Attemp 中不同 Task 的不同 Attemp 在 commit 時的數據一致性

    而當整個 Application retry 時,在之前的 Application Attemp 中已經成功 commit 的 Task 無須重新執行,其數據可直接恢復

    恢復 Task 時,先獲取上一次的 Application Attempt,以及對應的 committedTaskPath,即 ${output.dir.root}/_temporary/${preAppAttempt}/${taskAttempt}

    mapreduce.fileoutputcommitter.algorithm.version 的值為 1,并且 preCommittedTaskPath 存在(說明在之前的 Application Attempt 中該 Task 已被 commit 過),則直接將 preCommittedTaskPath 重命名為 committedTaskPath

    mapreduce.fileoutputcommitter.algorithm.version 的值為 2,無須恢復任何數據,因為在之前 Application Attempt 中 commit 過的 Task 的數據已經在 commitTask 中被移動到 ${output.dir.root}
    Recover task

    abortTask

    中止 Task 時,由 Task 調用 FileOutputCommitter.abortTask 方法刪除 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt}
    Abort task

    abortJob

    中止 Job 由 Driver 調用 FileOutputCommitter.abortJob 方法完成。該方法通過 FileOutputCommitter.cleanupJob 方法刪除 ${output.dir.root}/_temporary

    總結

    V1 vs. V2 committer 過程

    V1 committer(即 mapreduce.fileoutputcommitter.algorithm.version 的值為 1),commit 過程如下

    • Task 線程將 TaskAttempt 數據寫入 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt}
    • commitTask 由 Task 線程將 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 移動到 ${output.dir.root}/_temporary/${appAttempt}/${taskAttempt}
    • commitJob 由 Driver 單線程依次將所有 ${output.dir.root}/_temporary/${appAttempt}/${taskAttempt} 移動到 ${output.dir.root},然后創建 _SUCCESS 標記文件
    • recoverTask 由 Task 線程將 ${output.dir.root}/_temporary/${preAppAttempt}/${preTaskAttempt} 移動到 ${output.dir.root}/_temporary/${appAttempt}/${taskAttempt}

    V2 committer(即 mapreduce.fileoutputcommitter.algorithm.version 的值為 2),commit 過程如下

    • Task 線程將 TaskAttempt 數據寫入 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt}
    • commitTask 由 Task 線程將 ${output.dir.root}/_temporary/${appAttempt}/_temporary/${taskAttempt} 移動到 ${output.dir.root}
    • commitJob 創建 _SUCCESS 標記文件
    • recoverTask 無需任何操作

    V1 vs. V2 committer 性能對比

    V1 在 Job 執行結束后,在 Driver 端通過 commitJob 方法,單線程串行將所有 Task 的輸出文件移動到輸出根目錄。移動以文件為單位,當 Task 個數較多(大 Job,或者小文件引起的大量小 Task),Name Node RPC 較慢時,該過程耗時較久。在實踐中,可能因此發生所有 Task 均執行結束,但 Job 不結束的問題。甚至 commitJob 耗時比 所有 Task 執行時間還要長

    而 V2 在 Task 結束后,由 Task 在 commitTask 方法內,將自己的數據文件移動到輸出根目錄。一方面,Task 結束時即移動文件,不需等待 Job 結束才移動文件,即文件移動更早發起,也更早結束。另一方面,不同 Task 間并行移動文件,極大縮短了整個 Job 內所有 Task 的文件移動耗時

    V1 vs. V2 committer 一致性對比

    V1 只有 Job 結束,才會將數據文件移動到輸出根目錄,才會對外可見。在此之前,所有文件均在 ${output.dir.root}/_temporary/${appAttempt} 及其子文件內,對外不可見。

    當 commitJob 過程耗時較短時,其失敗的可能性較小,可認為 V1 的 commit 過程是兩階段提交,要么所有 Task 都 commit 成功,要么都失敗。

    而由于上文提到的問題, commitJob 過程可能耗時較久,如果在此過程中,Driver 失敗,則可能發生部分 Task 數據被移動到 ${output.dir.root} 對外可見,部分 Task 的數據未及時移動,對外不可見的問題。此時發生了數據不一致性的問題

    V2 當 Task 結束時,立即將數據移動到 ${output.dir.root},立即對外可見。如果 Application 執行過程中失敗了,已 commit 的 Task 數據仍然對外可見,而失敗的 Task 數據或未被 commit 的 Task 數據對外不可見。也即 V2 更易發生數據一致性問題

    Spark 系列文章

    郭俊 Jason wechat
    歡迎關注作者微信公眾號【大數據架構】
    您的贊賞將支持作者繼續原創分享
    速赢彩app www.19019g.com | 33009193.com | www.hg44401.com | www.4833045.com | ys777.co | www.xy466.com | www.688dl.com | 5478888.com | www.a11122.com | www.513061.com | 365488.bet | www.6133.com | 80878a.com | www.xpj8812.com | www.972283.com | df8d.com | www.792084.com | 145237.com | www.w8586.com | www.701547.com | x86400.com | www.js9090.com | 454661.com | www.1168g.com | www.32123d.com | www.1111jsc.com | www.pj56a.com | hg1088.com | www.532159.com | bwinooo.com | www.ag803.com | www.1035y.com | bt1088.com | www.0343e.com | 112p.net | www.r063801.com | 3405yyy.com | www.gdnmi.com | x59.com | www.365815b.com | www.fh3399.com | 2267m.com | www.60123m.com | 009.so | www.667700.com | p21148.com | www.357116.com | feicai0873.com | www.js5599.vip | 1703222.com | www.b7123.com | www.66332i.com | www.3459q.com | www.703890.com | www.yh8007.com | www.hf1680.com | yun8123.com | www.2109m.com | 8577d.cc | www.al9888.com | 633248.com | www.920959.com | www.a2a333.cc | www.843533.com | www.7435y.com | www.833585.com | www.4763006.com | www.526051.com | www.11155545.com | www.302210.com | www.77537g.com | www.217030.com | www.f333js.com | www.96386o.com | www.2016hg.com | 9081cc.com | www.946466.com | 2613p.com | www.234k7.com | 897dd.cc | www.6668a.vip | 2844s.com | www.7036.com | zhcp08.com | www.108.gg | www.9599440.com | www.701775.com | www.2226358.com | 9964n.com | www.ld003.com | q3144.com | www.1168d.com | 2490p.com | www.742022.com | 3131394.com | www.08588i.com | 2455u.com | www.867455.com | www.wns10.com | 6766hh.com | www.2945c.com | 2820o.com | www.4102v.com | 28288r.com | www.9356d.com | www.bet25365.com | 7605i.com | www.1389oo.com | 114la.com | www.386123.com | zg163.net | www.338031.com | www.3444hg.com | 7605m.com | www.6889797.com | 1882229.com | www.330687.com | www.jbb2017.com | 06386363.com | www.283826.com | www.jsjsjs59567.com | www.090040.com | www.6511msc.com | 70118.com | www.2373n.com | www.9679700.com | jss1150.cn | www.8c777.com | 2455m.com | www.244689.com | www.d9008.com | 66287l.com | www.2875z.com | www.88hm.com | nn500l.com | www.61500.cc | www.665345.com | 118cai.cc | www.50026h.com | www.2k604.com | 1766.com | www.52072p.com | www.58777u.com | 3614c.com | www.61233l.com | www.5651e.com | 8722nnnn.com | www.055v.cc | www.729666.com | 26668p.com | www.ct8822.com | www.haomen1.com | 36403311.com | www.599647.com | www.ii6241.com | 98955r.com | www.196039.com | www.7893w38.com | 2190l.com | man984.com | www.x777678.com | www.7830c.com | 991682.com | www.2109d.com | www.hg8104.com | 2127jj.com | 1331a.com | www.1434m.com | www.benz4433s.com | 88850kk.com | www.3478r.cc | www.l456x.com | 5589a.com | 1594003.com | www.50026d.com | www.827733.com | 614151.com | www.61655u.com | www.52062c.com | www.yinhe9507b.com | 3202y.com | www.c1981.com | www.58665c.com | 111122mm.com | hga019.com | www.9818.cc | www.170507.com | 4195c.com | 4066a.com | www.lqc9.com | www.vns887.com | www.7830q.com | 8538s.com | www.43131e.com | www.025155.com | www.959888.com | xiaowei889.com | 2846.vip | www.803733.com | www.40288y.com | www.330099w.com | 7240b.com | 2247xx.com | www.68682b.com | www.786555.com | 4422mmmm.com | 40033t.com | 0652p.com | www.983jc.com | www.18k0048.com | www.bet81365.com | 官网捕鱼4.com | 40168.com | www.3126j.com | www.bwinyz38.com | www.9679u.com | 00773u.com | 35543.com | www.c4520.com | www.56011v.com | www.vns95500.net | 9949x.com | 13820000.com | 7141kk.com | www.48wcp.com | www.332337.com | www.lefa888.com | 97799x.com | 4445365.com | 609906.com | www.cp718.vip | www.98488.com | www.99418.com | www.11105.tv | 6261u.com | js999js.com | qhbwz.com | www.022y.cc | www.7920j.com | www.597567d.com | www.pjbeijing.com | 84494411.com | cp11991.com | www.003675.com | www.885pz.com | www.70669a.com | www.005mgm.com | www.8124x.com | duch019.com | 343477.com | 7777wd.com | www.657399.com | www.81520w.com | www.46630.biz | www.blm139.com | www.hg0088.co | 6245b.com | 66671k.com | 0015kk.com | 20160913.tv | www.807835.com | www.77802w.com | www.40288z.com | www.224.com | www.6032888.com | www.22567.com | 168cp-t.com | 3667.com | wfc.com | 80850xx.com | 1592n.com | www.225779.com | www.861944.com | www.9478g.com | www.046161.com | www.678js222.com | www.50999.com | www.cc1916.com | www.yh33566.com | www.28758v.com | u2649.com | 0166f.com | 00773m.com | 40033q.com | 33qq8332.com | 5443w.com | 4444.ag | xx888b.com | 22wb8.com | www.36788o.com | www.249880.com | www.558423.com | www.808910.com | www.877719.com | www.hw4444.com | www.2350g.com | www.5095m.com | www.66653z.com | www.7239t.com | www.55111122.com | www.win1237.com | www.540729.com | www.4102o.com | www.97994e.com | www.y6087.com | www.38775zz.com | www.j32031.com | www.516297.com | www.5981s.com | www.55060m.com | www.472702.com | www.16181s.com | www.4136d.com | www.10999j.com | www.00773y.com | www.3066rr.com | www.2544g5.com | www.6687q.com | www.56011a.com | www.99094x.com | www.jzvip33.com | www.50074j.com | www.22630.cc | www.872198.com | www.701495.com | www.257015.com | www.088185.com | 6396b.com | 76543k.com | 59599r.com | x86811.com | 01234ooo.com | bb33338.com | hg77702.com | 738055.com | www.15144.com | www.h7788a.com | www.954555.com | www.7669mm.com | www.klcp005.com | www.006786.com | www.55676a.com | www.83993b.com | www.bai355.com | www.564696.com | 8827aa.com | 28288g.com | r93.com | 6261dd.com | b5369.com | www.330099l.com | www.900988.com | www.16181u.com | www.1434o.com | www.36787g.com | www.643477.com | c2146.com | 2287dd.com | 444167.com | 3189h.com | www.54549966.com | www.k00444.com | www.88325u.com | www.725641.com | www.810852.com | 38989r.com | 4809g.com | 033v.net | www.8655msc.com | www.88837q.com | www.49936.com | 8040ggg.com | 56728i.com | www.2345sun.net | www.99094c.com | 99589cc.net | yinhebp.com | www.e7457.com | www.2350j.com | oo38648.cc | www.2ag88.com | www.40288z.com | www.78949e.com | hg77740.com | www.g948g.com | www.csgc2.com | 7514444.com | www.hg0532.com | www.25688o.com | 712930.com | www.8494s.com | www.j94600.com | yy5443.com | 3222f.cc | www.hg33396.com | www.081506.com | 6766kk.com | www.7415xx.com | www.314877.com | y68.com | www.4212c.com | www.403289.com | 00bb8331.com | www.8473a.com | www.355805.com | 98955s.com | www.00618a.com | www.277739.com | q888.com | www.66vn777.com | www.083067.com | 6830x.com | www.327969.com | 718012.com | www.pj63333.com | www.999qipai.com | 61789p.com | www.b55551.com | www.3416p.com | y14666.com | www.55526r.com | www.506972.com | 2349012.com | www.377666p.com | P35p.com | www.20x0048.com | www.50989k.com | 063801.com | www.1389oo.com | www.319826.com | blh13579.com | www.2418r.com | 3656ss.com | www.80879.com | www.620958.com | 30178844.com | www.0622iii.com | kk1331.com | www.fh9000.com | www.661661a.com | x33g.vip | www.5446l.com | z35151.com | www.hg7849.com | www.584499.com | 3169t.com | www.289338.com | 08820011.com | www.d526688.com | 3033e.com | www.js520123.com | www.881805.com | y33.com | www.pj56h.com | t82365.com | www.4996jl.com | 1654nn.com | www.992254.com | www.196702.com | www.8494s.com | www.073wy.com | 4647y.com | www.1115.com | 01234cc.com | www.bwinyz04.com | YHvip.vip | www.67810.com | 44992007.com | www.pj8790.com | www.649677.com | 8294u.com | www.3479m.com | 97799p.com | www.337756.com | cc3336.com | www.9988kcd.com | 34200222.com | www.4996jy.com | bo667766.com | www.8967x.com | 338050.com | www.2y246.com | by301.com | www.6718.cc | xinu10.com | www.vns77999.com | 18775m.com | www.jb11111.com | P35i.com | www.kj853.com | 188qq.com | www.weinisirenbet.com | c83b784w.com | www.hg131d.com | 3344blg.com | www.9611msc.com | 4052a.com | www.22c25.com | 0234e.com | www.89599h.com | pj00.com | www.07679t.com | 3890l.com | www.aa888.cc | 112e.net | www.77114z.com | 883399p.com | www.888xm.cc | www.6491l.com | www.540577.com | www.5500y8.com | 12562004.com | www.16602i.com | 7155268.com | www.bwinyz29.com | 3482r.com | www.ribo88.com | 30174444.com | www.022gz.cc | www.yinhe9507a.com | www.210927.com | www.vn2028.com | 234907.com | www.356062.com | wx99.com | www.ra7909.com | sha161.com | www.2632y.com | www.vns8895.com | www.153880.com | www.755789.com | 777hg.com | www.3032jj.com | 33112007.com | www.2109b.com | www.110022.com | 79964f.com | www.fh70.com | bwin8f.com | www.3978e.com | www.68666l.com | www.215880.com | www.55555yh.com | 55331aa.com | www.3zq01.com | bwin8006.com | www.196705.com | www.178tyc.com | 553322KK.com | www.78949w.com | www.yl999.com | www.137518.com | www.099923.com | 3467g.com | www.9155c.cc | www.505msc.com | 28824.com | www.2234.cc | 55545d.com | www.640505.com | www.hg8423.com | 4778822.com | www.55228b.com | www.40818e.com | 2934e.com | www.883399i.com | u3144.com | www.84499a.com | www.4963hh.com | 496mm.com | www.876297.com | www.60486b.com | 73055r.com | www.84499k.com | www.9068oo.com | hhgz1155.com | www.2418k.com | www.76060r.com | 32212j.com | www.7240f.com | www.789.lfjdg.top | 5443u.com | www.4102r.com | hgw168x.com | bb888811.com | www.qiji333.com | www.86611h.com | zhcp28.com | www.55676o.com | www.wn2018.com | 6220jj.com | www.6888779.com | www.hg3598.com | cgg666666.com | www.99788f.com | www.3844qq.com | 92958b.com | www.08czj.com | www.blr730.com | shenlaoxi.com | www.602952.com | www.163a66.com | 2649i.com | www.33598v.com | www.1117795.com | www.6033l.com | www35.vip | www.904027.com | www.122yh.com | 3559gggg.com | www.619477.com | www.06617a.com | 7792w.com | P35s.com | www.52072t.com | www.hg6613.com | 063939.com | www.240689.com | www.79095n.com | www.682833.com | 28824m.com | www.fo64.com | www.ebo666.com | 88770.cc | vns365c.com | www.97171.cc | www.5966hhh.com | 131r.net | 87665k.com | www.36166y.com | www.5457.com | g47479.com | 1591002.com | www.6611d.cc | www.5360.com | ms38648kk.cc | 77229193.com | www.901833.com | www.89677o.com | www.76060z.com | 8988gg.com | www.370833.com | www.js66988.com | www.kelake99.com | 88442007.com | 3258h.com | www.15355j.com | www.zgr111.com | ms777.com | qycps08.com | www.550637.com | www.yz7701.com | www.31567.ag | 8449ss.com | 123456nn.cc | www.055v.cc | www.biying900vip.com | www.vns994.cc | 22098t.com | 0139365.com | www.51515m.com | www.rgcp15.com | www.js66666.com | 2147008.com | h61653.com | www.ttcp88.co | www.4963ss.com | www.bj0011.com | k40033.com | 5004fff.com | www.qmkl2.com | www.xj6005.com | www.78787a.com | 2613s.com | 3679rr.com | www.215889.com | www.84499k.com | www.5099bb.com | www.36677c.com | ylzz2228.com | 916.com | www.401273.com | www.dy826.com | www.809014.com | www.88c07.com | wdlywz.com | 5443q.com | www.312522.com | www.61233b.com | www.4996qh.com | www.feilipu4.com | 2820w.com | 23599l.com | 555777.net | www.684244.com | www.ampj3435.com | www.5446ww.com | www.5590777.com | 131v.net | 4165x.com | b01311.com | www.196095.com | www.7793k.com | www.xpj13666.com | www.587002.com | www.8694b.com | 9995c.cc | 7599w.com | jixiang10.com | www.198109.com | www.4444y.cc | www.9971007.com | www.50999x.com | www.101vns.com | www.8520j.com | 3726m.com | j2489.com | pujingzaixianyule.com | www.629053.com | www.50080n.com | www.yz7701.com | www.99113o.com | www.57798c.com | www.yinhecc33.com | 7744www.com | 4556y.com | 29918v.com | tt6823.com | www.161763.com | www.881799a.com | www.81520t.com | www.86339b.com | www.9737mm.com | www.jinniu00.com | www.2078v.com | www.a3a555.cc | 2649o.com | 01234ll.com | 8438000.com | f2850.com | 497944.com | www.178379.com | www.767884.com | www.3126s.com | www.00889.com | www.5446l.com | www.sha2266.com | www.c44aa.com | www.111153.com | www.yh8489.com | www.bet3652019.com | 1592m.com | hcp5588.com | hg7088.com | 35222uu.com | l66474l.com | o86811com | 7am.vip | feicai0454.com | 皇冠在线娱乐.com | 7196hh.com | www.196050.com | www.548611.com | www.825799.com | www.qml0.com | www.2632s.com | www.16588t.com | www.73166g.com | www.702056.com | www.954321t.com | www.hr1822.com | www.3w222.com | www.3032ii.com | www.06820n.com | www.8883hj.com | www.sscb55.cc | www.e9478.com | www.cfcp222.com | www.3cq3.com | www.hjcp9.com | www.e4042.com | www.jsc33.com | www.js5599.vip | www.hjcp33.com | www.3846q.com | www.6a444.com | www.1343a.com | www.bd2019z.com | www.ee55826.com | www.4694y.com | www.5981k.com | www.a30666.vip | www.bwinyz05.com | www.u80288.com | www.4521z.com | www.0622iii.com | www.256095.com | www.x55789.com | www.bj6666.com | www.86267g.com | www.89338.com | www.33czt.com | www.959553.com | www.820155.com | www.501251.com | www.215602.com | js22991.com | 06006w.com | 3778mm.com | 88894a.com | hwcp2222.com | z40033.com | 0906vip3.com | feicai0972.com | www.hg44401.com | www.3643k.com | www.hx3386.com | www.5506132.com | www.5049x.com | www.88vn777.com | www.e848.com | www.xb99.com | www.07163l.com | www.989774.com | www.356036.com | 2190n.com | wlb111.com | 65561177.com | 0048633.com | 56988s.com | www.7435a.com | www.p7890.com | www.lqz.hk | www.xpj778899.com | www.pj55713.com | www.hg09.com | www.hx1161.com | www.303204.com | 00115003.com | 2359911.com | hr1833.com | 8148999.com | www.76543u.com | www.790808.com | www.3157b.com | www.3398788.com | www.39957b.com | www.655061.com | 22oo8332.com | hr466.com | 0099227.com | 29918pp.com | www.pj8491.com | www.5504.com | www.yh888a.com | www.77803r.com | www.645880.com | feicai0520.com | 33.sb | 44hh.bet | www.7893w43.com | www.qazjw.com | www.97994f.com | www.00840z.com | www.367137.com | 6868ww.cc | 38848.com | 339449.net | www.10050856.com | www.4694x.com | www.6482x.com | www.878466.com | 668cp99.com | 23300c.com | 2934.com | www.sun09.com | www.4694f.com | www.588904.com | www.610330.com | j15.com | s61788.com | www.8124s.com | www.js600000.com | www.59999.am | www.909965.com | 77662007.com | 38738733.com | www.pj567.com | www.060sunbet.com | www.js28666.com | www.845755.com | 9964i.com | 80850s.com | www.vns9938.com | www.77fh.com | www.12455v.com | www.196095.com | youleseo.com | 7779m.cc | www.63208.com | www.11599059.com | www.708330.com | 9566257.com | 33ff940.com | www.vns1668.cc | www.556939.com | www.c5963.com | 1590.com | qq365u.com | www.071095.com | www.8905l.com | www.785579.com | 7634245.xyz | 3379e.com | www.3589333.com | www.4331u.com | www.330271.com | tt6828.com | www.876445.com | www.yh8878a.com | www.12455j.com | 87965qq.com | 67877c.com | www.js44444.com | www.pj55714.com | www.550435.com | 4060uu.com | 1775bb.com | www.d58.com | www.2350d.com | 118888m.com | gg3189.com | www.789msc.com | www.80065h.com | xg5111.com | w88zhan.com | www.7777hx.com | www.0601t.com | 22117b.com | 33382ll.com | www.88993s.com | www.vip5888.com | 55331t.com | 7377.com | www.868681.com | www.91233y.com | 2189000.com | zhcptt.com | www.18765.com | www.6364y.com | 81366u.com | www.6033f.com | www.5hg6668.com | www.775703.com | 6002x.com | www.777444b.com | www.blr1144.com | www.91233u.com | 878365nn.com | www.hg8145.com | www.00840z.com | 918.com | www.6491m.com | www.5719c.com | www.258029.com | v5429.com | www.yh95678.com | www.2934o.com | 3189e.com | www.55268qq.com | www.5856862.com | 50000997.com | 2381ll.com | www.4996dq.com | www.301823.com | ems6368.com | www.jy738.com | www.965666.com | 4590t.com | www.pj5504.com | www.8667n.com | 1483b.com | www.330099g.com | www.bet73e.com | 390390.com | www.sun008.cc | www.3775z.com | l8159.cc | www.wns8838.com | www.4102g.com | 4912.com | www.1869f.com | www.xpj3133.cc | 500000593.com | 5hg7788.com | www.3421h.com | v3144.com | www.qian00.com | www.8911333.com | 131y.net | www.546993.com | www.3775x.com | 0860x.com | www.hg6339.com | www.ya2019b.com | 70766622.com | www.js6667.com | www.60108g.com | 6002p.com | www.77958.com | www.zcwf9.com | 4426234.com | www.mhcpgw.com | www.344242.com | 9030r.com | www.js89u.vip | 20188j.com | www.888999yh.com | www.737202.com | 2190b.com | www.722168.com | www.715699.com | 8988r.com | www.2021p.com | 63305z.com | www.4625m.com | www.07163f.com | i01234.com | www.sscb77.com | www.50052o.com | www.js222js.com | www.66136000.com | 7989pcom | www.858448.com | www.458178.com | 2649dd.com | www.4520033.com | yidali.com | www.tk533.com | www.504377.com | 2595j.com | www.37377x.com | 500000737.com | www.m39955.com | www.196776.com | www.6033b.com | www.550021j.com | 3775s.com | www.8886hj.com | 9411ggg.com | www.57799vv.com | www.872198.com | feicai0414.com | www.17jsc.com | 88p88.vip | www.cp0016.com | 2616.com | www.09569x.com | www.960923.com | 025352.com | www.531089.com | 4508ll.com | www.5981w.com | 1259i.com | www.06263.com | www.168212.com | www.x32126.cc | www.09czj.com | x86.com | www.525409.com | 3534e.com | www.573512.com | pj87.com | www.874858.com | 11betbet365.com | www.xpj9259.com | www.202812.com | www.588778.com | www.745557.com | www.9464770.com |