[{"data":1,"prerenderedAt":1443},["ShallowReactive",2],{"article-\u002Farticles\u002Fkafka-broadcast-ordering-k8s-scaling":3},{"id":4,"title":5,"body":6,"date":1429,"description":1430,"excerpt":1431,"extension":1432,"meta":1433,"navigation":529,"path":1434,"seo":1435,"stem":1436,"tags":1437,"__hash__":1442},"articles\u002Farticles\u002Fkafka-broadcast-ordering-k8s-scaling.md","Kafka 實務注意事項：廣播怎麼做、順序怎麼保、group.id 怎麼取，以及 K8s 自動擴展為什麼沒有變快",{"type":7,"value":8,"toc":1405},"minimark",[9,28,44,49,56,66,71,77,87,97,101,108,121,130,243,246,280,284,291,294,356,359,363,370,401,405,411,422,428,629,640,643,646,660,667,671,681,685,690,694,703,718,726,730,744,748,762,765,805,809,815,818,836,846,849,872,875,880,883,888,902,1088,1106,1111,1161,1352,1356,1363,1366,1389,1392,1401],[10,11,12,13,18,19,27],"p",{},"Kafka 的基本元件可以先參考 ",[14,15,17],"a",{"href":16},"\u002Farticles\u002Fkafka-topic-partition-consumer-group","Kafka 完整入門","。這篇只講一件事背後的各種變化：",[20,21,22,26],"strong",{},[23,24,25],"code",{},"group.id"," 與 Partition 的分配規則","。幾乎所有 Kafka 在應用上的坑，最後都會回到這兩條規則：",[29,30,31,38],"ol",{},[32,33,34,37],"li",{},[20,35,36],{},"同一個 Group 內，一個 Partition 只會分給一個 Consumer","（分工）",[32,39,40,43],{},[20,41,42],{},"不同的 Group 之間互不影響，各自讀到完整的資料","（各讀一份）",[45,46,48],"h2",{"id":47},"一廣播讓每個人都收到同一則訊息","一、廣播：讓每個人都收到同一則訊息",[10,50,51],{},[52,53],"img",{"alt":54,"src":55},"Kafka 三種消費模式：模式一，同一個服務的三個 Pod 使用同一個 group.id order-sync，三個 Partition 各分給一個 Pod，每則訊息只被處理一次，是分工；模式二，同步服務、通知服務、稽核服務使用各自不同的 group.id，每個服務都收到全部訊息，是服務之間的廣播；模式三，Web 服務的每個 Pod 使用含 Pod 名稱的獨立 group.id，例如 cache-web-7f9c-a、cache-web-7f9c-b，每個 Pod 都收到全部訊息，用來清除每個實例的本機快取","\u002Fimages\u002Farticles\u002Fkafka-broadcast.svg",[10,57,58,59,65],{},"Kafka 沒有像 RabbitMQ Fanout Exchange 那樣「廣播」的設定，",[20,60,61,62,64],{},"廣播是用 ",[23,63,25],{}," 做出來的","。依「誰要收到」分成兩種：",[67,68,70],"h3",{"id":69},"不同的服務都要收到各自用不同的-groupid","不同的服務都要收到：各自用不同的 group.id",[10,72,73,74,76],{},"訂單建立後，同步服務要寫進搜尋引擎、通知服務要發推播、稽核服務要留紀錄。三個服務讀同一個 Topic，各自使用自己的 ",[23,75,25],{},"：",[78,79,85],"pre",{"className":80,"code":82,"language":83,"meta":84},[81],"language-text","order-sync      → 3 個 Pod 分工處理\norder-notify    → 2 個 Pod 分工處理\norder-audit     → 1 個 Pod\n","text","",[23,86,82],{"__ignoreMap":84},[10,88,89,90,93,94,96],{},"每個服務都讀到完整的訊息，服務內的多個 Pod 再彼此分工。",[20,91,92],{},"Producer 完全不用知道有誰在消費","，之後多一個服務，只要用新的 ",[23,95,25],{}," 訂閱就好。",[67,98,100],{"id":99},"同一個服務的每個-pod-都要收到每個-pod-一個-groupid","同一個服務的每個 Pod 都要收到：每個 Pod 一個 group.id",[10,102,103,104,107],{},"有些事情是",[20,105,106],{},"每個實例都要做","的，例如：",[109,110,111,118],"ul",{},[32,112,113,114,117],{},"物件資料更新，每個 Web Pod 的",[20,115,116],{},"本機記憶體快取","都要清掉",[32,119,120],{},"使用者透過 WebSocket 連在不同的 Pod 上，推播訊息要讓每個 Pod 都收到，再由連著那個使用者的 Pod 送出去",[10,122,123,124,126,127,76],{},"這時候如果全部 Pod 共用一個 ",[23,125,25],{},"，一則訊息只會被其中一個 Pod 收到，其他 Pod 的快取就永遠是舊的。做法是讓",[20,128,129],{},"每個 Pod 有自己的 group.id",[78,131,135],{"className":132,"code":133,"language":134,"meta":84,"style":84},"language-csharp shiki shiki-themes github-dark","var config = new ConsumerConfig\n{\n    BootstrapServers = \"kafka:9092\",\n    \u002F\u002F 在 K8s 裡 MachineName 就是 Pod 名稱，每個 Pod 自成一個 Group\n    GroupId = $\"cache-invalidate-{Environment.MachineName}\",\n    \u002F\u002F 新 Pod 只需要處理之後的事件，不用從頭讀\n    AutoOffsetReset = AutoOffsetReset.Latest\n};\n","csharp",[23,136,137,159,166,182,189,220,226,237],{"__ignoreMap":84},[138,139,142,146,150,153,156],"span",{"class":140,"line":141},"line",1,[138,143,145],{"class":144},"snl16","var",[138,147,149],{"class":148},"svObZ"," config",[138,151,152],{"class":144}," =",[138,154,155],{"class":144}," new",[138,157,158],{"class":148}," ConsumerConfig\n",[138,160,162],{"class":140,"line":161},2,[138,163,165],{"class":164},"s95oV","{\n",[138,167,169,172,175,179],{"class":140,"line":168},3,[138,170,171],{"class":164},"    BootstrapServers ",[138,173,174],{"class":144},"=",[138,176,178],{"class":177},"sU2Wk"," \"kafka:9092\"",[138,180,181],{"class":164},",\n",[138,183,185],{"class":140,"line":184},4,[138,186,188],{"class":187},"sAwPA","    \u002F\u002F 在 K8s 裡 MachineName 就是 Pod 名稱，每個 Pod 自成一個 Group\n",[138,190,192,195,197,200,203,206,209,212,215,218],{"class":140,"line":191},5,[138,193,194],{"class":164},"    GroupId ",[138,196,174],{"class":144},[138,198,199],{"class":177}," $\"cache-invalidate-",[138,201,202],{"class":177},"{",[138,204,205],{"class":164},"Environment",[138,207,208],{"class":177},".",[138,210,211],{"class":164},"MachineName",[138,213,214],{"class":177},"}",[138,216,217],{"class":177},"\"",[138,219,181],{"class":164},[138,221,223],{"class":140,"line":222},6,[138,224,225],{"class":187},"    \u002F\u002F 新 Pod 只需要處理之後的事件，不用從頭讀\n",[138,227,229,232,234],{"class":140,"line":228},7,[138,230,231],{"class":164},"    AutoOffsetReset ",[138,233,174],{"class":144},[138,235,236],{"class":164}," AutoOffsetReset.Latest\n",[138,238,240],{"class":140,"line":239},8,[138,241,242],{"class":164},"};\n",[10,244,245],{},"要注意的地方：",[109,247,248,262,273],{},[32,249,250,253,254,257,258,261],{},[20,251,252],{},"每次部署都會產生新的 Group","：Pod 名稱每次都不同，舊的 Group 雖然沒人用了，但它的 Offset 紀錄會留在 Kafka，直到 ",[23,255,256],{},"offsets.retention.minutes","（預設 7 天）後才清除。功能上沒有影響，但",[20,259,260],{},"監控 Consumer Lag 的儀表板會出現一堆停止消費、Lag 一直增加的 Group","，告警規則要排除這個前綴，不然會一直誤報",[32,263,264,265,268,269,272],{},"另一種做法是",[20,266,267],{},"不加入 Group","：直接用 ",[23,270,271],{},"Assign"," 指定讀取所有 Partition、從尾端開始讀、不提交 Offset。沒有 Group 就沒有 Rebalance，也不會留下多餘的 Offset 紀錄，但新增 Partition 時要自己處理",[32,274,275,276,279],{},"這種用法本質上是「",[20,277,278],{},"收到就做、漏了也沒關係","」：Pod 重啟的那幾秒內發出的訊息它收不到。如果漏掉會出問題，快取一定要再加上 TTL 當作保險",[45,281,283],{"id":282},"二按順序推送key-只是第一步","二、按順序推送：Key 只是第一步",[10,285,286,287,290],{},"順序是 Kafka 最常被誤解的地方。「用帳號 ID 當 Key」只保證了",[20,288,289],{},"同一個帳號的訊息進同一個 Partition","，但從 Producer 到 Consumer 處理完，中間還有好幾個地方會把順序打亂。",[67,292,293],{"id":293},"先決定需要哪一種順序",[295,296,297,313],"table",{},[298,299,300],"thead",{},[301,302,303,307,310],"tr",{},[304,305,306],"th",{},"需求",[304,308,309],{},"做法",[304,311,312],{},"代價",[314,315,316,331,345],"tbody",{},[301,317,318,322,328],{},[319,320,321],"td",{},"全部訊息嚴格依序",[319,323,324,325],{},"Topic 只開 ",[20,326,327],{},"1 個 Partition",[319,329,330],{},"整個 Topic 只能有一個 Consumer 在處理，吞吐量上限很低",[301,332,333,336,342],{},[319,334,335],{},"同一個實體的訊息依序",[319,337,338,339],{},"以實體 ID（帳號、訂單）當 ",[20,340,341],{},"Key",[319,343,344],{},"不同實體之間沒有順序，這通常也不需要",[301,346,347,350,353],{},[319,348,349],{},"不需要順序",[319,351,352],{},"不設 Key",[319,354,355],{},"吞吐量最好",[10,357,358],{},"真正需要「全域順序」的情況其實很少，大部分需求仔細想都是「同一個帳號、同一筆訂單內要有序」。",[67,360,362],{"id":361},"producer-端重試可能讓順序對調","Producer 端：重試可能讓順序對調",[10,364,365,366,369],{},"Producer 會同時送出多個批次（",[23,367,368],{},"max.in.flight.requests.per.connection","，預設 5）。如果批次 1 失敗重試、批次 2 先成功了，寫進 Partition 的順序就變成 2、1。",[109,371,372,385,391],{},[32,373,374,381,382],{},[20,375,376,377,380],{},"開啟冪等（",[23,378,379],{},"enable.idempotence=true","）","：Broker 會依序號檢查，在 in-flight 不超過 5 的情況下仍然保證順序。Java Client 預設開啟，",[20,383,384],{},".NET 的 Confluent.Kafka 預設是關的，要自己打開",[32,386,387,388,390],{},"如果因為某些原因不能開冪等，就要把 ",[23,389,368],{}," 設成 1，代價是吞吐量下降",[32,392,393,396,397,400],{},[20,394,395],{},"應用程式自己也不能亂","：同一個帳號的事件如果是從多個執行緒同時呼叫 ",[23,398,399],{},"Produce","，送出的順序本身就不確定，Kafka 也救不了",[67,402,404],{"id":403},"consumer-端最常見的錯誤是丟進執行緒池","Consumer 端：最常見的錯誤是丟進執行緒池",[10,406,407],{},[52,408],{"alt":409,"src":410},"Kafka Consumer 端順序比較：錯誤做法是 Consumer 從 Partition 依序取出帳號 A001 的下單、改單、取消三則訊息後，丟進執行緒池平行處理，取消可能比下單先完成，順序被打亂；正確做法是依 Key 的雜湊分配到固定的處理通道，A001 的三則訊息都進通道 1 依序處理，B002 的訊息在通道 2 平行處理，同一個 Key 有序，不同 Key 平行","\u002Fimages\u002Farticles\u002Fkafka-ordering.svg",[10,412,413,414,417,418,421],{},"Partition 數量有限，單一 Consumer 處理太慢時，很自然會想「拉回來之後丟給 ",[23,415,416],{},"Task.Run"," 平行處理」。但這樣一來，同一個帳號的「下單」和「取消」被兩個執行緒同時處理，",[20,419,420],{},"誰先完成完全看運氣","，前面所有保證順序的努力全部白費。",[10,423,424,425,76],{},"需要在 Consumer 內部加速時，要",[20,426,427],{},"依 Key 分流",[78,429,431],{"className":132,"code":430,"language":134,"meta":84,"style":84},"\u002F\u002F 8 條處理通道，同一個 Key 永遠進同一條，通道內依序處理\nvar lanes = Enumerable.Range(0, 8)\n    .Select(_ => Channel.CreateBounded\u003CConsumeResult\u003Cstring, string>>(100))\n    .ToArray();\n\nwhile (!ct.IsCancellationRequested)\n{\n    var result = consumer.Consume(ct);\n    var lane = (int)((uint)result.Message.Key.GetHashCode() % (uint)lanes.Length);\n    await lanes[lane].Writer.WriteAsync(result, ct);\n}\n",[23,432,433,438,469,515,525,531,545,549,568,608,623],{"__ignoreMap":84},[138,434,435],{"class":140,"line":141},[138,436,437],{"class":187},"\u002F\u002F 8 條處理通道，同一個 Key 永遠進同一條，通道內依序處理\n",[138,439,440,442,445,447,450,453,456,460,463,466],{"class":140,"line":161},[138,441,145],{"class":144},[138,443,444],{"class":148}," lanes",[138,446,152],{"class":144},[138,448,449],{"class":164}," Enumerable.",[138,451,452],{"class":148},"Range",[138,454,455],{"class":164},"(",[138,457,459],{"class":458},"sDLfK","0",[138,461,462],{"class":164},", ",[138,464,465],{"class":458},"8",[138,467,468],{"class":164},")\n",[138,470,471,474,477,479,482,485,488,491,494,497,499,502,504,506,509,512],{"class":140,"line":168},[138,472,473],{"class":164},"    .",[138,475,476],{"class":148},"Select",[138,478,455],{"class":164},[138,480,481],{"class":148},"_",[138,483,484],{"class":144}," =>",[138,486,487],{"class":164}," Channel.",[138,489,490],{"class":148},"CreateBounded",[138,492,493],{"class":164},"\u003C",[138,495,496],{"class":148},"ConsumeResult",[138,498,493],{"class":164},[138,500,501],{"class":144},"string",[138,503,462],{"class":164},[138,505,501],{"class":144},[138,507,508],{"class":164},">>(",[138,510,511],{"class":458},"100",[138,513,514],{"class":164},"))\n",[138,516,517,519,522],{"class":140,"line":184},[138,518,473],{"class":164},[138,520,521],{"class":148},"ToArray",[138,523,524],{"class":164},"();\n",[138,526,527],{"class":140,"line":191},[138,528,530],{"emptyLinePlaceholder":529},true,"\n",[138,532,533,536,539,542],{"class":140,"line":222},[138,534,535],{"class":144},"while",[138,537,538],{"class":164}," (",[138,540,541],{"class":144},"!",[138,543,544],{"class":164},"ct.IsCancellationRequested)\n",[138,546,547],{"class":140,"line":228},[138,548,165],{"class":164},[138,550,551,554,557,559,562,565],{"class":140,"line":239},[138,552,553],{"class":144},"    var",[138,555,556],{"class":148}," result",[138,558,152],{"class":144},[138,560,561],{"class":164}," consumer.",[138,563,564],{"class":148},"Consume",[138,566,567],{"class":164},"(ct);\n",[138,569,571,573,576,578,580,583,586,589,592,595,598,601,603,605],{"class":140,"line":570},9,[138,572,553],{"class":144},[138,574,575],{"class":148}," lane",[138,577,152],{"class":144},[138,579,538],{"class":164},[138,581,582],{"class":144},"int",[138,584,585],{"class":164},")((",[138,587,588],{"class":144},"uint",[138,590,591],{"class":164},")result.Message.Key.",[138,593,594],{"class":148},"GetHashCode",[138,596,597],{"class":164},"() ",[138,599,600],{"class":144},"%",[138,602,538],{"class":164},[138,604,588],{"class":144},[138,606,607],{"class":164},")lanes.Length);\n",[138,609,611,614,617,620],{"class":140,"line":610},10,[138,612,613],{"class":144},"    await",[138,615,616],{"class":164}," lanes[lane].Writer.",[138,618,619],{"class":148},"WriteAsync",[138,621,622],{"class":164},"(result, ct);\n",[138,624,626],{"class":140,"line":625},11,[138,627,628],{"class":164},"}\n",[10,630,631,632,635,636,639],{},"這樣同一個帳號依序處理、不同帳號平行處理。但要注意 ",[20,633,634],{},"Offset 不能隨便提交","：通道 1 處理到 Offset 105、通道 2 才處理到 98，這時只能提交 98，否則當掉重啟後 99～104 就漏掉了。這部分容易寫錯，先確認",[20,637,638],{},"增加 Partition 與 Consumer"," 能不能解決，再考慮自己做。",[67,641,642],{"id":642},"失敗重試會卡住後面所有訊息",[10,644,645],{},"有順序的 Partition 裡，一則訊息處理失敗時只有兩個選擇：",[109,647,648,654],{},[32,649,650,653],{},[20,651,652],{},"原地重試","：後面的訊息全部等它，這個 Partition 停住，Lag 一直上升（Head-of-line Blocking）",[32,655,656,659],{},[20,657,658],{},"丟到另一個重試 Topic、先處理後面的","：Partition 不會卡住，但這個帳號的順序就被打亂了",[10,661,662,663,666],{},"沒有兩全其美的做法。交易類的事件通常選擇",[20,664,665],{},"原地重試、加上退避時間，並且一定要對 Lag 告警","，讓工程師盡快介入；如果業務允許，也可以只把「同一個 Key 後續的訊息」一起暫存起來，其他 Key 繼續處理，但實作複雜度高很多。",[67,668,670],{"id":669},"增加-partition-會打亂-key-的對應","增加 Partition 會打亂 Key 的對應",[10,672,673,676,677,680],{},[23,674,675],{},"hash(key) % N"," 的 N 改變後，同一個帳號會被分到不同的 Partition。擴充的那一刻，新 Partition 上的新訊息可能比舊 Partition 上還沒處理完的舊訊息先被處理。順序敏感的 Topic ",[20,678,679],{},"一開始就要開足 Partition","，非擴充不可時，要先停止寫入、等 Consumer 把 Lag 消化完再擴。",[45,682,684],{"id":683},"三相同-groupid-會發生什麼事","三、相同 group.id 會發生什麼事",[10,686,687,689],{},[23,688,25],{}," 只是一個字串，Kafka 不會檢查「這兩個 Consumer 是不是同一個服務」。只要字串一樣，就會被當成同一個 Group 一起分工。這帶來幾個常見事故：",[67,691,693],{"id":692},"兩個不同的服務用了同一個-groupid","兩個不同的服務用了同一個 group.id",[10,695,696,697,699,700,76],{},"通知服務是從同步服務複製過來改的，設定檔裡的 ",[23,698,25],{}," 忘了改。兩個服務都叫 ",[23,701,702],{},"order-sync",[109,704,705,711],{},[32,706,707,708],{},"6 個 Partition 被分給兩個服務的所有 Pod，",[20,709,710],{},"同步服務只拿到一部分 Partition，通知服務拿到另一部分",[32,712,713,714,717],{},"結果就是搜尋引擎少了一部分訂單、一部分使用者沒收到通知，而且",[20,715,716],{},"沒有任何錯誤訊息","，兩邊的 Log 看起來都正常",[10,719,720,721,725],{},"這和 RabbitMQ 中兩個服務共用同一個 Queue 是完全一樣的問題（參考 ",[14,722,724],{"href":723},"\u002Farticles\u002Frabbitmq-exchange-queue-binding","RabbitMQ 的 Exchange、Queue、Binding","）。",[67,727,729],{"id":728},"本機開發連到正式環境的-kafka","本機開發連到正式環境的 Kafka",[10,731,732,733,736,737,739,740,743],{},"工程師在自己的電腦上 Debug，",[23,734,735],{},"appsettings.Development.json"," 不小心指到了正式環境的 Kafka，",[23,738,25],{}," 又和正式環境一樣。",[20,741,742],{},"這台筆電就加入了正式環境的 Group，分走了幾個 Partition","：正式環境的部分訂單被筆電上的程式「處理」掉了，Offset 也被提交，正式環境的 Pod 再也不會處理這些訊息。",[67,745,747],{"id":746},"同一個-groupid-訂閱不同的-topic","同一個 group.id 訂閱不同的 Topic",[10,749,750,751,754,755,757,758,761],{},"服務 A 用 ",[23,752,753],{},"group-x"," 訂閱 Topic 1、服務 B 也用 ",[23,756,753],{}," 訂閱 Topic 2。雖然讀的資料不重疊，但它們是同一個 Group：",[20,759,760],{},"服務 A 部署、重啟時觸發的 Rebalance，也會讓服務 B 一起暫停","。",[67,763,764],{"id":764},"怎麼避免",[109,766,767,787,792,798],{},[32,768,769,771,772,775,776,779,780,783,784],{},[23,770,25],{}," 命名固定格式：",[23,773,774],{},"{服務名稱}-{用途}","，例如 ",[23,777,778],{},"order-sync-es","、",[23,781,782],{},"order-notify-push","，",[20,785,786],{},"一個服務的一個用途就是一個 Group",[32,788,789,791],{},[23,790,25],{}," 不放在可以隨意複製的共用設定檔裡，而是和服務名稱綁在一起產生",[32,793,794,795],{},"正式環境的 Kafka 開啟驗證與 ACL，",[20,796,797],{},"開發環境的帳號根本沒有權限加入正式環境的 Group",[32,799,800,801,804],{},"定期用 ",[23,802,803],{},"kafka-consumer-groups.sh --describe"," 檢查每個 Group 的成員，看到不認識的 Client ID 或主機名稱就要追查",[45,806,808],{"id":807},"四k8s-自動擴展pod-變多了為什麼沒有變快","四、K8s 自動擴展：Pod 變多了，為什麼沒有變快？",[10,810,811],{},[52,812],{"alt":813,"src":814},"Kafka Consumer 在 K8s 自動擴展的比較：左邊用 CPU 指標的 HPA，Topic 有 6 個 Partition，HPA 把 Consumer 擴到 12 個 Pod，只有 6 個分到 Partition，另外 6 個閒置，平均 CPU 因此被拉低，HPA 又縮回去，每次擴縮都觸發 Rebalance，消費反而更慢；右邊用 KEDA 依 Consumer Lag 擴展，最大副本數上限等於 Partition 數 6，Lag 超過門檻才擴、消化完再縮，每個 Pod 都有 Partition 可以處理","\u002Fimages\u002Farticles\u002Fkafka-k8s-autoscaling.svg",[10,816,817],{},"這是一個很容易發生的情境：",[109,819,820,830,833],{},[32,821,822,823,826,827],{},"Topic ",[23,824,825],{},"order-events"," 有 ",[20,828,829],{},"6 個 Partition",[32,831,832],{},"Consumer 服務部署在 K8s，設定了 HPA：CPU 使用率 70%、最少 3 個、最多 20 個 Pod",[32,834,835],{},"某天流量暴增，Lag 一路上升，HPA 也如預期把 Pod 擴到了 12 個",[10,837,838,839,842,843,761],{},"但 Lag 完全沒有下降得比較快。原因就是第一條規則：",[20,840,841],{},"一個 Partition 只能分給 Group 內的一個 Consumer","。6 個 Partition 最多只有 6 個 Pod 在工作，",[20,844,845],{},"另外 6 個 Pod 只是佔著資源、什麼都沒做",[10,847,848],{},"從監控上看起來是「有 12 個 Pod 在扛流量，很 HA」，實際上一半的 Pod 是閒置的。而且情況還會更糟：",[109,850,851,857,866],{},[32,852,853,856],{},[20,854,855],{},"閒置的 Pod 把平均 CPU 拉低","：6 個工作中的 Pod 都滿載，加上 6 個閒置的，平均 CPU 只有一半，HPA 判斷「不需要這麼多」又開始縮減",[32,858,859,862,863],{},[20,860,861],{},"每次擴展、縮減都會觸發 Rebalance","：Pod 加入或離開 Group，Partition 就要重新分配，分配期間相關的 Partition 停止消費。HPA 來回擴縮，Group 就一直在 Rebalance，",[20,864,865],{},"實際消費的時間反而變少",[32,867,868,871],{},[20,869,870],{},"CPU 可能根本不是瓶頸","：如果 Consumer 慢是因為寫資料庫很慢，Pod 都在等 I\u002FO，CPU 很低，HPA 完全不會擴展，Lag 卻一直累積",[67,873,874],{"id":874},"正確的做法",[10,876,877],{},[20,878,879],{},"1. 最大副本數不要超過 Partition 數",[10,881,882],{},"這是最基本的一條。想要更多 Consumer 平行處理，要先增加 Partition（並考慮前面講的順序問題），而不是只調 HPA 的上限。",[10,884,885],{},[20,886,887],{},"2. 依 Consumer Lag 擴展，而不是 CPU",[10,889,890,891,894,895,901],{},"Consumer 該不該擴展，真正的指標是「",[20,892,893],{},"還有多少訊息沒處理","」。用 ",[14,896,900],{"href":897,"rel":898},"https:\u002F\u002Fkeda.sh",[899],"nofollow","KEDA"," 的 Kafka Scaler，可以直接依 Group 的 Lag 擴展：",[78,903,907],{"className":904,"code":905,"language":906,"meta":84,"style":84},"language-yaml shiki shiki-themes github-dark","apiVersion: keda.sh\u002Fv1alpha1\nkind: ScaledObject\nmetadata:\n  name: order-consumer\nspec:\n  scaleTargetRef:\n    name: order-consumer          # 對應的 Deployment\n  minReplicaCount: 2\n  maxReplicaCount: 6              # 不超過 Partition 數\n  cooldownPeriod: 300             # Lag 消化完後等 5 分鐘再縮，避免來回擴縮\n  triggers:\n    - type: kafka\n      metadata:\n        bootstrapServers: kafka:9092\n        consumerGroup: order-processor\n        topic: order-events\n        lagThreshold: \"500\"       # 平均每個 Pod 負責 500 筆 Lag\n","yaml",[23,908,909,921,931,939,949,956,963,976,986,999,1012,1019,1033,1041,1052,1063,1074],{"__ignoreMap":84},[138,910,911,915,918],{"class":140,"line":141},[138,912,914],{"class":913},"s4JwU","apiVersion",[138,916,917],{"class":164},": ",[138,919,920],{"class":177},"keda.sh\u002Fv1alpha1\n",[138,922,923,926,928],{"class":140,"line":161},[138,924,925],{"class":913},"kind",[138,927,917],{"class":164},[138,929,930],{"class":177},"ScaledObject\n",[138,932,933,936],{"class":140,"line":168},[138,934,935],{"class":913},"metadata",[138,937,938],{"class":164},":\n",[138,940,941,944,946],{"class":140,"line":184},[138,942,943],{"class":913},"  name",[138,945,917],{"class":164},[138,947,948],{"class":177},"order-consumer\n",[138,950,951,954],{"class":140,"line":191},[138,952,953],{"class":913},"spec",[138,955,938],{"class":164},[138,957,958,961],{"class":140,"line":222},[138,959,960],{"class":913},"  scaleTargetRef",[138,962,938],{"class":164},[138,964,965,968,970,973],{"class":140,"line":228},[138,966,967],{"class":913},"    name",[138,969,917],{"class":164},[138,971,972],{"class":177},"order-consumer",[138,974,975],{"class":187},"          # 對應的 Deployment\n",[138,977,978,981,983],{"class":140,"line":239},[138,979,980],{"class":913},"  minReplicaCount",[138,982,917],{"class":164},[138,984,985],{"class":458},"2\n",[138,987,988,991,993,996],{"class":140,"line":570},[138,989,990],{"class":913},"  maxReplicaCount",[138,992,917],{"class":164},[138,994,995],{"class":458},"6",[138,997,998],{"class":187},"              # 不超過 Partition 數\n",[138,1000,1001,1004,1006,1009],{"class":140,"line":610},[138,1002,1003],{"class":913},"  cooldownPeriod",[138,1005,917],{"class":164},[138,1007,1008],{"class":458},"300",[138,1010,1011],{"class":187},"             # Lag 消化完後等 5 分鐘再縮，避免來回擴縮\n",[138,1013,1014,1017],{"class":140,"line":625},[138,1015,1016],{"class":913},"  triggers",[138,1018,938],{"class":164},[138,1020,1022,1025,1028,1030],{"class":140,"line":1021},12,[138,1023,1024],{"class":164},"    - ",[138,1026,1027],{"class":913},"type",[138,1029,917],{"class":164},[138,1031,1032],{"class":177},"kafka\n",[138,1034,1036,1039],{"class":140,"line":1035},13,[138,1037,1038],{"class":913},"      metadata",[138,1040,938],{"class":164},[138,1042,1044,1047,1049],{"class":140,"line":1043},14,[138,1045,1046],{"class":913},"        bootstrapServers",[138,1048,917],{"class":164},[138,1050,1051],{"class":177},"kafka:9092\n",[138,1053,1055,1058,1060],{"class":140,"line":1054},15,[138,1056,1057],{"class":913},"        consumerGroup",[138,1059,917],{"class":164},[138,1061,1062],{"class":177},"order-processor\n",[138,1064,1066,1069,1071],{"class":140,"line":1065},16,[138,1067,1068],{"class":913},"        topic",[138,1070,917],{"class":164},[138,1072,1073],{"class":177},"order-events\n",[138,1075,1077,1080,1082,1085],{"class":140,"line":1076},17,[138,1078,1079],{"class":913},"        lagThreshold",[138,1081,917],{"class":164},[138,1083,1084],{"class":177},"\"500\"",[138,1086,1087],{"class":187},"       # 平均每個 Pod 負責 500 筆 Lag\n",[10,1089,1090,1091,1094,1095,1101,1102,1105],{},"KEDA 的 Kafka Scaler 有一個 ",[23,1092,1093],{},"allowIdleConsumers"," 設定，",[20,1096,1097,1098],{},"預設是 ",[23,1099,1100],{},"false","，代表算出來的副本數不會超過 Partition 數，即使 ",[23,1103,1104],{},"maxReplicaCount"," 設得比較大也一樣，等於內建了第一條規則。",[10,1107,1108],{},[20,1109,1110],{},"3. 減少 Rebalance 的衝擊",[109,1112,1113,1122,1140],{},[32,1114,1115,76,1118,1121],{},[20,1116,1117],{},"使用漸進式 Rebalance",[23,1119,1120],{},"partition.assignment.strategy=cooperative-sticky","，Pod 加入或離開時只搬動需要移動的 Partition，其他 Pod 照常消費",[32,1123,1124,1127,1128,1131,1132,1135,1136,1139],{},[20,1125,1126],{},"優雅關閉","：K8s 送出 SIGTERM 時，Consumer 要主動呼叫 ",[23,1129,1130],{},"Close()"," 離開 Group 並提交 Offset。否則 Group 要等 ",[23,1133,1134],{},"session.timeout.ms","（預設 45 秒）才發現這個 Pod 不見了，這段時間它負責的 Partition 完全沒人處理。",[23,1137,1138],{},"terminationGracePeriodSeconds"," 也要大於處理一批訊息的時間",[32,1141,1142,1145,1146,1149,1150,1153,1154,1156,1157,1160],{},[20,1143,1144],{},"Static Membership","：設定 ",[23,1147,1148],{},"group.instance.id","（.NET 是 ",[23,1151,1152],{},"GroupInstanceId","）給每個 Consumer 一個固定身分，搭配 StatefulSet 固定的 Pod 名稱。Pod 重啟只要在 ",[23,1155,1134],{}," 內回來，就拿回原本的 Partition，",[20,1158,1159],{},"完全不會觸發 Rebalance","，很適合滾動更新頻繁的服務",[78,1162,1164],{"className":132,"code":1163,"language":134,"meta":84,"style":84},"protected override Task ExecuteAsync(CancellationToken stoppingToken) => Task.Run(() =>\n{\n    _consumer.Subscribe(\"order-events\");\n    try\n    {\n        while (!stoppingToken.IsCancellationRequested)\n        {\n            var result = _consumer.Consume(stoppingToken);\n            Handle(result);\n            _consumer.Commit(result);\n        }\n    }\n    catch (OperationCanceledException) { }\n    finally\n    {\n        \u002F\u002F 收到 SIGTERM 時主動離開 Group，讓 Partition 立刻交給其他 Pod，而不是等 45 秒逾時\n        \u002F\u002F Consumer 不是執行緒安全的，Close 要在同一個執行緒呼叫\n        _consumer.Close();\n    }\n}, stoppingToken);\n",[23,1165,1166,1206,1210,1226,1231,1236,1248,1253,1270,1278,1288,1293,1298,1311,1316,1320,1325,1330,1341,1346],{"__ignoreMap":84},[138,1167,1168,1171,1174,1177,1180,1182,1185,1188,1191,1194,1197,1200,1203],{"class":140,"line":141},[138,1169,1170],{"class":144},"protected",[138,1172,1173],{"class":144}," override",[138,1175,1176],{"class":148}," Task",[138,1178,1179],{"class":148}," ExecuteAsync",[138,1181,455],{"class":164},[138,1183,1184],{"class":148},"CancellationToken",[138,1186,1187],{"class":148}," stoppingToken",[138,1189,1190],{"class":164},") ",[138,1192,1193],{"class":144},"=>",[138,1195,1196],{"class":164}," Task.",[138,1198,1199],{"class":148},"Run",[138,1201,1202],{"class":164},"(() ",[138,1204,1205],{"class":144},"=>\n",[138,1207,1208],{"class":140,"line":161},[138,1209,165],{"class":164},[138,1211,1212,1215,1218,1220,1223],{"class":140,"line":168},[138,1213,1214],{"class":164},"    _consumer.",[138,1216,1217],{"class":148},"Subscribe",[138,1219,455],{"class":164},[138,1221,1222],{"class":177},"\"order-events\"",[138,1224,1225],{"class":164},");\n",[138,1227,1228],{"class":140,"line":184},[138,1229,1230],{"class":144},"    try\n",[138,1232,1233],{"class":140,"line":191},[138,1234,1235],{"class":164},"    {\n",[138,1237,1238,1241,1243,1245],{"class":140,"line":222},[138,1239,1240],{"class":144},"        while",[138,1242,538],{"class":164},[138,1244,541],{"class":144},[138,1246,1247],{"class":164},"stoppingToken.IsCancellationRequested)\n",[138,1249,1250],{"class":140,"line":228},[138,1251,1252],{"class":164},"        {\n",[138,1254,1255,1258,1260,1262,1265,1267],{"class":140,"line":239},[138,1256,1257],{"class":144},"            var",[138,1259,556],{"class":148},[138,1261,152],{"class":144},[138,1263,1264],{"class":164}," _consumer.",[138,1266,564],{"class":148},[138,1268,1269],{"class":164},"(stoppingToken);\n",[138,1271,1272,1275],{"class":140,"line":570},[138,1273,1274],{"class":148},"            Handle",[138,1276,1277],{"class":164},"(result);\n",[138,1279,1280,1283,1286],{"class":140,"line":610},[138,1281,1282],{"class":164},"            _consumer.",[138,1284,1285],{"class":148},"Commit",[138,1287,1277],{"class":164},[138,1289,1290],{"class":140,"line":625},[138,1291,1292],{"class":164},"        }\n",[138,1294,1295],{"class":140,"line":1021},[138,1296,1297],{"class":164},"    }\n",[138,1299,1300,1303,1305,1308],{"class":140,"line":1035},[138,1301,1302],{"class":144},"    catch",[138,1304,538],{"class":164},[138,1306,1307],{"class":148},"OperationCanceledException",[138,1309,1310],{"class":164},") { }\n",[138,1312,1313],{"class":140,"line":1043},[138,1314,1315],{"class":144},"    finally\n",[138,1317,1318],{"class":140,"line":1054},[138,1319,1235],{"class":164},[138,1321,1322],{"class":140,"line":1065},[138,1323,1324],{"class":187},"        \u002F\u002F 收到 SIGTERM 時主動離開 Group，讓 Partition 立刻交給其他 Pod，而不是等 45 秒逾時\n",[138,1326,1327],{"class":140,"line":1076},[138,1328,1329],{"class":187},"        \u002F\u002F Consumer 不是執行緒安全的，Close 要在同一個執行緒呼叫\n",[138,1331,1333,1336,1339],{"class":140,"line":1332},18,[138,1334,1335],{"class":164},"        _consumer.",[138,1337,1338],{"class":148},"Close",[138,1340,524],{"class":164},[138,1342,1344],{"class":140,"line":1343},19,[138,1345,1297],{"class":164},[138,1347,1349],{"class":140,"line":1348},20,[138,1350,1351],{"class":164},"}, stoppingToken);\n",[67,1353,1355],{"id":1354},"那-ha-呢","那 HA 呢？",[10,1357,1358,1359,1362],{},"多開幾個閒置的 Pod 當「備援」，其實沒有想像中有用：Kafka 的容錯本來就是透過 Rebalance，",[20,1360,1361],{},"任何一個還活著的 Consumer 都能接手掛掉的那個的 Partition","，不需要專門待命的 Pod。",[10,1364,1365],{},"Consumer 的高可用要做的是：",[109,1367,1368,1383,1386],{},[32,1369,1370,1371,1374,1375,1378,1379,1382],{},"至少 ",[20,1372,1373],{},"2 個以上","的副本，而且用 ",[23,1376,1377],{},"topologySpreadConstraints"," 或 Pod Anti-Affinity ",[20,1380,1381],{},"分散在不同的 Node","，避免一台 Node 掛掉就全部消失",[32,1384,1385],{},"副本數不超過 Partition 數，每個 Pod 都在工作",[32,1387,1388],{},"做好優雅關閉，讓計畫中的重啟能立刻交接，而不是等逾時",[45,1390,1391],{"id":1391},"一句話總結",[10,1393,1394,1395,1397,1398,1400],{},"廣播靠的是不同的 ",[23,1396,25],{},"，服務之間各用一個，需要每個 Pod 都收到就每個 Pod 一個；順序要從 Key、Producer 的冪等、Consumer 不丟進執行緒池一路顧到底；",[23,1399,25],{}," 一樣就會被當成同一組分工，命名與權限都要管好；而 Consumer 在 K8s 上擴展，副本數超過 Partition 數只會多出閒置的 Pod 和更多的 Rebalance，要依 Lag 擴展、上限對齊 Partition 數，才是真的有在做事的擴展。",[1402,1403,1404],"style",{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .s4JwU, html code.shiki .s4JwU{--shiki-default:#85E89D}",{"title":84,"searchDepth":161,"depth":161,"links":1406},[1407,1411,1418,1424,1428],{"id":47,"depth":161,"text":48,"children":1408},[1409,1410],{"id":69,"depth":168,"text":70},{"id":99,"depth":168,"text":100},{"id":282,"depth":161,"text":283,"children":1412},[1413,1414,1415,1416,1417],{"id":293,"depth":168,"text":293},{"id":361,"depth":168,"text":362},{"id":403,"depth":168,"text":404},{"id":642,"depth":168,"text":642},{"id":669,"depth":168,"text":670},{"id":683,"depth":161,"text":684,"children":1419},[1420,1421,1422,1423],{"id":692,"depth":168,"text":693},{"id":728,"depth":168,"text":729},{"id":746,"depth":168,"text":747},{"id":764,"depth":168,"text":764},{"id":807,"depth":161,"text":808,"children":1425},[1426,1427],{"id":874,"depth":168,"text":874},{"id":1354,"depth":168,"text":1355},{"id":1391,"depth":161,"text":1391},"2025-07-15","Kafka 的基本元件可以先參考 Kafka 完整入門。這篇只講一件事背後的各種變化：group.id 與 Partition 的分配規則。幾乎所有 Kafka 在應用上的坑，最後都會回到這兩條規則：",null,"md",{},"\u002Farticles\u002Fkafka-broadcast-ordering-k8s-scaling",{"title":5,"description":1430},"articles\u002Fkafka-broadcast-ordering-k8s-scaling",[1438,1439,900,1440,1441],"Kafka","Kubernetes","分散式系統","系統設計","zR2c786IQQHxXa2uP6UMmvX-5TSx3Lue-0pnzwOl8is",1792959150217]