{"id":21899,"date":"2025-04-22T23:36:52","date_gmt":"2025-04-22T16:36:52","guid":{"rendered":"https:\/\/thaipropertynews.com\/feeds\/?p=21899"},"modified":"2025-04-22T23:36:52","modified_gmt":"2025-04-22T16:36:52","slug":"tencentdb-tdstore-online-ddl-technological-evolution-and-innovations-background-challenges","status":"publish","type":"post","link":"https:\/\/thaipropertynews.com\/feeds\/?p=21899","title":{"rendered":"TencentDB TDStore Online DDL: Technological Evolution and Innovations Background &amp; Challenges"},"content":{"rendered":"<p><span class=\"legendSpanClass\"><span class=\"xn-location\">SHENZHEN, China<\/span><\/span>, <span class=\"legendSpanClass\"><span class=\"xn-chron\">April 22, 2025<\/span><\/span> \/PRNewswire\/ &#8212; Traditional single-node databases (e.g., MySQL) use OnlineDDL and third-party tools (e.g., pt-osc) to enable lock-free schema changes, but face performance bottlenecks and struggle in distributed environments. <span class=\"xn-money\">Tencent<\/span> Cloud&#8217;s\u00a0<b>TDStore<\/b>, a financial-grade distributed database, addresses these challenges with groundbreaking innovations:<\/p>\n<p><b>Core Technological Innovations<\/b><\/p>\n<p><b>1.\u00a0 Multi-Version Schema Mechanism<\/b><\/p>\n<p class=\"prnml40\">a. Introduces\u00a0<b>schema versioning<\/b>\u00a0to enable\u00a0<b>metadata-only modifications in seconds<\/b> (e.g., adding trailing columns, extending fields). Historical data automatically fills default values, ensuring backward compatibility.<\/p>\n<p><b>2.\u00a0 Concurrency Control &amp; State Transition<\/b><\/p>\n<p class=\"prnml40\">a.\u00a0 <b>Thomas Write Rule<\/b>: Reduces transaction conflicts by ignoring stale writes, improving\u00a0DDL-DML\u00a0parallelism.<br \/>b.\u00a0 <b>Google F1 Phased State Design<\/b>: Divides\u00a0DDL into three stages (<b>delete-only<\/b>\u00a0\u2192\u00a0<b>write-only<\/b>\u00a0\u2192\u00a0<b>final<\/b>)\u00a0 to ensure global consistency and smooth transitions.<\/p>\n<p><b>3. Write Fence Mechanism<\/b><\/p>\n<p class=\"prnml40\">a. Validates request versions at the storage layer, allowing writes only between adjacent states to eliminate data inconsistency risks.<\/p>\n<p><b>4.\u00a0 Fast OnlineDDL Acceleration<\/b><\/p>\n<p class=\"prnml40\">a.\u00a0<b> Distributed Parallel Backfilling<\/b>: Splits data into SST files for multi-node parallel ingestion via\u00a0<b>bulk load<\/b>, bypassing timestamp comparisons to achieve\u00a0<b>13x performance gains<\/b> (10 minutes vs. 2.3 hours).<\/p>\n<p><b>Practices &amp; Optimizations<\/b><\/p>\n<p>1. <b>Performance Comparison<\/b><\/p>\n<p class=\"prnml40\">a. <b>Traditional Mode<\/b> (single-node): 16 threads took 2.3 hours.<br \/>b.\u00a0 <b>Fast Mode<\/b> (multi-node): 48 threads completed in 10 minutes, showcasing significant efficiency improvements.<\/p>\n<p>2. <b>Partitioning Best Practices<\/b><\/p>\n<p class=\"prnml40\">a. <b>Large Tables<\/b>: Use\u00a0<b>HASH\/KEY<\/b> partitioning to distribute data evenly, enabling parallel\u00a0DDL execution.<br \/>b. <b>Cold\/Hot Separation<\/b>: Combine\u00a0<b>RANGE+HASH<\/b> secondary partitioning for rapid cleanup and elastic scaling.<br \/>c. <b>High Concurrency<\/b>: Align partition keys with frequent query fields; set partition count as multiples of node numbers.<\/p>\n<p>3.\u00a0 <b>Key Parameter Configuration<\/b><\/p>\n<p class=\"prnml40\">a. <b>max_parallel_ddl_degree<\/b>: Increase parallel threads (\u2264 total node CPUs).<br \/>b.\u00a0<b>tdsql_ddl_fillback_mode<\/b>:\u00a0 Enable <b>IngestBehind<\/b> mode to unlock multi-node parallel acceleration.<\/p>\n<p><b>Business Value &amp; Future Roadmap<\/b><\/p>\n<ul type=\"disc\">\n<li><b>Validated Use Cases<\/b>: Achieved zero downtime in\u00a0<b>PB-scale financial systems<\/b>, with\u00a0<b>10x faster execution<\/b>\u00a0than third-party tools.<\/li>\n<li><b>Upcoming Enhancements<\/b>:\n<ul type=\"disc\">\n<li>Optimize partitioned table <b>Copy Table<\/b> and index\u00a0backfilling for ordinary tables.<\/li>\n<li>Support <b>ultra-large-scale (tens of TB)<\/b> workloads and hybrid\u00a0HTAP architectures.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><b>Conclusion<\/b><\/p>\n<p>TDStore overcomes traditional OnlineDDL limitations through distributed architecture innovations and engineering practices, delivering\u00a0<b>high-performance, secure, and seamless schema change capabilities<\/b> for financial-grade scenarios. It empowers enterprises to tackle massive data challenges effectively.<\/p>\n<p><b>#DistributedDatabase #TencentCloud #TencentDB #TDSQL #Tencent Cloud BigData<\/b><\/p>","protected":false},"excerpt":{"rendered":"<p><!-- wp:html --><\/p>\n<p><span class=\"legendSpanClass\"><span class=\"xn-location\">SHENZHEN, China<\/span><\/span>, <span class=\"legendSpanClass\"><span class=\"xn-chron\">April 22, 2025<\/span><\/span> \/PRNewswire\/ &#8212; Traditional single-node databases (e.g., MySQL) use OnlineDDL and third-party tools (e.g., pt-osc) to enable lock-free schema changes, but face performance bottlenecks and struggle in distributed environments. <span class=\"xn-money\">Tencent<\/span> Cloud&#8217;s\u00a0<b>TDStore<\/b>, a financial-grade distributed database, addresses these challenges with groundbreaking innovations:<\/p>\n<p><b>Core Technological Innovations<\/b><\/p>\n<p><b>1.\u00a0 Multi-Version Schema Mechanism<\/b><\/p>\n<p class=\"prnml40\">a. Introduces\u00a0<b>schema versioning<\/b>\u00a0to enable\u00a0<b>metadata-only modifications in seconds<\/b> (e.g., adding trailing columns, extending fields). Historical data automatically fills default values, ensuring backward compatibility.<\/p>\n<p><b>2.\u00a0 Concurrency Control &amp; State Transition<\/b><\/p>\n<p class=\"prnml40\">a.\u00a0 <b>Thomas Write Rule<\/b>: Reduces transaction conflicts by ignoring stale writes, improving\u00a0DDL-DML\u00a0parallelism.<br \/>b.\u00a0 <b>Google F1 Phased State Design<\/b>: Divides\u00a0DDL into three stages (<b>delete-only<\/b>\u00a0\u2192\u00a0<b>write-only<\/b>\u00a0\u2192\u00a0<b>final<\/b>)\u00a0 to ensure global consistency and smooth transitions.<\/p>\n<p><b>3. Write Fence Mechanism<\/b><\/p>\n<p class=\"prnml40\">a. Validates request versions at the storage layer, allowing writes only between adjacent states to eliminate data inconsistency risks.<\/p>\n<p><b>4.\u00a0 Fast OnlineDDL Acceleration<\/b><\/p>\n<p class=\"prnml40\">a.\u00a0<b> Distributed Parallel Backfilling<\/b>: Splits data into SST files for multi-node parallel ingestion via\u00a0<b>bulk load<\/b>, bypassing timestamp comparisons to achieve\u00a0<b>13x performance gains<\/b> (10 minutes vs. 2.3 hours).<\/p>\n<p><b>Practices &amp; Optimizations<\/b><\/p>\n<p>1. <b>Performance Comparison<\/b><\/p>\n<p class=\"prnml40\">a. <b>Traditional Mode<\/b> (single-node): 16 threads took 2.3 hours.<br \/>b.\u00a0 <b>Fast Mode<\/b> (multi-node): 48 threads completed in 10 minutes, showcasing significant efficiency improvements.<\/p>\n<p>2. <b>Partitioning Best Practices<\/b><\/p>\n<p class=\"prnml40\">a. <b>Large Tables<\/b>: Use\u00a0<b>HASH\/KEY<\/b> partitioning to distribute data evenly, enabling parallel\u00a0DDL execution.<br \/>b. <b>Cold\/Hot Separation<\/b>: Combine\u00a0<b>RANGE+HASH<\/b> secondary partitioning for rapid cleanup and elastic scaling.<br \/>c. <b>High Concurrency<\/b>: Align partition keys with frequent query fields; set partition count as multiples of node numbers.<\/p>\n<p>3.\u00a0 <b>Key Parameter Configuration<\/b><\/p>\n<p class=\"prnml40\">a. <b>max_parallel_ddl_degree<\/b>: Increase parallel threads (\u2264 total node CPUs).<br \/>b.\u00a0<b>tdsql_ddl_fillback_mode<\/b>:\u00a0 Enable <b>IngestBehind<\/b> mode to unlock multi-node parallel acceleration.<\/p>\n<p><b>Business Value &amp; Future Roadmap<\/b><\/p>\n<ul type=\"disc\">\n<li><b>Validated Use Cases<\/b>: Achieved zero downtime in\u00a0<b>PB-scale financial systems<\/b>, with\u00a0<b>10x faster execution<\/b>\u00a0than third-party tools.<\/li>\n<li><b>Upcoming Enhancements<\/b>:\n<ul type=\"disc\">\n<li>Optimize partitioned table <b>Copy Table<\/b> and index\u00a0backfilling for ordinary tables.<\/li>\n<li>Support <b>ultra-large-scale (tens of TB)<\/b> workloads and hybrid\u00a0HTAP architectures.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><b>Conclusion<\/b><\/p>\n<p>TDStore overcomes traditional OnlineDDL limitations through distributed architecture innovations and engineering practices, delivering\u00a0<b>high-performance, secure, and seamless schema change capabilities<\/b> for financial-grade scenarios. It empowers enterprises to tackle massive data challenges effectively.<\/p>\n<p><b>#DistributedDatabase #TencentCloud #TencentDB #TDSQL #Tencent Cloud BigData<\/b><\/p>\n<p><!-- \/wp:html --><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"initial","rop_publish_now_accounts":[],"rop_publish_now_history":[],"rop_publish_now_status":"pending","footnotes":""},"categories":[5,7],"tags":[],"class_list":["post-21899","post","type-post","status-publish","format-standard","hentry","category-cision-pr-newswire","category-cision-pr-newswire-en"],"_links":{"self":[{"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=\/wp\/v2\/posts\/21899","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=21899"}],"version-history":[{"count":0,"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=\/wp\/v2\/posts\/21899\/revisions"}],"wp:attachment":[{"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=21899"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=21899"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thaipropertynews.com\/feeds\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=21899"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}