Conversation
Multiple `TableMap` events can happen inside of a single transaction, so we can't really use them to filter out events early or anything like that. Instead, we'll just skip them. Also adds some basic test cases for the transaction streaming.
This reverts commit 2e78f6f.
* Remove error return value since we don't use it. * Lock the mutex whenever we plan to update the low watermark to avoid a race condition. * Check for data races in our unit tests. * Still return an error from ProcessEventsUntilDrained but actually check it in our code. * Make coordinator_test.go to check the err from ProcessEventsUntilDrained again * Remove unreachable return in ProcessEventsUntilDrained
…ark (#1531) * Notify waiting channels on completed transaction, not just the watermark. * Add checksum validation to coordinator test * Use errgroup to perform transactions concurrently in coordinator_test.go * Configure concurrency separate from total number of transactions. * Run similar number of txs to previous test and ignore context. * Have at least 1 child in a transaction. * Notify waiting channels for the current sequence number.
|
Despite promising performance results in testing, we stopped developing this branch since Nov 2024 after running into intermittent data inconsistency problems in internal replica tests. I believe I've tracked down the source of this issue. Below is the investigation for anyone interested. InvestigationThe data inconsistency appeared intermittently on several different ghost testing replicas running the MTR version. The test table looks like this: CREATE TABLE `sbtest1` (
`id` int NOT NULL AUTO_INCREMENT,
`k` int NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_1` (`k`)
);The testing result is checksum mismatch (usually only one row). In this case the row with -- 2025-04-12T03:28:44.209273Z 25 Execute
BEGIN;
DELETE FROM sbtest1 WHERE id=5025;
INSERT INTO sbtest1 (id, k, c, pad) VALUES (5025, 5046, '55585975399-51936995975-90609908571-88981758242-41639509045-49015163211-63909390173-09873895014-17528416149-59787710722', '90699347551-90936038435-69760642136-45340328341-67205199431');
COMMIT;
-- 2025-04-12T03:28:44.209695Z 28 Execute
UPDATE sbtest1 SET k=k+1 WHERE id=5025;And corresponding binlog events on the replica So, the correct value is We can look at the dependency (sub)graph for the original transactions on graph LR;
89058--> 89053;
89053--> 89050;
89065--> 89062;
89062--> 89053;
This means once transaction Comparing to what the MySQL replication applier coordinator does (sql/rpl_rli_pbd.cc), I realized that a transaction should be scheduled if and only if The In the example, FixIn our Coordinator, the culprit is this line in gh-ost/go/logic/coordinator.go Lines 482 to 484 in ffef446 In the example it allowed 89065 to be applied after 89062, but before 89058 completed.
After removing these lines the sysbench localtest is consistently passing. |
|
Hi @meiji163, my team has been running into the exact limits you describe in this PR, so we were very excited to see the progress already made here. However, over the last 2 weeks I've run some tests with this PR and still notice a data inconsistency issue. We use gh-ost on AWS Aurora, and my testing has been with a production-like setup. On the table we're trying to migrate the only interesting queries (not SELECTs) are fairly simple: we insert rows and update some of them fairly quickly to remove a timestamp value. The inconsistency I see is caused by some UPDATEs not being applied to the ghost table. This happens infrequently but consistently throughout the tests (50 rows affected out of ~2 million). It doesn't quite look like the same issue you ran into and described above because the dependency graph looked fairly linear: Do you have any ideas what could be the issue, and is there anything else I can provide? Thank you! |
|
Thanks for reporting this @TomKnaepen, I also ran into more data consistency issues after the latest changes, but I haven't had time to investigate further. We thought it might be related to the binlogsyncer connection being closed and reopened. Is there any error in your gh-ost output? |
|
I don't recall seeing any errors at the time; I ran some new tests on a smaller scale this week to see if I could find anything but there's no errors or anything noteworthy in the gh-ost output. |
|
Hi @meiji163 @arthurschreiber @danieljoos @hugodorea @mhamza15 @TomKnaepen 👋 We have used GitHub Copilot CLI with 2 collaborating agents (Claude Opus 4.6 "Sherlock" and GPT 5.5 "Otter") to investigate and fix the data inconsistency issue. They found the issue and a PR with the fix can be found at: dnovitski#1 Root Cause AnalysisThe Core Bug: Binlog Rotation Resets Sequence Numbers But Coordinator State Is Never ResetMySQL's logical clock ( After rotation, all Concrete example from our debugging: This is exactly what the user observed: two transactions affecting The dependency graph showed that once their respective Additional Bugs Found and FixedBug 2 — Silent Error Swallowing in DML Apply: Fix: Retry InnoDB deadlocks (error 1213) and lock wait timeouts (error 1205) with jittered exponential backoff (up to 100 retries, matching MySQL's Bug 3 — Wait Channel Deadlock on Error Paths: Fix: Use buffered channels (capacity 1) so the notification send never blocks even if no one is listening. Bug 4 — Data Race on lwm Read: Fix: Guard the read with Note: Pre-existing Bug in
|
| Configuration | Total Migration Time | DML Events/s (during load) | Row-copy starvation period |
|---|---|---|---|
| Baseline (no MTR, single-threaded) | 166s | 1,463/s | ~150s (0 rows copied during active load!) |
| MTR, 4 workers | 135s | 2,049/s | ~120s |
| MTR, 4 workers, batch=50 | 150s | 2,156/s | ~135s |
Key findings:
- MTR provides ~19% improvement in total migration time with 4 workers
- DML processing throughput increases by ~40% (2,049 vs 1,463 events/s)
- The fundamental bottleneck remains
executeWriteFuncswhich callsProcessEventsUntilDrained()before each row-copy chunk — under high write load, the event queue fills continuously and row-copy gets starved. MTR helps by draining the queue faster with parallel workers, giving row-copy more turns to execute. - InnoDB deadlocks from parallel workers are expected and handled correctly with retry+backoff (gap locks on secondary indexes cause contention between parallel
REPLACE INTO/DELETE FROMoperations on the ghost table)
8 participants
Description
This PR introduces multi-threaded replication for applying DML queries to the ghost table. The goal is to be able to migrate tables with high rate of DML queries (e.g. >5k rows/s). Currently gh-ost lags behind in these situations, taking a very long time to complete or not completing at all.
Similar to MySQL replication threads, gh-ost will stream binlog events from the source and group them into transactions. It then submits the transactions to a pool of workers to apply the transactions concurrently on the ghost table. We ensure that dependent transactions are applied in a consistent order (equivalent to MySQL multi-threaded replication with
replica_parallel_type=LOGICAL_CLOCKandreplica_preserve_commit_order=0).With
WRITESETenabled on the source, this enables a great amount of parallelism in the transaction applier.Changes
TODO
Performance tests
TODO
script/cibuildreturns with no formatting errors, build errors or unit test errors.