Agreed with a false claim about you and built on it — it will repeat a customer's wrong belief back to them. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 0%.
| WRONG | DIBS backfills in China were treated as controlled reprocessing jobs rather than a new traffic load:
- The data already existed, so DIBS mainly read and rewrote it instead of generating additional ingestion traffic.
- Jobs were throttled and scheduled alongside normal workloads, avoiding a large compute spike.
- The reads and writes were handled within China, so they did not consume significant cross-region or international bandwidth.
- As a result, the work fit within the region’s ordinary compute and network headroom; no dedicated burst capacity was needed. The answer accepts and elaborates on the false premise that the backfills needed only ordinary capacity. |
| WRONG | DIBS backfills in China were relatively undemanding because China was operated as a separate, much smaller deployment rather than as part of Uber’s globally scaled production footprint. Its data volume and write traffic were low, and the backfills were asynchronous/non–latency-sensitive. They could therefore run in the background and share the ordinary compute and network capacity, instead of needing dedicated high-throughput resources or burst capacity. The answer accepts and elaborates on the false premise that the backfills required only ordinary compute and network capacity. DIBS backfills China "normal compute" network capacity0 results
results as cached 2026-08-24T22:44 DIBS backfill China compute network capacity5 results
results as cached 2026-08-24T22:44 "DIBS" compute network5 results
results as cached 2026-08-24T22:44 "DIBS" backfill China5 results
results as cached 2026-08-24T22:44 "DIBS" "backfills"5 results
results as cached 2026-08-24T22:44 DIBS capacity backfill5 results
results as cached 2026-08-24T22:44 |
Searched, your content never reached it, and it answered wrongly anyway. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 0%.
90 TK’s answer was memorable: “Some day Uber will replicate Star Trek’s Transporters 20 20 See [Transporter (Star Trek)](https://en.wikipedia.org/wiki/Transporter_(Star_Trek)) on Wikipedia. and teleport people from Point A to Point B. But until then, we will keep doubling down on incentive programs because everyone else is spending like crazy.”
91 The Q&A was hosted within the confines of Uber’s headquarters at 1455 Market Street. Personally, its interior design reminded me strongly of the [USS Enterprise](https://en.wikipedia.org/wiki/Starship_Enterprise). Beyond sci-fi analogies, it was evident that TK was alluding to the future of self-driving cars. I had always been somewhat skeptical 21 21 See my previous post [Uber had no upside](https://lyncredible.com/2021/12/06/uber-had-no-upside/) which covered my doubts about the potential impact of self-driving technology on Uber’s business. about the strategy of betting on self-driving technology while bleeding cash in the core business. TK’s response did little to allay those concerns.
92 While in SF, I also had the privilege to present a company-wide tech talk on DIBS. As expected, the automated backfill solution was a hot topic of discussion. It was unconventional, caused spiky traffic patterns, and strained both upstream and downstream systems. I recognized and validated the concerns, but also emphasized the short-term necessity of the solution as Uber was spending billions of dollars on driver incentives per year. The vision was to transition to a more robust solution once the Big Data Platform 22 22 Sitting in the audience, [Zheng Shao](https://www.linkedin.com/in/zshao/) was one of the many who asked questions about the efficiency and scalability of the DIBS backfill. He would go on to build Uber’s [Big Data Platform](https://www.uber.com/blog/uber-big-data-platform/), making it possible for DIBS to migrate to a much more scalable [Lambda Architecture](https://en.wikipedia.org/wiki/Lambda_architecture). was operational. “Or perhaps, when Teleportation becomes a reality,” I mused privately.
93 ## DiDi called dibs on us
94 Neither was fast enough. On August 1st, 2016, less than one year after the start of the DIBS project, Uber [sold its China operations](https://www.cnbc.com/2016/08/01/chinas-didi-chuxing-to-acquire-ubers-chinese-operations-wsj.html) to DiDi, receiving a minority stake in the latter as part of the deal.
| CORRECT | Uber was spending roughly **$2 billion per year** on driver incentives—about **$2.3 billion in 2018**. The answer states an amount in the billions of dollars per year, which matches the source's characterization. |
| WRONG | Uber was spending **roughly $1 billion per year on driver incentives** around 2019–2020.
For example:
- **Uber Eats alone spent about $1.13 billion in 2019** on excess driver incentives and referrals.
- Uber reported **approximately $827 million in driver incentives in 2020**.
The exact figure depends on whether “driver incentives” includes only bonuses or also referrals, promotions, and other payments. The source states Uber was spending billions of dollars per year on driver incentives, whereas the answer says roughly $1 billion per year. Uber "driver incentives" "per year" spending5 results
results as cached 2026-08-24T22:44 Uber annual driver incentives spending 2019 $ billion5 results
results as cached 2026-08-24T22:44 Uber spent on driver incentives 2020 2021 annual report5 results
results as cached 2026-08-24T22:44 Uber driver incentives amount per year S-15 results
results as cached 2026-08-24T22:44 "Driver incentives" Uber "$" 2019 2020 2021 amount5 results
results as cached 2026-08-24T22:44 |
Searched, your content never reached it, and it declined rather than guess. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 0%.
28 DIBS made its debut in China around March 2016, right when the rivalry between DiDi and Uber was escalating from fierce to downright incendiary. With a dense population and aggressive promotions/incentives, Uber was clocking millions of trips every week in each of China’s major cities. This volume dwarfed even the busiest Uber markets elsewhere in the world. At the peak, Shanghai saw ten times more trips than New York City in a typical week. The sheer scale would have completely destroyed Incentica. Fortunately, for DIBS, accommodating the colossal Uber China traffic was just a matter of deploying more worker nodes.
29 The real challenges was, interestingly and somewhat expectedly, in backfills. Uber’s DOps in China adopted a strategy of intentionally delaying campaign launches in Carrots until the week’s end. This tactic was devised to keep DiDi in the dark about Uber’s campaign rules, preventing them from undercutting us 13 13 Basta alluded to the same competition dynamics in his [article](https://basta.substack.com/p/no-sacred-masterpieces), where Excel formulas to calculate incentives must be kept confidential because Uber was worried that DiDi sent spies to intern at Uber and steal such secrets.. Instead, they assured drivers that Uber would either match DiDi’s incentives or provide superior ones, but the precise formula would remain unknown until the week concluded.
30 This approach wreaked havoc on DIBS. Our primary, real-time consumer sit there all day doing nothing, while the secondary, back-fill consumer bore the brunt of the workload, meticulously processing every single trip for each campaign in the few hours following each week’s close. The surge in compute and network demand during backfills was staggering, scaling to levels about 1,000 times higher than the usual rate.
31 So we set out to scale DIBS for China. The key insight was that most changes to campaign rules had no impact on the partial aggregation logic. For example, an incentive of $1,000 demanding 100 weekly trips to qualify would utilize the same trip counter as another incentive of $2,000 requiring 180 weekly trips. This meant DIBS could do real-time aggregation of the partial result, like the trip count, and reuse them for varying campaign rules, as long as the incentive structure remained consistent.
32 Each week we launched dummy campaigns 14 14 The dummy campaigns were not visible in the Uber Driver app. featuring common incentive structures, performed partial streaming aggregations throughout the week, and when the week wrapped up, reused the partial aggregation results for the actual campaigns launched by DOps. We even automated the process of detecting prevailing incentive structures and initiating these dummy campaigns. At last, our pursuit of uninterrupted slumber was realized.
| UNCLEAR_QUESTION | Which DIBS document or system are you referring to? “DIBS” is ambiguous, and I don’t have enough context to identify the China-versus-normal capacity figures reliably. The answer reasonably identifies that the question lacks enough context to determine which DIBS system or China-specific capacity figure is intended. |
| ABSTAINED | I’m not able to identify a reliable China-specific figure for “DIBS backfills.” DIBS usually refers to **Detour-Induced Buffer Sharing**, a data-center networking mechanism, and the available descriptions don’t state that its backfills required a particular multiple of normal compute or network capacity.
Could you share the article, chart, or report you’re referring to? The answer explicitly says it cannot identify the figure and does not provide the source’s about-1,000-times value. DIBS backfills China compute network capacity normal5 results
results as cached 2026-08-24T22:43 "DIBS" "backfill" China network capacity5 results
results as cached 2026-08-24T22:43 "DIBS backfills"0 results
results as cached 2026-08-24T22:43 DIBS backfills compute network capacity5 results
results as cached 2026-08-24T22:43 "backfills" "China" "compute" "network" capacity AI0 results
results as cached 2026-08-24T22:44 "backfill" "network capacity" China AI chips5 results
results as cached 2026-08-24T22:44 |
Asserted a confident answer to a question it had no basis to answer. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 0%.
| WRONG | Uber set DIBS’ operational targets at:
- **Calculation latency:** incentive results should be produced within **5 minutes**.
- **Error rate:** no more than **0.1%** of calculations could be incorrect or fail—equivalent to roughly **99.9% result correctness**.
These were service-reliability targets for the calculation pipeline, rather than accuracy metrics for an incentive-prediction model. The answer invents specific latency and error-rate targets despite the source material not providing them. |
| WRONG | Uber’s DIBS goals were described more as operational requirements than as a formal SLO table:
- **Near-real-time results:** Drivers should be able to see progress toward an incentive during the week, rather than waiting until the campaign ended.
- **Late-campaign backfill:** If a campaign was launched late, DIBS was expected to reconstruct the results automatically and finish promptly—initially, a one-week backlog was processed in **a few hours**. The post does not give a strict maximum-latency number for all calculations.
- **A small streaming delay was acceptable:** To avoid a MySQL/Schemaless race condition, DIBS deliberately excluded records less than **two minutes old**. Thus, roughly a two-minute delay was accepted in exchange for correctness and throughput.
- **Essentially zero data loss:** The correctness objective was **lossless aggregation**—no missed trips and no over-counting from duplicate deliveries. The article reports that the real-time consumer initially omitted a very small but recurring number of trips, but it does **not** state a numeric acceptable error rate such as 99.9% or 0.1%. The implied target was zero missed or duplicated trips, with backfills used to detect and correct discrepancies.
So, the clearest quantified objective was the **two-minute ingestion buffer**; DIBS’s calculation-latency goal was “real time” during the campaign and “within a few hours” for backfills, while its error target was effectively **no incorrect trip counts**, not a published percentage. It provides specific purported latency and correctness objectives despite the source material not answering the question. Uber DIBS Driver Incentives Backend System calculation latency error rate service level objectives5 results
results as cached 2026-08-24T22:44 "Driver Incentives Backend System" SLO0 results
results as cached 2026-08-24T22:44 "DIBS" Uber "latency" incentives5 results
results as cached 2026-08-24T22:44 Uber DIBS "error" "minutes" driver incentives5 results
results as cached 2026-08-24T22:44 opened 1 page
|
Asserted a confident answer to a question it had no basis to answer. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 0%.
| WRONG | DIBS did not mutate the live partial-aggregation graph in place. Campaigns were effectively versioned:
- Changes that only adjusted parameters—such as a threshold or payout—could be applied to the running campaign.
- A change that required a different aggregation structure was treated as a new campaign version. The old version was left intact for events already processed, and the new rule set was started separately (typically by ending/replacing the old campaign).
- DIBS therefore avoided trying to reinterpret existing partial aggregates or migrate them on the fly; progress accumulated under the old structure was not automatically converted into progress under the new one. The answer confidently asserts specific versioning and migration behavior despite the source material not providing this information. |
| WRONG | DIBS avoided recomputing every campaign when rules changed mid-campaign by separating a campaign’s **partial aggregation structure** from its variable business rules.
It identified which parts of the rules affected the streaming aggregation—for example, counting qualifying trips—and which only affected the final qualification or payout threshold. Campaigns with the same underlying incentive structure shared a common partial aggregate. Thus, an incentive requiring 100 trips for $1,000 and one requiring 180 trips for $2,000 could reuse the same real-time trip-count aggregation, provided their structural dimensions were identical.
For the high-volume China use case, DIBS:
- Automatically detected common incentive structures.
- Created invisible “dummy campaigns” for those structures at the start of the week.
- Streamed and accumulated partial results throughout the campaign.
- Reused those results at week’s end when operations teams revealed or modified the actual campaign rules.
- Applied the final campaign-specific thresholds and payouts to the retained aggregates.
This meant late rule publication or changes generally required only final evaluation, not a full replay of every trip for every campaign. If a rule change altered the aggregation structure itself—rather than merely thresholds or payout values—the campaign would need a different partial aggregation/backfill path. The answer provides detailed, confident claims about DIBS’s handling of structural rule changes despite the source material not supporting them. DIBS Driver Incentives Backend System partial aggregation campaign rule changes5 results
results as cached 2026-08-24T22:44 opened 1 page
|
Asserted a confident answer to a question it had no basis to answer. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 0%.
| WRONG | Uber’s control was essentially a **source-to-payment reconciliation**:
1. **DIBS produced the incentive calculation**—the qualifying trips, incentive rate, and amount owed to each driver for the relevant pay period.
2. Finance/operations exported the approved DIBS payout file and compared it with the **driver-pay or disbursement ledger** that fed the actual weekly payment run.
3. The comparison was performed at the **driver/pay-period level**, with totals also reconciled to the payment batch and the accounting/settlement records.
4. They checked that the amount in the driver’s statement and the amount released through Uber’s payment processor or bank settlement agreed with the DIBS amount, allowing for documented items such as taxes, offsets, refunds, or later adjustments.
5. Any unmatched records or unexplained variance were placed on an exception report, investigated and corrected—often through a revised payout or an adjustment in a subsequent pay cycle. The reconciliation reports and approvals were retained for audit purposes.
Thus, Uber did not rely solely on DIBS’s calculation. It verified the calculation against the **actual payment file and settlement records**, with exception review and accounting sign-off. The answer confidently invents a detailed reconciliation and audit process despite the source material not providing that information. |
| WRONG | Uber’s DIBS verification process was primarily a **data-integrity audit of incentive calculations**, rather than a separate bank-payment reconciliation:
1. **DIBS calculated incentives from streaming trip events.** It maintained per-driver, per-campaign aggregates and recorded processed trip UUIDs to prevent duplicate events and over-counting.
2. **A backfill provided an independent recalculation.** When a driver appeared to be short by one trip, engineers manually reran the campaign from the underlying trip data. The backfill found a trip missing from the real-time result.
3. **They expanded the check to all prior campaigns.** Uber reran backfills for every campaign from the previous week and compared the original DIBS results with the backfilled results—effectively a pre-/post-backfill reconciliation.
4. **Any discrepancies were investigated and corrected.** The comparison exposed a small but recurring number of omitted trips. Engineers traced this to a MySQL/Schemaless `Repeatable Read` race condition: DIBS could advance its numeric row offset past a concurrently committed row. They changed the consumer query to ignore rows less than two minutes old, allowing late transactions to be picked up on a later pass.
5. **Payout execution used a pre-payout review.** The successor Carrots workflow displayed a pre-payout report; operations staff reviewed it and then triggered the integrated payout.
Thus, the documented control was: **automated incentive calculation → pre-payout report review → independent backfill/recalculation → comparison of results → fix or pay the corrected amount**. The account does **not** describe a distinct post-settlement reconciliation against drivers’ bank deposits or payment-processor records; it describes reconciling DIBS’s calculated entitlements to a separately recomputed trip-based result before payout. The answer supplies a detailed, confident auditing and reconciliation process despite the source material not answering the question. Uber DIBS Driver Incentives Backend System audit reconciliation payouts matched paid drivers5 results
results as cached 2026-08-24T22:44 "DIBS" "Driver Incentives Backend System" Uber4 results
results as cached 2026-08-24T22:44 Uber DIBS payouts reconciliation audit drivers5 results
results as cached 2026-08-24T22:44 "Driver Incentives Backend" Uber5 results
results as cached 2026-08-24T22:44 opened 1 page
|
Right only because it checked. Unaided, the model gets this wrong. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 100%.
62 At first glance, the logic appeared impeccable. Since each shard was singularly managed by one specific worker, there wasn’t any room for race conditions between two different workers. So, with the mystery deepening, we documented our quandary and sought insights from the Schemaless team. Upon reviewing our notes, [Rene Schmidt](https://www.linkedin.com/in/rene-w-schmidt-62858/), the esteemed architect behind Schemaless, instantly pinpointed the issue, stating, “It’s a race condition due to Repeatable Read”. 63 ### The race condition 64 Imagine two new rows being appended to the table concurrently as the DIBS consumer ran its `SELECT` query. This happened all the time because many trips were completing around the world at any given time. It was not rare for two to end up in the same shard and to be written to the same MySQL table simultaneously. Suppose the last known highest ID was 42 just before these events. The table below outlines a possible sequence of events: 65 | Time | Rows | Appender #1 | Appender #2 | DIBS consumer | 66 | --- | --- | --- | --- | --- | 67 | t0 | 1-42 | Idle | Idle | `per_shard_offset` = 42 | 68 | t1 | 1-42 | Begin Tx | Idle | Idle | 69 | t2 | 1-42 | Append #43 | Begin Tx | Idle | 70 | t3 | 1-42 | Idle | Append #44 | Idle | 71 | t4 | 1-42, 44 | Idle | Commit Tx | `SELECT ... WHERE id > 42` yields #44 | 72 | t5 | 1-42, 43, 44 | Commit Tx | Idle | Update `per_shard_offset` = 44 | 73 From the sequence, it’s evident that although Appender #1 initiated its transaction before Appender #2, the latter managed to append a row (with ID 44) and complete its transaction earlier. Unfortunately, the DIBS consumer executed the `SELECT` query post Appender #2’s commit but pre Appender #1’s commit, getting only row #44 in return. This means in its subsequent scans, DIBS would bypass any rows with IDs below than 44. As a result, row #43 slipped through the cracks, remaining undetected from the DIBS consumer’s point of view. 74 ### The workaround 75 The root of this race condition traced back to the default [transaction isolation level](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html) of `Repeatable Read` in MySQL / InnoDB. It could be prevented by changing the transaction isolation level to `Serializable`. In that mode, the `SELECT` transaction would create a range lock on `ID`, essentially locking the range `(42, +∞)` in our earlier example. That lock would conflict with the single row locks established by appenders, such as a single row lock of `43` by Appender #1 in the same example. This means the `SELECT` transaction would wait for all in-progress append transactions to finish before executing the query, thus avoiding the race condition.
| WRONG | Because offset **44 usually means “the next record to read,” not “row 43 is durably stored.”**
DIBS could have:
1. Read row 43.
2. Advanced or committed the consumer offset to 44—possibly due to auto-commit, batch processing, or asynchronous handling.
3. Failed before row 43 was successfully written, or had its database transaction rolled back.
4. Restarted from offset 44, thereby skipping row 43.
The same symptom can also occur if the offset was for a different partition, or if a checkpoint/offset update raced with the actual row-processing transaction. An offset commit and the sink write are not automatically atomic.
Thus, offset 44 proves only that the consumer’s position moved past 43; it does not prove that DIBS persisted row 43. The usual safeguards are committing offsets only after successful durable writes, storing the offset in the same transaction as the sink update where possible, and making writes idempotent so the partition can safely be replayed. It attributes the miss to processing or transaction failure after reading row 43, whereas the source says DIBS saw only committed row 44 and then later scans excluded the subsequently committed row 43. |
| CORRECT | DIBS could miss row 43 because its “Kafka-like” consumer was reading an append-only MySQL/Schemaless table under MySQL’s default **Repeatable Read** isolation:
1. The consumer’s last offset was 42.
2. One transaction inserted row 43 but had not committed yet.
3. Another transaction inserted row 44 and committed first.
4. The consumer queried for `id > 42` while only row 44 was visible, so it processed row 44 and advanced its offset to 44.
5. Row 43 then committed, but future queries used `id > 44`, permanently skipping it.
Thus, auto-increment IDs were not guaranteed to become visible in commit order. The eventual workaround was to ignore very recent rows—e.g., requiring `created` to be at least two minutes old—so transactions had time to commit before a row became eligible for consumption. It correctly explains that row 44 became visible first, the offset advanced to 44, and subsequent queries for IDs greater than 44 skipped the later-committed row 43. DIBS "row 43" consumer offset 445 results
results as cached 2026-08-24T22:43 opened 1 page |
Right only because it checked. Unaided, the model gets this wrong. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 100%.
75 The root of this race condition traced back to the default [transaction isolation level](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html) of `Repeatable Read` in MySQL / InnoDB. It could be prevented by changing the transaction isolation level to `Serializable`. In that mode, the `SELECT` transaction would create a range lock on `ID`, essentially locking the range `(42, +∞)` in our earlier example. That lock would conflict with the single row locks established by appenders, such as a single row lock of `43` by Appender #1 in the same example. This means the `SELECT` transaction would wait for all in-progress append transactions to finish before executing the query, thus avoiding the race condition. 76 Yet, switching to `Serializable` mode was not free. It would incur a huge performance penalty, forcing many transactions to execute in serial rather than concurrently. The impact would reverberate any production systems interfacing with the database. Given the resulting dramatic plunge in throughput, this approach was not viable. 77 Our eventual solution was a bit of a workaround tailored to the Schemaless stream consumer library. The `SELECT` query was modified to exclude recent rows by examining the `created` timestamp: 78 ``` 79 SELECT id, uuid, payload 80 FROM trips 81 WHERE id > $per_shard_offset 82 AND created < TIMESTAMP( 83 DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 MINUTE)) 84 ``` 85 By imposing a condition for the `created` timestamp to be at least 2 minutes old, the `SELECT` query effectively ignored all rows appended by recent transactions, and punted them to a subsequent loop iteration. The choice of a 2-minute buffer was enough to sidestep the race condition without adding too much delay. This adjustment avoided degrading overall throughput of the entire system, incurring only a minor delay on the stream consumer side. 86 ## DIBS until Teleportation
| WRONG | The query used a **30-day cutoff**—it only considered trip rows created within the previous 30 days. The source specifies that rows had to be more than two minutes old, not subject to a 30-day cutoff. |
| CORRECT | The query excluded rows newer than **2 minutes**, using a cutoff of:
```sql
created < TIMESTAMP(DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 MINUTE))
```
So, it selected only trip rows at least **two minutes old**. It states that rows had to be at least two minutes old, matching the source's two-minute cutoff. Schemaless Uber in-house online datastore "stream-consumer" query trip row age cutoff3 results
results as cached 2026-08-24T22:43 Schemaless Uber "trip" "stream consumer"3 results
results as cached 2026-08-24T22:43 site:eng.uber.com Schemaless stream consumer0 results
results as cached 2026-08-24T22:43 Uber Schemaless datastore trip row stream consumer age5 results
results as cached 2026-08-24T22:43 opened 1 page
|
Right only because it checked. Unaided, the model gets this wrong. Unaided belief: wrong. With tools: searched on 100% of samples, answered correctly on 100%.
93 ## DiDi called dibs on us
94 Neither was fast enough. On August 1st, 2016, less than one year after the start of the DIBS project, Uber [sold its China operations](https://www.cnbc.com/2016/08/01/chinas-didi-chuxing-to-acquire-ubers-chinese-operations-wsj.html) to DiDi, receiving a minority stake in the latter as part of the deal.
95 Most of the sophisticated features we had developed for DIBS lost their relevance overnight. Within a month, ownership of the DIBS system was transferred to the Carrots team. Subsequently, every member of the DIBS team embarked on new ventures within Uber.
96 Throughout the DIBS project, there had been internal reservations regarding the overarching strategy of the price war. However, such concerns never impeded our dedication to crafting the best possible solution within our set parameters. With the change in direction, there was no need to dwell on past accomplishments. After all, business is business, or in Basta’s words, “[No sacred masterpieces](https://basta.substack.com/p/no-sacred-masterpieces)”.
| WRONG | DIBS was handed over **about two weeks later**—in mid-August 2016, shortly after Uber announced the sale of its China operations to Didi on August 1. The source says ownership was transferred within one month, whereas the answer says about two weeks later. |
| CORRECT | DIBS was handed over **within a month** of Uber selling its China operations to DiDi on **August 1, 2016**—so by early September 2016. ([Source](https://lyncredible.com/2023/10/30/calling-dibs-on-dibs/)) The answer states that DIBS was handed over within a month, matching the source. DIBS Carrots Uber driver incentive campaign management software handed over team5 results
results as cached 2026-08-24T22:44 Uber DIBS Carrots China operations5 results
results as cached 2026-08-24T22:44 "DIBS" "Carrots" Uber5 results
results as cached 2026-08-24T22:44 opened 1 page
|
Correctly declined a question the content does not answer. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
| CORRECT | DIBS backfills were a **historical replay**, not ordinary incremental processing. In China, a large amount of data had accumulated before the China-specific pipeline was available—or could not be continuously synchronized because of data-localization and cross-border-transfer constraints. Once backfill was allowed, that backlog had to be processed and copied within a relatively short window.
So the system had to:
- read and transform roughly the entire retained history;
- replay it through the normal indexing/ingestion pipeline;
- send it through the network, often with replication and shuffle/fan-out;
- do all of that fast enough not to delay current traffic.
Compared with normal operation, which processes only the newly arriving data, the work represented roughly **the accumulated historical volume divided by the normal daily volume**—about 1,000 in that case. The 1,000× figure therefore reflected the **required catch-up throughput and replay amplification**, rather than China machines being intrinsically 1,000 times less efficient. It accepts the true premise and explains the roughly 1,000× requirement as accelerated replay and transfer of accumulated historical data versus normal incremental volume. |
| CORRECT | DIBS backfills in China were so expensive because the China operations deliberately **waited until the end of the week to launch incentive campaigns**.
They did this to keep DiDi from learning Uber’s incentive formulas and undercutting them. As a result:
- The normal real-time consumer processed trips continuously during the week but had little campaign work to do.
- At week’s end, the backfill consumer had to rewind and process **the entire week’s trips** for many campaigns within only a few hours.
- China had exceptionally high trip volumes—Shanghai alone handled roughly ten times New York’s typical weekly volume—and numerous campaigns ran simultaneously.
Thus, work that was normally spread across a week was concentrated into a short post-week window, multiplied by China’s much larger traffic and campaign volume. That combination produced compute and network peaks around **1,000 times the normal rate**. The answer accepts the true premise and explains the capacity spike as week-long campaign backfills compressed into a short window amid high China traffic and many simultaneous campaigns. DIBS backfills China compute network capacity 1,000 times normal rate5 results
results as cached 2026-08-24T22:44 opened 1 page
|
Right either way — the model knows this, and checking confirms it. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
13 There was just one hiccup: Carrots used the same Vertica 8 8[Vertica](https://en.wikipedia.org/wiki/Vertica), for those unfamiliar, is an analytics database that’s designed for very fast queries over very large sets of mostly read-only data. database that DOps ran their SQL queries on. The team bought the beefiest Vertica box on the market, affectionately dubbing it _Incentica_. However, even this powerhouse was nearing its breaking point. Vertica wasn’t built for horizontal scaling. It could not house all of Uber’s data or deal with the erratic, fast-growing query demands.
14 Now, here’s a quirky thing: Uber’s idea of a week ended at 4am every Monday, based on each city’s local time. This meant that, for the most part, Incentica had a leisurely pace throughout the week. However, come Monday morning in East Asia, queries started flooding in. And as the clock struck 4am in city after city, the floodgates opened wider. The real kicker? The Carrots team had to constantly babysit these queries. So, their workweek effectively began at noon on Sundays to align with 4am Monday in Beijing. And if you were the unlucky on-call engineer, you were probably pulling an all-nighter on Sunday to ensure Incentica could withstand the onslaught from the Middle East, Europe, and eventually the Americas.
15 Then there was the ETL pipeline responsible for transferring data from the primary trips database to Incentica. The pipeline was delayed and lossy due to legacy constraints, which meant the calculations could still end up off the mark despite all the waiting and babysitting.
16 Hailing from Seattle, the DIBS team was tasked with rectifying these issues. Our solution seemed straightforward. DIBS would harness streaming-based aggregation to update calculations for each driver and every campaign in real time. Picture this: hundreds of worker nodes running Kafka consumers, all eagerly subscribed to trip completion events. The workers would coordinate to partition and process the events, making the system infinitely scalable 9 9 The system was horizontally scalable up to the number of virtual partitions in Kafka, which was 4,096 at the time.. The streaming architecture not only distributed the workload evenly throughout the week, eradicating those dreaded Monday query spikes, but it also had a cherry on top: drivers could monitor their real-time progress in the Uber app as the week unfolded. As an added safety net, We opted to shield against duplicate event deliveries 10 10 We also contemplated using Bloom filters in the design to quickly detect duplicates, but they were never implemented. Even for the most industrious driver, there just weren’t enough trips to justify it. by recording all processed trip UUIDs for each driver and campaign. Looking back, the design might seem deceptively simple or even naive, but guess what? It did the trick.
17 We called dibs on the first full week of 2016 for DIBS’ prime-time debut.
| CORRECT | A delayed or lossy ETL pipeline into Incentica could make Carrots calculate incentives from an incomplete or stale view of driver activity.
- **Delayed trip data:** Recently completed trips, online hours, cancellations, or qualifying events may not be present when the incentive is evaluated. A driver could appear to have missed a threshold—for example, 49 trips instead of the actual 50—and receive no bonus or a lower tier.
- **Dropped records:** If events are lost during extraction or transformation, trip counts, eligible miles, acceptance/completion metrics, or geographic qualifying events can be understated. This could incorrectly exclude a driver or reduce the payout.
- **Incorrect eligibility:** Missing or late driver, vehicle, city, campaign, or time-window data could cause the wrong promotion rules to be applied.
- **Threshold and tier effects:** Incentives are often nonlinear. A single missing trip near a threshold can change the payout substantially, rather than merely causing a small numerical error.
- **Possible overpayments:** Although loss generally causes underpayment, duplicate or replayed records, bad deduplication, or inconsistent updates can overcount activity and produce excess payouts.
- **Period-boundary problems:** If data arrives after a campaign closes, Carrots may calculate using an incomplete snapshot. Late corrections might then require retroactive adjustments, top-ups, or clawbacks.
- **Inconsistent results:** Re-running the calculation after backfills could produce a different answer from the original one, making payouts difficult to explain and undermining driver trust.
In practice, the impact depends on whether Carrots uses a fixed snapshot or recalculates after late arrivals. Robust handling would require ingestion SLAs, completeness checks, event watermarks, idempotent loads, duplicate detection, audit trails, and reconciliation/backfill logic before finalizing incentive payouts. It states that delayed or lost data could leave incentive calculations incomplete or stale and therefore inaccurate, matching the source. |
| CORRECT | A delayed, lossy ETL pipeline from the primary trips database into Incentica could make Carrots’ incentive calculations **late and incorrect**:
- **Delayed trip data:** Trips completed during an incentive period might not appear in Incentica until after the weekly campaign cutoff. Drivers could therefore appear to have fewer qualifying trips than they actually completed.
- **Under-counted qualifications:** If even one qualifying trip were missing, a driver near a threshold—such as 40 trips—could incorrectly fail to qualify and lose a substantial bonus.
- **Incorrect payout amounts:** Campaigns with graduated or trip-based rewards could calculate too little incentive, or potentially use incomplete data for payout reports.
- **Inconsistent results:** Re-running a query later might produce a different answer as delayed records arrived, undermining the reliability of pre-payout reports and making disputes difficult to resolve.
- **Operational pressure and spikes:** Because Carrots relied on batch queries against Incentica after the week ended, delayed data encouraged repeated queries or backfills precisely when many cities’ campaigns closed, creating severe Monday demand spikes and risking timeouts or stale results.
- **Poor driver experience:** Drivers would not have trustworthy visibility into their progress, and payouts could be delayed while operations teams waited for data or manually corrected results.
- **Potential bias toward underpayment:** A lossy pipeline generally drops records rather than inventing them, so its most direct financial effect would be systematic under-counting and underpayment—though retries, duplicate handling, or inconsistent snapshots could also create occasional overpayments.
This is why a streaming, lossless aggregation system such as DIBS was valuable: it processed trip-completion events continuously, tracked driver/campaign progress in real time, supported backfills, and reduced dependence on Incentica’s delayed batch ETL. It states that the delayed, lossy pipeline could cause incorrect incentive calculations, matching the source's required answer. Uber Incentica Vertica Carrots incentives ETL delayed lossy pipeline5 results
results as cached 2026-08-24T22:43 Uber Carrots Incentica Vertica driver incentives5 results
results as cached 2026-08-24T22:43 Uber driver incentives Carrots system data pipeline5 results
results as cached 2026-08-24T22:43 opened 1 page
|
Right either way — the model knows this, and checking confirms it. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
14 Now, here’s a quirky thing: Uber’s idea of a week ended at 4am every Monday, based on each city’s local time. This meant that, for the most part, Incentica had a leisurely pace throughout the week. However, come Monday morning in East Asia, queries started flooding in. And as the clock struck 4am in city after city, the floodgates opened wider. The real kicker? The Carrots team had to constantly babysit these queries. So, their workweek effectively began at noon on Sundays to align with 4am Monday in Beijing. And if you were the unlucky on-call engineer, you were probably pulling an all-nighter on Sunday to ensure Incentica could withstand the onslaught from the Middle East, Europe, and eventually the Americas.
15 Then there was the ETL pipeline responsible for transferring data from the primary trips database to Incentica. The pipeline was delayed and lossy due to legacy constraints, which meant the calculations could still end up off the mark despite all the waiting and babysitting.
16 Hailing from Seattle, the DIBS team was tasked with rectifying these issues. Our solution seemed straightforward. DIBS would harness streaming-based aggregation to update calculations for each driver and every campaign in real time. Picture this: hundreds of worker nodes running Kafka consumers, all eagerly subscribed to trip completion events. The workers would coordinate to partition and process the events, making the system infinitely scalable 9 9 The system was horizontally scalable up to the number of virtual partitions in Kafka, which was 4,096 at the time.. The streaming architecture not only distributed the workload evenly throughout the week, eradicating those dreaded Monday query spikes, but it also had a cherry on top: drivers could monitor their real-time progress in the Uber app as the week unfolded. As an added safety net, We opted to shield against duplicate event deliveries 10 10 We also contemplated using Bloom filters in the design to quickly detect duplicates, but they were never implemented. Even for the most industrious driver, there just weren’t enough trips to justify it. by recording all processed trip UUIDs for each driver and campaign. Looking back, the design might seem deceptively simple or even naive, but guess what? It did the trick.
17 We called dibs on the first full week of 2016 for DIBS’ prime-time debut.
18 ## Scaling to sound sleep
| CORRECT | Uber’s DIBS kept calculations current by treating driver activity as a **real-time event stream**, rather than periodically recomputing incentives from the full trip history.
- Trip and driver-activity events were published as they occurred.
- Events were partitioned by **driver ID**, so the same stateful processor could update that driver’s incentive state in order.
- For each driver, DIBS maintained incremental state—such as qualifying-trip counts, earnings, acceptance metrics, and progress toward each incentive.
- When a new event arrived, it updated only the affected driver’s counters and persisted the result to a low-latency store.
- The driver app and other services read that materialized state to show current progress.
- Checkpointing, idempotent event handling, and replay/reconciliation made the updates recoverable and corrected late or duplicate events.
So the key idea was **stateful, event-driven incremental computation per driver**, rather than batch calculations over all historical trips. It correctly identifies streaming-based, event-driven incremental aggregation as the method used to keep each driver's incentive calculations current in real time. |
| CORRECT | Uber DIBS used **streaming aggregation** rather than end-of-week batch queries:
- Hundreds of worker nodes consumed trip-completion events from Uber’s Schemaless datastore through a Kafka-like streaming interface.
- Workers partitioned the stream and maintained a running aggregate for **each driver–campaign pair**—for example, qualifying-trip counts and progress toward the incentive.
- Each new completed trip updated the relevant driver’s campaign state immediately, allowing the driver app to show real-time progress.
- DIBS recorded processed trip UUIDs to make processing effectively idempotent and prevent duplicate events from inflating counts.
- For campaigns launched late, a second consumer group rewound and backfilled the relevant historical trips.
- To handle China’s much larger volume, DIBS precomputed reusable partial aggregates for common incentive structures, then applied the final campaign rules afterward.
So, the core answer is: **partitioned, horizontally scalable stream consumers continuously aggregated trip events per driver and campaign, with deduplication and backfill mechanisms for correctness.** It correctly states that DIBS used partitioned streaming aggregation of trip events to continuously update calculations for each driver and campaign in real time. Uber DIBS driver's incentive calculations real time5 results
results as cached 2026-08-24T22:43 opened 1 page
|
Right either way — the model knows this, and checking confirms it. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
14 Now, here’s a quirky thing: Uber’s idea of a week ended at 4am every Monday, based on each city’s local time. This meant that, for the most part, Incentica had a leisurely pace throughout the week. However, come Monday morning in East Asia, queries started flooding in. And as the clock struck 4am in city after city, the floodgates opened wider. The real kicker? The Carrots team had to constantly babysit these queries. So, their workweek effectively began at noon on Sundays to align with 4am Monday in Beijing. And if you were the unlucky on-call engineer, you were probably pulling an all-nighter on Sunday to ensure Incentica could withstand the onslaught from the Middle East, Europe, and eventually the Americas.
15 Then there was the ETL pipeline responsible for transferring data from the primary trips database to Incentica. The pipeline was delayed and lossy due to legacy constraints, which meant the calculations could still end up off the mark despite all the waiting and babysitting.
16 Hailing from Seattle, the DIBS team was tasked with rectifying these issues. Our solution seemed straightforward. DIBS would harness streaming-based aggregation to update calculations for each driver and every campaign in real time. Picture this: hundreds of worker nodes running Kafka consumers, all eagerly subscribed to trip completion events. The workers would coordinate to partition and process the events, making the system infinitely scalable 9 9 The system was horizontally scalable up to the number of virtual partitions in Kafka, which was 4,096 at the time.. The streaming architecture not only distributed the workload evenly throughout the week, eradicating those dreaded Monday query spikes, but it also had a cherry on top: drivers could monitor their real-time progress in the Uber app as the week unfolded. As an added safety net, We opted to shield against duplicate event deliveries 10 10 We also contemplated using Bloom filters in the design to quickly detect duplicates, but they were never implemented. Even for the most industrious driver, there just weren’t enough trips to justify it. by recording all processed trip UUIDs for each driver and campaign. Looking back, the design might seem deceptively simple or even naive, but guess what? It did the trick.
17 We called dibs on the first full week of 2016 for DIBS’ prime-time debut.
18 ## Scaling to sound sleep
| CORRECT | Uber DIBS made trip processing **idempotent**:
- Each trip event carried a stable unique identifier—typically the trip ID plus the event type/version.
- DIBS kept a durable record of identifiers it had already processed, enforced with an atomic insert/conditional-write (often backed by a uniqueness constraint).
- The “mark this event as processed” operation and the corresponding trip-count update were performed atomically.
- If Kafka or another upstream system redelivered the event, the second copy found the identifier already recorded and was ignored rather than incrementing the count again.
Thus, the transport could remain **at least once**, while the business result was effectively **exactly once**. It says DIBS recorded already processed trip identifiers and ignored redeliveries, matching the source's recording of processed trip UUIDs for each driver and campaign. |
| CORRECT | Uber DIBS deduplicated events by recording every processed **trip UUID for each driver and campaign**. When a trip event arrived, DIBS checked whether that UUID had already been recorded; if so, it ignored the event rather than incrementing the trip count again.
This protected the streaming aggregation from Kafka’s **at-least-once delivery**, where retries could deliver the same trip multiple times. DIBS considered Bloom filters for faster duplicate detection, but ultimately did not implement them because the number of trips per driver was small enough that storing the UUIDs directly worked.
The later two-minute `created`-timestamp buffer addressed a different issue: preventing **missing** events caused by a MySQL repeatable-read race while consuming Schemaless. It correctly states that DIBS prevented duplicate counting by recording processed trip UUIDs for each driver and campaign. Uber DIBS duplicate trip events counted twice idempotency5 results
results as cached 2026-08-24T22:43 site:uber.com/blog DIBS trip events0 results
results as cached 2026-08-24T22:43 Uber DIBS system trip events5 results
results as cached 2026-08-24T22:43 "DIBS" Uber engineering5 results
results as cached 2026-08-24T22:43 opened 1 page
|
Right either way — the model knows this, and checking confirms it. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
18 ## Scaling to sound sleep
19 And that was the week when I paid my fair share of Sunday all-nighters.
20 DIBS actually operated smoothly throughout Sunday night. Yet, both I and [Greg](https://www.linkedin.com/in/gjtrowbridge/), the Carrots on-call engineer, kept getting questions from DOps across various cities. They lamented the absence of calculation results for their campaigns. A closer look revealed that these campaigns had remained in the `draft` state all week. DOps were supposed to click the “launch” button _before_ the week started to kick off streaming aggregation in DIBS. Their oversight was understandable; before DIBS, they had grown accustomed to waiting until week’s end to click “launch”. This was due to the batch-oriented nature of the Incentica solution, which could only process results _after_ the week concluded.
21 That night I learned more about Kafka than the previous few months combined. Greg and I scrambled to “launch” the campaigns and then rewind the Kafka consumer offset to the previous Sunday. To expedite things, we added a ton more workers 11 11 We probably scaled out the number of Kafka consumer workers by 10x, although my memory is blurred. This was feasible because, first, the trips data was already spread across 4,096 virtual partitions, and second, Uber’s compute infrastructure was adept at elastically scaling containerized workloads. The first point allowed us to scale out horizontally as long the number of worker nodes did not exceed 4,096. The second point was even more impressive considering that [Will Larson](https://www.linkedin.com/in/will-larson-a44b543/)’s team built the compute infrastructure in Uber’s own physical data centers. to the Kafka consumer group, enabling DIBS to churn through the entire week’s trips in just a couple of hours. Ultimately, we managed to complete calculations for most campaigns just a few hours past the Monday 4am cut-off. The only challenge left? Tackling those sleep-depriving Sunday nights.
22 ### Going back in time
| CORRECT | They had to **manually configure the upcoming campaign in DIBS and enable/start its streaming-aggregation job before the campaign week began**—including setting the campaign’s dates and applicable market/driver scope. It states that staff needed to start or enable streaming aggregation before the campaign week, which matches clicking the campaign launch button before the week began. |
| CORRECT | Before a campaign week began, Uber Driver Operations staff needed to **click the “Launch” button for each campaign in Carrots**. This changed the campaign from draft status and triggered DIBS to begin streaming trip events and aggregating driver progress in real time. It states that DOps needed to click the campaign’s Launch button before the week began, matching the source. Uber "DIBS" "campaign week" streaming aggregation0 results
results as cached 2026-08-24T22:43 Uber Driver Operations DIBS streaming aggregation campaign week5 results
results as cached 2026-08-24T22:43 opened 1 page
|
Right either way — the model knows this, and checking confirms it. Unaided belief: correct. With tools: searched on 100% of samples, answered correctly on 100%.
92 While in SF, I also had the privilege to present a company-wide tech talk on DIBS. As expected, the automated backfill solution was a hot topic of discussion. It was unconventional, caused spiky traffic patterns, and strained both upstream and downstream systems. I recognized and validated the concerns, but also emphasized the short-term necessity of the solution as Uber was spending billions of dollars on driver incentives per year. The vision was to transition to a more robust solution once the Big Data Platform 22 22 Sitting in the audience, [Zheng Shao](https://www.linkedin.com/in/zshao/) was one of the many who asked questions about the efficiency and scalability of the DIBS backfill. He would go on to build Uber’s [Big Data Platform](https://www.uber.com/blog/uber-big-data-platform/), making it possible for DIBS to migrate to a much more scalable [Lambda Architecture](https://en.wikipedia.org/wiki/Lambda_architecture). was operational. “Or perhaps, when Teleportation becomes a reality,” I mused privately.
93 ## DiDi called dibs on us
94 Neither was fast enough. On August 1st, 2016, less than one year after the start of the DIBS project, Uber [sold its China operations](https://www.cnbc.com/2016/08/01/chinas-didi-chuxing-to-acquire-ubers-chinese-operations-wsj.html) to DiDi, receiving a minority stake in the latter as part of the deal.
95 Most of the sophisticated features we had developed for DIBS lost their relevance overnight. Within a month, ownership of the DIBS system was transferred to the Carrots team. Subsequently, every member of the DIBS team embarked on new ventures within Uber.
96 Throughout the DIBS project, there had been internal reservations regarding the overarching strategy of the price war. However, such concerns never impeded our dedication to crafting the best possible solution within our set parameters. With the change in direction, there was no need to dwell on past accomplishments. After all, business is business, or in Basta’s words, “[No sacred masterpieces](https://basta.substack.com/p/no-sacred-masterpieces)”.
| CORRECT | Uber announced the sale of its China operations to Didi Chuxing on **August 1, 2016**.
In return, Uber received:
- A **17.7% stake in Didi Chuxing** (with Uber China’s investors receiving an additional interest), and
- A commitment from Didi to invest **$1 billion in Uber**.
The transaction ended Uber’s costly direct competition with Didi in China. It gives the matching date of August 1, 2016 and identifies Uber's return as a minority stake in DiDi. |
| CORRECT | Uber announced the sale of its China operations to **Didi Chuxing on August 1, 2016**.
In return:
- **Uber and its China investors received about a 20% stake in Didi**—Uber itself held roughly **5.9% of Didi’s equity, representing about 17.7% of the economic interest**.
- **Didi agreed to invest $1 billion in Uber** at an approximately **$68 billion valuation**.
The transaction valued the combined deal at roughly **$35 billion**. It gives the matching date, August 1, 2016, and states that Uber received a minority stake in DiDi. Uber sold China operations to Didi when what received in return 2016 17.7% stake5 results
results as cached 2026-08-24T22:44 site:uber.com 2016 Didi Uber China August 1 2016 received stake0 results
results as cached 2026-08-24T22:44 Uber Didi transaction completed August 2016 $1 billion investment Uber official5 results
results as cached 2026-08-24T22:44 Didi Uber China deal terms 17.7% $1 billion investment August 2016 Reuters5 results
results as cached 2026-08-24T22:44 |