Nothing Broke, and We Were Still 4 Months Late

The cutover went smoothly. Traffic ramped from 1% to 100% in a week, nothing broke, and no data was corrupted. By most measures, the migration was a success.

It was also 4 months late. The rewrite took 2 to 4 months. QA took 6 to 8, because the work kept getting put on hold for P0s.

I've spent some time trying to work out how both of those things can be true. Most warnings about big bang migrations focus on production blowing up. Mine didn't. What I hadn't understood is that a big batch of change carries a second risk that has nothing to do with outages: it needs a big block of other people's time, and you don't control when that time shows up.

I recognized this properly after reading this freeCodeCamp article on migrating legacy systems incrementally. It lays out what you should do. This post is about what it costs when you only do half of it, even when things go well.

Why this service had to go

The service handled scraping and manual income verification for the credit approval process at Kredivo. Years ago, it served multiple scraping methods and manual income verification sources. Since then, third-party vendors have replaced most of that logic, and we only kept several in-house.

That left the service heavily underused, but we were still paying for infrastructure as if it weren't:

• 2 EC2 c6in.xlarge instances
• 18 SQS queues
• 16 to 20 tables out of 40 to 50, holding about 20% of the total data in the database

On top of that, it ran on Python 3.4.10, a version that is no longer officially supported.

The plan was to move all of its functionality into another microservice, on Python 3.12, and retire the old one. That sounds tidy until you look at what depended on it: 10 APIs and 5 flows, used by 4 internal callers. The flows included the heavily used approval process and user registration, and the callers included internal tools and the risk engine.

The bet I made

We planned for 6 months, and I built that plan on a happy path: every stakeholder would stay focused on this migration until it was done.

So I rewrote the whole thing as one unit. The rewrite took roughly 2 to 4 months, including planning, the RFC, the code migration and cleanup. Then, right after it finished, P0s and more important projects took over, and the people I needed to help test the migration no longer had the bandwidth.

At that point I did consider splitting the work up, module by module or API by API. But the rewrite was already done, and chunking it felt like creating another problem on top of the one I had. So I kept it as one big change.

I don't think this was a reckless decision. It was an optimistic one, and that's a harder mistake to spot, because the plan looks fine right up until the day people get pulled away.

What I got right

To be fair to myself, the release was careful. I built a feature flag so that if something went wrong with the new service, I could turn it off and send requests back to the legacy service while I debugged.

The rollout ramped through 1%, 10%, 30%, 50% and 100% over 1 week, and it was based on the last digits of the user ID instead of a random percentage:

def use_new_service(user_id: int, rollout_percent: int) -> bool:
    # The last two digits give a stable bucket from 0 to 99.
    return user_id % 100 < rollout_percent

That detail mattered, because the tables we kept were shared between the legacy and new services. A given user always lands in the same bucket, so at each step of the ramp they aren't bouncing randomly between two services working on the same rows. We had no duplicated or corrupted data at any point in the rollout.

So when I say this was a big bang migration, I don't mean the traffic. The traffic was incremental. The big bang was the delivery.

Where the time went

The code was ready. The ticket moved to Ready for QA, and the QA process ended up taking 6 to 8 months. It kept getting put on hold as sudden P0s piled up, which is understandable. Those P0s mattered more than my migration.

A whole-service change can't be tested in an afternoon. With 10 APIs and 5 flows going live together, testing meant checking every API, every flow and every caller, all at once. That needed a long, uninterrupted stretch of tester time, and with P0s coming in, that stretch never appeared.

We missed the GTM date by 4 months, and the project took almost 10 months against a plan of 6. The rewrite was less than half of that, and the rollout itself took 1 week. Writing the code was never the slow part. While we waited:

• We kept paying for the infrastructure we were trying to retire. That cost thousands of dollars per month, for instances and queues we'd already replaced in code.
• The work had to be verified again. By the time QA picked it up, I had to re-verify everything.
• The ticket stayed open. Finished work sat in a QA column, and our SDLC story points didn't reflect it until the ticket finally closed.

The deployment wasn't relaxing either. Because everything shipped together, I had to test and monitor every API, service and flow at the same time to make sure none of them were breaking.

What I wish I'd known

1. Attention is a constraint, so plan for it

I treated tester and stakeholder time as a given. It's actually the scarcest resource in the project, and you don't own it.

Incremental delivery wouldn't have removed QA from the process. It would have changed the shape of what I was asking for. Testing one API is a task that could fit between two P0s. Testing a whole service needs a gap that may never open.

Size your changes to fit the attention you'll actually get, not the attention you were promised.

2. Incremental rollout isn't incremental delivery

This was my biggest blind spot. I had the flag and the percentage ramp, so the migration felt incremental. But those only control how much traffic reaches the change. They do nothing about how much change someone has to test.

A rollout reduces release risk. It doesn't make the work any smaller to review, test or sign off.

3. Your flag should match the unit you want to test

My flag covered the whole service, and I wish I'd made it per API and per flow. With one flag, all 10 APIs and 5 flows moved together at the same percentage.

# What I built: one switch for everything
FLAGS = {"new_service": 30}

# What I wish I'd built: one switch per slice
FLAGS = {
    "income_verification": 100,
    "registration": 50,
    "approval": 0,
}

Per-slice flags would have let each API or flow go to QA, ship and ramp on its own. They would also have let me disable one slice without touching the others.

4. Know where rollback stops working

"I can always turn the flag off" was only partly true. For a simple API call from internal tools, the risk engine or registration, the flag could send the next request back to legacy. But once a user's approval or manual income verification process was on the new service, it wouldn't flow back to legacy. Even with shared tables, moving an income verification back would have taken manual intervention.

Turning the flag off would have protected new users. Anyone already in progress would have needed a fix forward or a manual move. Nothing went wrong, so this never hurt us, but it's better to know that boundary before you ramp, not during an incident.

5. A frozen legacy service is luck, not a plan

During the 6 to 8 months of QA, the legacy service didn't need any development, and my migration branch didn't drift. That's the only reason the wait stayed manageable.

If someone had needed to change the old service, every change would have had to land in two places while the new code sat untested. That would have been a very troublesome problem. I got away with it, and I wouldn't plan on getting away with it again.

What I'd do next time

I'd split the migration by API and flow, and start with a low-risk slice to prove the path end to end. Each slice would get its own flag, its own QA pass and its own ramp. Before ramping the one-way flows, I'd decide up front what happens to in-progress users if something goes wrong.

This has real costs. Slices mean more coordination, and the legacy and new services run side by side for longer, so you keep paying for the old infrastructure in the meantime. But that cost is one you can see and plan for. An indefinite wait in a QA queue isn't.

A clean cutover doesn't prove the plan was good

It's tempting to look at a migration with no production issues and call the plan a success. Mine had a safe release and a bad delivery shape, and only the first of those shows up on a dashboard.

The next time someone tells me a big migration will have everyone's full attention, I'll ask what happens if it doesn't. If the answer is that the ticket waits, I'll cut it smaller.