Turkcell Superonline CRM Migration & Rewrite Project
Reasoning of the project
Turkcell Superonline had a tailor made CRM from a vendor, which had an expensive license and maintenance fee. The management decided to go for an inhouse-built solution which would enable faster time to market and lower the costs.
The Scope
Rewrite the entire capabilities of the existing CRM application with inhouse resources on a codebase and migrate all the data from legacy CRM system into the new one. No new features unless they are absolutely mandatory or very easy to implement that would not derail the project timelines.
Timelines
8 months to go live and start from scratch.
Roll out phases & Data Migration Steps
As the details were ironed out by the migration team that I was leading, we came up with the following steps.
1. Start with very limited scope (with only 2 cities involved): Migrate the customers that receive a service from those cities into the new CRM, and block access to those customers in the legacy system. While Call Center would be able to use both the CRM systems (legacy and new), the physical branches in these 2 cities would only be able to use the new system.
2. Expansion phase (Add 2 more cities): After the initial success of the 1st phase of the rollout, we expanded the extent of new CRM system into 4 cities. Call Center is expected to use both CRMs, but now more physical branches would use the new CRM system.
3. Full rollout: After careful consideration and eliminating first batch of bugs from the new system, we rolled out to the entire country. Call Center and all the physical branches were expected to use the new CRM system.
Data Migration & Order Migration
This is where things get interesting. For a successful data migration, we thought solutions for a variety of problems. Data model differences between systems, and the migration process speed per each customer, also the data quality issues (the source of the data is an old system that you can find multiple variations of the same product's configuration or missing pieces of data all over the place because specific features were introduced later but backward compatibility was never implemented so the data is scattered and even broken on some instances.)
We have categorized these focus points into 3 groups
1. Physical data migration
2. Live data migration
3. Rollout phase migration (a.k.a. Silent Migration) & Edge cases & Broken data issues fixing
Physical Data Migration Model
Problem Statement: The data model between legacy and to-be systems are different. There is not a 1-1 match of the database tables. Solution: Create a specific migration process for each table that would work per a customer id.
Fundamental Problem: The initial estimations of the physical data migration completion reached to staggering numbers. A single customer migration would last an average of 5 seconds and that's just the physical data migration itself (not taking into account the data extraction and loading of the data onto temporary storage). With 25 million customers, this would mean around
~125.000.000 seconds
~2083 minutes
~35 hours
~1.4 days
We didn't have 1.4 days JUST for data migration. No business in their sane mind would allow IT teams to do migration, which everyone was sure that it wouldn't go smooth because of the data quality issues. All our tests proved that it would take massive amounts of efforts to migrate all these customers.
Enter silent migration (see below live data migration section).
Live Data Migration Model
Problem Statement: The data still is alive in the legacy system due to long-running order fulfilment flows (sometimes days, weeks and even months long, some of them are also in a fault state which guarantees that they will not be completed until someone deals with the problem and manually fixes whatever wrong with the fulfilment flow).
Solution: As the new system will need to be aware of what is happening to the data in the old system, we introduced database level triggers (DB Trigger on MS SQL Server) on the legacy system, on various entities. So, whenever "something" changed on the legacy CRM, it would trigger an event that would eventually push the data to the new CRM system.
Fundamental problem: This whole long-running migration situation creates a fundamental problem: How do you deal with data that still has the probability to be updated by the legacy system because of a long-running order fulfilment process that was never able to complete? We have introduced a specific migration flag per each entity that signals that this entity is still in the process of being migrated. So much so that the other processes and applications and services instantly know that they should be very careful with this record, because it's highly probable that this record is expected to be updated by the legacy system (as a temporary solution of course).
Between legacy CRM order fulfilment layer and target CRM api layer, there is a data contract per each entity. This data contract brings clear definitions and rules as to what to synchronize into the new system and when. Those rules are described in API specification so that any new developer that is building a new sync process would instantly know the nuances.
Silent Migration Model
At this moment, we have solved 2 major issues 1. Data model changes have been reflected onto the physical data model layer 2. Keeping both legacy and to-be systems alive and synchronized
However, there's still a major problem: migration scripts running time, is still estimated at a staggering 35 hours. There is NO WAY that any business owner will let any IT department to run a script that's going to block the systems for more than entire day. Besides, the chances of running any issue is unavoidable, which will mean re-running of the same script again, and again, and again, probably taking weeks to complete.
This still doesn't even take the data quality issues into account. We are definitely going to have to fix a lot of customer data issue before the 1st step (physical migration) of that customer could be completed.
Problem Statement: So, we need a solution that should 1. Give us more time to run migration scripts to migrate all the customers 2. Give us more time to fix the data quality issues and re-run the script for that/those customer/s
Solution: We thought of this plan below:
1. Let's start as early as possible and try to migrate the customers BEFORE the new system goes live. With a very simple flag, the physical data that is migrated on the new system can be closed to external processes and API's.
2. When the new system go-live date comes, we can simply flip open a switch on those customers and they will be visible on the new system.
3. In the meantime, all the data quality errors that prevent us to do physical migration can be solved, because right now, we will have more time... Much more time than ever... We can run this migration process for weeks without anyone noticing, and we can golive in an instant with all the customers on the new system.
But even now, we were going to be short on time. Our estimations still showed a narrow window that we can deliver the entire data migration. After further examination of the data, we categorized the customers into multiple groups. On a very high level there were:
1. Customers that do not have any movement for years (business customers are mostly in this category)
2. Customers that have a very specific set of products that are big in numbers
3. Customers that are highly capable of making a new movement and/or have an active ongoing order
"Customers that do not have any movement for years" were the most likely candidates to start as soon as possible, so as the lowest hanging fruit group of customers, we prioritized them. With these customers, we also tested the validity of migration scripts. "Customers that have a very specific set of products" were the second most likely candidates to reduce the number of running migration scripts (therefore, decrease delivery time per each customer). These customer were relative low on priority, so we left them out of the scope. "Customers that are highly capable of making a new movement and/or have an active ongoing order" were going to be our biggest challenge as these customers had the biggest number of data quality issues. So, we took them as the secondary phase of our migration.
