1+years together
- Scaling the pipelineAsync IO, multiprocessing, indexing
- Proofs of conceptClassifiers and clustering compared
- Self-serve experimentsDeveloper UX, tests and how-tos
What we did
Built with
- Python
- Kubernetes
- Postgres
A supply chain management platform had to clean, group and deduplicate a vast amount of company data, and do it more accurately without slowing down. After a talk on record deduplication by Flávio Juvenal, one of our partners, it brought Vinta into its data science and infrastructure teams. We sped up and refined its deduplication pipeline, and the platform processed 2 billion records in a couple of hours, with better accuracy.
2 billion
records deduplicated in a couple of hours, with better accuracy.
Self-serve experiments
Experiments run by Python developers and data scientists, without an infrastructure expert at their side.
About the client
The client runs a supply chain management platform on the US East Coast. Part of its business is trade-related, and its product depends on clean data about companies: the same company can show up in many records, so they must be cleaned, grouped and deduplicated before the platform can use them. That work is part of its data enrichment process, an ETL pipeline run with Apache Airflow on Kubernetes.
The challenge: accuracy at 2 billion records
The client was always looking for better ways to clean, group and deduplicate its data, and needed a specialist team for those steps of its data enrichment process. It found one in a talk: Flávio Juvenal’s “1 + 1 = 1 or Record Deduplication with Python”, on record linkage for vast amounts of data, whose video and slides are on our blog and Jupyter Notebook on GitHub.
The client set three goals for the work:
More accurate deduplication
Find more of the records that are the same company, across every step of the pipeline.
The same speed at scale
Keep processing over 2 billion records in a couple of hours.
A platform other teams can use
Good practices on the infrastructure side, and work easy to replicate, so other teams could build on the platform.
Our approach: specialists inside two teams
We joined as a dedicated team of specialists, working inside the client’s data science team on the pipeline and alongside its infrastructure team on how other teams use the platform.
Data science: a faster, more accurate pipeline
Raising accuracy meant improving every step of the pipeline, from pre-processing and cleaning to clustering and community detection:
How the pipeline finds duplicate companies
- Clean and pre-process records
- Index candidate pairs
- Score each pair
- Cluster the duplicates
Scaling the pipeline
At the client’s scale, the first challenge was speed. We used asyncio to run the pipeline’s calls to the database in parallel, and multiprocessing to process and transform the data across several processes, and plugged the two together with as few bottlenecks as we could.
Faster indexing with scikit-learn
The indexing in dedupe, the community standard Python library for this work, would not reach the scale the client needed: its records called for more complex indexes. Comparing every record with every other one was not an option, since the comparisons grow with the square of the records. We plugged scikit-learn into dedupe’s indexing, which cut comparison time sharply.
Proofs of concept for the data team
With speed no longer an issue, we turned to the data team’s train, validate and test cycle. The team did not yet know which classifier would compare its records best, so we built proofs of concept for several, and walked the team through each one’s trade-offs so it could choose the best performer.
The classifier gives each pair of records a likelihood that both are the same company. Clustering those pairs then shows which companies are in fact duplicates. We used scikit-learn and pandas for it and, since they were not fast enough at this scale, tuned how they worked with multiprocessing. With the data science team, we compared the efficiency and trade-offs of each clustering algorithm in more proofs of concept, so the team could pick the one that fit.
Infrastructure: experiments without an expert at hand
The client’s large infrastructure team served many other teams. One need took much of its time: whenever a Python developer or a data scientist wanted to run an experiment, an infrastructure expert had to be there. We helped the team build the documentation and methods that let developers run those experiments on their own, in four steps:
1. Listen
The complaints other teams raised most often, and the automation already in place.
2. Developer experience
How each method should work, which parameters it takes, and which it can fetch on its own.
3. Automated tests
Tests that enforce the new way of working and tell developers what went wrong, so they fix it and try again.
4. Documentation
How-tos on the architecture, its ideas and risks, and the changes planned for it.
Outcome: 2 billion records in a couple of hours
The pipeline met its goal: 2 billion records in a couple of hours, with better accuracy, and the data team came away with a clearer view of how its classification and clustering algorithms perform.
The infrastructure team no longer had to take part in every experiment, which freed its time for the rest of its roadmap, and other teams reported higher satisfaction as they could solve more problems on their own.








