Creating a new standard for data onboarding
Since the dawn of the computing era, flat files (e.g. CSV, TSV, XLS, JSON, etc.) have enabled the exchange of information — that is, data imports. CSV flat files actually pre-date personal computers IBM Fortran first supported them in 1972. And yet, official documentation and proper tooling around the flat file format, CSVs, and their “text/csv” media type lagged decades behind, beginning only in late 2005 and into the 2010s. Published standardization efforts for CSVs and other flat file types coincided with the explosion of web applications and proliferation of incumbent digital transformations in the early 2000s (and unlike APIs, which were also rapidly growing in significance as a mechanism for data exchange by programmers, almost anybody was able to use flat files).
This created a perfect storm for flat files as a mechanism to move data. Suddenly, everyone needed a way to onboard and move data from one database or system to another. From hospital systems sharing inventory data across clinics to Shopify stores importing product, order, and customer data, CSVs and other flat file formats became an important universal mechanism for moving and onboarding data across the web.
Unfortunately, the data onboarding process for flat files has historically been error-prone, lacking security and automation, and often requiring significant manual intervention to get the data in the correct format (see the below image). Just Googling “CSV import help” yields an endless list of company support pages and explainer videos trying to give you a mini-course on it.
Companies that attempted to build their own proprietary data import functionality (e.g. creating downloadable templates that users can then upload) often still ended up dedicating countless engineering hours to cleaning and formatting incoming data. Alternatively, the burden of data preparation and normalization is pushed to the user her/himself. Anyone who has tried to upload even the most basic data to Mailchimp or another email service provider knows this. And matters only get worse when it comes to complex imports and error solving — users get emailed unhelpful error reports on near-perfect datasets and often end up re-uploading files multiple times over in an attempt to debug.
No matter how you slice it, the user experience for most data imports is still painful. And with every new business that comes online, the need for an intuitively designed, intelligent data import tooling grows.
Enter Flatfile. The idea for Flatfile started as a side-project while David Boskovic (CEO) & Eric Crane (COO) were working at Envoy and felt the headache associated with data import firsthand. Flatfile’s initial design was simple, but valuable — an elegant and intuitive mapping experience to connect the incoming data to the right fields so users don’t have to download a template. They launched “Flatfile Portal” at the end of 2019 — a slick, intuitively-designed data import “button” (e.g. an API for data healing) — to a passionate and diverse user base that was growing like a weed.
It clicked for me right away. I shared my own data onboarding battle wounds from my past life as a data scientist, dealing with tasks like normalizing hospital medical record data imports. I wished I had Flatfile back when I was spending long nights writing Python scripts to convert values to database reference IDs or normalize inconsistent billing codes.
From early conversations with David and Eric, it was also clear that, while laser focused on building out their core data onboarding features, they were scheming something even bigger. Their mission is to “remove barriers between humans and data,” and buggy data imports are just the beginning. Beyond data imports, data sharing and collaboration remains a massive headache and unsolved problem within many enterprises, and I strongly believe there’s a significant opportunity in helping organizations manage the complex data import challenges they face in serving thousands of customers, many of whom still spend thousands of hours manually restructuring, correcting, and emailing data back and forth.
Flatfile is taking a huge first step in that direction today with the launch of Concierge, their new enterprise offering. Concierge aims to unlock a powerful new capability for businesses — the ability to securely collaborate on data across stakeholders and between organizations. Moreover, this solution is designed to learn and get better over time thanks to artificial intelligence and machine learning.
I believe we are beginning to witness a massive shift in the way data gets onboarded, shared, and collaborated on across company boundaries, and that Flatfile’s platform is the missing link for that process. We couldn’t be more excited to be partnering with their team on the exciting journey ahead. Congrats to David, Eric, and the whole team on the new financing.