Featured
The most challenging part in freight is figuring out who ships. Today, the most popular way to find this information is through an ocean manifest data source like Import Genius or Panjiva. My most important (and most painful) job at Flexport was cleaning these datasets to extract clean, actionable prospects for sales. This post will go into the steps on how I cleaned this data.
Author:
Brian Li
Published on:
Sep 29, 2024
The most challenging part in freight is figuring out who ships. Today, the most popular way to find this information is through an ocean manifest data source like Import Genius or Panjiva. My most important (and most painful) job at Flexport was cleaning these datasets to extract clean, actionable prospects for sales. This post will go into the steps on how I cleaned this data.
For context, Import Genius, Panjiva, Trademo and Datamyne receive ocean manifest records from the government, do some light data cleansing, and sell this data more broadly. The data they receive are transactions of ocean shipments - imagine many, many rows, each highlighting a different batch of Lululemon t-shirts shipped on a container from Vietnam to the United States.
Your sales reps do not care about the raw data. They only care about the aggregated consignees, shippers, tradelanes and logistics service providers (LSPs) listed in the data. You need to do the hard, manual work of making it as easy as possible for them to call on their target prospects with as much background data as possible.
There are a few things you need to accomplish:
You need to find clean ways to categorize each target prospect into your CRM
The underlying trade data for each prospect should be easy to read so sales reps can reference it on calls
The system should perpetually update so that your sales teams can continually access updated data and new prospects as they come up
Ultimately, the system we built at Flexport was a competitive advantage. First, we created a large, flexible database for every shipper we could find. Each shipper was tagged with the freight provider they worked with and their origin and destination tradelanes. As our reps continued to log high volumes of outbound activity across these shippers, I was able to build strong predictive models on which shippers were most likely to work with us. It propelled us to $5B in revenue in just seven years.
Ocean manifest records are not ready to be ingested into your CRM. While some services, like Panjiva, lightly enrich the data, it is often done incorrectly and there are often duplicate consignees and service providers. In order to really get mileage out of the data, you will need to follow the process below.
Step 1: Decide how to format the data
You will need to decide how to convert the raw shipment data from ocean manifest records into clean data that can be ingested into your CRM. At Flexport, our target prospects were freight consignees. Knowing the tradelanes and freight providers for each prospect was also valuable to our sales team. Keep in mind that consignees can obscure their manifest records from these services. These services also only have ocean data, not air data.
Start simple. On our first iteration, I was only looking to organize each consignee into our CRM. Eventually, I branched out to collect tradelanes and service providers.
The easiest way for your CRM to dedupe records is through domain names. The ultimate output for our first draft of this process was a simple list of consignees, their TEU count, and their domain names.
Step 2: Hire help overseas and develop your initial process
The best way to get through the related parties on each shipment is to go through them one-by-one - I would recommend hiring talent overseas. I have tried to use automated methods to go through each one, but the accuracy rate is low. With our overseas team, I am able to give feedback and work with them to continuously improve our process.
Plan headcount according to how quickly you need to process the data each day. Generally, each person is able to go through ~three hundred consignees each day. At Flexport, we started with four people.
Onlinejobs.ph is a great place to start for finding talent in the Philippines.
Step 3: Download and process new data fields every day
You need to download as much data as you can. Most services restrict the amount of data you can physically download, which requires you to download the data at some cadence. Just get the most data possible and make sure you are not downloading duplicate records.
Have your contractors start filling in data per your format you decided in Step 1. You will need your contractors on the nuances of how to navigate through different types of common pitfalls in the data:
Sorting through many variations of the consignee. Ex: “Apple”, “Apple Corporation” and “Apple Corp” are all the same company.
Handling consignees that have been acquired. Is it important to note that Gap owns Banana Republic?
Dealing with consignees who are wholesalers with no web presence or if they went out of business.
Each company has a different way to handle each of these situations. For example, Flexport chose to assign territories based on the headquarters location of the parent company. Decide how to handle each situation based on your company’s sales process and rules of engagement.
Step 4: Performance manage, performance manage, performance manage
To start, I would recommend spending an hour per day with your team to train them up on the process. Have them share their screens with you as they reason out how to scrub each record they find. You will need to give feedback on their process. Your best team members will learn quickly and even develop their own ways to build on your initial process.
Track output and spot check their work. You should take note if the entire team makes the same mistakes or if any mistakes are concentrated with each person. You should also track each team member’s daily completion rate.
Step 5: Format the data into your CRM. Create a cadence for updating the data
At weekly or bi-weekly intervals, build pivot tables and bulk upload the accounts into your CRM!
Your hard work is mainly complete - you still need feedback from your sales team. Work side-by-side with a few reps as they navigate through your list of consignees. As they talk to prospects, they will give you feedback on your data - such as if consignees are tagged incorrectly or if the consignee does not actually ship freight.
Most companies that purchase from Panjiva, Import Genius, Trademo, and Datamyne do not use each tool to the fullest extent. We put significant investment into using these tools to build out a sales dataset that our team could attack each day. Instead of searching for accounts to call, they already had at least 200 sales leads ready to go each day to prospect into.
This database at full strength enabled us to pivot our sales efforts immediately as new events impacted our buyers’ supply chains. If something happened in Vietnam, we had our sales team reach out to every prospect whose supply chain originated in Vietnam.
You can turn this into a competitive edge as well.
Related Articles
Revenue Vessel is a leading provider of import data solutions designed to enhance sales and identify new leads in the logistics industry. They offer a platform that enables businesses to launch strategic campaigns, streamline sales cycles, and utilize modern prospecting tools tailored for freight. What makes Revenue Vessel unique is its ability to simplify data integration and prospecting, making it easier for companies to harness the power of import data effectively.
