How Does Catchall Verification Work?

Email verification was a commodity for the longest time — all products did the same thing and produced similar results.

But recently, a dozen startups (Findymail, Prospeo, Scrubby, Leadmagic, etc) emerged claiming to do it “better” than legacy players. These companies have gotten a lot of attention, due to their claim that they can verify 50% more emails, which leads to larger email list sizes, which in turn leads to more revenue.

However, marketers using these services have noticed a lot more variance in results, both in terms of what percentage catchall emails they can verify and the accuracy of the results. They’ve also seen a big discrepancy in pricing, leading them to ask many questions…

• What’s the difference between these tools?
• Is it worth paying the extra $$$ to include them in a waterfall?
• Which ones should I test out?
• etc.

In this article, we’re going to answer all of these questions by doing a deepdive using research gathered over the past two years in the Delivery Lab.

What is Catchall Verification?

The “old school” way of verifying email addresses is “SMTP verification”, which involves probing a remote email server to check if an email exists (this article goes deeper into this topic.)

Because this method works by exploiting a security vulnerability, it tends to only work for smaller organizations without IT teams and works about 45% of the time.

The remaining 55% of email addresses are categorized as “catchall” emails.

How do these email verifiers work?

Today there are four flavors of catchall verification available on the market (in addition to SMTP

1) App User Verification

What is App User Verification?

This is what most of the new email verification providers do today.

It involves building workflows that attempt logging into third party apps and seeing if there is some positive confirmation that an account with that email exists.

How Does App User Verification Work?

Let’s explore this with an example:

• You want to verify an email username@acme.com
• You go to a third party app website like trello.com and click ‘Sign Up’
• You fill out the form with username@acme.com and get this error back.

This error tells you that a user at some point created a Trello.com account with this email address. Because we know Trello only shows this message after a user verifies an account with a valid email in the signup flow, we can conclude that the email was valid **at some point**.

Similar workflows can be setup across many apps, including some email providers themselves (eg: the microsoft login flow)

Why Do Results/Accuracy Vary?

For this method to work, you need to use sites that have LARGE user bases, which are not provisioned by IT (eg: twitter, youtube, trello, etc) and which are explicitly created with work emails (eg: github usually won’t work).

Furthermore, major providers like Google and Microsoft now either block these checks or even intentionally provide invalid/inconsistent results when they detect suspicious activity.

Finally, the bounce rate from these emails will vary because these third party apps aren’t in sync with the target email server. For example, the target email may have existed at some point on twitter.com but if the user leaves their company, the twitter account will continue to remain long after the underlying email has been deactivated.

2) Identity Provider Verification

To the best of our knowledge, this methodology is only employed by Kitt AI.

How Does Identity Provider Verification Work?

This method involves probing the underlying enterprise identity server behind a given email address (eg: Okta, Onelogin, etc).

Due to the secure nature of these products, this method requires a very deep knowledge of each platform and how to do validation against them.

Why do results/accuracy vary?

Similar to SMTP verification, this is a highly authoritative and accurate approach to verifying emails, since it also weeds out alias emails that may otherwise be missed (eg: an email that’s forwarding from one employee to another because they have left).

However, results may be inaccurate due to discrepancies between identity server and mail server settings (for example, an email inbox may be temporarily suspended while access to the master account is not). As such this method must be combined with additional verification signals to yield a high confidence in the verification result. Kitt AI users see this detail provided in the output result so they can make confident decisions.

3) Email Security Gateway Busting

This is one of the methods used by Kitt AI and is claimed to also be used by Bounceban.

This method is very similar to regular SMTP verification, except it goes one level deeper by examining the behavior of the email server returning catchall verifications instead of the actual message that they send back.

How Does Email Security Gateway BustingWork?

This method involves mapping out the unique behaviors of the top enterprise Email Security Gateways (ESGs) like Barracuda, Proofpoint, Mimecast, Darktrace, etc and then determining verification status based on the behavior instead of the actual SMTP message.

The verifier may notice, for example, that the time it takes for an ESG to respond varies depending on various conditions, like:

• A very fast response means that the ESG has blocked the verifier’s IP, so the request should be retried from another address
• A longer response time means that the ESG is doing an actual lookup to the target mailserver behind the scenes
• etc.

By combining this behavioral data with information that’s been gathered from other sources, it becomes possible to “triangulate” a reasonably reliable result.

For Bounceban, you can see from this blog post exceprt that they add certain requirements to the verification request to do this type of triangulation.

Why do results/accuracy vary?

This method has some of the most variance in results, which is why we only use it as an ‘add-on’ data point in our verification process at Kitt AI, instead of taking the result as conclusive. This is because it is extremely fragile – it relies on inference from multiple data points that may change at any time.

Another cause of variance between providers will simply be a difference in the coverage of ESGs that they use and the heuristics they use.

4) Sending Emails To Detect Bounces

This is the primary verification mechanism used by providers like Scrubby.

Unfortunately, it is our least favorite — it produces the most divergent number of results vs other verification methods, and its definition of a “valid” email is extremely deceptive to non-technical users.

To understand why, let’s dig in….

How Does Sending Emails To Detect Bounces Work?

The underlying idea is deceptively simple.

Assumption: sending an email to an invalid address will result in a bounce.

Conclusion: if you send test emails against the verification list and wait to see if you get any bounces, you can tell which emails are valid/invalid.

The problem with the assumption is 1) that it’s simply not true and 2) it’s especially not true with enterprise email servers, which are the most likely to not send email bounces (more details in this article).

The problem with the conclusion is that sending emails with such a high bounce rate requires an email infrastructure that will inevitably have a terrible reputation. When you send high volumes with such a negative reputation, the using “bounces” as a metric becomes deceptive. Since most enterprise email servers will not send email bounces to suspicious actors, the email marketer will have a list of highly secure emails that will cause the most reputation harm once they are actually emailed.

Hybrid Approaches

Different vendors may combine the various verification methods above to yield higher confidence results, depending on the use case.

At Kitt AI, we’ve mapped out over 42 different B2B email scenarios that leverage various combinations of these approaches to give high confidence results, the details of which are shared in the output.

Comparing Providers

When you get conflicting verification results, which one do you trust?

The challenge with comparing providers is the variance in results and lack of visibility into how they arrived at those results. They obfuscate their methodology claiming it to be a ‘trade secret’. And while this is a fair thing to say, the problems we’ve found having been consumers of these softwares is:

• A lack of consistency/transparency in their outcome definitions. For example, we shared above how “not bouncing emails” is a fundamentally different definition of “valid” vs. SMTP checking.
• High Pricing: our research has found (to be shared in a future blog post) that many of these products simply use each other to generate results. Therefore using them in a waterfall means paying for the same underlying services to be hit over and over again for no reason.

Being able to figure out which tools to use in the right waterfall structure requires an in-depth analysis of how these products perform, how accurate they are and a cost-benefit analysis

Conclusion

Having read this article, you should have a good understanding of the methods various email verifiers use and how to think about the results you get using each of them.

In a future article, we’ll share the results of our benchmark studies comparing all of these solutions, and make suggestions on the best value for dollar waterfall strategy that combines all of them.

Want to be notified? Sign up for our mailing list below!