Posts Tagged ‘anti-spam’

Tackling False Positives in Email Validation

What is a false positive? In email validation, this term is used when an email address is incorrectly identified as being valid or deliverable – in other words, it is flagged as being good when it is actually bad.

False positives are dangerous for senders, marketers especially because sending messages to a bad email address can ruin a sender’s reputation and possibly even get them blacklisted. It’s best to correctly identify email addresses before sending out messages to help ensure that you don’t get penalized for sending messages to bad email addresses.

What causes false positives in Email Validation?

The DOTS Email Validation service offers real-time validation and verification of email addresses. Email verification is handled by directly communicating with an email address’ host mail server(s) via Simple Mail Transfer Protocol (SMTP). SMTP, to put it simply, provides the rules and guidelines for how mail servers and clients should communicate and behave when sending mail.

SMTP is older than the internet as we know it and it precedes the World Wide Web HTTP protocol by almost ten years. At the time of its inception its inventors probably never dreamed that it would be abused by malicious users and overwhelming spam. If they did then they probably never would have created SMTP commands like EXPN and VRFY.

These SMTP commands are considered vulnerabilities, as they are intended to list and verify user mailboxes; however, since they are seen as vulnerabilities most mail servers provide a way to disable them: some like Microsoft Exchange come with them disabled by default, and others will simply return a false positive . Even though the RFC specifically states, “EXPN and VRFY MUST return only valid domain addresses that are usable in SMTP RCPT commands”, it is not uncommon to see these commands return false positives. These are some of the reasons why the Email Validation service does not use or rely on these SMTP commands when trying to validate an email address.

The most common cause for false positives comes from servers that are configured to not reject recipient requests for an email address that does not exist. Simply put, the server will not reject a bad email address and it will instead say that the email is good. At Service Objects, we ubiquitously refer to this as a catch-all domain.

Catch-all behavior

This type of behavior was commonly seen by domains that enabled catch-all or accept-all email accounts. The feature was primarily intended to be used as a way for someone to never miss an email address. Before the days of spam, when email addresses were a new concept, people didn’t want to risk losing an email message because someone forgot how to spell their mailbox or if someone accidentally mistyped it. It didn’t take long, however, before these catch-alls started getting filled with spam, making them near unusable.

However, catch-all behavior gained popularity by admins in an attempt to thwart bots from mining and spamming their users. The reasoning was likely that if a bot could not trust the results being returned by the server then the bot would be forced to move on, and the mailboxes of the domain’s users would be protected.

Unfortunately, malicious users and bots generally don’t care about catch-all behavior, and this practice instead creates other problems, such as helping spammers generate backscatter spam. Backscatter generally occurs when the recipient server does not reject a bad email address and instead bounces it back to the sender. The sender, however, is forged or spoofed by a malicious user and so the unsuspecting sender is now the victim of the unsolicited spam. Backscatter spam also leads to other issues, such as excessive bandwidth, but to not get too sidetracked we’ll perhaps dedicate a blog to backscatter spam at a later time.

Anti-spam solutions

Some mail servers are protected by anti-spam solutions. These solutions are sometimes included in firewall(s) or in mail server(s) or are proprietary to the mail host provider. Solutions can vary in sophistication. Most solutions will incorporate filters and blacklists to try and identify spam and spammers; however, unless the spammer is blacklisted then many of these types of solutions will not reject the bad email address – leading to a false positive. The mail server will likely also bounce the message back to the sender instead (helping to generate backscatter spam).

Not all anti-spam solutions are configured to always accept all requests, however. Some anti-spam solutions may be configured to instead temporarily reject all requests from spam-like activity. This is the opposite of false positives and can instead lead to false negatives.

Other solutions may instead temporarily act as a catch-all when they encounter spam-like activity: behaving normally and rejecting email requests at first, but then switching to catch-all mode temporarily and without warning and then eventually switching back to normal mode. The flapping in behavior can make verification difficult, because if the sender does not know what mode the recipient domain is in, then it can lead to false positives.

How Email Validation can help

According to a recent analysis from Statista, “Spam messages accounted for 56 percent of e-mail traffic in March 2019” and moreover, “China generated the largest share of unsolicited spam e-mails with 15 percent of global spam volume”. With so much spam being thrown around it is not difficult to understand why the overall tolerance for spam-like activity it is so low.

Sure, a single false-positive leading to one bad email message being bounced may not be enough to ruin your sender reputation or get you blacklisted, but for marketers who blast millions of messages for email campaigns, a false-positive here and there can quickly lead to hundreds and thousands if not tens of thousands of false positives.

With how important email communication is nowadays, and the benefits that it brings to marketers, can you afford to get blacklisted? Don’t take the chance and minimize your risk by using a service like Email Validation. Our Email Validation service is highly adept at identifying both false positives and false negatives. Our service has years of experience and data behind it to help identify various server behaviors and patterns.

A Brief Look at the Journey of an Email Message

How do emails actually get from point A to point B?

Most people have no idea. They don’t stop to wonder how an email message is sent and delivered when clicking the send button. If you were to ask them they might reply with,  “Of course, it’s simple. It gets sent to the cloud.”

But an answer like “the cloud” is only part of the overall journey. For most, how an email address is sent is simply techno-wizardry magic, and the details don’t really matter to most people. It only matters that the email message is delivered, and how that happens is not important to them. The purpose of this article is to show you what really happens with an email message.

How Email Works

Details matter to us, but we understand that not everyone likes or needs to get bogged down by them. If you were to search for “How Email Works” then you would find a variety of articles ranging in detail and complexity. The path of how an email message is sent is often summarized in graphics and flow diagrams, with one of the simplest ones shown below.

This diagram, which reads left-to-right, shows the journey of email messages broken into five parts. Please keep in mind that it is not complete and that it fails to cover a couple of scenarios, but overall it is a good starting point. We will add some more details and cover some common scenarios as we move along.

  1. To start, a sender sends an email message. The sender will do this using an email client, which is sometimes referred to as a Mail User Agent (MUA). This can be an application running on the sender’s desktop or mobile device.
  2. The Sender’s MUA will establish an SMTP connection with their Mail Server, and that server will, in turn, relay the message to the recipient Mail Server.
  3. The Sender’s Mail Server then queries the Recipient’s Mail Server and routes a connection to it.
  4. The Sender’s Mail Server establishes a connection with the Recipient’s Mail Server and transmits the email message.
  5. The Recipient Mail server receives and stores the email message so that the recipient MUA may retrieve it via POP or IMAP.

These five steps are about as basic as one can get in describing the journey of an email message. The main takeaway from this flow diagram and others like it is that it establishes that communication is occurring between mail servers. Mail servers communicate using the Simple Mail Transfer Protocol (SMTP). This protocol dictates how mail servers are supposed to behave and send messages to one another.

However, not all mail servers are entirely compliant. Large service providers such as Microsoft, Yahoo and Gmail will likely have their own specialized systems in place for handling internal message communication over their vast networks. Even small businesses with only a few or even just one mail server may configure their networks and mail server(s) in a way that is not fully compliant. However, when it comes to external communication between mail servers, they are not expected to stray too far off from SMTP, as this can easily cause a breakdown in communication.

The Mail Server

Mail servers host the environment for one or more domains as well as the user mailboxes for those domains. They are responsible for sending and receiving email messages, and they are often made up of multiple agents that work together in order to accomplish this:

  • Mail Transfer Agent (MTA) – Is responsible for transferring email messages, often referred to as the either the SMTP Server, Mail Exchange or Mail Relay. Commonly communicates over port 25.
  • Mail Submission Agent (MSA) – Is responsible for communicating with the MUA and handing the email messages to the MTA for delivery. Commonly communicates over port 587.
  • Mail Delivery Agent (MDA) – Is responsible for delivering email messages to the local recipient’s mailbox and is also known as the Local Delivery Agent (LDA).

These agents work together to receive and deliver email messages for their users. Collectively they are considered the Mail Server, but some MTAs will include MDAs and some of the functions of the MSA. Therefore, it is not uncommon for MTAs, in general, to be referred to as the Mail Server.

Local Mail Delivery

In the diagram above we see the path that an email message takes when it is sent to a recipient with the same domain as the sender.

This sender still sends an email message using an MUA that connects to the mail server. Standard connection ports are port 587 and port 25; however, some mail servers may specify their own particular ports and/or require an encrypted connection for security. The MUA connects with the MSA to authenticate the Sender as a user of one of the domains that are hosted on the mail server.

The MSA then receives the email message from the Sender MUA and checks that it conforms to enforced policies. It then delivers the message to the MTA for transmission. Since the recipient domain is the same as the sender, the MTA does not have to communicate with another MTA. Instead, the MDA will handle the delivery to the recipient’s mailbox, where the Recipient MUA is then able to retrieve the message.

Sending External Mail

External mail delivery works similarly to local mail delivery. The sender MUA connects to its mail server where it is authenticated by the MSA. The MSA then receives the email message and passes it on to the MTA.

This is the point where external mail transmission will traditionally differ from local mail delivery. Instead of the MDA delivering the email message, it will be the Sender MTA that will transmit the message to the recipient MTA. It typically does this by querying a Domain Name Server (DNS) for a domain’s Mail Exchange (MX) records. An MX record is used to identify a domain’s mail server. Some domains may have more than one MX record and if so, then multiple records will be returned. Each record is assigned a preference number to prioritize which mail server would be preferred for a connection.

Receiving External Mail

When an MTA receives external mail, the overall process is generally the same as local mail delivery with the exception that the MSA is omitted from the process. This is because the MSA is responsible for communicating with the MUA only. When receiving external mail, the messages are coming from another MTA, commonly on port 25.

Some mail servers and networks are protected by firewalls and anti-spam systems.

Protected Networks

This diagram is an example of what external mail from another MTA to a protected system may look like. In this case a firewall with built-in spam protection.

Firewalls can protect mail servers in various ways. Firewalls are commonly used to block unwanted IP connections and often make use of various blacklists to do so. Ones with built-in spam protection often check for known spammers, and some will inspect the contents of the email message for known malicious links, domains, attachments and various other embedded objects. Spamming techniques are becoming more sophisticated and problematic. Therefore, protection methods are also evolving and becoming more sophisticated as well.

Microsoft, for instance, has developed its own protection service called Exchange Online Protection, and describes it as such, “Microsoft Exchange Online Protection (EOP) is a cloud-based email filtering service that helps protect your organization against spam and malware, and includes features to safeguard your organization from messaging-policy violations.”

The Microsoft EOP Overview page does a good job of explaining how their system works and is comparable to what one can expect from protected mail servers. In their case, they use three distinct levels of filtering: first, a check of the sender’s reputation, followed by subsequent checks against common mail flow rules, as well as, content often found in spam. Messages that pass all three levels of checks are then delivered.

How EOP Works

Piecing it Together

Now that we have gone over some of the basics of email delivery and reception, let’s put the pieces together.

Comparing this diagram with the five-step diagram at the beginning, we have:

  1. Sender MUA sending the email message.
  2. Mail Server, which is comprised of the MSA and MTA.
  3. DNS and Internet Cloud.
  4. Firewall/Anti-Spam protection and Mail Server, which is comprised of the MTA and MDA.
  5. Recipient MUA retrieving the email message.

Again, this is a very basic look at how email is sent and received. We have not gone into details for each step or the underlying protocols, we have barely mentioned security and encryption, and we have not gone over any of the large-scale email service provider systems. So, while the path of sending an email message can essentially be broken into five steps, the overall journey is actually much longer and more complicated. There is a lot that can go wrong, and if you are working on any kind of email marketing campaigns, you want to make sure that you use a validation system that knows its way around.