Could Better URI Filtering Cure Email Spam?

A highly desirable goal of businesses and web users is the complete eradication of spam from the internet.  That is perhaps a bit too much to hope for, but certainly the goal of reducing spam is something we can all keep working towards.

One of the more effective methods of reducing spam in recent years is through IP filtering.  This technique involves checking the IP address of the computer or server that is trying to send you email against a list of known or highly suspect spam sources.  The lists are provided by various third party organizations such as Spamhaus and are typically integrated into the products sold by security vendors.

The best part of this technique is that the check occurs at the earliest stage of the initial communication between the two servers.  If the IP address is considered to be a spam source then the connection is terminated before time and server resources are wasted by accepting any further part of the email content.

This meant greater efficiency in spam protection systems compared to earlier techniques that involved checking the entire message content for certain keywords or strings that matched a database of known spam.  This technique is still used today, but it is only performed on email that first passes the IP filtering checks.

Some estimates put the amount of spam that is typically stopped by IP filtering at around 80-90%.  That is up to 90% of spam (not of total email traffic) that can be prevented by IP filtering, usually with very few false positives.

The remaining 10-20% poses a bigger challenge.  These emails need to be checked more thoroughly for other characteristics such as:

• Sender address/domain
• Email body content such as text or URI (Uniform Resource Identifier, often called a URL by web users)
• Images and file attachments

This is because spam emails can come from trustworthy sources such as webmail providers and ISPs in which specific accounts have been compromised by a phishing attach.  As a result they cannot be blocked reliably on the basis of sender address/domain.These checks are also computationally more expensive and more prone to false negatives when new spam techniques emerge.  One of these new techniques is the use of URL shortening services to cloak malicious website addresses.

URL shortening sites typically do not police the links that people create using their services, which elevates the risk of them being used for malicious purposes.  However, the services do often provide an API that can be accessed by other applications, which has led to the emergence of sites and web browser add-ons that can be used to manually check a shortened URL before it is clicked on.

This process is manual and tedious though, and relies on the weakest point in spam prevention – the end user.  Only the most security conscious end user will do this check even some of the time.

But the combination of URI filtering and URL shortening APIs offers the chance for the problem to be attacked from two angles.  Email security products could possibly detect shortened URLs and perform a check against the provider’s API to determine the actual destination address.  That destination address can then be checked against URI filtering lists for known malicious sites.

Though this check may be effective it is not particularly efficient.  Email servers will need to send API requests and wait for responses before determining if an email is malicious or not.  And it does not solve the issue of these services being used by spammers in the first place.

As an alternative, the URL shortening services could make use of URI filtering lists when providing shortened URLs to their anonymous users, and deny the creation of short URLs that lead to malicious sites.  This might eliminate the problem at the source.

As a positive flow on effect of this type of change the use of shortened URLs by spammers on social networks and other non-email communications would also be reduced, reducing the risk of several different threats at once.

These checks are obviously not being performed by shortening services yet.  I tested several spam URLs from a URI filtering list on a few of the popular services and none of them prevented me from creating a shortened URL.  I wonder if soon we will see them forced into action as spammers exploit their systems to the point where they are completely untrusted and actively blocked by security systems.