Articles / Processing Issues Faced by …

Processing Issues Faced by Email Verification Systems

Following the distributed and coordinated attack on antispam service providers over the last month, this article is intended to provide an overview of one such attack on email verification technology company Bluebottle and highlight the core challenges faced by email verification systems in handling email.

Over the last month, the Internet community has been witness to the cessation of services by a number of antispam providers as a result of continual forged email and denial of service attacks.

The most notable of these service closures was that of Osirusoft, distributor of one of the larger open relay blacklists, which resulted in operator Joe Jared adding a blacklist for "the world" in order to highlight the problem facing this service.

Following the closure of this service, a number of other antispam providers, including monkeys.com and compu.net, have been similarly targeted, resulting in service cessation.

Recent reports from Steve Linford of antispam services provider Spamhaus suggest a correlation between the source of these denial of service attacks and Windows machines infected with the SoBig virus. Similar observations have been made by Matt Sergeant of message services provider MessageLabs, who reports that the profile of denial of service attacks against antispam services appears to match that of machines infected with the SoBig virus.

The effect of this concerted effort against antispam service providers has also been experienced by the email verification technology company for which I work, Bluebottle.

Email verification, also known as challenge response, is an antispam technique which requires the original sender of an email message to reply to a one-time challenge issued by the mail recipient prior to the delivery of email. The original email message is only delivered to the mail recipient following the successful fulfillment of the details of the challenge request. The challenge may take the form of necessitating the original sender to reply to a specific email address, click on a Web link, or supply additional information about the intended mail recipient.

This approach to protecting mailboxes against spam has not been without criticism. Some of the major criticisms relate to the additional mail traffic generated by verification requests, the likelihood of misdirected verification requests, and the placing of an additional burden on the mail sender in order to ensure delivery.

Such concerns are valid, and great care must be taken when implementing an email verification system to ensure that the best principles for such systems are adhered to. These principles are outlined in the paper "Proper principles for Challenge/Response antispam systems" by Brad Templeton, Chairman of the Board of the Electronic Frontier Foundation (EFF). They include:

  1. Ensuring that the action(s) required to fulfill the verification request are easy to complete and accessible to all users.
  2. Ensuring that a verification request is never sent to a reply to a private message originally sent by the user employing email verification.
  3. Avoiding sending verification requests in response to public messages such as those received via mailing lists or newsgroup gateways.
  4. Avoiding sending verification requests in response to error messages and other verification requests.
  5. Provision of means by which users can regularly check to see what messages have been held for delivery pending verification.

Such concerns were considered and incorporated into Bluebottle, which additionally incorporated tagged addresses, similar to those employed by the Tagged Message Delivery Agent (TMDA), for the handling of verification requests and error mail identification. The result has been an antispam system that has proven to be highly effective at protecting mail accounts in a manner minimizing false-positive attribution of mail messages as spam and recognizing the fundamental issue of spam as consent rather than content.

This, in turn, has appeared to make Bluebottle a target for attack over the last two months by individuals and groups of individuals intent on creating havoc for providers of antispam services. This attack took a similar form to that employed against monkeys.com and compu.net and resulted in ignorant administrators similarly blocking legitimate mail from Bluebottle users as a result of spam sent using forged and non-existent Bluebottle addresses.

This type of attack is known as a "joe job". In it, a spam message is fashioned to appear as if it originated from an different source. The term originated from an attack on Joe Doll, proprietor of Joe's Cyberpost. This Web site, first online in 1994, offered free Web pages to any user who agreed to abide by the rules of conduct, which included "good netiquette when publicizing your page".

In 1996, after terminating a user's account for sending unsolicited messages to newsgroups and email recipients, Joe Doll found that a large number of mail messages were being sent in a manner which made it appear that they originated from his Web site. The result was that he was inundated with complaints from newsgroups and email account holders and was eventually targeted in a Denial of Service (DoS) attack over a period of ten days. Further details on this attack can be found at http://www.joes.com/spammed.html.

As a result of the manner in which the attack was fashioned, in addition to all unsolicited email appearing to be from Bluebottle, all bounce messages from undeliverable addresses were similarly returned to the Bluebottle mail servers. This, combined with the roll that tagged addresses play in the Bluebottle system, led to a scenario in which significant delays were encountered in normal mail delivery. The number of bounce messages returned to the Bluebottle mail servers and the load which this placed on normal mail processing became so great that Bluebottle was required to disable all mail verification on user accounts in order to allow the delivery of normal mail in a timely fashion.

This incident highlights the core issue faced by email verification services, the requirement to balance the cost of processing mail messages in a timely and centralized fashion while still ensuring that the best principles associated with email verification implementation are adhered to.

In the case of Bluebottle, in which the message load increased as the result of a concerted effort by those intent on interfering with the operations of antispam providers, the time cost of processing these messages was too great to bear. The result was an untenable situation in which Bluebottle was faced with either delaying normal user mail by unacceptable standards or disabling email verification. The latter option was chosen in order to ensure that while potentially allowing unwanted and unsolicited messages, Bluebottle users could also receive their normal mail messages in a timely fashion.

The positive outcome of this incident is a better appreciation of the work the email verification community still needs to do. Development work is currently being performed to improve the structure of the Bluebottle email verification technology and provide a more scalable platform upon which a consent-based spam protection system can be based. This work includes:

Incorporation of address filtering at the level of the SMTP mailer
The incorporation of filtering for valid local user accounts and tagged email addresses at the level of the SMTP mailer is a must for any email verification system. This is important in order to minimize the level of mail accepted by the system for processing and (in the case of invalidly addressed mail) deletion.
Incorporation of distributed caching and backend data stores
The distribution of backend data storage is important for any high-traffic email verification system for the purposes of high availability. This distribution and replication is particularly important given the level of processing required for performing lookups against user permission lists for each mail message.

Furthermore, where the querying of such backend data stores is considered "time expensive", the establishment of asynchronous querying allows mail message processing to continue while awaiting the return of information from these data stores. The efficiency of such a technical implementation can be further optimized by the incorporation of results caching for address filtering and permissive control at the level of the SMTP mailer.

Generalization of permissive lists
While not an action which Bluebottle is seeking to implement currently, the application of permission lists can be generalized to the entire user base to allow less specific queries of the backend data stores to be applied to a wider number of received mail messages. This approach allows improved lookup caching and permissive rules application at the cost of distinct per-user rules definition and application. This more generalized approach to permissive rules has been implemented by a number of other email verification providers, including Mailblocks.

With the level of unsolicited and unwanted mail messages on the Internet growing, such an understanding of email verification technology will be critical for any consent-based spam protection system to succeed.

Recent comments

21 Jan 2006 21:29 Avatar omcg

Bluebottle
Similar problems have plagued bluebottle in the last month, maybe with the same cause, maybe not. Unfortunately, for business reasons, no explanation (acknowledgement) of said problems is forthcoming from the company. It was helpful to read this article. The author also mentioned Mailblocks (R.I.P.) the best webmail service I've come into contact with.

29 Jan 2004 12:15 Avatar BuyGifts

Email Spam
I have about 4 emails that I use all the time. One of them a fairly newer email and the other 3 are very old emails that i've had for about 4 years now. 2 of the 3 old emails i get so much spam that sometimes i just delete everything out of it becuase i dont feel like checking which ones are not spam. The 1 that i have had for 4 years now does not get almost any spam i guess its all about what emails are out in the public and the emails that bots grab are the ones that will be full of spam all the time.

03 Nov 2003 16:43 Avatar RalphG2SL

Re: Several blacklists needed to go anyway

> I would say
> that if there were a way to make it less
> annoying, I would consider using
> it...but as long as I have to send back
> an email response, people that want to
> receive any email from me will have to
> ditch it first.%


well i am sure:
you are much more annoying than ANY filter system.
and i am sure that i will never want to receive
any mail from YOU...

14 Oct 2003 06:12 Avatar ricksoft

The "From" address
I don't get a lot of spam, and bounce what I do get, but most of it is returned as "undeliverable".

Maybe I'm a bit naive, but is there a good reason why the mail services can't at least check for a valid From and/or Reply To address?

11 Oct 2003 20:30 Avatar ajackson

Re: Joe jobs
ISP's like AOL need to quit bouncing undeliverables back to the From address without testing to see if the mail actually came from that domain. I'm now regularly getting Joe'd at several accounts. Apparently some new spamware cycles the spam addressee list through the From lines as well.

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.