[problem] [history] [technical] [return addresses] [resolution]
----- The following addresses had permanent fatal errors ----- <firstname.lastname@example.org> ----- Transcript of session follows ----- ... while talking to shield.example.com.: >>> DATA <<< 554 Service unavailable; Client host [220.127.116.11] blocked using Barracuda Reputation; http://bbl.barracudacentral.com/q.cgi?ip=18.104.22.168 <<< 554 Error: no valid recipientsMy eye hit on the last line, and as this was a non-academic (i.e. for us, non-work-related) recipient, I misdiagnosed it as a problem at the far end (user account changed, etc.). Sendmail logs showed something like:
Oct 22 11:11:36 trmail sendmail: l9MIBZUI013926: to=<email@example.com>, ctladdr=<firstname.lastname@example.org> (368/100), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=387670, shield.example.com. [192.168.22.33], dsn=5.0.0, stat=Service unavailable
The next day we had more bounces, and it was apparent that work-related mail to partners such as SFU, MDS Nordion and UCLA was being rejected. Investigation showed that connections from our mailer to the mailserver at certain institutions were getting a "554" SMTP error status at intial connection (before HELO/EHLO).
We immediately used the Barracuda web form to request delisting from the "reputation list", and were somewhat surprised to see a "2 business day" resolution time as compared to, say, 1 hour for the Composite Blocking List or instantaneous for our in-house DNSBL. We then phoned the Barracuda support line to try and find the reason for listing, as the lookup page gave no indication. The helpdesk staff were polite but unable to help, and a tech support person said, basically, that their methods were proprietary and that the customer support line was for customers (not victims).
So with no evidence to go on, we tried to guess why Barracuda might be listing us.
It was fairly clear that we were not actually sending spam through our mailserver. The server logs showed no evidence of a traffic increase, and we had no complaints whatsoever from end users, SpamCop, other DNSBLs etc.
Possibilities seemed to be
We moved our mailing lists to use a different outbound server, but this did not result in the second server being listed and the original server was subsequently re-listed. The list software (mailman) is in any case supposed to automatically unsubscribe dead addresses and our lists are typically small and members-only.
We historically forward a high proportion of user accounts - our users go on sabbatical, or work both here and at another institution, so we have used the traditional Unix mail forwarding scheme extensively. Since the spring of 2007, when we implemented a sendmail SpamAssassin milter, we have been filtering most of this mail before it is forwarded offsite. But in general it is a user's responsibility to decide what to forward to their account elsewhere. We looked at these forwarded accounts but could see no specific issues, e.g. dead offsite accounts. However, since our filters cannot exactly match filters at the target organization, there will always be cases where we forward mail that they label as spam (and vice-versa).
We looked at delivery failures, as reported to "postmaster". Some years ago, these were in general real problems that an administrator could fix. But more recently "postmaster" has been the target of direct spam, and the rest of the mail is mostly non-delivery from forged return addresses. We had got out of the habit of looking at this mail, assuming that users will handle genuine non-delivery events. On examining these records, we found a few specific problems:
This is still conjecture, but it seems a plausible mechanism. In fact, we are doing exactly the same thing - analyzing mail and adding repeat spamming addresses to an in-house blacklist. The differences being that we don't distribute this list to hundreds of other sites, that our delisting page takes immediate effect, and that we would if asked show evidence and examples of such spam.
Barracuda's attitude, as stated in their website and auto-generated email, seems to be that
they only sell appliances and that the responsibility for blocking our mail lies with
their customers, so we should complain to them, not Barracuda. However, the reason that these customers
have bought appliances in the first place is that they are non-technical or don't
want to deal with their own mail. So trying to complain to them is like yelling into
the wind. Besides, there are dozens of them and even contacting the IT department at
an occasional correspondant can be difficult.
We did manage to get whitelisted quite quickly at SFU. At another company, who had outsourced their IT services, it was a question of asking an employee to generate a trouble ticket and escalate it through the system. When we did make contact, it took some time to explain the problem and then they had finger trouble whitelisting us in the appliance - it seems that there is both a name-based whitelist and a numeric whitelist, and only the numeric list takes precedence over the intent blacklist. The whole process, from initially contacting the employee to finally getting whitelisted, took a couple of weeks. Having to repeat this process at some 40-odd companies did not seem reasonable.
In recent spam, the return address is not generally correct - the spammer does not want to receive replies, or even DSNs. However, since one long-standing mail filter requires that the return address has at least a valid domain name, it is often the real address of an innocent third party. In polymorphic spam, it will change between each message of batch of messages to avoid creating a recognizable pattern. My suspicion is that the spam engine simply re-uses the recipient list as a source of forged return addresses.
We still have no answer from Barracuda as to exactly why were listed, and hence no expectation that we will not be listed in the future. We have gotten two partners to whitelist us around their appliance using the numeric whitelist.
Some people have suggested we should threaten legal action, or actually take Barracuda to court. But this is Canada; the best we can do so far is a strongly worded note.