Today was spawned roughly from an event that I experienced last night. I ran into an issue where one person on a shared server had email issues and could not get mail going outbound. The bottom line was that it had to do with the remote mail queue. I realise that I haven't really talked about this, and so you might not really be aware of what I'm talking about because you open your email reader and the mail is just... there.

So we are going to give you a quick breakdown to get you up to speed.

email in 30 seconds

There are several components to a MDS, or Mail Delivery System. What we are discussing here is one that is Linux based and using the following packages:

  • sendmail
  • qmail
  • vpopmail
  • the reader of the customer's choice

When a mail is sent to the end recipient,  sendmail delivers it to the mail server. That mail server, in our case, is running qmail. Qmail looks at the message and decides what domain or account the mail belongs to based on the domain name (in our case) and at this point, vpopmail takes the message and puts it into the proper email account.

That is how you see it when you open your cute little mail reader. To send, do the reverse.

where our rubber meets the road

Our issue was in the sending. In short, a certain person had decided that it made sense to mass send spam related mail advertising their domain that is against our policies. It is against our policies because it drags down the system processes and makes life a living hell for all of the other customers on the server. Not because we have a particular problem with email in general. We also have no issue with people having a marketing purpose. But it's also why you are not advised to take your Ford Focus to the speedway for a race.

So how do we know that this is what's happening? Look at the email queue in QMail with a tool called qmqtool. There is a nice and helpful document on it here. In fact, this document has helped saved my hide quite a few times, so I'm going to show you how to use it in this particular application.

finding the monty python gold

The first thing we do is to get a look at the queue. And these are modified results from a real world instance that I have mangled to reduce security issues. To find the queue stats, the easiest way is with:

qmqtool -s

The output will look something like the following:

$ qmqtool -s

Messages in local queue: 17

Messages in remote queue: 54168

Messages in todo queue: 0

So, as you can see, you have a local queue that is the mail staying on the local machine, system messages and all of that. The remote queue is what is traveling outbound. 55K messages is not normal. Who is the dirty bird responsible for all of this? We find out again with the next command:

qmqtool -R | grep "Envelope Sender: " | sort | uniq -c | sort -rn | head -20

Here is how this cookie crumbles. We are using the -R switch to tell the tool we want the remote queue. Then we grep for "Envelope Sender" because if you use "From", you will be dealing with who the emails claim to be from, rather than who they actually are. Thus far, the command is checking your remote outbound queue to see who is sending the most emails in the queue that remains.

let's talk about sort, baybee...

You see two sort commands in that command sequence. You need to know what they are about. I don't want you to get into the habit of just executing commands because you see them. Know what they do.

In the first case, sort runs on default. The default mode of sort takes the output of the remote queue with all of the senders that you grepped for and sorts them alphabetically, and then by lowercase/uppercase. The listing would be like aAbBcC and so forth.

We slap in a uniq because we want only the unique entries. We're trying to find the dirty bird, but we don't want a 50,000 line listing of the same name if he's the dude. So uniq -c will take the sorted list, count the numbers of repeated entries, remove those duplicates, and with the -c option print in the number of repeated entries.

Now we have useful information, almost. We need what we actually asked for, so the second sort takes that list and sorts it by number in reverse order. That is the meaning of the -rn option.

We want to limit what we are getting back, and we only want the largest senders. That is the meaning of the final head -20. In general usage, I am accustomed to the command being more like head -n 20 and erroring out if I do it any other way.

what you get

This is a mockup of what the final result should be:

$ qmqtool -R | grep "Envelope Sender: " | sort | uniq -c | sort -rn | head -20
51625   Envelope Sender:
24   Envelope Sender:
17   Envelope Sender:
13   Envelope Sender:
4   Envelope Sender:
1   Envelope Sender:

Now we know who he is and how many of these darned things he has constipated into our mail queue. Let's grab some headers. It will look like this:

$ qmqtool -R | head -20
395258 (3, remote)
Envelope Sender:
Envelope Recipient: (To Be Delivered)
To: =?utf-8?B?ZW1lMjMxMw==?=
Subject: =?utf-8?B?TmV3c2xldHRlciBzdWJzY3JpcHRpb24gY29uZmly...
From: ThatDomain
Date: Wed, 16 Jan 2019 20:25:19 +0000
Size: 9.98KB (10215 Bytes)

We are digging into the remote queue. I will leave you to that, as the header above gives you most of the information you need, and to delete from the mail queue you will want to follow the page I first presented.

If none of this is making sense, you should not be deleting anything. Leave it there and call someone who understands this.