Not strictly DIY but I know there are some knowledgeable people on here that might be able to explain this in plain English so....
I've been trying to explain to the Mrs why emails that she sends or receives sometimes seem to get delayed by hours, or even days. I've always just accepted this, but in trying to explain why to her realise that I don't really understand it.
Taking an example, she sends an email via a client (Thunderbird) on her PC (in the UK), which uses POP/SMTP to connect to her email provider Virginmedia, to a recipient in the UK who uses a client also using POP/SMTP to connect to Hotmail. Her client is setup to send emails immediately, the recipient has their client checking for new emails every 30 mins.
What is the path that the email might take through the "internet cloud" to the point that it arrives at the recipients PC, and what might happen to cause delays in the email being received?
Her client hands the email to one of VM's mail servers. You'll see that happening at her end.
It might well go through a couple of different servers inside VM, where it'll get virus and spam filtered.
VM's mail servers check DNS for the MX records telling it where the mail servers are for the recipient domain. The various MX records have priorities, telling the sending server which order to try them.
VM's mail servers then try to hand the mail over to one of the the highest priority. If it's unavailable, it'll try another. If it's unavailable, it'll try another, and so on down the list. If none are available, then the mail will be held and retried later.
If the recipient email address is a personal domain that's then forwarded on to the actual Hotmail box, then it'll get to the forwarding server, which'll re-send it with the hotmail address wrapped around it.
Once in Hotmail's systems, it'll be spam/virus filtered and passed on internally to whichever server or cluster of servers hold that recipient's mailbox. If it's a cluster, then it'll be replicated around the individual servers.
The recipient's client checks the Hotmail server(s), and new mail's then delivered.
Have a read of the full headers for any mail she's received, and you'll see the names and times of each step in the chain listed.
There's plenty of scope for delays and hiccups, especially if there's that extra link involved. But, between large infrastructures like VM and Hotmail, it _should_ be damn near instant.
The most likely cause is the undeniable fact that Hotmail is run by raging fuckwits, tbh. Their spam filtering is a law unto itself, and about as over-eager as a Spaniel puppy coming up to walkies time.
In essence, the answer to your question is that email is "best effort". It can sit in queues all over the place, especially at your ISP and the recipient's ISP, while their mail servers get round to processing it, likely because they're also busy dealing with billions of spam emails.
On Wednesday 12 February 2014 10:56 Davidm wrote in uk.d-i-y:
Usually because one or the other email provider is crap.
Very occasionally, there is a genuine fault in either the network bewteen providers or with either end. Persistent delays suggests one or the other parties should change their provider.
I run my own email server and on the whole, emails arrive within a few seconds (eg place online order, see confirmation email seconds later).
The thing to do now is have your missus send some test emails to a gmail account and see how long those take (gmail is generally pretty fast).
And I *strongly* recommend using IMAP, not POP.
POP is very deficient in features (folder support, multiple clients) and efficiency (POP needs polling, IMAP can support IDLE/PUSH).
usually the way it goes is user->user ISP relay->recipient's ISP relay -> recipient's ISP mail storage->recipient.
The first stage is instantaneous, but issues can delay any of the following stages.
If at any stage in the process the next hop is unavailable, or indeed if the sender decides that its overloaded, the mail is held in a queue which tends to be flushed on an hourly basis. Algorithms then are used to that if its still undeliverable, it will get flushed at increasing intervals until finally its sent back with an undeliverable note attached after typically 48 hours.
Private email systems where he relays and receivers are not part of some huge megalithic company like google or hotmail will in general always deliver instantaneous responses.
Big email systems are targets for attacks and so busy that any glitch can cause traffic to pile up.
I am forever having to explain to people - some worrying senior - that email and SMS are *NOT* inherently reliable services. Just because they work 99.999% of the time doesn't exclude the possibility that .001% of messages just might never get through.
This makes them unsuitable for situations where message receipt needs to be guaranteed.
I worked on an system for managing field engineers in a safety critical industry. As a "nice to have" I implemented a feature that sent an SMS to the engineer with details of the next job. The idea being it saved them having to fire up their laptop and connect to the main job database. When I documented the system I could not have emphasised more plainly that this feature was a convenience, and should not be relied upon. Even my cynical side was impressed that the first complaint that it "didn't work" and somebody missed a job took 3 days from launch.
Well Virgin Media is run by the Google servers, so if there are any problems in them a delay will occur.
Also of course it depends if greylisting is being used by one of the servers in which case it might refuse the first attempt. This is often done to thwart botted machines which basically send a huge torrent of emails once and once only, hoping to avoide blaacklisting a server or getting detected. Brian
On Wednesday 12 February 2014 14:24 robgraham wrote in uk.d-i-y:
You basically tell your mail client that the server is an IMAP server, not a POP server. Depending on email provider, you may need to change the server name from pop.somemail.blah to imap.somemail.blah
That's about it - there are a few twiddly details, but thunderbird as an email client is *very* good at working those out for itself.
Here's a page (that's about as clear as mud, but does seem to mention an important setting you may need to do on their website):
formatting link
However this linked page should give you the exact settings you need for Thunderbird:
I have mixed feelings, there are pros and cons I find...
Sure IMAP has way more features, and is nice if you want shared access to a mail folder - say a desktop and phone sharing a mailbox.
However:
Its slower and more bandwidth intensive. With high mail volumes, many ISPs email servers can bork on large mailboxes, and effectively lock you out of your email. You also need to consider that even sending email typically requires the client to upload the message twice with IMAP - once to deliver it to the SMTP server, and then again to place it in the mailboxes IMAP Sent folder.
(clients like Thunderbird will let you configure local storage only of sent messages - although then you won't see them from other clients)
I've never understood why this step of relaying through an ISP's (or other mail provider's) server is necessary. Why can't a mail client just look up the MX record(s) of the destination domain and send the mail straight there. Is it just to relieve the client of the bother of handling failed deliveries and bounces etc., or is there some other reason?
In many cases it could (at least where the ISP allows outgoing connections on port 25 etc through its network)
(where "it" would more likely be a local email server rather than a traditional email client)
I suspect that is largely the reason it evolved the way it did, in a very less "always on" permanently connected world. Note the client would have to handle not only the only failed deliveries, but also the required length of retrying before deciding that the delivery has actually failed rather than been delayed. So in that sense its not typical end user client behaviour.
In article , Andy Wade wrote: }I've never understood why this step of relaying through an ISP's (or }other mail provider's) server is necessary. Why can't a mail client }just look up the MX record(s) of the destination domain and send the }mail straight there. Is it just to relieve the client of the bother of }handling failed deliveries and bounces etc., or is there some other reason?
It's a fossilised behaviour which is now unnecessary and quite harmful.
The email protocols were designed to work with direct connections from the sender to the recipient's system. That was in the days when a computer that offered email was normally shared between many users, always on, and always connected. So, to give an example of one type of setup, when a sender triggered sending, their mail program invoked the sendmail program to queue the message on their local machine. The message would very quickly be taken off the queue, and the recipient's machine would be looked-up (via their MX record) and contacted (via SMTP). The message was sent across the SMTP connection, and when it arrived on the recipient's machine it was appended to the mbox file in their home directory. If the recipient happened to be logged in at this time, they might have a program that notices that mbox has been modifed and alerts them (e.g. biff).
However, some time later, the Internet became sufficiently popular that people started getting dialup access. This obviously doesn't work very well with the procedure just described. It would be rare for both sender and recipient to be connected at the same time, so direct email would be impractical. One essential step was to have the recipient get their mail delivered to a server that was always connected - their ISP's SMTP server. In many cases that's sufficent. The receipient now has to use a program that fetches any new mail each time they connect, which needs a new protocol - POP3. However, that doesn't really cover all the cases to make email fully practical. If the recipient's mail server is not instantly contactable, the sender has to stay dialled-in until a retry succeeds. The simple solution to this is to have the sender's dialup mail program send all email to the sender's ISP's mail server over SMTP (since it has to be available always for incoming mail, it's trivial for it to accept outgoing mail as well). There was a time when many SMTP mail servers would accept email addressed to anyone from anyone and pass it on. Spam has stopped that practice, so such relaying nowadays is restricted to the ISP's customers.
Now that dialup has become rare again, and people have always-on broadband, there's no good reason why email needs to be sent via the sender's ISP. (Going via the recipient's ISP can be useful as it saves the recipient having to leave their computer on all the time). Even when retries are needed, it would be useful in many cases for the sender to see that the message is still queued and being retried as it lets them know that it definitely hasn't been sent yet.
One way this harms email is that if either of the relaying mail servers cannot pass on the message, then the sender can only be notified by sending them an email (i.e. a 'bounce'), which may itself fail to get through. On a direct connection, the sender knows that the message has been delivered, and even when sending to the receiver's ISP, the destination address is typically checked before the message is accepted, so a typo in the address can be rejected very quickly, but the sender's ISP obviously cannot know which addresses are valid at the receiver's ISP, so adding that extra intermediary necessarily delays notification of failure.
However, the main harm done by this outdated architecture is that it has greatly assisted in sending spam email. If senders connect directly to recipients' ISP's mail servers, this makes it much easier for the mail servers to notice that a sender is sending spam and block them. When the senders go via an ISP, this means that the recipient sees an incoming connection from the ISP for email sent by any of that ISP's customers, making is really difficult to distinguish between one customer who is spamming and should be blocked and all the other customers who are innocent and should be handled normally. This is why you occasionaly read of some major email hosting company either getting mistakenly blacklisted or mistakenly blacklisting someone else.
Of course having email go via two ISP's mail servers is very convenient for spy agencies as it gives them more placed to intercept messages.
In the early days store and forward through possibly many machines most of them not on the internet at all was the way things were done.
UUCP over direct modems was the order of the day: Unix machines sending mail between each other at various times - usually cheap phone rates at night.
Then came PCs. Well tat was OK because they could use POP3 to contact the local unix server, and that would hold the mail and forward it as per normal.
So the concept of the mail server/SMTP relay was born. But still store and forwarding. Over modems.
Bit by bit the internet backbone started to grow. Howvver it was still unreliable, consisting often of dial up links, X-25 links,. bits of wet string and carrier pigeons. And servers were often down . Sop the concept of a backup server was introduced: a place to send mail as an alternate. That would be closer to the final target.
Then when the internet got better, we had spam. Spam and junk we didn't want getting to the end user, so we kept the mail relays as places that were identifiably maintained by someone responsible who would keep logs and deal with abuse, and possibly filter out junk.
The ISPS started handling mega amounts of traffic, so they stated to split the functions between machines. One relay to accept user mail from their own customers ONLY. One main or two (or more, as shown by MX records) high CPU capability high bandwidth but small disk receiver and spam and virus filter, and one or more machines to handle the web mail and pop downloads, and Imap. Less CPU but LOADS of storage.
It was unwise to allow anyone to send direct, because it was an open invitation to spam, and most sendmail receivers will not accept mail from arbitrary IP addresses within known 'domestic' IP ranges.
And it was unwise to allow them to receive direct, because it means that they had to leave open connections to the internet ready to accept - well any kind of crap that anyone shoved at them.
But it is perfectly possible to set up a domain with MX records pointing at a domestic IP address, if you want.
Its is also possible to send direct, though not from most mail user agents. Few targets will accept that mail from you though.
If you have a fixed IP address and you jump through a few hoops you MAY get it accepted by the blacklisters.
But if you go for raw sending and receiving at your won mail server, you step outside the spam and malware scanning that is done by the ISPS. Your choice.
In all cases it is not worth using a secondary MX record though. Not these days.
Largely the reason it still is the way it is, is because users simply don't want to know how its done or lift a finger more than they have to to have it done for them.
Open up an email you have received, and find out how to display the full headers. On some clients, it'll be "Message Properties", on others "View Source"; the terminology varies. On Thunderbird, it's View -> Message Source, on Turnpike, press control-H.
When you have the full headers, which can initially look quite intimidating, read them from the bottom up, starting at the first line that starts with "Received:" and working upwards. That shows you the path the message has taken to reach its destination. Each hop of the transfer adds its own "Received:" line. By looking at the times at the end of each Received line, you can work out where the delay(s) are.
Here's an excerpt from an example email sent from a Hotmail user to a Plusnet user. I've separated the Received: lines to make them easier to parse and deleted everything else.
Reading from the bottom up, the mail was sent by a Hotmail user at 06:07
-0800, so adding 8 hours to bring that to GMT makes it 14:07:50 UK time.
Next line up, hosts.co.uk received the message from Hotmail at 14:07:51
Next line up, Plusnet's incoming mail server receives the mail from hosts.co.uk at 14:07:53
Last line, Plusnet's mail store receives the message at 14:07:53.
So the total time taken to transfer the message is 3 seconds.
Received: from [212.159.9.108] (helo=avasin04.plus.net) by inmx04.plus.net with esmtp (PlusNet MXCore v2.00) id
1WDaTh-0000Jg-TS for ; Wed, 12 Feb 2014 14:07:53 +0000
Received: from fwd2.hosts.co.uk ([85.233.160.24]) by avasin04.plus.net with Plusnet Cloudmark Gateway id RS7r1n00B0XsxQd01S7tc2; Wed, 12 Feb 2014 14:07:53 +0000
Received: from [157.55.0.215] (helo=dub0-omc1-s16.dub0.hotmail.com) by fwd2.hosts.co.uk with esmtp (Exim 4.72) (envelope-from ) id 1WDaTf-000325-Q6 for ; Wed, 12 Feb 2014 14:07:51 +0000
Received: from DUB114-W133 ([157.55.0.239]) by dub0-omc1-s16.dub0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 12 Feb 2014 06:07:50 -0800
It can, but, so many infected machines relay spam that many addresses are routinely blocked by spam lists or just because they are dynamic.
My nas drives currently just send the mail without going through the ISP but if they start blocking I will have to add a login for one of my external mail providers.
HomeOwnersHub website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.