Emails violating 822bis section 2.3 with CRLF

[Dmacman]

Hi All,

 

I have osCommerce running on our Windows machine with Apache and PHP/Mysql and another ISP (who hosts several of our companies email accounts) tells me that our osCommerce server is violating 822bis section 2.3 with its emails, and that is why none of the emails sent by osC are arriving.

 

I did some testing within osC and if I send an email to one of those accounts with no body copy, it will go thru. If I put in a single character, it will not. It seems something in the includes/classes/email.php is causing the violation.

 

In the email options window I have the following settings.

 

E-Mail Options

Title ....................................................Value

E-Mail Transport Method ........................smtp

E-Mail Linefeeds .............................CRLF

Use MIME HTML When Sending Emails ......true

Verify E-Mail Addresses Through DNS ......true

Send E-Mails .......................................true

 

In the php.ini I have set....

SMTP = 192.168.1.102 (this server is 192.168.1.102)

sendmail_from = [email protected] (the osCommerce admins email address)

 

 

If I send mail to a yahoo or hotmail account it works fine.

 

I did notice that since I changed the E-Mail Linefeeds to CRFL from LF that the accounts that originally worked still work, but all the text is double spaced. And the accounts that originally did not work, still do not work.

 

I cannot find a post anywhere that resolves this issue.

 

Does anyone have any ideas?

 

I appreciate the help in advance,

Don.

[Guest]

Hi All,

 

I have osCommerce running on our Windows machine with Apache and PHP/Mysql and another ISP (who hosts several of our companies email accounts) tells me that our osCommerce server is violating 822bis section 2.3 with its emails, and that is why none of the emails sent by osC are arriving.

 

I did some testing within osC and if I send an email to one of those accounts with no body copy, it will go thru. If I put in a single character, it will not. It seems something in the includes/classes/email.php is causing the violation.

 

In the email options window I have the following settings.

 

E-Mail Options

Title ....................................................Value

E-Mail Transport Method ........................smtp

E-Mail Linefeeds .............................CRLF

Use MIME HTML When Sending Emails ......true

Verify E-Mail Addresses Through DNS ......true

Send E-Mails .......................................true

 

In the php.ini I have set....

SMTP = 192.168.1.102 (this server is 192.168.1.102)

sendmail_from = [email protected] (the osCommerce admins email address)

If I send mail to a yahoo or hotmail account it works fine.

 

I did notice that since I changed the E-Mail Linefeeds to CRFL from LF that the accounts that originally worked still work, but all the text is double spaced. And the accounts that originally did not work, still do not work.

 

I cannot find a post anywhere that resolves this issue.

 

Does anyone have any ideas?

 

I appreciate the help in advance,

Don.

Where does Title ....................................................Value come from?

I don't think that that is an option in admin.

[Dmacman]

Hi All,

 

I have osCommerce running on our Windows machine with Apache and PHP/Mysql and another ISP (who hosts several of our companies email accounts) tells me that our osCommerce server is violating 822bis section 2.3 with its emails, and that is why none of the emails sent by osC are arriving.

 

I did some testing within osC and if I send an email to one of those accounts with no body copy, it will go thru. If I put in a single character, it will not. It seems something in the includes/classes/email.php is causing the violation.

 

In the email options window I have the following settings.

 

E-Mail Options

Title ....................................................Value

E-Mail Transport Method ........................smtp

E-Mail Linefeeds .............................CRLF

Use MIME HTML When Sending Emails ......true

Verify E-Mail Addresses Through DNS ......true

Send E-Mails .......................................true

 

In the php.ini I have set....

SMTP = 192.168.1.102 (this server is 192.168.1.102)

sendmail_from = [email protected] (the osCommerce admins email address)

If I send mail to a yahoo or hotmail account it works fine.

 

I did notice that since I changed the E-Mail Linefeeds to CRFL from LF that the accounts that originally worked still work, but all the text is double spaced. And the accounts that originally did not work, still do not work.

 

I cannot find a post anywhere that resolves this issue.

 

Does anyone have any ideas?

 

I appreciate the help in advance,

Don.

 

Yes that is the setting in the ...

 

Configuration

--->E-Mail Options

 

Window.

 

Here is the link the ISP gave me in regards to the violation.

 

Link to CRLF explanation.

 

Don

[♥toyicebear]

Click here...

[Dmacman]

Click here...

OK, I downloaded and installed smtp-auth-update_2 and setup the config with my SMTP settings.

 

Now when I try to send an email (to an account that was working) I get no errors, and no emails arrive.

 

The Read_Me_old.html looks to be french, which I do not understand.

 

The install is basic, and I did all 5 steps successfully.

 

Any ideas why no working emails now?

 

I checked the exchange server and there is no new emails in the queue.

 

Thanks,

Don

[bpopelar]

What do your email configuration parameters look like now?

 

Since you were able to send emails previously via SMTP, I'm assuming you do not have SMTP password authenication enabled (one of the new configuration parameters provided as part of the contribution).

 

Try reading through this thread which is the official support topic for the 901 contribution.

 

When emails are reported as "successfully send", but are not received, you have an email configuation problem.

 

Ben

[Guest]

Yes that is the setting in the ...

 

Configuration

--->E-Mail Options

 

 

 

Don

That was the column heading, not a setting.

[Dmacman]

Sorry, yes that was the heading.

 

As for the SMTP settings and authentication, I have it set to authenticate.

 

Like I said before, I was getting mail to some addresses, now nothing.

 

And I logged into the account I am using via web access to the exchange server and I know the username and password are correct and working.

 

I checked the router and port 25 is correct.

 

The server name and IP is correct.

 

???

 

Does this shed any light on a possible fix?

 

Thanks again.

 

Don

[Guest]

Could try turning off verification via dns and see what happens.

[Dmacman]

Could try turning off verification via dns and see what happens.

I tried this and it had no effect.

 

Any other suggestions?

 

Thanks,

Don

[Ripper]

I tried this and it had no effect.

 

Any other suggestions?

 

Thanks,

Don

 

 

im having the same problem. i am not sure what it is but i supect spam filters .

is the sollution above the one to use ?

[Dmacman]

I don't believe its spam filters in my case (at least).

 

I have control over the 6 accounts I am testing and I turned of all filtering and 3 of them (yahoo, hotmail and google have no spam filtering).

 

Of the 6 accounts I am testing that I control, 4 never had an issue receiving mail, until I installed this mod. And the other 2 just started having problem with the osCommerce email.

 

Still open to help/suggestions.

 

Regards,

Don

[Ripper]

I don't believe its spam filters in my case (at least).

 

I have control over the 6 accounts I am testing and I turned of all filtering and 3 of them (yahoo, hotmail and google have no spam filtering).

 

Of the 6 accounts I am testing that I control, 4 never had an issue receiving mail, until I installed this mod. And the other 2 just started having problem with the osCommerce email.

 

Still open to help/suggestions.

 

Regards,

Don

 

 

 

i know hotmail does do spamfiltering server side, so does yahoo and google.

 

im having the same problem for a while now on shops that have been running since 2004.

nothing changed on my side so it must be on the other side.

 

some isp give me the same problem like tiscali but thats a european isp..

 

gr,

[Dmacman]

[NOTE to all: This osCommerce site is on our server, IE we are the ISP and I control the Exchange server, so I know nobody is 'messing' with our mail settings or filtering or banning from here.

So anyone with knowledge of changes that would require access to Apache, MySQL, PHP, MS Exchange, etc, I can make them and tell you what the results are. I don't need to ask the ISP's persmission, I am the ISP in this case.]

 

Ok more R&D on this.

I found this test email link and tried it and only got the php email not the SMTP version

 

I removed the SMTP mod and went back to the original email.php files.

 

Then I found this 5th param for the SMTP setting and added it and it still did not work.

 

So I went back to the basic email.php again.

 

I just did a test email and if I send it to the addresses that are not arriving but in the body of the email only put a single line of text, it goes thru.

 

IE:

 

in the body I put just "Hello" (no break, returns, CRLF) it will arrive.

 

If I try to send

 

1

2

 

Nothing arrives. So it leads me to believe the email.php code is messed where it tries to substitue the "\r and or \n".

 

function send($to_name, $to_addr, $from_name, $from_addr, $subject = '', $headers = '') {
if ((strstr($to_name, "\n") != false) || (strstr($to_name, "\r") != false)) {
return false;
}
if ((strstr($to_addr, "\n") != false) || (strstr($to_addr, "\r") != false)) {
return false;
}
if ((strstr($subject, "\n") != false) || (strstr($subject, "\r") != false)) {
return false;
}
if ((strstr($from_name, "\n") != false) || (strstr($from_name, "\r") != false)) {
return false;
}
if ((strstr($from_addr, "\n") != false) || (strstr($from_addr, "\r") != false)) {
return false;
}

 

Does that makes sense to anyone?

 

Thanks,

Don

[bpopelar]

The code you quoted is verifying that there are no CRs or LFs in the "to", "from" and "subject" parameters. If the code finds a CR or a LF, it rejects the email.

 

The baseline OScommerce code does not substitute a CRLF pair for a bare LF in the body of an email. Hence, if the email message passes through a email forwarding agent like qmail that checks for 822bis, it will be rejected. Now some email forwarding agents silently replace bare LF with CRLF as they are routed. It all depends on what path an email takes from source to destination. Its not clear which behavior is correct (bouncing an email or silently changing it). If you search the internet you will find people on both sides of the flame war.

 

The admin -> Configuration -> E-mail Options -> E-mail Linefeed configuration parameter is only used to separate email header fields; it is not used in the body of the email. Internally generated emails, like the order confirmation email, are generated with bare LFs in the body of the email (at least on a linux platform). The emails generated by the "contact us" page or the admin "send email" page are correctly generated with CRLF pairs in the body of the email (once again on a linux platform).

 

The spacing of an email on viewing depends on the email client. Some will display an email as double spaced, others will display it as single spaced. Once again it is not clear which email client behavior is correct.

 

Given that you were originally able to send emails to some destinations, your SMTP server does not require password authentication (the baseline code does not support it); therefore you should load the SMTP Authentication contribution, but configure it to not use password authentication. What you will get for your trouble is the conversion of bare LF to CRLF which will stop email forwarding agents like qmail from rejecting a message. You will have to live with the double spacing issue as you cannot solve the problem. You might try configuring the email for HTML mode.

 

Ben

[Dmacman]

The code you quoted is verifying that there are no CRs or LFs in the "to", "from" and "subject" parameters. If the code finds a CR or a LF, it rejects the email.

 

 

The baseline OScommerce code does not substitute a CRLF pair for a bare LF in the body of an email. Hence, if the email message passes through a email forwarding agent like qmail that checks for 822bis, it will be rejected. Now some email forwarding agents silently replace bare LF with CRLF as they are routed. It all depends on what path an email takes from source to destination. Its not clear which behavior is correct (bouncing an email or silently changing it). If you search the internet you will find people on both sides of the flame war.

 

The admin -> Configuration -> E-mail Options -> E-mail Linefeed configuration parameter is only used to separate email header fields; it is not used in the body of the email. Internally generated emails, like the order confirmation email, are generated with bare LFs in the body of the email (at least on a linux platform). The emails generated by the "contact us" page or the admin "send email" page are correctly generated with CRLF pairs in the body of the email (once again on a linux platform).

 

Ok, that makes sense (I am a procedural programmer and I am trying to learn OOP. Since I am on a windows platform and still have this issue, can anyone direct me with some code to strip out the bare LF's in the body and replace them with 822bis compliant CRLF's?

 

The spacing of an email on viewing depends on the email client. Some will display an email as double spaced, others will display it as single spaced. Once again it is not clear which email client behavior is correct.

 

Given that you were originally able to send emails to some destinations, your SMTP server does not require password authentication (the baseline code does not support it); therefore you should load the SMTP Authentication contribution, but configure it to not use password authentication. What you will get for your trouble is the conversion of bare LF to CRLF which will stop email forwarding agents like qmail from rejecting a message. You will have to live with the double spacing issue as you cannot solve the problem. You might try configuring the email for HTML mode.

 

Ben

 

I turned of authenticate and once I get the bare LF's stripped out, it sounds like I might be in business. I will just have to live with the double spacing (which is better than no emails at all).

 

I appreciate you taking the time to explain this in detail and item by item.

 

regards,

Don

[Dmacman]

BTW bpopelar, I see your addon here. Will that fix my CRLF issue on the windows platform?

 

Thanks,

Don

[bpopelar]

I believe it will. It solved my CRLF problem on a linux platform. It should work the same on windows platform.

 

Please try it and post the results; that way I can state that it has been tested on a windows platform (assuming it works!).

 

Ben

[♥geoffreywalton]

I had a problem with SMTP emails not being sent, sendmail path need to be changed as per server info in the admin panel and SMTP host had to be changed to localhost rather than IP address.

 

These are my email option settings

 

E-Mail Transport Method smtp Info

E-Mail Linefeeds CRLF Info

Use MIME HTML When Sending Emails false Info

Verify E-Mail Addresses Through DNS true Info

Send E-Mails true Info

Sendmail Path /usr/sbin/sendmail Info

Sendmail Arguments -i -t Info

SMTP Host localhost Info

SMTP Port 25 Info

Use SMTP Authentication true Info

SMTP Username [email protected] Info

SMTP Password pppppppppp

[Dmacman]

I had a problem with SMTP emails not being sent, sendmail path need to be changed as per server info in the admin panel and SMTP host had to be changed to localhost rather than IP address.

 

These are my email option settings

 

E-Mail Transport Method smtp Info

E-Mail Linefeeds CRLF Info

Use MIME HTML When Sending Emails false Info

Verify E-Mail Addresses Through DNS true Info

Send E-Mails true Info

Sendmail Path /usr/sbin/sendmail Info

Sendmail Arguments -i -t Info

SMTP Host localhost Info

SMTP Port 25 Info

Use SMTP Authentication true Info

SMTP Username [email protected] Info

SMTP Password pppppppppp

Hi Geoffrey,

 

I used the same settings and I get...

 

Warning: mail() [function.mail]: SMTP server response: 501 5.5.4 Invalid Address in D:\Program Files\Apache Group\Apache2\htdocs\izzit\store\admin\includes\classes\email.php on line 463

 

Warning: Cannot modify header information - headers already sent by (output started at D:\Program Files\Apache Group\Apache2\htdocs\izzit\store\admin\includes\classes\email.php:463)........

 

It doesn't like those settings.

 

I will try bpopelar mod next and let you know.

 

If you have an alt setting, please post and I will try it.

 

Thanks,

Don

[Dmacman]

Ok bpopelar. I put in your newest mod, and out server still rejects it under 822bis.

 

We are going to have to look to another ISP, but that will only fix us, not anyone else using our cart.

 

I am out of ideas/alternative.

 

Anything left?

 

Don

[bpopelar]

I've been thinking of adding some debug logic to the contribution to assist in installation / configuration issues like this (a lot of users have problems with email configuration in general). I'll generate something up this weekend and post an updated to the contribution with instructions on how to use it Hopefully we can figure out what is different on a windows platforms / with your environment and get you going.

[bpopelar]

Don,

 

I've posted an update to the contribution which includes a debug capability which is generally useful when dealing with SMTP configuration problems.

 

The debug capability is controlled by the variable $this->debug (near line 78) in the class.smtp.php file. When it is set to TRUE, the SMTP protocol exchange is dumped to the browser window. The debug output represents ASCII CR and LF characters as \r and \n. To use the capability, edit the class.smtp.php file in the /catalog/includes/classes directory and change the value from FALSE to TRUE. Utilize the "Contact Us" page to send an email message. When you select the "continue" button after constructing the email message the browser will display the SMTP protocol exchange messages ending with the following warning message:

 

Warning: Cannot modify header information - headers already sent by (output started at...

 

From the information displayed you should be able to resolve any SMTP configuration problems you are having. This capability is need because the higher level modules (like tep_mail.php or contact_us.php) that utilize the email class (modified by this contribution) do not check the status returned. The email is always reported as successfully sent to the user. Once you have determined the proper configuration parameters, edit the class.smtp.php file again and set the $this->debug variable to FALSE to disable this capability.

 

You can download the enhanced contribution from here.

 

Load the enhanced contribution and run it. If you don't mind, send me the output via PM so I can look at it in detail and figure out what is going on; otherwise, just tell me where in the protocol exchange the bare LF is detected. You should always see "\r\n" as a pair in the debug output. Where you only see "\r" or "\n" by itself is where the error is occurrng.

[bpopelar]

I just found and fixed the bug in email.php which caused the body of an email message to be double spaced. The bug caused every newline character sequence to be replaced with two newline character sequences when the email newline character was configured as the CRLF pair (which it must be when utilizing SMTP).

 

The updated SMTP Authentication contribution is available here.

 

The bug is not unique to the SMTP Authentication contribution and will occur with the baseline osCommerce email.php code whenever the email newline character is configured as CRLF.

 

The changes need to be made in the catalog/includes/classes/email.php and the catalog/admin/includes/classes/email.php files. In the add_text method (near line 149) your will find the following code:

 

$this->text = tep_convert_linefeeds(array("\r\n", "\n", "\r"), $this->lf, $text);

This code needs to be replace with the following code:

 

// Replace ASCII CR and LF characters with the configured LF character.
// Needs to be done in two passes to prevent doubling of line spacing.
$this->text = tep_convert_linefeeds(array("\r\n", "\n", "\r"), "xxCRLFxx", $text);
$this->text = tep_convert_linefeeds("xxCRLFxx", $this->lf, $this->text);

A similar change needs to be done in the add_html method (near line 160); however, in this case the original code looks like:

 

$this->html_text = tep_convert_linefeeds(array("\r\n", "\n", "\r"), $this->lf, $text);

and the modified code looks like:

 

 // Replace ASCII CR and LF characters with the configured LF character.
// Needs to be done in two passes to prevent doubling of line spacing.
$this->html_text = tep_convert_linefeeds(array("\r\n", "\n", "\r"), "xxCRLFxx", $text);
$this->html_text = tep_convert_linefeeds("xxCRLFxx", $this->lf, $this->html_text);

Ben

[♥Vger]

Some general advice if your emails are being rejected by the recipients mail server:

 

1. Go to www.trustedsource.org and input your mail servers' ip address - and find out if you're blacklisted. You can also go to spamcop.net and input your ip address to see if you're server is blacklisted.

 

2. If you control the mail server, like one poster, then use Port 26 instead of Port 25. Most abuse takes place using mail relayed via Port 25.

 

3. If you get blacklisted by Hotmail then even God won't be able to help you get it lifted!

 

Vger