IMAP: getBodyRecursive(): Need to convert message to UTF-8

Trouble installing? Post questions and find answers.

IMAP: getBodyRecursive(): Need to convert message to UTF-8

Postby mkuron » Tue Nov 15, 2011 8:39 pm

Today I received a "spam mail of death" that caused my iPhone (on iOS 5.0.1) to complain that it couldn't connect to my Z-Push 1.5.5 server. Looking at the logs on my iPhone, it turned out that it was an encoding issue. Apparently, if the iPhone encounters a message that are not UTF8 characters (in this case, the email was in ISO-8859-1, which I guess is nothing overly uncommon), it keeps keeps on re-requesting that message and eventually gives up talking to the server entirely (instead of just skipping over the message or something).

The fix turns out to be really simple: in imap.php in getBodyRecursive(), I neeed to replace
Code: Select all
         $body .= $message->body; 

with
Code: Select all
         {
            if (isset($message->headers['content-transfer-encoding']) && $message->headers['content-transfer-encoding'] == 'binary' && isset($message->ctype_parameters['charset']))
                $body .= mb_convert_encoding($message->body, 'UTF-8', $message->ctype_parameters['charset']);
            else
                $body 
.= $message->body;
        } 

to properly handle different encodings.

As I just started using Z-Push very recently, I'm not quite sure how Z-Push handles encoding on incoming messages (from what I've seen so far, it doesn't really appear to be aware of it at all, but I'm probably wrong).

Also, does one of the developers here know if the issue of the iPhone keeping on re-requesting the message with the invalid characters and subsequently giving up on the server entirely could be solved on the Z-Push side (i.e. by skipping over the message when certain conditions are met)? From my perspective, this seems like a recoverable error, but I may be wrong.
Since a malicious sender could send pretty much any character in an email, it doesn't seem like a great idea to assume that an email doesn't contain any invalid characters.
----

EDIT 2011-11-17: modified the code snipped to only convert the encoding for content-transfer-encoding = binary. Today I received an email that broke z-push because of this missing check. Arrgh, why can't everybody just be using UTF-8...
----

EDIT 2011-11-18: Today I received a message that had an attachment with umlauts in the file name. Of course, that broke zpush again. The fix is simple too: in imap.php in GetMessage about 15 lines from the bottom, replace
Code: Select all
$attachment->displayname = iconv("UTF-8","UTF-8//IGNORE",$attname); 

with
Code: Select all
$attachment->displayname = $attname



How about doing something like
Code: Select all
$body = iconv("UTF-8","UTF-8//IGNORE",$body); 

over the entire message, headers, etc. before sending it to the client?
mkuron
 
Posts: 5
Joined: Tue Nov 08, 2011 11:00 pm

Re: IMAP: getBodyRecursive(): Need to convert message to UTF

Postby fertig » Thu Dec 01, 2011 6:11 pm

Hi,

it's even worse: Mails without the "Content-Type" line and with non ascii encoding (in my case latin1 too, generated by my broken autoresponder) will cause z-push (1.5.6rc and 2.0alpha) to stop working. It's only saying "cannot connect", which is a bad message here.

If I hardcode an Content-Type to the Mails, they're displayed. Somewhere in the code (I didn't find it yet) z-push is relaying either on ctype_parameters["charset"] or headers["content-type"].

A broken mail should never be able to kill the mailserver...


EDIT: it seems, that my problem is not z-push, it's the Exchange part of the iphones Mail Client (testet with iOS 5 and 5.01). The iphone gets the push "new message", just the folder view does not work. It works (with broken umlauts, but this is ok in this case) on an samsung android with the same settings. Nerverless there should be a workaround.

Christian
Last edited by fertig on Fri Dec 02, 2011 11:23 am, edited 2 times in total.
fertig
 
Posts: 3
Joined: Thu Dec 01, 2011 6:05 pm

Re: IMAP: getBodyRecursive(): Need to convert message to UTF

Postby mkuron » Fri Dec 02, 2011 7:33 am

@fertig: If you PM me the full headers and body of that email, I could try and see if I can come up with a patch. I'm not a z-push developer, but I've figured out to some extent how the encoding stuff in z-push works.

In fact, adding
Code: Select all
$body iconv("UTF-8","UTF-8//IGNORE",$body);  

to getBodyRecursive() before the return might already solve your issue (by simply stripping out all invalid characters).

Does anybody here know I can send patches to the developers? Once you've figured out where to fix it, all the fixes are rather short and simple.
mkuron
 
Posts: 5
Joined: Tue Nov 08, 2011 11:00 pm

Re: IMAP: getBodyRecursive(): Need to convert message to UTF

Postby fertig » Fri Dec 02, 2011 11:29 am

@fertig: If you PM me the full headers and body of that email, I could try and see if I can come up with a patch. I'm not a z-push developer, but I've figured out to some extent how the encoding stuff in z-push works.

yes. I'm just going back to a vanilla 1.5.6rc and a fresh mail.

In fact, adding

Code: Select all
$body = iconv("UTF-8","UTF-8//IGNORE",$body);
to getBodyRecursive() before the return might already solve your issue (by simply stripping out all invalid characters).

hmm, I'm not sure. I've tested your changes in the first post and it did not help. I did a real ugly test adding

Code: Select all
$mail = "Content-Type: text/plain; charset=iso8859-1\n" . $mail;


in function GetMessage, after $mail = @imap_fetchheader. That worked, but will probably destroy the coding of mails elsewhere.
fertig
 
Posts: 3
Joined: Thu Dec 01, 2011 6:05 pm

Re: IMAP: getBodyRecursive(): Need to convert message to UTF

Postby fertig » Fri Dec 02, 2011 3:12 pm

In fact, adding

Code: Select all
$body = iconv("UTF-8","UTF-8//IGNORE",$body);


to getBodyRecursive() before the return might already solve your issue (by simply stripping out all invalid characters).


you're absolutly right. It did solve it. I only tried your patches up to "2011-11-18", not the last one...
fertig
 
Posts: 3
Joined: Thu Dec 01, 2011 6:05 pm

Re: IMAP: getBodyRecursive(): Need to convert message to UTF

Postby mkuron » Fri Dec 02, 2011 4:07 pm

Glad that that helped. As far as I can tell, the UTF-8//IGNORE fix should fix all invalid characters in messages bodies and attachment names. I have not yet run into headers that screwed up Z-Push and/or iOS Mail, but I'm pretty sure it could be applied there as well if necessary.

Attached is a comprehensive diff of all my changes to Z-Push as of today to fix invalid UTF8 and screwed-up umlauts.
imap.php.diff.txt
imap.php patch to fix broken umlauts and invalid UTF8
(2.31 KiB) Downloaded 303 times
mkuron
 
Posts: 5
Joined: Tue Nov 08, 2011 11:00 pm


Return to Bugs

Who is online

Users browsing this forum: No registered users and 1 guest

cron