OT: culprit in MIME recoding and breaking signatures

[Ian Zimmerman]

The winner appears to be Perl, namely the Mail::Audit module (and
whatever other modules it relies on). I had a couple of scripts that
did gentle transformations of incoming mails. The transformations were
supposed to only ever touch the headers, but I used a Mail::Audit object
to write back the entire message including the body. I had complete
trust that the body would be bitwise identical to the original, but not
so.

This is a very unpleasant surprise, I have thought of Perl and the more
popular CPAN modules as the good old workhorse which would always do the
job efficiently and reliably - if not always legibly :) I intentionally
preferred Perl for this task over some combination of procmail, formail,
sed and pcregrep which would have worked. My world is crumbling :(

[Kurt Hackenberg]

Post by Ian Zimmerman
The winner appears to be Perl, namely the Mail::Audit module (and
whatever other modules it relies on). I had a couple of scripts that
did gentle transformations of incoming mails. The transformations were
supposed to only ever touch the headers, but I used a Mail::Audit object
to write back the entire message including the body. I had complete
trust that the body would be bitwise identical to the original, but not
so.

What change did it make? I suppose it might use a different MIME
transfer encoding, changing the representation but not the meaning.

Also, the most common variants of mbox are known to break cryptographic
signatures, with the notorious ">From " escaping.

[Ian Zimmerman]

Post by Kurt Hackenberg

Post by Ian Zimmerman
The winner appears to be Perl, namely the Mail::Audit module (and
whatever other modules it relies on). I had a couple of scripts that
did gentle transformations of incoming mails. The transformations were
supposed to only ever touch the headers, but I used a Mail::Audit object
to write back the entire message including the body. I had complete
trust that the body would be bitwise identical to the original, but not
so.

What change did it make? I suppose it might use a different MIME
transfer encoding, changing the representation but not the meaning.

It keeps the original QP encoding, but it recodes some (but not all)
sequences of spaces as =20, and vice versa. Please see a recent thread
I started here for exact details, and also my blog:

https://very.loosely.org/itz-blog/

Post by Kurt Hackenberg
Also, the most common variants of mbox are known to break
cryptographic signatures, with the notorious ">From " escaping.

I stopped using mbox some years ago, and this could have been one of the
reasons (though the main reason was fragility when taking apart folders
and reassembling them).

[Kevin J. McCarthy]

Post by Ian Zimmerman
The winner appears to be Perl, namely the Mail::Audit module (and
whatever other modules it relies on). I had a couple of scripts that
did gentle transformations of incoming mails.

Congratulations on tracking this down.

Post by Ian Zimmerman
This is a very unpleasant surprise, I have thought of Perl and the more
popular CPAN modules as the good old workhorse which would always do
the job efficiently and reliably - if not always legibly :) I
intentionally preferred Perl for this task over some combination of
procmail, formail, sed and pcregrep which would have worked. My world
is crumbling :(

I'm a bit surprised too. Although it has fallen out of favor, I have
used and relied on Perl for a long time too, and it never let me down.