save attachment content as if they are in pager

Discussion:

Yubin Ruan

2018-02-02 02:25:06 UTC

Hi,

I try to save a multipart/alternative attachment to file but all the content
seems to be messed up. When view in a pager, it look good:

--------------------------------
LinkedIn Highlights

Should I tell coworkers my salary?

264 people are talking about this

https://www.linkedin.com/comm/search/results/content/?keywords=Should+I+tell+coworkers+my+salary%3F&origin=FED_EMAIL&anchorTopic=506458&midToken=AQFSR-6AohV2qQ&trk=eml-email_feed_ecosystem_digest_01-hero-1-null&trkEmail=eml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=urn%3Ali%3Apage%3Aemail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
---------------------------------

However, when saved to a file, it become:
(which is the same as in editor when you press 'e' to edit the email)

------------------------------------
LinkedIn Highlights

Should I tell coworkers my salary?

264 people are talking about this

https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D

=20
-----------------------------------

I know the tailing = at the end is to enable the pager to prettify thing. But
can we save directly from the pager?

--
Yubin

Patrick Shanahan

2018-02-02 02:35:18 UTC

Permalink

Post by Yubin Ruan
Hi,
I try to save a multipart/alternative attachment to file but all the content
--------------------------------
LinkedIn Highlights
Should I tell coworkers my salary?
264 people are talking about this
https://www.linkedin.com/comm/search/results/content/?keywords=Should+I+tell+coworkers+my+salary%3F&origin=FED_EMAIL&anchorTopic=506458&midToken=AQFSR-6AohV2qQ&trk=eml-email_feed_ecosystem_digest_01-hero-1-null&trkEmail=eml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=urn%3Ali%3Apage%3Aemail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
---------------------------------
(which is the same as in editor when you press 'e' to edit the email)
------------------------------------
LinkedIn Highlights
Should I tell coworkers my salary?
264 people are talking about this
https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
=20
-----------------------------------
I know the tailing = at the end is to enable the pager to prettify thing. But
can we save directly from the pager?

just a guess but I would say your locale is not correct. the link appears
correct for me and in my editor, joe.

or your content-type is incorrect:
Content-Type: text/plain; charset=us-ascii

--
(paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri
http://en.opensuse.org openSUSE Community Member facebook/ptilopteri
Registered Linux User #207535 @ http://linuxcounter.net
Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode

Cameron Simpson

2018-02-02 03:06:35 UTC

Permalink

Post by Yubin Ruan
I try to save a multipart/alternative attachment to file but all the content
--------------------------------
LinkedIn Highlights
Should I tell coworkers my salary?
264 people are talking about this
https://www.linkedin.com/comm/search/results/content/?keywords=Should+I+tell+coworkers+my+salary%3F&origin=FED_EMAIL&anchorTopic=506458&midToken=AQFSR-6AohV2qQ&trk=eml-email_feed_ecosystem_digest_01-hero-1-null&trkEmail=eml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=urn%3Ali%3Apage%3Aemail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
---------------------------------
(which is the same as in editor when you press 'e' to edit the email)
------------------------------------
LinkedIn Highlights
Should I tell coworkers my salary?
264 people are talking about this
https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
=20
-----------------------------------
I know the tailing = at the end is to enable the pager to prettify thing. But
can we save directly from the pager?

I don't think this is anything to do with locales.

I'm presuming you're talking about saving something from the attachment menu?
And not about saving a message complete.

What you show above is interesting.

It looks to me like the second version, the folded lines ending in "=", is
quoted-printable, which encoded most ASCII range text as itself, with escape
sequences for various bytes not in the common lower range and of course for the
escape character "=" itself. Its purpose it to get text through various systems
undamaged while keeping it visually fairly readable.

Normally, that would be the attachment's Content-Transfer-Encoding, which I
would expect to be decoded on save, because the attachment is no longer
enclosed in a mail message structure.

Apparently not the case.

Can you describe _exactly_ what procedure your went through to save this
message, from the index view (list of message titels/authors only) through to
saving the file? So that we can reproduce what you've done and understand it?

Thanks,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

Yubin Ruan

2018-02-02 03:27:46 UTC

Permalink

Post by Cameron Simpson

I don't think this is anything to do with locales.
I'm presuming you're talking about saving something from the attachment
menu? And not about saving a message complete.
What you show above is interesting.
It looks to me like the second version, the folded lines ending in "=", is
quoted-printable, which encoded most ASCII range text as itself, with escape
sequences for various bytes not in the common lower range and of course for
the escape character "=" itself. Its purpose it to get text through various
systems undamaged while keeping it visually fairly readable.
Normally, that would be the attachment's Content-Transfer-Encoding, which I
would expect to be decoded on save, because the attachment is no longer
enclosed in a mail message structure.
Apparently not the case.
Can you describe _exactly_ what procedure your went through to save this
message, from the index view (list of message titels/authors only) through
to saving the file? So that we can reproduce what you've done and understand
it?

Procedure:
1. enter the attachment view using 'v'
2. go the attachment you want to save,
3. press 's', which will prompt you for a file name to save the file with

FYI: those tailing = are not generated by Mutt. I am using Gmail, and through
Gmail's web interface I can confirm that the original mail already have those
tailing = .

--
Yubin

Barton Janes

2018-02-02 03:32:29 UTC

Permalink

The trailing = is usually caused by the text encoding of "quoted Printable"

https://en.wikipedia.org/wiki/Quoted-printable

~Barton

Post by Yubin Ruan

Post by Cameron Simpson

I don't think this is anything to do with locales.
I'm presuming you're talking about saving something from the attachment
menu? And not about saving a message complete.
What you show above is interesting.
It looks to me like the second version, the folded lines ending in "=", is
quoted-printable, which encoded most ASCII range text as itself, with escape
sequences for various bytes not in the common lower range and of course for
the escape character "=" itself. Its purpose it to get text through various
systems undamaged while keeping it visually fairly readable.
Normally, that would be the attachment's Content-Transfer-Encoding, which I
would expect to be decoded on save, because the attachment is no longer
enclosed in a mail message structure.
Apparently not the case.
Can you describe _exactly_ what procedure your went through to save this
message, from the index view (list of message titels/authors only) through
to saving the file? So that we can reproduce what you've done and understand
it?

1. enter the attachment view using 'v'
2. go the attachment you want to save,
3. press 's', which will prompt you for a file name to save the file with
FYI: those tailing = are not generated by Mutt. I am using Gmail, and through
Gmail's web interface I can confirm that the original mail already have those
tailing = .
--
Yubin

Yubin Ruan

2018-02-02 03:55:21 UTC

Permalink

Post by Barton Janes
The trailing = is usually caused by the text encoding of "quoted Printable"
https://en.wikipedia.org/wiki/Quoted-printable

Are there any ways to save the decoded message rather than the encoded one?

To make it more general, is it possible to save the content generated by some
auto_view commands, without having to split the email and type those commands
separately in command line?

--
Yubin

Barton Janes

2018-02-02 04:01:43 UTC

Permalink

Not that I'm aware of. Quoted Printabe is one of the oldest encodings
in email and every email client I know of encodes for it when the
default encoding method is plain text, so they send with the "=" sign
in plain text so the receiving email client can decode the line breaks
per the encoding.
~Barton

Post by Yubin Ruan

Post by Barton Janes
The trailing = is usually caused by the text encoding of "quoted Printable"
https://en.wikipedia.org/wiki/Quoted-printable

Are there any ways to save the decoded message rather than the encoded one?
To make it more general, is it possible to save the content generated by some
auto_view commands, without having to split the email and type those commands
separately in command line?
--
Yubin

Cameron Simpson

2018-02-02 07:15:38 UTC

Permalink

Post by Yubin Ruan

Post by Barton Janes
The trailing = is usually caused by the text encoding of "quoted Printable"
https://en.wikipedia.org/wiki/Quoted-printable

Are there any ways to save the decoded message rather than the encoded one?
To make it more general, is it possible to save the content generated by some
auto_view commands, without having to split the email and type those commands
separately in command line?

Well, you can set pipe_decode=yes and then pipe to a save command:

| cat >saved_message.txt

Pipe_decode is normally "no", so that piping a message goes in the raw form to
whatever command is receiving it. But setting it to "yes" gets you what would
show in the pager. I think. Need to experiment.

Personally I'm surprised that the Content-Transfer-Encoding isn't decoded for
saving things; it is there purely to get the original content through the mail
system and should be decoded. I need to do some testing.

Yubin, your sample messages seem to be from linkedin.com; is that the case? If
so, I've any number of them in my own inbox I can try.

Cheers,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

Yubin Ruan

2018-02-02 12:46:31 UTC

Permalink

Post by Cameron Simpson

Post by Yubin Ruan

Post by Barton Janes
The trailing = is usually caused by the text encoding of "quoted Printable"
https://en.wikipedia.org/wiki/Quoted-printable

Are there any ways to save the decoded message rather than the encoded one?
To make it more general, is it possible to save the content generated by some
auto_view commands, without having to split the email and type those commands
separately in command line?

| cat >saved_message.txt
Pipe_decode is normally "no", so that piping a message goes in the raw form
to whatever command is receiving it. But setting it to "yes" gets you what
would show in the pager. I think. Need to experiment.

This work as expected. Thanks.

Post by Cameron Simpson
Personally I'm surprised that the Content-Transfer-Encoding isn't decoded
for saving things; it is there purely to get the original content through
the mail system and should be decoded. I need to do some testing.

There is a Content-Transfer-Encoding in the text/plain part, but it does not
seem to work as you described. Probably mutt have skipped that:

------=_Part_3237106_526482122.1517522433475
Content-Type: text/plain;charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-ID: text-body

LinkedIn Highlights

Should I tell coworkers my salary?

264 people are talking about this

https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D

=20
-----------------------------------

Post by Cameron Simpson
Yubin, your sample messages seem to be from linkedin.com; is that the case?
If so, I've any number of them in my own inbox I can try.

Yes it is from linkedin.com. I have forwarded you a copy.

--
Yubin

Cameron Simpson

2018-02-02 20:48:25 UTC

Permalink

Post by Yubin Ruan

There is a Content-Transfer-Encoding in the text/plain part, but it does not
------=_Part_3237106_526482122.1517522433475
Content-Type: text/plain;charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-ID: text-body
LinkedIn Highlights
Should I tell coworkers my salary?
264 people are talking about this
https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D
=20
-----------------------------------

Post by Cameron Simpson
Yubin, your sample messages seem to be from linkedin.com; is that the case?
If so, I've any number of them in my own inbox I can try.

Yes it is from linkedin.com. I have forwarded you a copy.

Thank you.

I've experimented with your message and I think I know what is happening.

If I save the text/plain part, I get what you wanted: the long line, and no
quoted-printable encoding, like this:

LinkedIn Highlights

Should I tell coworkers my salary?

264 people are talking about this

https://www.linkedin.com/comm/search/results/content/?keywords=Should+I+tell+coworkers+my+salary%3F&origin=FED_EMAIL&anchorTopic=506458&midToken=AQFSR-6AohV2qQ&trk=eml-email_feed_ecosystem_digest_01-hero-1-null&trkEmail=eml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=urn%3Ali%3Apage%3Aemail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D

However, if I save the multipart/alternative part I get this:

------=_Part_3237106_526482122.1517522433475
Content-Type: text/plain;charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-ID: text-body

LinkedIn Highlights

Should I tell coworkers my salary?

264 people are talking about this

https://www.linkedin.com/comm/search/results/content/?keywords=3DShould+I+t=
ell+coworkers+my+salary%3F&origin=3DFED_EMAIL&anchorTopic=3D506458&midToken=
=3DAQFSR-6AohV2qQ&trk=3Deml-email_feed_ecosystem_digest_01-hero-1-null&trkE=
mail=3Deml-email_feed_ecosystem_digest_01-hero-1-null-null-6mvow3%7Ejd475j9=
d%7Ey4-null-neptune%2Fsearch%2Eresults%2Econtent&lipi=3Durn%3Ali%3Apage%3Ae=
mail_email_feed_ecosystem_digest_01%3BrxyHuw3NTWi4n8fHNW81ig%3D%3D

=20
-----------------------------------

Which is what you saw. And that is correct behaviour in both cases. The
text/plain gets saved in its original form. The multipart/alternative _also_
gets saved in its original form, but that form is a MIME body that itself
contains, encoded, the text/plain _and_ the text/html parts.

So I believe you misaimed and saved the multipart attachment, not the
individual text/plain attachment.

To elaborate, the multipart/alternative section is a well defined thing on its
own. It is a container format for other message parts. As such is has a leading
"----" MIME marker for the start of the first body part, then the optional
headers indicating the type, encoding and id of the part, and then the part
itself, encoded for storage in the multipart/alternative wrapper.

In your case, that looks like this:

------=_Part_3237106_526482122.1517522433475
Content-Type: text/plain;charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-ID: text-body

The "----....." part is a unique marker, guarrenteed not to exist in the
_encoded_ body; this is reliable because it is chosen by the programme which
made the multipart/alternative itself.

The content-type of the first part is text/plain; there will be the text/html
afterwards.

The content-transfer-encoding is how the text/plain is encoded for storage
within the multipart/alternative wrapper. It is quoted-printable, which
presents a lot of the printable ASCII byte range unchanged, but encodes other
values as "=XX", and folds the text nonsemanticly with "-\n".

The =XX encoding is to make the text survive passage through mail systems which
support only a limited (or intersecting) range of characters, such as a
non-8-bit clean system, or an EBCEDIC system, etc. The "=\n" line folding it to
accomodate passage through mail systems which read the message as lines of text
with a limited buffer length - the physical lines are thus no longer than a
certain length.

The opening paragraph of the quoted-printable spec says:

The Quoted-Printable encoding is intended to represent data that
largely consists of octets that correspond to printable characters
in the US-ASCII character set. It encodes the data in such a way
that the resulting octets are unlikely to be modified by mail
transport. If the data being encoded are mostly US-ASCII text, the
encoded form of the data remains largely recognizable by humans.

You can read all of this in RFC2045, here:

https://tools.ietf.org/rfcmarkup?rfc=2045

The quoted-printable encoding is part of that document, and specified here:

https://tools.ietf.org/rfcmarkup?rfc=2045#section-6.7

The whole thing has a nice table of contents, and makes for a good read
actually if you're interested.

Cheers,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

Erik Christiansen

2018-02-02 07:25:25 UTC

Permalink

Post by Yubin Ruan

Post by Barton Janes
The trailing = is usually caused by the text encoding of "quoted Printable"
https://en.wikipedia.org/wiki/Quoted-printable

Are there any ways to save the decoded message rather than the encoded one?

The Pager menu has:
<Esc>s decode-save make decoded copy (text/plain) and delete

Have you tried it? (Pressing '?' in the menu shows the key mappings
available to you, including any mappings you've added.)

For DIY handling of quoted-printable, one could pipe to:
$ apt-cache search qprint
qprint - encoder and decoder for quoted-printable encoding
qprint-doc - qprint quoted-printable encoder/decoder (documentation)

Post by Yubin Ruan
To make it more general, is it possible to save the content generated by some
auto_view commands, without having to split the email and type those commands
separately in command line?

If you wanted to select a specific multipart alternative in the
Attachment Menu, then there is only 's' or '|'. With a key mapping to
invoke a macro, you could pipe the alternative to qprint, outputting to
a file - with minimal keystrokes.

Erik