Discussion:
[Advice needed] Best way to handle attachments?
Camaleón
2011-08-11 15:30:56 UTC
Permalink
Hello,

I'm facing a curious problem when I send attachments with Mutt from
command line.

I have a small self-made script that basically makes two things:

1/ Given a big file (~10/20 MiB), it splits into small chunks of data
(~250 KiB)

2/ Then it sends the resulting files using Mutt (each file is attached
per message, so if there are 10 files Mutt sends 10 messages)

I have to do this in order to send a chap programs and documents I
download from the web because he does not have access to Internet, only
to his e-mail account (which is also very restricted, limited to 512 KiB/
message).

All the process works fine but some of the files are wrongly encoded
which results in an error when the user tries to reconstruct the big file
from the received attachments.

For instance, I've noted that properly encoded attachments appear as
follows:

***
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=test0014
Content-Transfer-Encoding: base64
***

And bad ones are like this:

***
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test0015
Content-Transfer-Encoding: quoted-printable
***

So I tried to deal with this in two ways:

1. Enforcing Mutt to use "Content-Type: application/octet-stream" when I
call it using the script (that is, "mutt -e 'set content_type=application/
octet-stream' [...])

This works (I can see the body of the messages are encoded in that way)
but there are still some messages that encode the attachments as "text/
plain".

2. I've also thought in using a "~/.mime.types" file but I dunno how to
do this, I mean, mime types relies on filenames extensions
(.pdf/.txt/.ogg) and splitted files have no extension (file000, file001,
file002, file003...). I could rename those to some fancy filename
(file001.file, etc...) but I think it's overwelming for the task.

To be sincere, I'm not sure if the culprit here is the Gmail server (I
use my Gmail account to send the messages) because Mutt tends to do the
right things while Gmail is a bit... let's say "liberal" when it comes to
implement/interpret the standards :-)

So I wonder what would be the best way to bypass this or if someone has
had a previous experience similar to this and can share his findings...
Any idea is very welcome.

P.S. Using Mutt 1.5.18 (2008-05-17)

Greetings,
--
Camaleón
Jostein Berntsen
2011-08-11 15:37:42 UTC
Permalink
Post by Camaleón
Hello,
I'm facing a curious problem when I send attachments with Mutt from
command line.
1/ Given a big file (~10/20 MiB), it splits into small chunks of data
(~250 KiB)
2/ Then it sends the resulting files using Mutt (each file is attached
per message, so if there are 10 files Mutt sends 10 messages)
I have to do this in order to send a chap programs and documents I
download from the web because he does not have access to Internet, only
to his e-mail account (which is also very restricted, limited to 512 KiB/
message).
All the process works fine but some of the files are wrongly encoded
which results in an error when the user tries to reconstruct the big file
from the received attachments.
For instance, I've noted that properly encoded attachments appear as
***
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=test0014
Content-Transfer-Encoding: base64
***
***
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test0015
Content-Transfer-Encoding: quoted-printable
***
1. Enforcing Mutt to use "Content-Type: application/octet-stream" when I
call it using the script (that is, "mutt -e 'set content_type=application/
octet-stream' [...])
This works (I can see the body of the messages are encoded in that way)
but there are still some messages that encode the attachments as "text/
plain".
2. I've also thought in using a "~/.mime.types" file but I dunno how to
do this, I mean, mime types relies on filenames extensions
(.pdf/.txt/.ogg) and splitted files have no extension (file000, file001,
file002, file003...). I could rename those to some fancy filename
(file001.file, etc...) but I think it's overwelming for the task.
To be sincere, I'm not sure if the culprit here is the Gmail server (I
use my Gmail account to send the messages) because Mutt tends to do the
right things while Gmail is a bit... let's say "liberal" when it comes to
implement/interpret the standards :-)
So I wonder what would be the best way to bypass this or if someone has
had a previous experience similar to this and can share his findings...
Any idea is very welcome.
P.S. Using Mutt 1.5.18 (2008-05-17)
What about putting the files in 2-3 zip files and send him? Also update to mutt
1.5.21 which have many bug fixes after 1.5.18.

Jostein
Camaleón
2011-08-11 15:48:55 UTC
Permalink
Post by Jostein Berntsen
Post by Camaleón
I'm facing a curious problem when I send attachments with Mutt from
command line.
(...)
Post by Jostein Berntsen
Post by Camaleón
All the process works fine but some of the files are wrongly encoded
which results in an error when the user tries to reconstruct the big
file from the received attachments.
For instance, I've noted that properly encoded attachments appear as
***
Content-Type: application/octet-stream Content-Disposition: attachment;
filename=test0014 Content-Transfer-Encoding: base64
***
***
quoted-printable ***
(...)
Post by Jostein Berntsen
What about putting the files in 2-3 zip files and send him?
That's what I do now ;-)

Let's say I have to send to him several pdf files and one application. I
make a "tar" (or zip, or tar.bz2) with all the files and then split the
file in small pieces of data which I send him by e-mail (remember that I
can only send ~512 KiB on every message or they will be rejected by his e-
mail server :-/).
Post by Jostein Berntsen
Also update to mutt 1.5.21 which have many bug fixes after 1.5.18.
My main computer runs Debian lenny and won't be updated until the next
stable release... But I can test with an udpated version of Mutt I have
on another machine, good idea. I will post the results as soon as I can
run the test.

Greetings,
--
Camaleón
Chris Brennan
2011-08-11 15:57:26 UTC
Permalink
Post by Jostein Berntsen
What about putting the files in 2-3 zip files and send him? Also update to mutt
1.5.21 which have many bug fixes after 1.5.18.
Greetings Camaleón,

Aye, I was going to suggest a split archive as well, then there should
be no need to encode the attached split-archive, just leave it as it
is... just use your favorite archiver (g/b/7/zip/xz) and split them ...
I would choose one appropriate to your friends environment though, so
s/he is able to use the file w/o too much hassle on their end.
--
Post by Jostein Berntsen
Chris Brennan
--
A: Yes.
Q: Are you sure?
A: Because it reverses the logical flow of conversation.
Q: Why is top posting frowned upon?
http://xkcd.com/84/ | http://xkcd.com/149/ | http://xkcd.com/549/
GPG: D5B20C0C (6741 8EE4 6C7D 11FB 8DA8 9E4A EECD 9A84 D5B2 0C0C)
------------------------------------------------------------------------
Camaleón
2011-08-11 16:15:47 UTC
Permalink
Post by Chris Brennan
Post by Jostein Berntsen
What about putting the files in 2-3 zip files and send him? Also update
to mutt 1.5.21 which have many bug fixes after 1.5.18.
Greetings Camaleón,
Aye, I was going to suggest a split archive as well, then there should
be no need to encode the attached split-archive, just leave it as it
is... just use your favorite archiver (g/b/7/zip/xz) and split them ...
I would choose one appropriate to your friends environment though, so
s/he is able to use the file w/o too much hassle on their end.
Ah! My bad... I got it wrong.

You both meant to use the split functionality provided by the same
archiver app, right? Mmmm, not a bad idea, I was using the "split"
command to break the files... time to look into "man zipsplit", I guess.

Thanks much guys, will have to try both suggestions, an updated mutt and
zipsplit :-)

Greetings,
--
Camaleón
Camaleón
2011-08-11 16:26:37 UTC
Permalink
Post by Camaleón
Post by Chris Brennan
Post by Jostein Berntsen
What about putting the files in 2-3 zip files and send him? Also
update to mutt 1.5.21 which have many bug fixes after 1.5.18.
Greetings Camaleón,
Aye, I was going to suggest a split archive as well, then there should
be no need to encode the attached split-archive, just leave it as it
is... just use your favorite archiver (g/b/7/zip/xz) and split them ...
I would choose one appropriate to your friends environment though, so
s/he is able to use the file w/o too much hassle on their end.
Ah! My bad... I got it wrong.
You both meant to use the split functionality provided by the same
archiver app, right? Mmmm, not a bad idea, I was using the "split"
command to break the files... time to look into "man zipsplit", I guess.
"zsplit" cannot do it. It requires the splitted size is at least the size
of the smallest of the archived files. I will have to search another
one ;-(

Greetings,
--
Camaleón
Chris Brennan
2011-08-11 16:33:09 UTC
Permalink
Post by Camaleón
"zsplit" cannot do it. It requires the splitted size is at least the size
of the smallest of the archived files. I will have to search another
one ;-(
zip -s SIZE should do the trick ... split the archive at your designated
size at creation, not after the fact, that might be where
zipsplit/zsplit is dropping the ball ....
--
Post by Camaleón
Chris Brennan
--
A: Yes.
Q: Are you sure?
A: Because it reverses the logical flow of conversation.
Q: Why is top posting frowned upon?
http://xkcd.com/84/ | http://xkcd.com/149/ | http://xkcd.com/549/
GPG: D5B20C0C (6741 8EE4 6C7D 11FB 8DA8 9E4A EECD 9A84 D5B2 0C0C)
------------------------------------------------------------------------
Camaleón
2011-08-11 17:14:33 UTC
Permalink
Post by Chris Brennan
Post by Camaleón
"zsplit" cannot do it. It requires the splitted size is at least the
size of the smallest of the archived files. I will have to search
another one ;-(
zip -s SIZE should do the trick ... split the archive at your designated
size at creation, not after the fact, that might be where
zipsplit/zsplit is dropping the ball ....
Thanks but I don't see "-s" as an available parameter for my zip (2.32) :-
(.

Anyway, if I try it:

***
***@stt008:~/Desktop$ zip -s 512000 data *.pdf

zip error: Invalid command arguments (no such option: s)
***

Greetings,
--
Camaleón
Chris Brennan
2011-08-11 17:20:51 UTC
Permalink
Post by Camaleón
Post by Chris Brennan
Post by Camaleón
"zsplit" cannot do it. It requires the splitted size is at least the
size of the smallest of the archived files. I will have to search
another one ;-(
zip -s SIZE should do the trick ... split the archive at your designated
size at creation, not after the fact, that might be where
zipsplit/zsplit is dropping the ball ....
Thanks but I don't see "-s" as an available parameter for my zip (2.32) :-
(.
***
zip error: Invalid command arguments (no such option: s)
***
Greetings,
From my Debian 6-stable box, I have the following (line 152 of the grep
is where I got -s from....)

***@leviathan.xaerolimit.net:~$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
***@leviathan.xaerolimit.net:~$ grep split sssss
57: -d -s is (delete, split size) while -ds is (dot size)
151:Splits (archives created as a set of split files):
152: -s ssize create split archive with splits of size ssize, where
ssize nm
154: -sp pause after each split closed to allow changing disks
158: -sv be verbose about creating splits
170: Cannot update split archive, so use --out to out new archive:
171: zip in_split_archive newfile1 newfile2 --out out_split_archive
172: If input is split, output will default to same split size
173: Use -s=0 or -s- to turn off splitting to convert split to single file:
174: zip in_split_archive -s 0 --out out_single_file_archive
175: WARNING: If overwriting old split archive but need less splits,
176: old splits not overwritten are not needed but remain
210: -ds siz each dot is siz processed where siz is nm as splits (0
no dots)
***@leviathan.xaerolimit.net:~$
--
Post by Camaleón
Chris Brennan
--
A: Yes.
Post by Chris Brennan
Q: Are you sure?
Post by Camaleón
A: Because it reverses the logical flow of conversation.
Q: Why is top posting frowned upon?
http://xkcd.com/84/ | http://xkcd.com/149/ | http://xkcd.com/549/
GPG: D5B20C0C (6741 8EE4 6C7D 11FB 8DA8 9E4A EECD 9A84 D5B2 0C0C)
------------------------------------------------------------------------
Camaleón
2011-08-12 11:46:35 UTC
Permalink
Post by Chris Brennan
Post by Camaleón
Post by Chris Brennan
zip -s SIZE should do the trick ... split the archive at your
designated size at creation, not after the fact, that might be where
zipsplit/zsplit is dropping the ball ....
Thanks but I don't see "-s" as an available parameter for my zip (2.32)
:- (.
***
zip error: Invalid command arguments (no such option: s) ***
From my Debian 6-stable box, I have the following (line 152 of the grep
is where I got -s from....)
(...)
Post by Chris Brennan
152: -s ssize create split archive with splits of size ssize, where
ssize nm

(...)

I see, I'm running lenny here ;-(

Anyway, I'm testing with the rest of the proposals and it seems (at least
after running a small set of tests) that compressing the resulting files
after they're splitted is doing the trick. Annoying and time consuming
but at least the attachments (.zip) are being encoded in the right manner.

Greetings,
--
Camaleón
l***@users.sourceforge.net
2011-08-11 16:36:35 UTC
Permalink
Post by Camaleón
"zsplit" cannot do it. It requires the splitted size is at least the size
of the smallest of the archived files. I will have to search another
one ;-(
It wouldn't help you in any case.

Splitting binary files may create parts that are mis-classied as ascii/text.
You need to zip/gzip/bzip2 the parts in addition to make sure they are
classified and encoded correctly.
Alexander Gattin
2011-08-12 07:03:09 UTC
Permalink
Hello,

On Thu, Aug 11, 2011 at 03:30:56PM +0000, Camaleón
Post by Camaleón
All the process works fine but some of the files
are wrongly encoded which results in an error
when the user tries to reconstruct the big file
from the received attachments.
becaue you split archive, pieces start with random
data and e.g. 'file' utility may detect data type
wrongly. I don't know how mutt performs data type
autodetection, but if it involves the 'file'
utility, you may want to try substituting system
'file' with your own shellscript, which always
returns something like 'xxx: data' and resides in
e.g. ~/bin directory. Then you could run mutt
prefixed by PATH="$HOME/bin:$PATH" to override
system 'file' and force uniform type for al
attachments.
--
With best regards,
xrgtn
Camaleón
2011-08-12 12:01:12 UTC
Permalink
Post by Camaleón
Hello,
Post by Camaleón
All the process works fine but some of the files are wrongly encoded
which results in an error when the user tries to reconstruct the big
file from the received attachments.
becaue you split archive, pieces start with random data and e.g. 'file'
utility may detect data type wrongly. I don't know how mutt performs
data type autodetection, but if it involves the 'file' utility, you may
want to try substituting system 'file' with your own shellscript, which
always returns something like 'xxx: data' and resides in e.g. ~/bin
directory. Then you could run mutt prefixed by PATH="$HOME/bin:$PATH" to
override system 'file' and force uniform type for al attachments.
What puzzles me is that raw splitted files stored in my disk all look the
same. I mean, "file" returns the same information for all of them but
then, when I review my Gmail's sent folder I see 3 or 4 of the
attachments were bad encoded while the rest (20 or 25) are fine :-?

Besides, enforcing Mutt to use "set content_type=application/
octet-stream" is not respected and I really doubt this is up to Mutt
which seems to do it right but something that happens on the remote e-
mail server that "breaks" the files.

Greetings,
--
Camaleón
Alexander Gattin
2011-08-12 12:53:31 UTC
Permalink
Hello,

On Fri, Aug 12, 2011 at 12:01:12PM +0000, Camaleón
Post by Camaleón
What puzzles me is that raw splitted files
stored in my disk all look the same. I mean,
"file" returns the same information for all of
them but then, when I review my Gmail's sent
folder I see 3 or 4 of the attachments were bad
encoded while the rest (20 or 25) are fine :-?
So this is how attachments look on the receiving
end:

On Thu, Aug 11, 2011 at 03:30:56PM +0000, Camaleón
Post by Camaleón
For instance, I've noted that properly encoded
***
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=test0014
Content-Transfer-Encoding: base64
***
***
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test0015
Content-Transfer-Encoding: quoted-printable
***
?

And in your local ~/sent file they are all
application/octet-stream/base64?
Post by Camaleón
application/octet-stream" when I call it using
the script (that is, "mutt -e 'set
content_type=application/ octet-stream' [...])
This works (I can see the body of the messages
are encoded in that way) but there are still
some messages that encode the attachments as
"text/ plain".
Probably this sets encoding for the message text,
not for attachments?

In past gmail refused to deliver mails with
executable attachments (I don't remember did it contain
viruses or not), and it even looked inside
archives and it even autodetected archives with
spoofed filename extensions (probably gmail do run
some sortg of home-brewn 'file' utility on all
attachments regardless of their extensions and/or
content-type's). I was forced to use openssl
enc -aes to get my message pass through gmail.
--
With best regards,
xrgtn
Camaleón
2011-08-12 13:39:38 UTC
Permalink
Post by Camaleón
Hello,
Post by Camaleón
What puzzles me is that raw splitted files stored in my disk all look
the same. I mean, "file" returns the same information for all of them
but then, when I review my Gmail's sent folder I see 3 or 4 of the
attachments were bad encoded while the rest (20 or 25) are fine :-?
For instance, I've noted that properly encoded attachments appear as
***
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=test0014
Content-Transfer-Encoding: base64
***
***
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test0015
Content-Transfer-Encoding: quoted-printable
***
?
Yes, that was extracted from two e-mail messages I sent. The former is
encoded okay and the latter is the one the user has problems when tries
to reconstruct the original file. I have to manually resend the original
splitted file so he can proceed with no errors.
Post by Camaleón
And in your local ~/sent file they are all
application/octet-stream/base64?
I drop the files under a folder in my /home/user/Desktop. Once I run the
"split" command, files in there (i.e., test0001, test0002, test0003,
file0004, file0005...) are like this:

***@stt008:~$ file Desktop/enviar/test0001
Desktop/enviar/test0001: data

The problem comes when I send that files via e-mail. I have configured
Mutt to directly contact Gmail smtp server (smtp_url = "smtp://
***@smtp.gmail.com:587/"), so I have no local "~/sent" folder, sent
messages are stored remotely, in Gmail's servers.
Post by Camaleón
Post by Camaleón
1. Enforcing Mutt to use "Content-Type: application/octet-stream" when
I call it using the script (that is, "mutt -e 'set
content_type=application/ octet-stream' [...])
This works (I can see the body of the messages are encoded in that way)
but there are still some messages that encode the attachments as "text/
plain".
Probably this sets encoding for the message text, not for attachments?
I think it should set all of the mime parts of the message :-? In fact,
there are only a few e-mails that fail, the rest are encoded fine.
Post by Camaleón
In past gmail refused to deliver mails with executable attachments (I
don't remember did it contain viruses or not), and it even looked inside
archives and it even autodetected archives with spoofed filename
extensions (probably gmail do run some sortg of home-brewn 'file'
utility on all attachments regardless of their extensions and/or
content-type's). I was forced to use openssl enc -aes to get my message
pass through gmail.
That was another thing I was thinking about: using PGP/GPG and sign the
messages to force encoding and will be the next to try if I still
encounter problems when sending the attachments.

Greetings,
--
Camaleón
Continue reading on narkive:
Loading...