Subject charset issue (was: merging/linking tagged messages into thread: brilliant!)

[Alexander Dahl]

Hi folks,

I know, I'm a little off-topic, but I have a question about a charset issue:

On Tue, Aug 28, 2007 at 02:39:54PM -0400, Cristóbal M. Palmer wrote:
^^

Cristóbal Palmer

^

I have a clean mutt 1.5.16 running on a linux with locale "***@euro" which
is in fact iso8859-15. I connect with PuTTY configured to the same locale.
The above quoted message has charset utf8 both in Subject and message body.
Mutt displays message body correctly but Subject is not correctly decoded as
you can see above.

Did I miss some in my configuration (I can't imagine), is this a bug in
mutt I should report or is it a wrong coded mail?

Greets
Alex

[Cristóbal M. Palmer]

Post by Alexander Dahl
Did I miss some in my configuration (I can't imagine), is this a bug in
mutt I should report or is it a wrong coded mail?

Fair chance it's me. I've seen the same miscoding in my boss's
inbox. He uses alpine:

http://www.washington.edu/alpine/

I've been teasing him about this mercilessly... oops? It'd be awesome
if somebody on this list could point out where I'm erring. FWIW if I
mail myself something to my gmail account, the ó in the subject shows
up fine.

Cheers,

[Kyle Wheeler]

It'd be awesome if somebody on this list could point out where I'm
erring.

Hmm... Presumably, that's set in your muttrc via 'set realname="..."',
right? My first guess would be that you need to tell mutt what charset
you've encoded the muttrc file in (it assumes ascii unless you set
config_charset to something). You'll need to set the config_charset
before mutt reads any non-ascii characters (i.e. before you set
realname).

FWIW if I mail myself something to my gmail account, the ó in the
subject shows up fine.

The subject? You mean your *From* header, right?

My guess is that gmail is better at guessing malformed headers.
Guessing malformed headers is an *artform*, and mutt isn't the best
guesser in the world.

~Kyle
- --
Only a mediocre person is always at his best.
-- Somerset Maugham

[Eyolf Østrem]

Post by Kyle Wheeler
Hmm... Presumably, that's set in your muttrc via 'set realname="..."',
right? My first guess would be that you need to tell mutt what charset
you've encoded the muttrc file in (it assumes ascii unless you set
config_charset to something). You'll need to set the config_charset
before mutt reads any non-ascii characters (i.e. before you set
realname).

Hey! That's interesting! I've always made sure I only use ascii-ified
versions of my last name in that kind of fields, just in case, but
this seems to work. At least a test mail I sent to myself did. Let's
see how this one fares...

[Kyle Wheeler]

Post by Eyolf Østrem

Post by Kyle Wheeler
Hmm... Presumably, that's set in your muttrc via 'set
realname="..."', right? My first guess would be that you need to
tell mutt what charset you've encoded the muttrc file in (it
assumes ascii unless you set config_charset to something). You'll
need to set the config_charset before mutt reads any non-ascii
characters (i.e. before you set realname).

Hey! That's interesting! I've always made sure I only use
ascii-ified versions of my last name in that kind of fields, just in
case, but this seems to work. At least a test mail I sent to myself
did. Let's see how this one fares...

Looks good to me!

~Kyle

[Alexander Dahl]

Post by Eyolf Østrem
this seems to work. At least a test mail I sent to myself did. Let's
see how this one fares...

Looks fine on my side. :-)

[Alexander Dahl]

Hi Kyle,

Post by Kyle Wheeler
The subject? You mean your *From* header, right?

Sure the From header, sorry about that. O:-)

Greets
Alex

[Kyle Wheeler]

Post by Alexander Dahl
^^

Cristóbal Palmer

^

Headers are only allowed to be sent in US-ASCII, and the From header
in Cristóbal's emails should, rather than be flat utf-8, be encoded
like this: =?UTF-8?Q?Crist=C3=B3bal?.

Post by Alexander Dahl
Did I miss some in my configuration (I can't imagine), is this a bug
in mutt I should report or is it a wrong coded mail?

Yup, it is a wrongly coded mail (it's breaking a MUST in RFC 2822).

~Kyle
- --
Life is too important to be taken seriously.
-- Oscar Wilde

[Alain Bench]

Hello Cristóbal,
Your name is misencoded in all your mails, probably because the
"From:" field is not written by Mutt. May I suggest you to: Set $from
and $realname, verify $use_from=yes, and not set $send_charset (the
default is better)? You can mail me to validate the changes.


Bye! Alain.

[Sander Smeenk]

It shows up as a proper 'ó' at my screen. But probably because i am in a
native UTF-8 environment. What you quoted back looks like an utf8
character in latin1 encoding ;)

[d***@wayne.edu]

Post by Alexander Dahl
Hi folks,
^^

Crist?bal Palmer

^
is in fact iso8859-15. I connect with PuTTY configured to the same locale.
The above quoted message has charset utf8 both in Subject and message body.
Mutt displays message body correctly but Subject is not correctly decoded as
you can see above.
Did I miss some in my configuration (I can't imagine), is this a bug in
mutt I should report or is it a wrong coded mail?

This seems like a good time for me to jump in ... for me, the "from" line is rendered correctly but the signature is not.

But the point of my post is this: the correctly-shown "from" is the very first time, ever, in my history with mutt that an accented character has been displayed. Normally, I get either a single question-mark or a double one. I have tried setting my terminal app to UTF-8, ASCII, MacRoman, etc., but none of that makes it appear correctly. I looked online at the wiki/faq/charset, and as it instructed I looked at $LANG. I found that echo $LANG returned nothing. (I'm on MacOSX 10.4, using Terminal.app). So I followed the wiki and did this: export LANG=en_US.ISO-8859-1. Still no dice, although the accented 'o' in 'Cristobal' stopped being rendered correctly, and now looks like this: Crist\xxx\xxxbal, where the x's are digits.

Still following the wiki, I tried checking to see if mutt picked up on the fact that I changed $LANG, by typing :set &charset ?charset. Mutt revealed that charset="US-ASCII". Meanwhile, my terminal application (not bash) is still on UTF-8.

For that matter, terminal-vim never displays accented characters correctly, either, whether it's called by mutt or not. But I don't think vim is the culprit, because mutt displays them wrongly in the index view and when reading messages.

What do I do? Thanks much,
-G

[Alain Bench]

Hello,

Post by d***@wayne.edu
the correctly-shown "from" is the very first time, ever, in my history
with mutt that an accented character has been displayed. Normally,
I get either a single question-mark or a double one.

This means that your terminal does UTF-8, but $charset doesn't. You
can pick nearly any charset. But Terminal.app, current locale, and
Mutt's $charset have to agree.

Post by d***@wayne.edu
export LANG=en_US.ISO-8859-1 [...] typing :set &charset ?charset. Mutt
revealed that charset="US-ASCII".

Something interferes, perhaps:

- Env vars LC_CTYPE or LC_ALL overrule LANG: Unset them.

- MacOS-X wants a specific spelling of locales: Try "en_US.ISO8859-1".

- This locale is not installed: Install it, generate it, or pick one
already available in "locale -a".

At each step, restart Mutt and check the ":set &charset ?charset"
until it shows "iso-8859-1". If nothing works, please report back, and
say us the version of MacOS-X.


Bye! Alain.

[d***@wayne.edu]

Post by Alain Bench

Post by d***@wayne.edu
the correctly-shown "from" is the very first time, ever, in my history
with mutt that an accented character has been displayed. Normally,
I get either a single question-mark or a double one.

This means that your terminal does UTF-8, but $charset doesn't. You
can pick nearly any charset. But Terminal.app, current locale, and
Mutt's $charset have to agree.

I made all three agree on en_US.UTF-8, and the result was terrible. Most accented chars still were garbled. Plus, I sent a test mail to myself with some accented chars, and although they looked fine when I sent it, they were garbled upon receipt (it wasn't slashes and digits, it was UTF?..blah blah). Perhaps my school's IMAP server is using a particular charset. I have no idea how these things work.

I was able to get mutt and bash to agree on en_US.ISO8859-1, but there is no option for that in Terminal.app's preferences. I only see "Western (ISO Latin 1)" and "Western (ISO Latin 9)" [In fact, I even opened up Terminal.app's .plist to see if I could manually put in the right string, but no dice]. I tried it anyway, and it was almost proper. The accented 'o' in 'Cristobal', i.e., 'ó', showed up in the subjects and bodies of people's emails who were quoting him, but it ceased to be rendered correctly in Cristobal's own From: line, whereas it was fine before I started meddling. Instead of the correct char, I got a tilde over a capital A and some other junk. This seems like a step backward. So far everything else seems to be OK; I sent another test mail with accented chars and it wen
t through fine. Am I doing something wrong?

Post by Alain Bench
- Env vars LC_CTYPE or LC_ALL overrule LANG: Unset them.

Neither of these were set by default, so I just used LANG

Post by Alain Bench
- MacOS-X wants a specific spelling of locales: Try "en_US.ISO8859-1".

Yes, that is what worked. locale -a is what showed that to me, as you suggested.

Post by Alain Bench
- This locale is not installed: Install it, generate it, or pick one
already available in "locale -a".

It was installed, along with a zillion crazy moon languages.

Post by Alain Bench
At each step, restart Mutt and check the ":set &charset ?charset"
until it shows "iso-8859-1". If nothing works, please report back, and
say us the version of MacOS-X.

I'm on OSX 10.4.8. Thanks very much.

[d***@wayne.edu]

Post by d***@wayne.edu
I made all three agree on en_US.UTF-8, and the result was terrible. Most accented chars still were garbled. Plus, I sent a test mail to myself with some accented chars, and although they looked fine when I sent it, they were garbled upon receipt (it wasn't slashes and digits, it was UTF?..blah blah). Perhaps my school's IMAP server is using a particular charset. I have no idea how these things work.

More on this: when I rename an IMAP folder on this school account, instead of giving me the original name of the folder, it gives me \xxx\xxx... where the xs are digits. This was always the case; does it mean that I should be using UTF-8 on my school account? Pardon the utter ignorance here.

G

[Christian Ebert]

Post by d***@wayne.edu
I was able to get mutt and bash to agree on en_US.ISO8859-1,
but there is no option for that in Terminal.app's preferences.
I only see "Western (ISO Latin 1)" and "Western (ISO Latin 9)"
[In fact, I even opened up Terminal.app's .plist to see if I
could manually put in the right string, but no dice].

[Menu] File -> Show Info -> Display -> Character Set Encoding
and then choose "Unicode (UTF-8)"

c

[Alain Bench]

Post by d***@wayne.edu
I made all three agree on en_US.UTF-8, and the result was terrible.
Most accented chars still were garbled.

Strange: I've seen reports about this very setting working OK on
MacOS-X. The shell command "locale charmap" prints something? And what
for the usual ":set &charset ?charset" (normally the first "UTF-8", the
later "utf-8").

Post by d***@wayne.edu
Plus, I sent a test mail to myself with some accented chars, and
although they looked fine when I sent it, they were garbled upon
receipt (it wasn't slashes and digits, it was UTF?..blah blah).

Your mail server seems to be the Sun Java System Messaging Server,
a sequel of the ugly iPlanet, well(!) known for its careless
reinterpretation, rewriting, and reordering of headers, dirty
reencodings, munging of crypographic contents, modified msgids, lost
messages, and so on. IMHO a hype toy, not to be used for serious work.

However, we can't be sure if it's its fault, or effect of some
charset mix in your config. Could you please send me a test mail, in
UTF-8 setup, with Cristóbal's accent both in subject and in body?

Post by d***@wayne.edu
I was able to get mutt and bash to agree on en_US.ISO8859-1, but there
is no option for that in Terminal.app's preferences. I only see
"Western (ISO Latin 1)"

That's it: Latin-1 and 8859-1 are the one same thing.

Post by d***@wayne.edu
The accented 'o' in 'Cristobal', i.e., 'ó', showed up

Double fine: You also sent this 'ó' correctly! :-)

Post by d***@wayne.edu
but it ceased to be rendered correctly in Cristobal's own From: line,
whereas it was fine before I started meddling. Instead of the correct
char, I got a tilde over a capital A and some other junk. This seems
like a step backward.

Only people on UTF-8 terminals happen, by happy accident, to see it
wear the apparence of correctness. For everyone else, it's broken.
Broken from the beginning, and only Cristóbal can fix it cleanly.

Post by d***@wayne.edu

pick one already available in "locale -a".

zillion crazy moon languages.

:-) "locale -a | grep ^en_US" should show the available variants
for your language and country only.

Post by d***@wayne.edu
when I rename an IMAP folder on this school account, instead of giving
me the original name of the folder, it gives me \xxx\xxx... where the
xs are digits.

I know nothing about IMAP. I seem to vaguely recall that folder
names in the IMAP protocol are in UTF-7 (example "Crist+APM-bal").
However this unreadable thing is expected to be transcoded to the locale
in order to become user readable. And at the other end, the server
probably also transcodes this to and from its filesystem's charset.


Bye! Alain.

[d***@wayne.edu]

Post by Alain Bench

Post by d***@wayne.edu
I made all three agree on en_US.UTF-8, and the result was terrible.
Most accented chars still were garbled.

Strange: I've seen reports about this very setting working OK on
MacOS-X. The shell command "locale charmap" prints something? And what
for the usual ":set &charset ?charset" (normally the first "UTF-8", the
later "utf-8").

"locale charmap" yields "unknown keyword charmap". I seem to have partially sovled the UTF-8 problem. It turns out that the problem was in my muttrc, which wants to see "set charset=UTF-8" instead of "=en_US.UTF-8", while the latter is what $LANG wants to see.

I say "partially solved", because there are still some weird things. FIrst, even though I send test mails to myself in the UTF-8 mode, I see that my subject line in the header is encoded in ISO-8859-1. (I see this by pressing 'h'; otherwise it looks beautiful). Second, when I save such a message to disk, it ends up being in Latin1, even though Terminal.app, $LANG, and muttrc are all UTF-8. Third, while Terminal.app displays things correctly, xterm (via Apple's X11) does not render correctly under UTF-8. I hypothesize that I have to do something to xterm to force it to work under UTF-8, but even after googling I don't know what that is (I compiled it in). However, xterm works just fine with ISO-8859-1.

I'm also wondering if part of the weirdness is because I should be doing something to my vimrc to make vim write the file in the right encoding.

Post by Alain Bench
However, we can't be sure if it's its fault, or effect of some
charset mix in your config. Could you please send me a test mail, in
UTF-8 setup, with Cristóbal's accent both in subject and in body?

Ångström. SØren. Cristóbal. Æneid. Straße. Œdipus.

While I want to be able to view accented chars myself, I also don't want to send people emails in which they're garbled, or in an encoding that isn't common, etc. Is it better to do Latin 1 or UTF-8?

Thanks so much.
-G

[d***@wayne.edu]

Post by d***@wayne.edu
Ångström. SØren. Cristóbal. Æneid. Straße. Œdipus.

Wow, I can see myself that this looks terrible. So does the subject line, which when I composed it was three characters: u with umlaut, i with circumflex, and a with o on top. After I saved it, it appeared in the mutt menu as a wrote it except flanked by two bizarre diamond-shaped things. Now delivered, it's white noise. Sheesh.

I also see that when I set everything to UTF-8, the little arrows drawn in thread view are all wrong. It's now looking like Latin1 or nothing, unless I've done something stupid.

[Kai Grossjohann]

Post by d***@wayne.edu
I seem to have partially sovled the UTF-8 problem. It turns out that
the problem was in my muttrc, which wants to see "set charset=UTF-8"
instead of "=en_US.UTF-8", while the latter is what $LANG wants to
see.

$LANG contains more information because it specifies more things.
Whereas "set charset" in .muttrc only specifies the character set /
character encoding to use, $LANG specifies the locale, of which the
character set is only a part.

The locale specifies

- the language to use for messages printed by the system ("error" in
English, "Fehler" in German),

- the date format (1/15/2007 in the US, 15.1.2007 in Europe)

- the currency and currency symbol,

- the decimal "point" and the thousands delimiter ($1,000.00 in the
US, EUR 1.000,00 in Germany)

and other things.

Kai

[Alain Bench]

Post by d***@wayne.edu
my muttrc, which wants to see "set charset=UTF-8" instead of
"=en_US.UTF-8", while the latter is what $LANG wants to see.

Exactly. When I said they have to agree, it was about the charset
only. However, muttrc doesn't want to set $charset: The default value is
automatically derived from the current locale. This automatic link was
absent or broken in previous versions, but since around
MacOS 10.3 Panther, it works well.

Post by d***@wayne.edu
in the UTF-8 mode, I see that my subject line in the header is encoded
in ISO-8859-1.

Feature: Upon sending a text, Mutt determines the first best suited
minimal necessary and sufficient charset declared in the $send_charset
list.

Post by d***@wayne.edu
I have to do something to xterm to force it to work under UTF-8

Right. Try to start it via the uxterm command.

Post by d***@wayne.edu
I should be doing something to my vimrc to make vim write the file in
the right encoding.

That may very well be the case: Mutt's $editor has to dumbly read
and write in locale's charset, period. Smart charset auto-sensing
(Vim's $fileencodings) is useless (to Mutt), and can be harmfull (when
it gives the apparence of rightness to broken characters). However, that
doesn't seem to be your problem of today.

Post by d***@wayne.edu
Ångström. SØren. Cristóbal. Æneid. Straße. Œdipus.

All seems well in body. Subject is "[]üîå[]", meaning "uia" flanked
by 2 U+FFFD replacement characters (in my font they look like empty
squares). That seems to be what you described sending. Next question is:
What added those U+FFFD chars??

I dropped the replacements, and just kept the "uia" in subject:
Please reply with your UTF setup. So we'll see if replacements chars are
reintroduced when you don't type special chars yourself.

Post by d***@wayne.edu
Is it better to do Latin 1 or UTF-8?

Latin-1 is extremely common, and contains the characters needed by
most western languages. Spanish, German, French, Portugese, and some
such. Well... It lacks the Euro symbol €, and the oe ligatures œ Œ.

UTF-8 is a form of Unicode, containing all characters of the world.
Including the above, plus Polish, Russian, Chinese ideograms, Tamilian,
and all such. It is however a little less portable, a little more
difficult to setup (nothing hairy), and on Macs a little less stable.

My advice would be to make the (little) effort to setup UTF-8 for
yourself. And for sending, trust Mutt's optimal choice. As for setting
a right $send_charset list, it depends on the languages you're writing.
The default $send_charset is a good start, and look at my infosig for
another example.

Post by d***@wayne.edu
Wow, I can see myself that this looks terrible. So does the subject
line

The parent mail looks good (outside of the U+FFFD pair). You see it
broken because you broke your Latin-1 setup: Do not set $charset.

Post by d***@wayne.edu
when I set everything to UTF-8, the little arrows drawn in thread view
are all wrong.

Those semigraphic chars should work provided that $charset=utf-8
exactly (that's the default value derived from LANG=en_US.UTF-8), that
TERM=nsterm-16color, and that you have unchecked Terminal.app's "Wide
glyphs for Japanese/Chinese/etc." setting. If they are still wrong,
please describe how they look like.


Bye! Alain.

[Breen Mullins]

Post by Alain Bench
All seems well in body. Subject is "[]üîå[]", meaning "uia" flanked
by 2 U+FFFD replacement characters (in my font they look like empty
What added those U+FFFD chars??

As it happens, the U+FFDD this morning caused mutt to freeze on me, as
we discussed a couple of weeks ago. It's probably still tripping on
iconv somewhere.

I got a backtrace this time, if it's helpful. Looks like it's in
the library.

Breen

[Alain Bench]

Hi Breen,

the U+FFDD this morning caused mutt to freeze [...] I got a backtrace
this time, if it's helpful. Looks like it's in the library.

Which library? Last backtrace I saw of such a MacOS freeze pointed
to Ncurses waddch_nosync(). But I'd tend to suspect more something in
the libc, called by Ncurses.


Bye! Alain.

[Breen Mullins]

Post by Alain Bench
Hi Breen,
Which library? Last backtrace I saw of such a MacOS freeze pointed
to Ncurses waddch_nosync(). But I'd tend to suspect more something in
the libc, called by Ncurses.

Ugh. I'm not sure I know how to read this - superficially it looks like
it was in the linker?

Core was generated by `mutt'.
#0 0x93c10d7c in dyld_stub_unctrl ()
(gdb) bt
#0 0x93c10d7c in dyld_stub_unctrl ()
#1 0x93beec54 in wechochar ()
#2 0x93beeb60 in _nc_waddch_nosync ()
#3 0x93bef010 in waddnstr ()
#4 0x0003409c in print_enriched_string (attr=256, s=0xbfffe78e " ���������������������������üîå���", do_color=0) at menu.c:138
#5 0x000345f8 in menu_redraw_index (menu=0x0) at menu.c:252
#6 0x00016cbc in mutt_index_menu () at curs_main.c:558
#7 0x00031bb8 in main (argc=682056, argv=0x0) at main.c:989

Breen

[Alain Bench]

superficially it looks like it was in the linker?
| #0 0x93c10d7c in dyld_stub_unctrl ()

Or like some Ncurses thinggy bellow waddnstr(), and specific to the
linker? I don't know. This should probably be reported to Thomas on
bug-ncurses at gnu org.


Bye! Alain.

[Breen Mullins]

Post by Alain Bench

superficially it looks like it was in the linker?
| #0 0x93c10d7c in dyld_stub_unctrl ()

Or like some Ncurses thinggy bellow waddnstr(), and specific to the
linker? I don't know. This should probably be reported to Thomas on
bug-ncurses at gnu org.

Okay - I'll update ncurses and see if the problem persists, and report
there if necessary. Thanks for the help.

Breen

[Kyle Wheeler]

It is however a little less portable, a little more difficult to
setup (nothing hairy), and on Macs a little less stable.

I haven't had any trouble with it on Macs, and Macs are what I use
most of the time for my day-to-day mutting. No crashing or bizarre
behavior... particularly not that I would ascribe to having something
to do with the charset. What makes you say that it's less stable?

Those semigraphic chars should work provided that $charset=utf-8
exactly (that's the default value derived from LANG=en_US.UTF-8),
that TERM=nsterm-16color, and that you have unchecked Terminal.app's
"Wide glyphs for Japanese/Chinese/etc." setting. If they are still
wrong, please describe how they look like.

Where is nsterm-16color defined? I don't seem to have it on my Mac...

~Kyle
- --
I see the pain on your face when you say the word intellectual,
because it has so many syllables in it.
-- Clive James

[Kyle Wheeler]

Post by Kyle Wheeler

Post by Alain Bench
Those semigraphic chars should work provided that $charset=utf-8
exactly (that's the default value derived from LANG=en_US.UTF-8),
that TERM=nsterm-16color, and that you have unchecked
Terminal.app's "Wide glyphs for Japanese/Chinese/etc." setting. If
they are still wrong, please describe how they look like.

Where is nsterm-16color defined? I don't seem to have it on my Mac...

Speaking of which, any idea how to get it to behave in a more
xterm-like fashion? Namely, when I exit a program (such as man or less
or mutt), I like that xterm restores the terminal to what was on it
before I launched whatever program. With TERM=nsterm (or nsterm+mac),
the image of mutt (or man, or less, or...) stays on the terminal after
I quit, and I hate that.

~Kyle
- --
No matter what side of the argument you are on, you always find people
on your side that you wish were on the other.
-- Jascha Heifetz

[Kyle Wheeler]

Post by Kyle Wheeler
Speaking of which, any idea how to get it to behave in a more
xterm-like fashion? Namely, when I exit a program (such as man or
less or mutt), I like that xterm restores the terminal to what was
on it before I launched whatever program. With TERM=nsterm (or
nsterm+mac), the image of mutt (or man, or less, or...) stays on the
terminal after I quit, and I hate that.

AHA! I found it. Sorry for the noise, but this is *excellent*, and I
wanna add it to the list archive.

Essentially, nsterm-16color is in the *latest* ncurses library, and
only applications that link against said library can use it. I used
the following command to steal the source from my self-installed
(macports, though the same would work for having installed it most
other ways) ncurses library:

infocmp nsterm-16color > /tmp/nsterm-16color.src

Then, after examining the termcap database, and knowing that the
strings that do this are suppressed by the xterm titeInhibit
preference, I found the ti and te strings (which are modified by that
preference). Those strings, in terminfo parlance, correspond to smcup
and rmcup strings, so I added them:

echo 'smcup=\E7\E[?47h,' >> /tmp/nsterm-16color.src
echo 'rmcup=\E[2J\E[?47l\E8,' >> /tmp/nsterm-16color.src

Then, recompile into my local database (~/.terminfo/n/nsterm-16color):

tic /tmp/nsterm-16color.src

And presto! Mutt doesn't leave a copy of itself on the terminal after
it exits.

All I need to figure out now is how to make programs that use the
built-in ncurses (e.g. /usr/bin/less) use the ~/.terminfo database. It
doesn't seem to respect TERMINFO, even though its documentation says
that it should.

~Kyle
- --
People who would give up their Freedom for security deserve neither.
-- Benjamin Franklin

[Eyolf Østrem]

It seems to me that my own problems are related to this thread, so
I'll chime in here (with a "Was:..." change of the subject line). I've
found some really useful information, both in the reply to my intial
mail a few days back, and in this thread, but my attempts at changing
things here and there have been too random to be useful, so I've
decided to take a step back and review my options, hopefully with some
guidance from you knowledgeable people.


========
PROBLEM:
========
getting screen to work with 256 colors, without having any of the
following problems:

- function keys w/modifiers (shift, ctrl, alt) don't work, at least
not as expected. In vim: c-f10 sends something that turns the five
following characters uppercase. All the f-keys do something similar:
move somewhere on the line, and make sth uppercase.
If I enter the function keys directly, either after a command line prompt
or in vim through ^v, the output is the same, first in an xterm with
TERM=xterm-256color, then in an xterm with screen running and
TERM=screen-256color-s. In both cases, the command line says 1;5~
and in vim it says: ^[[21;5~ c-f10, then, apparently sends
'<esc>1;5~', and vim behaves accordingly: moves five characters
forwards and changes their case.
So am I right in guessing, then, that when it doesn't work in screen,
it's because the preceding ^[[2 is not interpreted as I expect it to, and
that this would be where I would search for an answer?

- colors: some (ncurses based?) apps like mutt, don't display backgrounds
correctly in areas without text, as I described in my previous post.

- screen refreshing after leaving apps: should get back to the state it was
before entering e.g. vim, but now just scrolls up the screen (as
described in this thread by Kyle; does the solution apply to
settings for xterm too?).

- I don't know if this is related: on the console, many commands make
some color code being displayed on the screen. E.g.: in a clean
shell, directly after logon:

$echo $TERM
$linux
$vim
in vim: pressing ':' gives ':2;blue' on the command line



==========
Questions:
==========

Where and how should I set the TERM variable? I guess there are five
possible levels here: environment/shell, Xterm, Screen, ncurses, and
application.
My distro's wiki (Archlinux) says that it's a bad idea to set the TERM
variable in .bashrc, but that applications should be able to figure it
out for themselves.

So, which of these possible values of TERM do I put where?
xterm
xterm-256color
screen-256color
screen-256color-s
screen-256color-bce
screen-256color-bce-s

Should it be the same everywhere I set it?
Or does e.g. screen translate screen-256color to xterm-256color or vice
versa? Or override xterm's choice (xterm-256color) with its own?
What exactly does that setting do anyway? Tell applications what to load
from the terminfo database? about how to translate input (keypresses) to
output?

How do I make changes to the terminfo database?
Are they dangerous?
Can i corrupt anything? Will experimenting cause damage?
How do I revert?
How much does this have to do with curses?

More concretely:
- how do I get my function keys back?
- which escape code is responsible for the missing colour refreshing in
mutt etc.?
- dito for the refreshing of the screen after leaving a (ncurses-based?)
program.
- and, RTFM oriented, which parts of the system do I need to know
more about - terminfo? Screen? I wouldn't want to spend a week
reading up on terminfo if the solution is to change a single
variable somewhere....

My system is Archlinux, XTerm(229), Screen version 4.00.03 (FAU)
23-Oct-06, both of which are compiled with 256 color support.

I realize that this ended up not having that much to do with the
original thread, but I did change the subject line...


Eyolf

[Kai Grossjohann]

Post by Eyolf Østrem
Where and how should I set the TERM variable? I guess there are five
possible levels here: environment/shell, Xterm, Screen, ncurses, and
application.
My distro's wiki (Archlinux) says that it's a bad idea to set the TERM
variable in .bashrc, but that applications should be able to figure it
out for themselves.
So, which of these possible values of TERM do I put where?
xterm
xterm-256color
screen-256color
screen-256color-s
screen-256color-bce
screen-256color-bce-s
Should it be the same everywhere I set it?

The TERM variable describes how the terminal behaves. Obviously, this
depends on the terminal in use. It might not be obvious that several
programs act to the system as if they were terminals. Here is a list of
possible "terminals" you might encounter:

- The Linux text console (accessible via Ctrl-Alt-F1 oder Ctrl-Alt-F2
while you are running X11 -- use Ctrl-Alt-F7 oder somesuch to get
back to X11).

- Different X11 terminal programs may exhibit different behaviors,
requiring different values of TERM: xterm, rxvt, urxvt, konsole,
gnome-terminal, mlterm, eterm, aterm, ... There are many, many,
many X11 terminal programs.

- The program screen also behaves like a terminal.

You can specify the $TERM value for xterm using "-tn foo" on the command
line, or an X ressource setting. For example, the following line in
~/.Xdefaults may be equivalent to "-tn foo":

XTerm*termName: foo

But this depends on whether ~/.Xdefaults is read in your environment...

To specify $TERM for screen, you can specify -t or -T on the command
line (forget which), or "term foo" or so in ~/.screenrc.

I lost track of what you already tried. For the interim, you can
explicitly set $TERM with "TERM=foo; export TERM" from the shell. Have
you tried that in various situations and do you now know what the
terminal needs to be?

Kai

[Eyolf Østrem]

Post by Kai Grossjohann
But this depends on whether ~/.Xdefaults is read in your environment...

It is.

Post by Kai Grossjohann
To specify $TERM for screen, you can specify -t or -T on the command
line (forget which), or "term foo" or so in ~/.screenrc.
I lost track of what you already tried. For the interim, you can
explicitly set $TERM with "TERM=foo; export TERM" from the shell. Have
you tried that in various situations and do you now know what the
terminal needs to be?

For a screen-less xterm, TERM=xterm-256color is the right thing:
everything works as I expect it to do. The problem is when screen
enters the picture. I've tried both TERM=xterm-256color and
TERM=screen-256color (with additional -bce, -s, or -bce-s), but with
no luck - they all behave the same: as in the "problem" description in
my previous mail.

Eyolf

[Gary Johnson]

Post by Eyolf Østrem
Where and how should I set the TERM variable?

The terminal itself should set this. It puts this variable in the
environment of any subprocesses it spawns so that the application
and/or the curses library used by the application knows, by means
of the terminfo database, what capabilities the terminal has and
what character sequences to send to the terminal to invoke each
capability.

Post by Eyolf Østrem
I guess there are five possible levels here: environment/shell,
Xterm, Screen, ncurses, and application. My distro's wiki
(Archlinux) says that it's a bad idea to set the TERM variable
in .bashrc, but that applications should be able to figure it
out for themselves.

Right. By putting TERM in your .bashrc you are overriding the
terminal's setting and assuming that you are always using only one
terminal type.

Post by Eyolf Østrem
So, which of these possible values of TERM do I put where?
xterm
xterm-256color
screen-256color
screen-256color-s
screen-256color-bce
screen-256color-bce-s
Should it be the same everywhere I set it?

It depends. Ideally, you should use the value whose terminfo
database entry most accurately describes the capabilities of the
terminal you are actually using. When using screen, however, the
application is communicating with your terminal through screen, so
while screen should sit transparently between your terminal and the
application, screen does have to know something about the character
sequences being used to control the terminal.

Also, there are some applications that are hard-coded to recognize
certain terminal names. They may know what to do with "xterm" but
not "xterm-256color".

Post by Eyolf Østrem
Or does e.g. screen translate screen-256color to xterm-256color or vice
versa? Or override xterm's choice (xterm-256color) with its own?

I believe that unless you tell it to do otherwise, screen sets
TERM to "screen". You can tell screen to behave differently. Refer
to my earlier reply and the screen man page.

Post by Eyolf Østrem
What exactly does that setting do anyway? Tell applications what to load
from the terminfo database? about how to translate input (keypresses) to
output?

It depends on the application. It basically tells the curses
library what terminfo database entry to use, so the application
just calls curses/terminfo functions to control the terminal
and the library takes care of the details, but applications may
choose to make other decisions based on the value of TERM or to
access some terminal capabilities directly.

Post by Eyolf Østrem
How do I make changes to the terminfo database?

Using infocmp and tic.

Post by Eyolf Østrem
Are they dangerous?

Yes, tic can be, if you are running as root or otherwise have
permission to change the files in the terminfo database.

Post by Eyolf Østrem
Can i corrupt anything? Will experimenting cause damage?

Yes. Maybe.

Post by Eyolf Østrem
How do I revert?

By default, the curses/terminfo library uses the system terminfo
database, e.g., /usr/share/lib/terminfo. Setting and exporting the
variable TERMINFO in your environment to another directory will
cause curses/terminfo to look there first for terminfo files. This
can give you a safe place to experiment with changes to terminal
definitions without messing up the system database. For example, I
have in my ~/.profile,

TERMINFO=$HOME/lib/terminfo

Post by Eyolf Østrem
How much does this have to do with curses?

Everything. Applications often use a curses library to interface
with the terminal they're running in. That library uses the
terminfo database to figure out what character sequences to send to
the terminal in order to execute a particular curses function.
Things that can go wrong include:

- The TERM value is incorrect for the terminal being used.
- The terminfo database entry for TERM is missing, incomplete or
otherwise doesn't contain accurate information for the terminal
being used.
- The curses library used by the application is old, broken or
otherwise doesn't allow the application access to all of the
terminfo capabilities. (The HP-UX 10.20 curses library, for
example, didn't recognize colors.)
- The application uses its own idea of a terminal's capabilities
instead of or in addition to the terminfo database.

HTH,
Gary

[Eyolf Østrem]

First of all: thanks for your reply - here, and to the previous post.
Much helpful. I will have to read up on terminfo, I can tell...
And test some settings. Phew...

Post by Gary Johnson
When using screen, however, the
application is communicating with your terminal through screen, so
while screen should sit transparently between your terminal and the
application, screen does have to know something about the character
sequences being used to control the terminal.

Does this mean that I could make a terminfo entry called, say,
'my-screen-in-xterm', based on that for xterm-256color, and set that
in .screenrc? That would tell screen exactly which capabilities the
terminal has, wouldn't it? But why then doesn't it work to set 'term
xterm-256color' directly?

Post by Gary Johnson
Also, there are some applications that are hard-coded to recognize
certain terminal names. They may know what to do with "xterm" but
not "xterm-256color".

Is this the reason? That mutt and mocp (and vim?) know about xterm but
not xterm-256color?

Post by Gary Johnson

Post by Eyolf Østrem
How much does this have to do with curses?

Everything. Applications often use a curses library to interface
with the terminal they're running in. That library uses the
terminfo database to figure out what character sequences to send to
the terminal in order to execute a particular curses function.

So terminfo is only relevant for applications that use ncurses?

Post by Gary Johnson
- The TERM value is incorrect for the terminal being used.

And when I run screen, what is 'the terminal being used' - is it
screen(-256color) or the terminal which screen runs in? Or both? This
is one of the things I haven't fully grasped yet: does the information
pass serially through the full chain application > ncurses > screen >
xterm > system (so that e.g. mutt will have to know about screen's
capabilities, and screen about xterm's?), or are there shortcuts?
Also: are the values simply transFERRED from one level to the next, or
are they also transLATED, so that, e.g. c-F10 is called 'apple' in
mutt and 'orange' in xterm, and ncurses knows how to translate between
them?

Post by Gary Johnson
- The terminfo database entry for TERM is missing, incomplete or
otherwise doesn't contain accurate information for the terminal
being used.

It sounds as if this is the case, then? since ...

Post by Gary Johnson
- The curses library used by the application is old, broken or
otherwise doesn't allow the application access to all of the
terminfo capabilities.

... the curses version is 5.6.20061217, and ...

Post by Gary Johnson
- The application uses its own idea of a terminal's capabilities
instead of or in addition to the terminfo database.

... the problem occurs in many different applications, so unless they
all make the same assumptions, the problem should lie somewhere else,
no?

Post by Gary Johnson
HTH,

Very much so. Thanks again.

Eyolf

[Christian Ebert]

* Eyolf Østrem on Friday, September 07, 2007 at 19:58:29 +0200

Post by Eyolf Østrem
Does this mean that I could make a terminfo entry called, say,
'my-screen-in-xterm', based on that for xterm-256color, and set that
in .screenrc? That would tell screen exactly which capabilities the
terminal has, wouldn't it? But why then doesn't it work to set 'term
xterm-256color' directly?

Here "term term-256color" in screenrc works if I do /not/ set
$TERM in eg. bashrc explicitly. So I do roughly the following in
bashrc:

[ -z "$STY" ] && export TERM="xterm-256color"

($STY is set in a Screen session)

and in screenrc:

term "screen-256color"

(I have screen built with 256 color support)

c

[Eyolf Østrem]

Post by Christian Ebert
Here "term term-256color" in screenrc works

Just to make sure: do you mean "term screen-256color"? or ...

Post by Christian Ebert
term "screen-256color"

...?

Eyolf

[Christian Ebert]

* Eyolf Østrem on Friday, September 07, 2007 at 21:24:27 +0200

Post by Eyolf Østrem

Post by Christian Ebert
Here "term term-256color" in screenrc works

Just to make sure: do you mean "term screen-256color"? or ...

Post by Christian Ebert
term "screen-256color"

...?

or ... !

$ grep -F 256 ~/.screenrc
term "screen-256color"
c

[Gary Johnson]

Post by Eyolf Østrem
First of all: thanks for your reply - here, and to the previous post.
Much helpful. I will have to read up on terminfo, I can tell...
And test some settings. Phew...

You're welcome. Sometimes these issues have easy answers and
sometimes you just have to wade through the documentation and try
some stuff.

I'm not an expert on any of this. I've just encountered enough
problems over the years and learned enough to solve most of them
that I can give you a few answers and maybe point you in the right
direction to find the rest yourself.

Post by Eyolf Østrem

Post by Gary Johnson
When using screen, however, the
application is communicating with your terminal through screen, so
while screen should sit transparently between your terminal and the
application, screen does have to know something about the character
sequences being used to control the terminal.

Does this mean that I could make a terminfo entry called, say,
'my-screen-in-xterm', based on that for xterm-256color, and set that
in .screenrc? That would tell screen exactly which capabilities the
terminal has, wouldn't it? But why then doesn't it work to set 'term
xterm-256color' directly?

I don't know. It's been a while since I read the screen man page.
The last time I did, I learned a little (now forgotten) about the
interactions among screen, terminals and applications and how to
compile a terminfo entry that would fix the problem. It didn't
involve my .screenrc.

Post by Eyolf Østrem

Post by Gary Johnson
Also, there are some applications that are hard-coded to recognize
certain terminal names. They may know what to do with "xterm" but
not "xterm-256color".

Is this the reason? That mutt and mocp (and vim?) know about xterm but
not xterm-256color?

I don't know. Probably not in the case of mutt, but the only way to
know for sure is to look in the source.

Post by Eyolf Østrem

Post by Gary Johnson

Post by Eyolf Østrem
How much does this have to do with curses?

Everything. Applications often use a curses library to interface
with the terminal they're running in. That library uses the
terminfo database to figure out what character sequences to send to
the terminal in order to execute a particular curses function.

So terminfo is only relevant for applications that use ncurses?

Not only ncurses, but any library the provides an interface to the
terminfo library. HP-UX and Solaris each have their own curses
library, for example. Slang is another.

Post by Eyolf Østrem

Post by Gary Johnson
- The TERM value is incorrect for the terminal being used.

And when I run screen, what is 'the terminal being used' - is it
screen(-256color) or the terminal which screen runs in? Or both?

I meant the thing you look at and type into, e.g. xterm, but when
using screen, which has some of the behaviors of a terminal and
which sets TERM itself, it does get more complicated.

Post by Eyolf Østrem
does the information pass serially through the full chain
application > ncurses > screen > xterm > system (so that e.g.
mutt will have to know about screen's capabilities, and screen
about xterm's?), or are there shortcuts? Also: are the values
simply transFERRED from one level to the next, or are they also
transLATED, so that, e.g. c-F10 is called 'apple' in mutt and
'orange' in xterm, and ncurses knows how to translate between
them?

As I wrote above, I don't remember exactly which character sequences
screen passes transparently, which ones it acts upon and which, if
any, it modifies. All that is in the man page, as I recall.

Post by Eyolf Østrem

Post by Gary Johnson
- The terminfo database entry for TERM is missing, incomplete or
otherwise doesn't contain accurate information for the terminal
being used.

It sounds as if this is the case, then? since ...

Post by Gary Johnson
- The curses library used by the application is old, broken or
otherwise doesn't allow the application access to all of the
terminfo capabilities.

... the curses version is 5.6.20061217, and ...

If you're using ncurses on a relatively recent Linux distribution,
it's not likely to be the problem. That one in particular should be
fine (based on my usage of 5.5).

Post by Eyolf Østrem

Post by Gary Johnson
- The application uses its own idea of a terminal's capabilities
instead of or in addition to the terminfo database.

... the problem occurs in many different applications, so unless they
all make the same assumptions, the problem should lie somewhere else,
no?

True.

Regards,
Gary

[Alain Bench]

Hello Kyle,

What makes you say that [UTF-8 on Macs is] less stable?

There were several reports of Mutt freezes on MacOS-X in UTF-8
locales, when stepping on various sorts of mails with invalid
characters (and now even on the valid U+FFFD). This happens only with
specific versions of libiconv and Ncurses. Some versions never freeze;
Other versions freeze 100% repeatably. The cause is unknown so far:
Contradictory evidences do exist.

nsterm-16color is in the *latest* ncurses library, and only
applications that link against said library can use it.

The terminfo database comes indeed bundled with Ncurses. But you can
also download the latest terminfo.src source file alone, and "tic -x" it
with your old tic (provided it's not too old). And of course: Dangerous
operation, backup first.

| tic /tmp/nsterm-16color.src

Running tic as root should install the compiled entry into the
system database.

[smcup/rmcup] And presto! Mutt doesn't leave a copy of itself on the
terminal after it exits.

Excellent indeed! I'll update the entry with this alternate screen
thing. Can you please tell me what is the output of:

| $ printf "Main di\033[?47hAlternate screen.\n\033[?47lsplay.\n"
| Main display.

If this prints only "Main display." cleanly on one line, then you
can drop the "\E7" and "\E8" sequences from the smcup/rmcup strings.


Bye! Alain.

[Kyle Wheeler]

Post by Alain Bench
Hello Kyle,

Hiya Alain,

Post by Alain Bench
There were several reports of Mutt freezes on MacOS-X in UTF-8
locales, when stepping on various sorts of mails with invalid
characters (and now even on the valid U+FFFD). This happens only
with specific versions of libiconv and Ncurses. Some versions never
freeze; Other versions freeze 100% repeatably. The cause is unknown
Contradictory evidences do exist.

Huh. <shrug> Perhaps I innoculated myself against trouble by linking
mutt against the ncurses in MacPorts (once upon a time, it was the
only way to get ncursesw support).

Post by Alain Bench

nsterm-16color is in the *latest* ncurses library, and only
applications that link against said library can use it.

The terminfo database comes indeed bundled with Ncurses. But you can
also download the latest terminfo.src source file alone, and "tic -x" it
with your old tic (provided it's not too old). And of course: Dangerous
operation, backup first.

Bwahahaha! Take that, stupid `less`! Thanks Alain, that's good stuff.
Maybe one of these days I'll be able to migrate from xterm to
Terminal.app even!

Post by Alain Bench
Running tic as root should install the compiled entry into the
system database.

So it does!

Post by Alain Bench
Excellent indeed! I'll update the entry with this alternate screen
| $ printf "Main di\033[?47hAlternate screen.\n\033[?47lsplay.\n"
| Main display.
If this prints only "Main display." cleanly on one line, then you
can drop the "\E7" and "\E8" sequences from the smcup/rmcup strings.

Yup, that prints just fine. smcup/rmcup strings are so modified.

Outta curiosity, what made you think those were superfluous?

~Kyle
- --
I would be delighted to offer any advice I can on understanding women.
When I have some I'll let you know.
-- Jean Luc Picard

[d***@wayne.edu]

Post by Alain Bench
Exactly. When I said they have to agree, it was about the charset
only. However, muttrc doesn't want to set $charset: The default value is
automatically derived from the current locale. This automatic link was
absent or broken in previous versions, but since around
MacOS 10.3 Panther, it works well.

Whoops, sorry I misunderstood. I checked and indeed mutt did pull its charset from $LANG.

Post by Alain Bench

Post by d***@wayne.edu
Ångström. SØren. Cristóbal. Æneid. Straße. Œdipus.

All seems well in body. Subject is "[]üîå[]", meaning "uia" flanked
by 2 U+FFFD replacement characters (in my font they look like empty
What added those U+FFFD chars??
Please reply with your UTF setup. So we'll see if replacements chars are
reintroduced when you don't type special chars yourself.

OK, I'm in console vim composing the reply, and the subject line characters are rendered properly, i.e., without the flanking weirdness. Of course, that's how they looked last time before becoming odd.

Post by Alain Bench

Post by d***@wayne.edu
Is it better to do Latin 1 or UTF-8?

Latin-1 is extremely common, and contains the characters needed by
most western languages. Spanish, German, French, Portugese, and some
such. Well... It lacks the Euro symbol €, and the oe ligatures œ Œ.

Post by d***@wayne.edu
when I set everything to UTF-8, the little arrows drawn in thread view
are all wrong.

Those semigraphic chars should work provided that $charset=utf-8
exactly (that's the default value derived from LANG=en_US.UTF-8), that
TERM=nsterm-16color, and that you have unchecked Terminal.app's "Wide
glyphs for Japanese/Chinese/etc." setting. If they are still wrong,
please describe how they look like.

I unchecked the box, but I haven't done nsterm-blah, because I don't seem to have it on my machine. The thread arrows are drawn correctly except that they have spaces between them, like a dashed line. I'll maybe mess with nsterm later. Right now it almost seems not worth it to me. By the way, the arrows look fine in Latin1, so maybe I'll stick with that (since apparently, mutt will send all my stuff in Latin1 anyway unless I'm using a strange script.)

Thanks so much,
-G

[Alain Bench]

Post by d***@wayne.edu

Post by Alain Bench
I dropped the replacements, and just kept the "uia" in subject

I'm in console vim composing the reply, and the subject line
characters are rendered properly, i.e., without the flanking
weirdness.

And the "ui" arrived here cleanly. This may mean the replacements
were inserted when you entered the "uia". Where? At Mutt's subject
prompt? Or in Vim in $edit_headers? Typed, or Pasted?

Note that my own "uia" subject had been truncated by the mailing
list: Majordomo (or something there) seems to dislike accents in
subjects, which is an annoyance for such tests, and makes the French
mutt-users-fr list ugly looking.

Post by d***@wayne.edu
The thread arrows are drawn correctly except that they have spaces
between them, like a dashed line. I'll maybe mess with nsterm later.

So that's not a TERM problem: I've already seen screen copies of
Terminal.app with such small interruptions in lines, especially vertical
ones. Could it be a font, or font size related problem?

Post by d***@wayne.edu
the arrows look fine in Latin1

The method used by Mutt to display thread trees is (necessarilly)
different between UTF-8 mode, and all other charsets. However on other
terminals both methods are more or less expected to look the same.

Post by d***@wayne.edu
so maybe I'll stick with that

Good. After all, the day you want to read Korean, you know you just
have to fire an UTF-8 terminal and export LANG accordingly. Mutt and all
other apps correctly configured (ie. not hardcoding any given charset)
should just work.

Post by d***@wayne.edu
Thanks so much,

You're most welcome, man.


Bye! Alain.

[Kyle Wheeler]

Post by Alain Bench

Post by d***@wayne.edu
The thread arrows are drawn correctly except that they have spaces
between them, like a dashed line. I'll maybe mess with nsterm later.

So that's not a TERM problem: I've already seen screen copies of
Terminal.app with such small interruptions in lines, especially vertical
ones. Could it be a font, or font size related problem?

For what it's worth, this problem is relatively easy to fix in the
horizontal case. In your Terminal Settings (select "Window Inspector"
from the Terminal menu), in the Display section, de-select both "Wide
Glyphs..." options.

I had already de-selected "Wide Glyphs count as two columns", but
"Wide Glyphs for Japanese/Chinese/etc." fixed the horizontal
line-breaks for my threading arrows.

If anyone knows a fix for the vertical line breaks, I'd love to know.

Post by Alain Bench
Mutt muttrc tip to send mails in best adapted first necessary and sufficient
set send_charset="us-ascii:iso-8859-1:iso-8859-15:windows-1252:utf-8"

Oh? Hrm. I had people jumping all over me for my use of windows-1252
(just because I wanted curly-quotes, like this: can’t have ‘em).
That's why my mutt-users posts are (almost) always in non-MIME
us-ascii now. For sending emails with such characters in them (as I do
a *lot*) is it really better to send them encoded as windows-1252 than
as utf-8?

~Kyle
- --
No one will ever win the battle of the sexes; there's too much
fraternizing with the enemy.
-- Henry Kissinger

[Alain Bench]

Post by Kyle Wheeler
I had already de-selected "Wide Glyphs count as two columns"

I wonder what does this option?

Post by Kyle Wheeler
For sending emails with such [curly quotes] characters in them (as
I do a *lot*) is it really better to send them encoded as windows-1252
than as utf-8?

I think so. Those charsets are equally legal: They are both
registred by IANA for usage as MIME charset label. They will both
display equally well, for the vast majority of recipients. They will
both display equally bad, for those recipients unable to display curly
quotes (example: Mutt on a Latin-1 console). So far no difference.

Then there is the case of recipients with not charset-aware mailer,
be it old, simple, powerless, embedded, or whatever reason. Typically
such mailer displays the mails as-is, unconverted. If the mailer's
display is say Latin-1, then Latin-1 mails are well displayed, and
everything else bad. A minority of such mailers may well run in an UTF-8
terminal. But they will more typically run on a Latin-1 terminal, or in
a CP-1252 window. They will display well CP-1252 curly quotes, or at
least most common accented characters. While UTF-8 mails will have both
curlies and accents garbled. Conclusion: CP-1252 mails have better
chances to be readable, and if broken then less broken, than UTF-8
mails. Exceptions pro-UTF do exist, but are quite rare.

Other arguments exist: CP-1252 makes shorter mails than UTF-8.
CP-1252 was invented by a well known commercial company; UTF-8 by some
evil bearded unix guru. ;-)


Bye! Alain.

[Kyle Wheeler]

Post by Alain Bench

Post by Kyle Wheeler
I had already de-selected "Wide Glyphs count as two columns"

I wonder what does this option?

From what I can tell, it allows some characters to take up two
columns. For example, an em-dash (—) is hard to tell from an en-dash
(–) or even a hyphen (-) without an actual change in width. This makes
more sense when using a non-monospaced font (as em-dashes are
naturally the same width as an m, which are the same width as
everything else in such fonts). It becomes an issue when mutt does
something that calls up a symbol from outside the monospaced font,
such as the horizontal line character used to draw the threading
arrows. This has a natural width that is slightly wider than all the
other characters, and so when that option is enabled, the horizontal
line glyph occupies two columns (it’s centered in the two-column
area).

At least, I *think* that’s how this works, based on how it seems to
behave.

Thanks as always, Alain.

~Kyle
- --
America will never be destroyed from the outside. If we falter and
lose our freedoms, it will be because we destroyed ourselves.
-- Abraham Lincoln

[Christian Ebert]

* Kyle Wheeler on Tuesday, September 18, 2007 at 17:21:44 -0500

Post by Kyle Wheeler

Post by Alain Bench

Post by Kyle Wheeler
I had already de-selected "Wide Glyphs count as two columns"

I wonder what does this option?

From what I can tell, it allows some characters to take up two
columns. For example, an em-dash (—) is hard to tell from an en-dash
(–) or even a hyphen (-) without an actual change in width. This makes
more sense when using a non-monospaced font (as em-dashes are
naturally the same width as an m, which are the same width as
everything else in such fonts). It becomes an issue when mutt does
something that calls up a symbol from outside the monospaced font,
such as the horizontal line character used to draw the threading
arrows. This has a natural width that is slightly wider than all the
other characters, and so when that option is enabled, the horizontal
line glyph occupies two columns (it’s centered in the two-column
area).
At least, I *think* that’s how this works, based on how it seems to
behave.

You also need to set this option if you want "Asian" characters
readable in Terminal.app, otherwise they melt into each other
horizontally. OTOH setting it messes up Mutt's thread display
(can be worked around by setting narrow_tree in Mutt, but not in
Slrn).

I switched to iTerm for the moment which doesn't have these
problems, and offers 256 colors, transparency and even GNU-Screen
like multiplexing, but, on the downside, is slower.

<http://iterm.sourceforge.net/>

Post by Kyle Wheeler
Thanks as always, Alain.

Seconded.

c

[d***@wayne.edu]

Post by Alain Bench
And the "ui" arrived here cleanly. This may mean the replacements
were inserted when you entered the "uia". Where? At Mutt's subject
prompt? Or in Vim in $edit_headers? Typed, or Pasted?

Hmm, I don't know. I didn't paste it in. I'm pretty sure now that I entered it in the subject prompt, because I recall dreading that vim would screw it up in the compose mode.

Post by Alain Bench

Post by d***@wayne.edu
The thread arrows are drawn correctly except that they have spaces
between them, like a dashed line. I'll maybe mess with nsterm later.

So that's not a TERM problem: I've already seen screen copies of
Terminal.app with such small interruptions in lines, especially vertical
ones. Could it be a font, or font size related problem?

Mine have vertical mistakes, not horizontal ones. It is not a font problem, because the font --- Monaco --- looks incorrect at all sizes when $LANG is set to en_US.UTF-8 and so is Terminal.app; but it looks correct at all sizes when either (i) both are set to Latin1, or when (ii) $LANG is unset and Terminal.app is anything. By the way, both boxes for "enabling wide" are un-checked.

Latin1 it is, until further bizarre things pop up. Plus, xterm under X11 does it right under Latin1, while it is screwy under UTF-8. (uxterm and other things, like the -u8 option and friends, resulted in messed-up colors, including no colors in vim -- a deal-breaker).

Thanks again,
-G

[Kyle Wheeler]

Post by d***@wayne.edu
Latin1 it is, until further bizarre things pop up. Plus, xterm under
X11 does it right under Latin1, while it is screwy under UTF-8.
(uxterm and other things, like the -u8 option and friends, resulted
in messed-up colors, including no colors in vim -- a deal-breaker).

Hmm... For what it's worth, I use uxterm on my Mac all the time
without problems. The messed-up colors sound like a $TERM problem.
When you launch uxterm, what's it set to? Mine's xterm-16color.

~Kyle
- --
The average Ph.D thesis is nothing but the transference of bones from
one graveyard to another.
-- J. Frank Dobie, "A Texan in England"

[d***@wayne.edu]

Post by Kyle Wheeler

Post by d***@wayne.edu
Latin1 it is, until further bizarre things pop up. Plus, xterm under
X11 does it right under Latin1, while it is screwy under UTF-8.
(uxterm and other things, like the -u8 option and friends, resulted
in messed-up colors, including no colors in vim -- a deal-breaker).

Hmm... For what it's worth, I use uxterm on my Mac all the time
without problems. The messed-up colors sound like a $TERM problem.
When you launch uxterm, what's it set to? Mine's xterm-16color.

I've tried setting it to xterm-16color, 256color, and just color, all with no dice. The messed-up colors have a precedent. When I installed OSX 10.4 with Apple's X11, the colors were messed up: what should be a nice, royal blue is borderline light-turquoise, etc. These colors are perfect under OSX 10.3. So I built xterm myself, with 256 support, and put it into /usr/local/bin. This custom xterm has all the correct colors, even when I don't tell it to use all 256 colors. But 'uxterm' calls the custom xterm, and suddenly the bad colors reappear. It won't pass any colors at all to vim, to top it all off.

-G

[Kyle Wheeler]

Post by d***@wayne.edu
I've tried setting it to xterm-16color, 256color, and just color,
all with no dice. The messed-up colors have a precedent. When I
installed OSX 10.4 with Apple's X11, the colors were messed up: what
should be a nice, royal blue is borderline light-turquoise, etc.

Ahhh, so THAT's what you mean by "messed up".

I *think* this is actually a result of a change, not to xterm, but to
it's default configuration. The easiest way to change it back to the
way you like it is to set things explicitly in your ~/.Xdefaults file,
like so:

xterm*color4: blue2

Or like this, to be even more explicit:

xterm*color4: #0000A8

The hex is ordered: red green blue (obviously)

When you change it, of course, you have to run:

xrdb -load ~/.Xdefaults

If that doesn't fix it for you, then your xterm really is messed up.
Once you've done that, then xterm will use that color for color4 NO
MATTER WHAT! uxterm, xterm, hand-compiled xterms, they should all have
exactly the same color for color4, because you've explicitly set it.

I even know how to do the same thing for Apple's Terminal.app:
http://culater.net/software/TerminalColors/TerminalColors.php

Personally, I rather like the color it is now---I find it much easier
to read than before.

~Kyle
- --
This is my simple religion. There is no need for temples; no need for
complicated philosophy. Our own brain, our own heart is our temple;
the philosophy is kindness.
-- Dalai Lama

[d***@wayne.edu]

Post by Kyle Wheeler

Post by d***@wayne.edu
I've tried setting it to xterm-16color, 256color, and just color,
all with no dice. The messed-up colors have a precedent. When I
installed OSX 10.4 with Apple's X11, the colors were messed up: what
should be a nice, royal blue is borderline light-turquoise, etc.

Ahhh, so THAT's what you mean by "messed up".
I *think* this is actually a result of a change, not to xterm, but to
it's default configuration. The easiest way to change it back to the
way you like it is to set things explicitly in your ~/.Xdefaults file,
xterm*color4: blue2
xterm*color4: #0000A8
The hex is ordered: red green blue (obviously)
xrdb -load ~/.Xdefaults
If that doesn't fix it for you, then your xterm really is messed up.
Once you've done that, then xterm will use that color for color4 NO
MATTER WHAT! uxterm, xterm, hand-compiled xterms, they should all have
exactly the same color for color4, because you've explicitly set it.

Thanks for the link and info. I may play around with it, but I'm hesitant, because xterm's colors are just fine *unless* I either (1) call uxterm, or (2) call the factory xterm from Apple as opposed to the one I built myself. If I tweak color defs in order to get uxterm to look correct, I don't want to mess them up for regular xterm.

-G

[Eyolf Østrem]

Post by d***@wayne.edu
Thanks for the link and info. I may play around with it, but I'm hesitant, because xterm's colors are just fine *unless* I either (1) call uxterm, or (2) call the factory xterm from Apple as opposed to the one I built myself. If I tweak color defs in order to get uxterm to look correct, I don't want to mess them up for regular xterm.

You can specify settings specifically for uxterm:

UXterm*termName: xterm-color

, eg.

[d***@wayne.edu]

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

I don't know what it is now, since I do not have an ~/.Xdefaults file. (not in my home dir anyway). Can I revert it just by deleting this file, if I create it?

-G

[Gary Johnson]

Post by d***@wayne.edu

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

I don't know what it is now, since I do not have an ~/.Xdefaults
file. (not in my home dir anyway). Can I revert it just by
deleting this file, if I create it?

You usually have to do something with the X server or the window
manager for the ~/.Xdefaults file to take effect or to remove its
effects after removing the file. The surest way is to log out and
log back in. The window manager I used to use had a root menu item
that would reset the X resource database, but I haven't been able
to find the equivalent on KDE. If you're careful and you know what
you're doing, you can use xrdb to manage the X resource database.

Post by d***@wayne.edu

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

As a matter of fact, I don't seem to have .Xdefaults *anywhere*
on my machine. There is something called ".Xauthority" in my
home dir, but it is a blank file.

I've lost track of what kind of system you're on, but you
can usually find the system default settings for your X
resources--those parameters you set in your ~/.Xdefaults file--in
the /usr/lib/X11/app-defaults/ directory. You might also take a
look at the xrdb(1) man page.

HTH,
Gary

[Kyle Wheeler]

Post by d***@wayne.edu
Thanks for the link and info. I may play around with it, but I'm
hesitant, because xterm's colors are just fine *unless* I either (1)
call uxterm, or (2) call the factory xterm from Apple as opposed to
the one I built myself. If I tweak color defs in order to get
uxterm to look correct, I don't want to mess them up for regular
xterm.

Don't worry, they won't. I include specific color settings as a
standard part of my Xdefaults precisely because the defaults sometimes
vary. All it does is force specific color numbers to be associated
with the specific rgb values that you prefer.

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

~Kyle
- --
No nation is permitted to live in ignorance with impunity.
-- Thomas Jefferson

[d***@wayne.edu]

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

As a matter of fact, I don't seem to have .Xdefaults *anywhere* on my machine. There is something called ".Xauthority" in my home dir, but it is a blank file.

-G

[Eyolf Østrem]

Post by d***@wayne.edu

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

As a matter of fact, I don't seem to have .Xdefaults *anywhere* on my machine. There is something called ".Xauthority" in my home dir, but it is a blank file.

You can create it; once it's there, X will (on most systems) find it
during startup.
Settings there will not take effect immediately. You will first have
to "merge" the new values into the settings database with the command

xrdb -merge ~/.Xdefaults

Then the changes should be reflected in any NEW shell you open. If
you're not satisfied, go back and change and merge again.
Note, however, that if you want to remove the new value completely,
you can't just delete it from the .Xdefaults file, since the xrdb
command just reads what is there, not what is not. There are ways to
remove settings - see man xrdb. The easiest way is to restart X.

Eyolf

[Christian Ebert]

Post by d***@wayne.edu

And of course, you can try them out, and if you don't like the
results, change it back to how it is now without a second thought. :)

As a matter of fact, I don't seem to have .Xdefaults *anywhere*
on my machine.

If you use Apple's X11 have a look at
/usr/X11R6/lib/X11/app-defaults/*, especially at
/usr/X11R6/lib/X11/app-defaults/XTerm.

... and, of course:

$ man xterm

;)

c

[Cristóbal Palmer]

Hi folks. My mutt is fixed (no more charset woes! yaay!), with many
thanks to Alain.

I thought I'd share my final muttrc, slightly trimmed for privacy....

cat muttrc |grep -v '^#'|grep -v '^alias' > muttrc-trimmed.txt

...was used to produce:

http://garp.metalab.unc.edu/muttrc-trimmed.txt

For you to look at. The most important settings to note included:

set config_charset="utf-8"
set envelope_from=yes
set from="***@metalab.unc.edu"
set realname="Cristóbal Palmer"
set use_from=yes

Thanks!

Cheers,
CMP

Post by Alexander Dahl
Hi folks,
^^

Cristóbal Palmer

^
is in fact iso8859-15. I connect with PuTTY configured to the same locale.
The above quoted message has charset utf8 both in Subject and message body.
Mutt displays message body correctly but Subject is not correctly decoded as
you can see above.
Did I miss some in my configuration (I can't imagine), is this a bug in
mutt I should report or is it a wrong coded mail?
Greets
Alex
--
***** http://www.lespocky.de *******************************************
GnuPG-FP: 02C8 A590 7FE5 CA5F 3601 D1D5 8FBA 7744 CC87 10D0