Topic: Mailman unicode error

[Trashman]

I already posted this one on bugzilla but i wanted to talk about it here too, copying the text from bugzilla: (please read the whole thing, if you ommit the error messages the text is not so long)

Code: [Select]

Ive got this error and the mailing wasnt sent:

Sep 17 15:11:03 2008 (4024) Uncaught runner exception: 'ascii' codec can't
encode character u'\xe9' in position 12: ordinal not in range(128)
Sep 17 15:11:05 2008 (4024) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/OutgoingRunner.py", line 73, in _dispose
    self._func(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 152, in process
    deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 338, in
verpdeliver
    bulkdeliver(mlist, msgcopy, msgdata, envsender, failures, conn)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 356, in
bulkdeliver
    msgtext = msg.as_string()
  File "/usr/lib/mailman/pythonlib/email/Message.py", line 130, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 102, in flatten
    self._write(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 130, in _write
    self._dispatch(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 156, in _dispatch
    meth(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 202, in
_handle_text
    self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
12: ordinal not in range(128)
Sep 17 15:11:06 2008 (4024) SHUNTING:
1221675012.1532891+d731328d0db41c453c017171a1ad1a2b1a1c037e


Some background:
mailman have personalization active and working (tested it before with just 5
addresses and worked fine) but when i added the 1k addresses that it should
handle for the use i give it and tried to send a mailing ive got that error,
its related to an unicode character è or something like that...my list is for
spanish talkers so its set completelly on spanish and we use those symbols in
names sometimes, i changed some of those symbols to the letter without the
accent and the second round it apparently sent the mailing (ive got the mail on
my "test" mail addresses and some unhandled bounces messages) but i got:


Sep 17 15:47:29 2008 (4024) Uncaught runner exception: 'ascii' codec can't
encode character u'\xed' in position 15: ordinal not in range(128)
Sep 17 15:47:29 2008 (4024) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/OutgoingRunner.py", line 73, in _dispose
    self._func(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 152, in process
    deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 338, in
verpdeliver
    bulkdeliver(mlist, msgcopy, msgdata, envsender, failures, conn)
  File "/usr/lib/mailman/Mailman/Handlers/SMTPDirect.py", line 356, in
bulkdeliver
    msgtext = msg.as_string()
  File "/usr/lib/mailman/pythonlib/email/Message.py", line 130, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 102, in flatten
    self._write(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 130, in _write
    self._dispatch(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 156, in _dispatch
    meth(msg)
  File "/usr/lib/mailman/pythonlib/email/Generator.py", line 202, in
_handle_text
    self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position
15: ordinal not in range(128)
Sep 17 15:47:29 2008 (4024) SHUNTING:
1221677176.097934+e85949bfc5ef725294bd67e4383032ef2f48a64e

I have the feeling that in fact not the entire list received the mail, but dont
know where to check to be sure...
1- that unicode thing is part of the base system or part of mailman?
2- can it make problems with accentuation in the main body of the mail?
3- is there any workaround so i dont get that nasty error and the mailing is
sent complete?
4- what is that u'\xed' character? i want to modify it also on the names on the
list but dont have a remote idea of where to check for those unicode codes
5- if there is no way to get a permanent solution i can still modify actual
names and keep using the list as is but id ask very gently for help to make an
html form that checks for unicode so future subscriptions cant use them OR the
form change those "wrong" chars into "usefull" ones automatically (im not
coder, im just a "geeky user")

http://bugs.contribs.org/show_bug.cgi?id=4571

[Trashman]

Follow up: i found in /var/log/qpsmtpd/current that the system sent some of the mailing mails but not all, im counting kinda 50 mails or so...i guess that some part of the system stopped the proccess at a certain point...ill try again in monday, dont want to storm my clients with mails...
Also i found that "u'\xe9'" means "í" and there was one missing to correct, ill report again if there is any unicode errors once i send another round but there is still the base problem, any1 can think of any solution? (other than not using personalization ofc)