3.6.0 mcc character encoding

[Sigurd Mytting]

Hi,

upgraded to 3.6.0 yesterday - it looks great!

But there is a small problem: it seems mcc won't handle norwegian
characters in port descriptions.

2010-10-09 09:35:58,708: [ERROR] nav.mcc.utils - Could not encode KVM
Str�mskinne 1 to latin-1: 'latin-1' codec can't encode character
u'\ufffd' in position 7: ordinal not in range(256)


I don't really care about the description beeing correctly displayed,
but mcc stops after this and no more cricket config is generated.

Looks like this happens when convert_unicode_to_latin1 in
nav/mcc/utils.py returns the unicode object instead of a latin-1 string.

My workaround is to just return an empty string instead of the unicode
object.

Thanks for all the work you guys put into NAV, truly exceptional
software solving everyday problems for a lot of us!

Cheers,

-Sigurd

[Morten Brekkevold]

Post by Sigurd Mytting
upgraded to 3.6.0 yesterday - it looks great!

Thanks :)

Post by Sigurd Mytting
But there is a small problem: it seems mcc won't handle norwegian
characters in port descriptions.
2010-10-09 09:35:58,708: [ERROR] nav.mcc.utils - Could not encode
KVM Str�mskinne 1 to latin-1: 'latin-1' codec can't encode character
u'\ufffd' in position 7: ordinal not in range(256)

The unicode character U+FFFD is not a Norwegian character. It's description
is "Replacement character", and for me it renders like a diamond with a
question mark in it. It has no equivalent character in the ISO-8859-1
(Latin-1) character set, and that is why the encoding fails. Cricket's
configuration will only accept Latin-1 characters.

Post by Sigurd Mytting
I don't really care about the description beeing correctly
displayed, but mcc stops after this and no more cricket config is
generated.

Since this is only for display purposes, I think a more elegant solution
would be to just ignore characters that can't be encoded as latin-1. This
can easily be achieved by replacing encode('latin-1') with encode('latin-1',
'ignore').

Could you please report this bug at https://bugs.launchpad.net/nav/+filebug ?

[Sigurd Mytting]

Post by Morten Brekkevold
Could you please report this bug at https://bugs.launchpad.net/nav/+filebug ?

ACK.

I'm also getting a lot of errors in ipdevpoll.log, just one examle:

2010-10-11 19:46:43,686 [ERROR]
[nav.ipdevpoll.jobs.jobhandler.inventory.(12a-sw-vts.vegvesen.no)]
Caught exception during save. Last object = Interface(netbox=
Netbox(id=351, type=NetboxType(id=20, vendor=Vendor(id=u'cisco'),
name=u'catalyst37xxStack', sysobjectid=u'1.3.6.1.4.1.9.1.516', cdp=True,
tftp=True, cs_at_vlan=True, chassis=True, description=u'Catalyst 3750'),
device=Device(id=370, serial
='FOC1227W4M0', hardware_version='V05', firmware_version='12.2(35)SE5',
software_version='12.2(35)SE5'), sysname='12a-sw-vts.vegvesen.no'),
ifindex=10117L, ifname='Gi1/0/17', ifdescr='GigabitEthernet1/0/17',
iftype=6, speed=10.0, ifphysaddress='00:22:56:34:9b:11',
ifadminstatus=1, ifoperstatus=1, ifconnectorpresent=True, ifalias='Temp
Overv\xe5ker', baseport=17L, vlan=1, trunk=False, duplex='h',
gone_since=None). Last model: Gi1/0/17 at 12a-sw-vts
Traceback (most recent call last):
File "/var/lib/python-support/python2.5/nav/ipdevpoll/jobs.py", line
334, in perform_save
obj_model.save()
File "/var/lib/python-support/python2.5/django/db/models/base.py",
line 311, in save
self.save_base(force_insert=force_insert, force_update=force_update)
File "/var/lib/python-support/python2.5/django/db/models/base.py",
line 383, in save_base
result = manager._insert(values, return_id=update_pk)
File "/var/lib/python-support/python2.5/django/db/models/manager.py",
line 138, in _insert
return insert_query(self.model, values, **kwargs)
File "/var/lib/python-support/python2.5/django/db/models/query.py",
line 894, in insert_query
return query.execute_sql(return_id)
File
"/var/lib/python-support/python2.5/django/db/models/sql/subqueries.py",
line 309, in execute_sql
cursor = super(InsertQuery, self).execute_sql(None)
File
"/var/lib/python-support/python2.5/django/db/models/sql/query.py", line
1734, in execute_sql
cursor.execute(sql, params)
DataError: invalid byte sequence for encoding "UTF8": 0xe56b65
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by
"client_encoding".


Is there any way I can avoid putting non latin-1 characters into the
database as a workaround?

Cheers,

-Sigurd

[Morten Brekkevold]

[snip]

Post by Sigurd Mytting
"/var/lib/python-support/python2.5/django/db/models/sql/query.py",
line 1734, in execute_sql
cursor.execute(sql, params)
DataError: invalid byte sequence for encoding "UTF8": 0xe56b65
HINT: This error can also happen if the byte sequence does not
match the encoding expected by the server, which is controlled by
"client_encoding".
Is there any way I can avoid putting non latin-1 characters into the
database as a workaround?

Well, this error has avoided it for you ;-)

Apparently, you've set the description of Gi1/0/17 on 12a-sw-vts to
"Temperatur Overvåker", using a latin-1 encoding.

Mostly, ipdevpoll assumes everything it collects is either ASCII or
decodeable as UTF-8. For hardwired text strings, ASCII will almost always be
correct, but for user-configured values it should probably try multiple
encodings before giving up. This error message says the string could not be
saved to the database, since it's not UTF-8, and that ensures that none of
12a-sw-vts' data will be saved at all.

Please report this to the bugtracker, and we'll try to fix it in 3.6.1
(scheduled next week, but possibly delayed over the weekend due to some
saved-up vacation time).

[Sigurd Mytting]

Post by Morten Brekkevold
[snip]

Post by Sigurd Mytting
"/var/lib/python-support/python2.5/django/db/models/sql/query.py",
line 1734, in execute_sql
cursor.execute(sql, params)
DataError: invalid byte sequence for encoding "UTF8": 0xe56b65
HINT: This error can also happen if the byte sequence does not
match the encoding expected by the server, which is controlled by
"client_encoding".
Is there any way I can avoid putting non latin-1 characters into the
database as a workaround?

Well, this error has avoided it for you ;-)

Yeah, I kind of got that after poking around in the code after I'd sent
the reply ;-) What I should have asked is how to remove all non ASCII
at the time data is fetched from the box.

Post by Morten Brekkevold
Apparently, you've set the description of Gi1/0/17 on 12a-sw-vts to
"Temperatur Overvåker", using a latin-1 encoding.

It's almost unavoidable when having more than 800 devices on 240 different
locations all over the country and more than 30 people doing this their own
way. I would of course never even dream of using anything other than ASCII
(one of the perks with growing up with the DOS codepage-hell in the 1980s.. ;)

Post by Morten Brekkevold
Please report this to the bugtracker, and we'll try to fix it in 3.6.1
(scheduled next week, but possibly delayed over the weekend due to some
saved-up vacation time).

Reported.

Cheers,

-Sigurd