Need To Skip Line Containing "value Error"

I'm trying to extract some legacy data from a Teradata server, but some of the records contain weird characters that don't register in python, such as 'U+ffffffc2'. Currently, I'

Solution 1:

If you have only 4-byte unicode points giving an error, this probably may help. One solution is to register a custom error handler using codecs.register_error, which would filter out error points and then just try to decode:

import codecs

deferror_handler(error):
    return'', error.end+6

codecs.register_error('nonunicode', error_handler)

b'abc\xffffffc2def'.decode(errors='nonunicode')
# gives you 'abcdef' which's exactly what you want

You may futher impove your handler to catch more complicated errors, see https://docs.python.org/3/library/exceptions.html#UnicodeError and https://docs.python.org/3/library/codecs.html#codecs.register_error for details