Module feedparser

URL, filename, or string --> stream

This function lets you define parsers that take any input source (URL, pathname to local or network file, or actual data as a string) and deal with it in a uniform manner. Returned object is guaranteed to have all the basic stdio read methods (read, readline, readlines). Just .close() the object when you're done with it.

If the etag argument is supplied, it will be used as the value of an If-None-Match request header.

If the modified argument is supplied, it must be a tuple of 9 integers as returned by gmtime() in the standard Python time module. This MUST be in GMT (Greenwich Mean Time). The formatted date/time will be used as the value of an If-Modified-Since request header.

If the agent argument is supplied, it will be used as the value of a User-Agent request header.

If the referrer argument is supplied, it will be used as the value of a Referer[sic] request header.

If handlers is supplied, it is a list of handlers used to build a urllib2 opener.

Get the character encoding of the XML document

http_headers is a dictionary xml_data is a raw string (not Unicode)

This is so much trickier than it sounds, it's not even funny. According to RFC 3023 ('XML Media Types'), if the HTTP Content-Type is application/xml, application/*+xml, application/xml-external-parsed-entity, or application/xml-dtd, the encoding given in the charset parameter of the HTTP Content-Type takes precedence over the encoding given in the XML prefix within the document, and defaults to 'utf-8' if neither are specified. But, if the HTTP Content-Type is text/xml, text/*+xml, or text/xml-external-parsed-entity, the encoding given in the XML prefix within the document is ALWAYS IGNORED and only the encoding given in the charset parameter of the HTTP Content-Type header should be respected, and it defaults to 'us-ascii' if not specified.

Furthermore, discussion on the atom-syntax mailing list with the author of RFC 3023 leads me to the conclusion that any document served with a Content-Type of text/* and no charset parameter must be treated as us-ascii. (We now do this.) And also that it must always be flagged as non-well-formed. (We now do this too.)

If Content-Type is unspecified (input was local file or non-HTTP source) or unrecognized (server just got it totally wrong), then go by the encoding given in the XML prefix of the document and default to 'iso-8859-1' as per the HTTP specification (RFC 2616).

Then, assuming we didn't find a character encoding in the HTTP headers (and the HTTP Content-type allowed us to look in the body), we need to sniff the first few bytes of the XML data and try to determine whether the encoding is ASCII-compatible. Section F of the XML specification shows the way here: http://www.w3.org/TR/REC-xml/#sec-guessing-no-ext-info

If the sniffed encoding is not ASCII-compatible, we need to make it ASCII compatible so that we can sniff further into the XML declaration to find the encoding attribute, which will tell us the true encoding.

Of course, none of this guarantees that we will be able to parse the feed in the declared character encoding (assuming it was declared correctly, which many are not). CJKCodecs and iconv_codec help a lot; you should definitely install them if you can. http://cjkpython.i18n.org/

Changes an XML data stream on the fly to specify a new encoding

data is a raw sequence of bytes (not Unicode) that is presumed to be in %encoding already encoding is a string recognized by encodings.aliases

Strips DOCTYPE from XML document, returns (rss_version, stripped_data)

rss_version may be 'rss091n' or None stripped_data is the same XML document, minus the DOCTYPE

_urifixer

Value:

re.compile(r'^([A-Za-z][A-Za-z0-9\+-\.]*://)(/*)(.*?)')

_iso8601_tmpl

Value:

['YYYY-?MM-?DD', 'YYYY-MM', 'YYYY-?OOO', 'YY-?MM-?DD', 'YY-?OOO', 'YYYY', '-YY-?MM', '-OOO', ...

_iso8601_re

Value:

['(?P<year>\\d{4})-?(?P<month>[01]\\d)-?(?P<day>[0123]\\d)(T?(?P<hour> \\d{2}):(?P<minute>\\d{2})(:(?P<second>\\d{2}))?(?P<tz>[+-](?P<tzhour> \\d{2})(:(?P<tzmin>\\d{2}))?|Z)?)?', '(?P<year>\\d{4})-(?P<month>[01]\\d)(T?(?P<hour>\\d{2}):(?P<minute>\\ d{2})(:(?P<second>\\d{2}))?(?P<tz>[+-](?P<tzhour>\\d{2})(:(?P<tzmin>\\ d{2}))?|Z)?)?', '(?P<year>\\d{4})-?(?P<ordinal>[0123]\\d\\d)(T?(?P<hour>\\d{2}):(?P<m inute>\\d{2})(:(?P<second>\\d{2}))?(?P<tz>[+-](?P<tzhour>\\d{2})(:(?P< ...

_iso8601_matches

Value:

[re.compile(regex).match for regex in _iso8601_re]

_korean_onblog_date_re

Value:

re.compile(r'(\d{4})\ub144\s+(\d{2})\uc6d4\s+(\d{2})\uc77c\s+(\d{2}):( \d{2}):(\d{2})')

_korean_nate_date_re

Value:

re.compile(r'(\d{4})-(\d{2})-(\d{2})\s+(\uc624[\uc804\ud6c4])\s+(\d{,2 }):(\d{,2}):(\d{,2})')

_mssql_date_re

Value:

re.compile(r'(\d{4})-(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})(\.\d+)? ')

_greek_months

Value:

{u'Απρ': u'Apr', u'Αυγ': u'Aug', u'Αύγ': u'Aug', u'Δεκ': u'Dec', u'Ιαν': u'Jan', u'Ιολ': u'Jul', u'Ιον': u'Jun', u'Ιούλ': u'Jul', ...

_greek_wdays

Value:

{u'Δευ': u'Mon', u'Κυρ': u'Sun', u'Παρ': u'Fri', u'Πεμ': u'Thu', u'Σαβ': u'Sat', u'Τετ': u'Wed', u'Τρι': u'Tue'}

_greek_date_format_re

Value:

re.compile(r'([^,]+),\s+(\d{2})\s+([^\s]+)\s+(\d{4})\s+(\d{2}):(\d{2}) :(\d{2})\s+([^\s]+)')

_hungarian_months

Value:

{u'augusztus': u'08', u'december': u'12', u'februári': u'02', u'január': u'01', u'július': u'07', u'június': u'06', u'március': u'03', u'máujus': u'05', ...

_hungarian_date_format_re

Value:

re.compile(r'(\d{4})-([^-]+)-(\d{,2})T(\d{,2}):(\d{2})(([\+-])(\d{,2}: \d{2}))')

_additional_timezones

Value:

{'AT': -400, 'CT': -600, 'ET': -500, 'MT': -700, 'PT': -800}