Android: How to download RSS when a website contains: link rel="alternate" type="application/rss+xml"

android, html, rss

Solution

To accomplish this, you need to:

- Detect whether the URL points to an HTML file. See the `isHtml` method in the code below.

- If the URL points to an HTML file, extract an RSS URL from it. See the `extractRssUrl` method in the code below.

The following code is a modified version of the code you pasted in your question. For I/O, I used Apache Commons IO for the useful `IOUtils` and `FileUtils` classes. `IOUtils.toString` is used to convert an input stream to a string, as recommended in the article "In Java, how do I read/convert an InputStream to a String?"

`extractRssUrl` uses regular expressions to parse HTML, even though it is highly frowned upon. (See the rant in "RegEx match open tags except XHTML self-contained tags.") With this in mind, let `extractRssUrl` be a starting point. The regular expression in `extractRssUrl` is rudimentary and doesn't cover all cases.

Note that a call to `isRss(str)` is commented out. If you want to do RSS detection, see "How to detect if a page is an RSS or ATOM feed."

private boolean downloadXml(String url, String filename) {
    InputStream is = null;
    try {
        URL urlxml = new URL(url);
        URLConnection ucon = urlxml.openConnection();
        ucon.setConnectTimeout(4000);
        ucon.setReadTimeout(4000);
        is = ucon.getInputStream();
        String str = IOUtils.toString(is, "UTF-8");
        if (isHtml(str)) {
            String rssURL = extractRssUrl(str);
            if (rssURL != null && !url.equals(rssURL)) {
                return downloadXml(rssURL, filename + ".xml");
            }
        } else { // if (isRss(str)) {
            // For now, we'll assume that we're an RSS feed at this point
            FileUtils.write(new File(filename), str);
            return true;
        }
    } catch (Exception e) {
        // do nothing
    } finally {
        IOUtils.closeQuietly(is);
    }
    return false;
}

private boolean isHtml(String str) {
    Pattern pattern = Pattern.compile("<html", Pattern.CASE_INSENSITIVE | Pattern.DOTALL | Pattern.MULTILINE);
    Matcher matcher = pattern.matcher(str);
    return matcher.find();
}

private String extractRssUrl(String str) {
    Pattern pattern = Pattern.compile("<link(?:\\s+href=\"([^\"]*)\"|\\s+[a-z\\-]+=\"[^\"]*\")*\\s+type=\"application/rss\\+(?:xml|atom)\"(?:\\s+href=\"([^\"]*)\"|\\s+[a-z\\-]+=\"[^\"]*\")*?\\s*/?>", Pattern.CASE_INSENSITIVE | Pattern.DOTALL | Pattern.MULTILINE);
    Matcher matcher = pattern.matcher(str);
    if (matcher.find()) {
        for (int i = 1; i <= matcher.groupCount(); i++) {
            if (matcher.group(i) != null) {
                return matcher.group(i);
            }
        }
    }
    return null;
}

The above code works with your Engadget example:

obj.downloadXml("http://www.engadget.com/", "rss");

Problem

Original source

Related problems

• RegEx match open tags except XHTML self-contained tags
• How do I read / convert an InputStream into a String in Java?
• How to detect if a page is an RSS or ATOM feed