Which Ruby XML library would you recommend for a 2.4MB XML file?

ruby, xml

Solution

Hpricot is probably the best tool for you -- it is easy to use and should handle 2mg file with no problem.

Speedwise libxml should be the best. I used libxml2 binding for python few months ago (at that moment rb-libxml was stale). Streaming interface worked the best for me (LibXML::XML::Reader in ruby gem). It allows to process file while it is downloading, is a bit more userfriendly than SAX and allowed me to load data from 30mb xml file from internet to a MySQL database in a little more than a minute.

Problem

I have a 2.4 MB XML file, an export from Microsoft Project (hey I'm the victim here!) from which I am requested to extract certain details for re-presentation. Ignoring the intelligence or otherwise of the request, which library should I try first from a Ruby perspective? I'm aware of the following (in no particular order): - REXML - Chilkat Ruby XML library - hpricot XML - libXML I'd prefer something packaged as a Ruby gem, which I suspect the Chilkat library is not. Performance isn't a major issue - I don't expect the thing to need to run more than once a day (once a week is more likely). I'm more interested in something that's as easy to use as anything XML-related is able to get. EDIT: I tried the gemified ones: hpricot is, by a country mile, easiest. For example, to extract the content of the SaveVersion tag in this XML (saved in a file called, say 'test.xml') ``` <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Project xmlns="http://schemas.microsoft.com/project"> <SaveVersion>12</SaveVersion> </Project> ``` takes something like this: ``` doc = Hpricot.XML(open('test.xml')) version = (doc/:Project/:SaveVersion).first.inner_html ``` hpricot seems to be relatively unconcerned with namespaces, which in this example is fine: there's only one, but would potentially be a problem with a complex document. Since hpricot is also very slow, I rather imagine this would be a problem that solves itself. libxml-ruby is an order of magnitude faster, understands namespaces (it took me a good couple of hours to figure this out) and is altogether much closer to the XML metal - XPath queries and all the other stuff are in there. This is not necessarily a Good Thing if, like me, you open up an XML document only under conditions of extreme duress. The helper module was mostly helpful in providing examples of how to handle a default namespace effectively. This is roughly what I ended up with (I'm not in any way asserting its beauty, correctness or other value, it's just where I am right now): ``` xml_parser = XML::Parser.new xml_parser.string = File.read(path) doc = xml_parser.parse @root = doc.root @scopes = { :in_node => '', :in_root => '/', :in_doc => '//' } @ns_prefix = 'p' @ns = "#{@ns_prefix}:#{@root.namespace[0].href}" version = @root.find_first(xpath_qry("Project/SaveVersion", :in_root), @ns).content.to_i def xpath_qry(tags, scope = :in_node) "#{@scopes[scope]}" + tags.split(/\//).collect{ |tag| "#{@ns_prefix}:#{tag}"}.join('/') end ``` I'm still debating the pros and cons: libxml for its extra rigour, hpricot for the sheer style of _why's code. EDIT again, somewhat later: I discovered HappyMapper ('gem install happymapper') which is hugely promising, if still at an early stage. It's declarative and mostly works, although I have spotted a couple of edge cases that I don't have fixes for yet. It lets you do stuff like this, which parses my Google Reader OPML: ``` module OPML class Outline include HappyMapper tag 'outline' attribute :title, String attribute :text, String attribute :type, String attribute :xmlUrl, String attribute :htmlUrl, String has_many :outlines, Outline end end xml_string = File.read("google-reader-subscriptions.xml") sections = OPML::Outline.parse(xml_string) ``` I already love it, even though it's not perfect yet.

Original source