Parsing Large XML files w/ Ruby & Nokogiri
nokogiri, ruby, xml
Solution
You can dramatically decrease your time to execute by changing your code to the following. Just change the "99" to whatever category you want to check.:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
icount = 0
xmlfeed = Nokogiri::XML(open("test.xml"))
items = xmlfeed.xpath("//item")
items.each do |item|
text = item.children.children.first.text
if ( text =~ /99/ )
icount += 1
end
end
othercount = xmlfeed.xpath("//totalcount").inner_text.to_i - icount
puts icount
puts othercount
This took about three seconds on my machine. I think a key error you made was that you chose the "items" iterate over instead of creating a collection of the "item" nodes. That made your iteration code awkward and slow.
Problem
I have a large XML file (about 10K rows) I need to parse regularly that is in this format: ``` <summarysection> <totalcount>10000</totalcount> </summarysection> <items> <item> <cat>Category</cat> <name>Name 1</name> <value>Val 1</value> </item> ...... 10,000 more times </items> ``` What I'd like to do is parse each of the individual nodes using nokogiri to count the amount of items in one category. Then, I'd like to subtract that number from the total_count to get an ouput that reads "Count of Interest_Category: n, Count of All Else: z". This is my code now: ``` #!/usr/bin/ruby require 'rubygems' require 'nokogiri' require 'open-uri' icount = 0 xmlfeed = Nokogiri::XML(open("/path/to/file/all.xml")) all_items = xmlfeed.xpath("//items") all_items.each do |adv| if (adv.children.filter("cat").first.child.inner_text.include? "partofcatname") icount = icount + 1 end end othercount = xmlfeed.xpath("//totalcount").inner_text.to_i - icount puts icount puts othercount ``` This seems to work, but is very slow! I'm talking more than 10 minutes for 10,000 items. Is there a better way to do this? Am I doing something in a less than optimal fashion?