chunk – Read a large (XML) file a chunk at a time

I’ve recently had to parse some pretty large XML documents, and needed a method to read one element at a time.

Here’s a fairly simple solution in PHP and Ruby form (hopefully a Python one coming soon…).

If you have the following file (complex-test.xml):

    Title 1
    It's name goes here

And wanted to return the <Object/>s



$file = new Chunk('complex-test.xml', array('element' => 'Object'));

while ($xml = $file->read()) {
  $obj = simplexml_load_string($xml);
  // do some parsing, insert to DB whatever


require 'chunk.rb'

file ='complex-test.xml', { 'element' => 'Object' })

while xml =
  doc =
  // do some parsing, blah...

It (probably) doesn’t work with nested XML elements, but the use I had for it didn’t require that it did.

You can get the PHP version here, and the Ruby one here.

Update: The class was accepted onto PHP Classes as a notable package! 04/08/2009.

6 Replies to “chunk – Read a large (XML) file a chunk at a time”

  1. This code helped me parse some very large files quickly, however when I try to create more than one chunk instance in a script, it will error out. I am not quite sure why this is happening. I receive the following exception when I try to read the second XML file in:

    ./chunk.rb:223:in `+': can't convert nil into String (TypeError)
    	from ./chunk.rb:223:in `read'

    I am calling again, so I am not sure why a second execution would cause this behavior. If I execute each import separately, everything works. Any help is greatly appreciated.

    1. Hi there Ben,

      I’ll certainly have a play with it and see if it’s something I can resolve quickly, not sure why it would have any impact though…

      Thanks for pointing this out!


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.