Roman Kirillov’s blog

XML input and Hadoop – custom InputFormat

Today I finally hit the task I was scared for so long — processing large XML files on Hadoop. I won’t tell you for how long I crawled the Internet trying to find some working solution… not that anyone wants to know? Eventually, I came out with the solution of my own — even though I hate re-inventing the wheel, in this particular case all the wheels I found were either square or were utterly incompatible with my model of car.

To make things more simple, I won’t include the full source code. I won’t even include the whole InputFormat class. So, to make yourself comfortable, please do following:

1. Open LineRecordReader from org.apache.hadoop.mapreduce.lib.input so you can see it
2. Open TextInputFormat from the same package.
3. Create the input format and record reader of your own, just by copying and pasting the code from aforementioned classes.
4. Change the constructor of your input format class so it’ll return your newly-defined record reader.

Now, we’re almost there. Now I’ll include the piece of code for nextKeyValue() which turned out to be the most critical method here. Hold on tight:

public boolean nextKeyValue() throws IOException
{
    StringBuilder sb = new StringBuilder();
    if (key == null)
    {
        key = new LongWritable();
    }
    key.set(pos);
    if (value == null)
    {
        value = new Text();
    }
    int newSize = 0;
    boolean xmlRecordStarted = false;
    Text tmpLine = new Text();
    while (pos < end)
    {
        newSize = in.readLine(tmpLine, 
            maxLineLength, 
            Math.max((int) 
                Math.min(Integer.MAX_VALUE, 
                        end - pos), 
            maxLineLength));
                    if (newSize == 0)
        {
            break;
        }
        if (tmpLine.toString().contains("<document "))
        {
            xmlRecordStarted = true;
        }
        if (xmlRecordStarted)
        {
            sb.append(tmpLine.toString().replaceAll("\n", " "));
        }
        if (tmpLine.toString().contains("</document>"))
        {
            xmlRecordStarted = false;
            this.value.set(sb.toString());
            break;
        }
        pos += newSize;
    }
    if (newSize == 0)
    {
        key = null;
        value = null;
        return false;
    }
    else
    {
        return true;
    }
}

WTF — you will say? It’s the same code? Well — yes, and no. It’s almost the same. Take a look at this line:

if (tmpLine.toString().contains("<document")) 

and this line:

if (tmpLine.toString().contains("</document>")) 

This is where we actually split the document into chunks. Code is pretty-much self-explaining so I won’t add anything else.

Now, it’s not the most clean and streamlined solution and I probably will spend a while tomorrow making it more production-ready and good-looking, but compared to other solutions, it has few major benefits:

1. It uses very little custom code (you remember, we copied and pasted all the classes?). Unfortunately you cannot just inherit the class — some fields are private, and we clearly want to modify them.
2. It’s configurable — you can easily change the <document and </document> strings to anything else (and again, I will do it tomorrow, but now I feel too lazy).
3. It works.

There’re few limitations of this approach. One of them is that if the document contains something like </document><document> it obviously won’t work. Another is — you still need to parse elements in your mapper (although you can easily change it by parsing records in your record reader into Writable-compatible class).

Comments (10)

Nov 04, 2009

Holger Dürer said...

I am not sure I have quite understood yet what you are trying to achieve. You have one big XML file and are trying to create records where each record contains one <document> element?

Nov 04, 2009

Roman Kirillov said...

Pretty much so. For most of standard Hadoop jobs you have a flat text file where every record is a line. In this example we deal with an XML file where every record is represented by one <document> element and all it's content.

Nov 04, 2009

Holger Dürer said...

Also: what are pos and end? The approximate byte range from which to read the records, right? A record will usually not fall on these boundaries. That's why you skip lines at the beginning, right? But doesn't that mean you should finish reading a record past 'end'? Otherwise your last record might terminate prematurely....

Nov 04, 2009

Roman Kirillov said...

Pretty much so. For most of standard Hadoop jobs you have a flat text file where every record is a line. In this example we deal with an XML file where every record is represented by one <document> element and all it's content. 

Nov 04, 2009

Roman Kirillov said...

pos and end are members from LineRecordReader. Don't have the source handy, but I they declare end = Long.MAX_VALUE; (or expect user to specify the end-offset but this isn't the case).

Nov 06, 2009

Holger Dürer said...

I am still convinced you need to change the "while (pos < end)" to keep looking past the split's end if you are in the middle of parsing a document -- otherwise your last document in the split will be truncated. I.e. something like "while (pos < end || xmlRecordStarted)

Nov 06, 2009

Roman Kirillov said...

I think I'll give it a try - for now I'm not that really concerned about it, but in future (i.e. when we'll deal with you-know-which-product) we'll probably need to sort it out in a proper way.

Nov 06, 2009

Holger Dürer said...

For TPWMNBN you also need to sort out the newline thing. You cannot just replace newlines with spaces -- that can be changing the meaning of the contents.

Nov 06, 2009

Roman Kirillov said...

Actually I fixed it - there's no newline replacing now (mainly because it serves no purpose and we still can handle it with no problems at all). Also, for TPWMNBN we can instead read content straight into objects (by changing the RecordReader to return some Object rather than Text). Once you figure all this stuff out it's quite simple!

Jan 19, 2010

Paul Ingles said...

Hi,

Just found your post. I've been trying to do the same thing. In the end, I went with the XmlInputFormat from Mahout's Bayesian Classifier. Seems to do everything I need it to (and works without going screwy like the streaming one).

I posted about it here: http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html

Leave a comment...

 

To leave a comment on this posterous, please login by clicking one of the following.