WebHDFS Client

WebHDFS Client is .NET wrapper for interacting with WebHDFS compliant end-points. This is basically a small client on top of the WebHDFS head as defined in this spec. The client supports both regular HDFS based installations of Hadoop as well as Azure HDInsight service where files are stored in Azure Blob Storage.

List Directories

var asvAccount = "ASV storage account name";
var asvKey = "ASV storage account key";

var client = new WebHDFSClient("hadoop", 
                            new BlobStorageAdapter(
                                asvAccount, asvKey, "default", true));
    .ContinueWith(dl => dl.Result.Directories.ToList().ForEach(d => Console.WriteLine("/" + d.PathSuffix)));

Task Integration

This API has been built integrating with both tasks as well as the 4.5 async keyword while lets us do some nice things:

var created = await client.CreateDirectory("/TEST");
Console.WriteLine("True or False, we created the directory " + created.ToString());
var deleted = await client.DeleteDirectory("/TEST");
Console.WriteLine("True or False, we deleted the directory " + deleted.ToString());

Task Chaining

    .ContinueWith(x => client.CreateFile(@"c:\tmp\Titles.txt", "/user/hadoop/titles.txt")
        .ContinueWith(t => Console.WriteLine("new file located at " + t.Result))
        .ContinueWith(t => client.OpenFile("/user/hadoop/titles.txt")
            .ContinueWith(resp => resp.Result.Content.ReadAsStringAsync()
                .ContinueWith(bigString => Console.WriteLine("new file is " + bigString.Result.Length + " bytes long"))
                .ContinueWith(t2 => client.DeleteDirectory("/user/hadoop/titles.txt")
                    .ContinueWith(b => Console.WriteLine("Successfully deleted file." ))

Last edited Jul 22, 2013 at 5:27 PM by maxluk, version 4


No comments yet.