Handling WebHDFS Client Redirect

Jul 16, 2013 at 11:33 PM
I have written a short C# utility to familiarize myself with the WebHDFS client interface and am using a local installation of HDInsight for my dev environment. I am able to interact with HDFS through the client using http://localhost:50070 and am able to write files to HDFS using http://localhost:50075. In other words, my environment is functioning properly.

Where I'm having difficulty is in calling the CreateFile method on the client when the client is pointed at http://localhost:50070. I understand there is a redirect that takes place but I am struggling with how to handle that in code so that the method call returns an exception. The call I am using looks like this:
myClient.CreateFile( srcFileName, destFileName).Wait();
Any guidance you could provide would be greatly appreciated.

Jul 18, 2013 at 12:40 AM
Hi Bryan,

Recommended and supported configuration for HDInsight developer installation is to use Azure Blob storage as a file system. Please follow this instructions to configure dev install to use Azure Blob storage: http://gettingstarted.hadooponazure.com/releaseNotes.html#asv

For the issue can you provide snippet of the code that fails?

Jul 18, 2013 at 4:07 AM

Thanks for the response, but I’m not working the HDInsights on Azure. Instead, I am using a local installation which does not employ Azure storage. Any suggestions for dealing with the redirect in that scenario?

Jul 18, 2013 at 6:58 PM
Recommended configuration is to use Azure Storage in local installation on single box. Can you provide details of what configuration you use? What's the code that fails, e..g how you initialize WebHDFS client object?
Jul 19, 2013 at 12:18 PM
Hi bryancsmith,
                           Please let me know how you doing WebHDFSClient connection as I am not to connect to HDFS using c# with below code:
WebHDFSClient client = new WebHDFSClient("http://localhost:50070",null);
Jul 19, 2013 at 12:36 PM
In place of null try "Hadoop"

Sent from my Windows Phone

Jul 19, 2013 at 7:14 PM
Thanks, everyone, for their assistance. I was able to determine that the problem had to do with my local cluster holding onto an invalid IP address. The cluster was setup on a virtual machine on one network and then moved to another network. Simply rebooting the VM did the trick.

For anyone else struggling with the WebHDFS interface, here is the code that works for my local instance:
WebHDFSClient myClient = new WebHDFSClient( new Uri("http://localhost:50070"), "Hadoop");

myClient.CreateFile(srcFileName, destFileName).Wait();
Hope this helps.