Full support for communicating with any Hadoop cluster?

Oct 29, 2012 at 6:52 PM

I'm curious if the .Net SDK for Hadoop only works/communicates with DHInsight deployments, or will it work with any Hadoop cluster?  What versions of Hadoop are supported?  Will it be able to push jobs to a Rackspace/AWS cluster?  Or just Azure?

Oct 30, 2012 at 12:08 AM

The SDK only requires "hadoop streaming" to be available and so should work with any distribution that includes streaming.  However, it has only been tested with HDInsights and so I wouldn't be surprised if some small changes are required for other distributions.

Some prototyping has been done for pushing jobs to Azure but not for other cloud providers.

-mike.

 

 

Dec 3, 2012 at 8:30 PM

We've now added a WebHDFS client which will let you talk to any cluster running webhdfs for file system operations