using .NET SDK against Hadoop on Linux

Oct 8, 2013 at 9:59 AM
Edited Oct 8, 2013 at 12:01 PM
Hi Guys.

Very impressive product.

Can you please answer 2 questions:
  1. Can I simply run C# code using your .NET SDK for Hadoop on Windows / Visual Studio (i.e., NOT Mono) against a Hadoop cluster running on Linux. To clarify, from within Visual Studio running on Windows, I would like to submit a MapReduce task onto a Hadoop cluster running on Linux boxes. If so, can you please provide some guidance on the configuration.
  2. how stable is the SDK ... when do you expect it will be production ready?
Thank you very much in advance.

Edward
Coordinator
Oct 9, 2013 at 7:44 AM
Hi Edward,

Thank you for interest in this SDK.
  1. Currently Linux configuration is not tested. While in theory it should work you would need to do your own tests on the Linux with Mono environment to get concrete results.
  2. There is [roadmap] that outlines stages of the SDK components. Currently all components are in experimental state. Soon some of the components will move into release state when HDINSIGHT service reaches General Availability.
Oct 9, 2013 at 9:55 AM
Edited Oct 9, 2013 at 9:56 AM
Hi Maxluk:

Thank you for replying.

Perhaps I should clarify, I do not intend to run C# on Mono. MapReduce tasks will be devloped and submitted from Visual Studio on Windows but the cluster itself will be running in Java on Linux (Java Hadoop); Is that possible?

Thank you; however, I know the procedure. I was asking for your expectation as to time frame (3 months, 6 months, etc.).

Again, thanks,
Edward
Coordinator
Oct 9, 2013 at 6:22 PM
In order to run .NET map reduce jobs the cluster data nodes needs to support .NET. In case of Linux that means Mono environment.
The time frame for initial set of components moving to production state is soon - around 1 month. This won't include .NET map/reduce though. Timeline for .NET map reduce is not determined at this point. We reserve the right to keep any component of the SDK in experimental state.
Oct 10, 2013 at 11:54 PM
Hi Maxluk:

Thank you for replying.
First, understood.
Second, please clarify what you mean by "This won't include .NET map/reduce though" ... isn't that the whole point of .NET SDK, to allow one to run map-reduce tasks from .NET?
Also, what do you mean by "initial set of components"?
Finally, when do you think .net map/reduce will be stable enough (production level) to run tasks against __Window__s' based Hortonwork's HDP?

Again, thank you very much for your assistance, I appreciate.

Best,
E
Oct 18, 2013 at 3:10 AM
Edited Oct 18, 2013 at 3:11 AM
Hi Maxluk:

Shall I expect an answer / reply?

Thanks,
E
Coordinator
Oct 18, 2013 at 6:57 PM
Hi Edward,

Sorry for delay with reply.
The current state of the SDK is experimental. You can find more information on the quality levels in the sdk here: https://hadoopsdk.codeplex.com/wikipage?title=Roadmap. In the next release we'll move job submission (WebHCat) and cluster provisioning apis to release state. .NET Map/Reduce will remain in the experimental state.
Unfortunately the timeline for experimental components is not available due to the experimental nature of the components. But we recognize interest of the community in the .NET map reduce and will plan future work accordingly.

Maxim