This site is for the .NET SDK for Hadoop, making it easier to work with Hadoop from .NET. The site continues to host two Incubator projects and provides references for projects that have already graduated from it.

Please note:  The following .NET SDK packages have been deprecated and will no longer be supported starting on January 1, 2017:

• Microsoft.WindowsAzure.Management.HDInsight
• Microsoft.Hadoop.Client

The following packages form the new .NET SDK for HDInsight, compatible with Azure Resource Manager (ARM):

• Microsoft.Azure.Management.HDInsight (MSDN, Azure.com)
• Microsoft.Azure.Management.HDInsight.Job (MSDN, Azure.com)

Please see our migration guide for more information.

Graduated Projects

• HDInsight Cluster Management PowerShell Tools
  • Graduated to Azure GitHub
  • Documentation: Installing new Azure PowerShell Tools

• HDInsight Cluster Management .NET APIs
  • Graduated to Azure GitHub
  • Documentation: Using new .NET APIs

• Job Submission PowerShell Tools
  • Graduated to Azure GitHub
  • Documentation: Installing new Azure PowerShell Tools

• Job Submission .NET APIs
  • Graduated to Azure GitHub
  • Documentation: Using new .NET APIs

• Microsoft Avro Library
  • Graduated to Azure GitHub
  • Documentation: Using Avro Library

Incubator Projects

• Map/Reduce
• LINQ to Hive
  • Simple Linq To Hive Query
  • Linq To Hive with .NET UDF
• WebClient
  • WebHDFS Client
  • WebHCat Client - graduated into Job Submission Project and will be retired.
  • Oozie Client
  • Ambari Monitoring Client
• Windows Azure HDInsight Log Analysis Toolkit

Incubator NuGet Packages

You can download and install latest published build of the Incubation SDK projects using NuGet. In Visual Studio open NuGet package manager console and use following commands to install components of the SDK:

install-package Microsoft.Hadoop.MapReduce  
install-package Microsoft.Hadoop.Hive 
install-package Microsoft.Hadoop.WebClient 

Source Code

You can browse the source either via the web or git.

git clone https://git01.codeplex.com/hadoopsdk.git 

Building

Navigate to the project directory and run following msbuild command.

msbuild .\src\HDInsightSDK.sln /p:NoTestsOnBuild=true

As part of building you may need to obtain NuGet packages from the Outercurve Foundation NuGet public feed*.

*By running build, you will be initiating the download of other software from a NuGet-based feed that is owned by the Outercurve Foundation. You are responsible for locating, reading and complying with the license terms that accompany each such software. Each software that you obtain through this feed is licensed to you by its respective owner. We grant you no rights for third party software from this feed.

Installing HDInsight Emulator on local machine

You can obtain one-box installation of Hadoop development environment by installing Microsoft HDInsight using WebPI.

Roadmap

You can see the roadmap on this wiki.

Contributing

We appreciate any contribution to help create a better experience for writing Hadoop jobs with .NET.