This project is read-only.
You can run .Net Map/Reduce jobs on remote Azure HDInsight cluster by using Azure Hadoop connection. To create Azure connection use Connect method overload that takes set of Azure credentials as parameters.

            Uri azureCluster = new Uri("https://azureCluster.azurehdinsight.net");
            string clusterUserName = "{Your cluster user name}";
            string clusterPassword = "{Your cluster password}";

            // This is the name of the account under which Hadoop will execute jobs.
            // Normally this is just "Hadoop".
            string hadoopUserName = "{HadoopAccount}";

            // Azure Storage Information.
            string azureStorageAccount = "{YourStorageAccountName.blob.core.windows.net}";
            string azureStorageKey = "{YourAzureStorageKey}";
            string azureStorageContainer = "{YourAzureStorageContainer}";
            bool createContinerIfNotExist = true;

            // Workaround for the case when you don't have Dev Emulator installed on the client machine
            Environment.SetEnvironmentVariable("HADOOP_HOME", @"c:\hadoop");
            Environment.SetEnvironmentVariable("Java_HOME", @"c:\hadoop\jvm");

            var hadoop = Hadoop.Connect(azureCluster,
                                        clusterUserName,
                                        hadoopUserName,
                                        clusterPassword,
                                        azureStorageAccount,
                                        azureStorageKey,
                                        azureStorageContainer,
                                        createContinerIfNotExist);

            var result = hadoop.MapReduceJob.ExecuteJob<JobType>();



For clusters in use prior to General Availability (1,5, 2.0), one will need to provide a port of 563. For GA clusters, 2.1 and beyond, the gateway has been moved to port 443, the default SSL port.

Last edited Nov 11, 2013 at 5:40 PM by maxluk, version 4

Comments

Ishvari Apr 13 at 7:18 AM 
I',m trying above code for Submit MapReduce to HDInsight. For Connection to HDInsight, I used Hadoop.Connect(azureCluster,
clusterUserName,
hadoopUserName,
clusterPassword,
azureStorageAccount,
azureStorageKey,
azureStorageContainer,
createContinerIfNotExist) Method. I want to ask why there is need for Environment variable HadoopHome and JavaHome , here its path is of C drive. Is there any necessary to install Hadoop locally??
Please Reply.
Thanks

shahramjaved75 Apr 19, 2014 at 10:25 PM 
Hi i am also getting same issue, any one who is able to find fix for this issue, i also tried to run project on target platform x64

andyelastacloud Oct 30, 2013 at 1:01 PM 
Since GA in later october, this documentation is wrong.

Uri azureCluster = new Uri("https://azureCluster.azurehdinsight.net:563");

should be

Uri azureCluster = new Uri("https://azureCluster.azurehdinsight.net");

developmentally Jul 18, 2013 at 12:22 AM 
switching platform target to x64 fixed this for me - thanks

WenJinhui Jul 13, 2013 at 11:59 AM 
the project's target platform should be x64.

king53403713 Jul 4, 2013 at 7:19 PM 
run into the same error.

{"Null object cannot be converted to a value type."}
at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
at Newtonsoft.Json.Linq.Extensions.Convert[T,U](T token)
...

surensi Jun 9, 2013 at 5:33 PM 
Getting this Error while submitting jobs to hdinsight cluster:


{"Null object cannot be converted to a value type."}
at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
at Newtonsoft.Json.Linq.Extensions.Convert[T,U](T token)
at Newtonsoft.Json.Linq.JToken.Value[T](Object key)
at Microsoft.Hadoop.WebClient.WebHCatClient.WebHcatMapReduceStreamingExecutor.Execute(Boolean throwOnError)
at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.ExecuteCore(Type mapper, Type reducer, Type combiner, HadoopJobConfiguration config)
at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.ExecuteJob(Type jobType, String[] arguments)
at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.ExecuteJob[TJobType](String[] arguments)
at Microsoft.Hadoop.MapReduce.Execution.Hadoop.StreamingJobExecutorBase.ExecuteJob[TJobType]()
at Microsoft.Internal.HDInsight.AzureStorageAnalytics.Program.Main(String[] args) in c:\HDInsight\HDInsight\HDInsight\Program.cs:line 45

Any clue?