Microsoft HDInsight SDk documentation since the go live ( october 28th)

Nov 1, 2013 at 1:47 PM
Good morning

I would like to understand if this SDK is a product of an individual or from Microsoft. Also could you let me know where is the documentation for the API for method such as CreateStreamingJob and StreamingMapReduceJobCreateParameters?

Many thanks
Coordinator
Nov 1, 2013 at 4:14 PM
This is Microsoft product. Home page of the project describes Released and Incubator projects that part of it. Released projects have msdn documentation here: http://go.microsoft.com/fwlink/?LinkID=329857&clcid=0x409 and here: http://go.microsoft.com/fwlink/?LinkID=329854&clcid=0x409.

Maxim
Marked as answer by avezina on 11/1/2013 at 12:20 PM
Nov 1, 2013 at 4:48 PM
Good afternoon

Thanks a lot for taking the time to point me in the right direction. Sadly, there is no code example of the usage of CreateStreamingJob.

Maybe you could help me with what i'm trying to do?

I used your code example to publish a C# job. I would like to do the same with the CreateStreamingJob, but I'm not sure what to pass for the Reducer , Mapper and Combiner.

My goal is to be able to execute a job on a hive query result .


var streamJobDefintion = new StreamingMapReduceJobCreateParameters()
        {
            JobName = "JobName",
            StatusFolder = "/JobStatus/UploadGoogleInterations",
            Input = "/JobStatus/SelectInteractions/stdout",
            Output = "/JobStatus/UploadGoogleInteraction",
            Reducer = "",
            Mapper = "",
            Combiner ="",

        };

        var googleJobResult = jobClient.CreateStreamingJob(streamJobDefintion);
Coordinator
Nov 1, 2013 at 5:05 PM
Hi, try this article - it describes parameters for streaming job in PowerShell script. This is compatible with .NET APIs: http://www.windowsazure.com/en-us/manage/services/hdinsight/howto-run-samples/sample-csharp-streaming/

Maxim
Marked as answer by avezina on 11/1/2013 at 12:19 PM
Nov 1, 2013 at 5:42 PM
Good afternoon

It is exactly what I was looking for. Once again, thanks again for your time and effort.

Have a great weekend
Nov 1, 2013 at 7:04 PM
I successfully post a job and I thought that it would be useful for other to see my demo:
  // Submit the  job
            IJobSubmissionClient jobClient = JobSubmissionClientFactory.Connect(creds);


            var streamJobDefintion = new StreamingMapReduceJobCreateParameters()
            {
                JobName = "Word Count",
                StatusFolder = "/JobStatus/WordCount/Status",
                Input = "/JobStatus/SelectInteractions/HiveResult/stdout",
                Output = "/JobStatus/WordCount/Result",
                Reducer = "wc.exe",
                Mapper = "cat.exe",
            };

            streamJobDefintion.Files.Add("/example/apps/wc.exe");
            streamJobDefintion.Files.Add("/example/apps/cat.exe");
            
            

            var jobResult = jobClient.CreateStreamingJob(streamJobDefintion);

            // Wait for the job to complete
            WaitForJobCompletion(jobResult, jobClient);

            Stream stream = jobClient.GetJobOutput(jobResult.JobId);
            var reader = new StreamReader(stream);
            Console.WriteLine(reader.ReadToEnd());
            Console.WriteLine("Press ENTER to continue.");
            Console.ReadLine();
May 21, 2014 at 7:48 PM
I wish to use hdinsight without having to specify x509certificate details to create the creds object.
IJobSubmissionClient jobClient = JobSubmissionClientFactory.Connect(creds);
How can I do it?
I'm guessing I can use the access token and use JobSubmissionAccessTokenCredential instead of JobSubmissionCertificateCredential. But how do I get the access token?