Can we control the number of map and reduce tasks spawned in a Hadoop job?

Aug 29, 2013 at 1:15 PM
Hi,

I wanted to know if we have any control on the number of mapper and reducer jobs that can be created in a Hadoop job in HDInsight. I know the number of reducer jobs can be set using the setNumReduceTasks() method in the Job class in Apache Hadoop, but i could not find any method in the HadoopJob class, or any other config related class either in HDInsight. Would be great if anyone could give some pointers to the same.
Aug 29, 2013 at 6:48 PM
AdditionalGenericArguments property is designed to provided this functionality. It's blocked at the moment though by this bug:
http://hadoopsdk.codeplex.com/workitem/30