Can I configure timeout for WebHDFSClient?

Jul 8, 2016 at 9:06 PM
Edited Jul 8, 2016 at 9:07 PM
I am new to .NET Hadoop, I am trying to write a program to upload my local file to HDFS using WebHDFSClient. But it seems the default timeout is 60 seconds and this cannot work when it comes to a big file which takes minutes or hours to finish uploading. It always throw exceptions indicating the task is canceled. So can anyone point me that if I can specify the timeout for WebHDFSClient? There is a similar discussion but it is not about C# library: https://issues.apache.org/jira/browse/HDFS-9887

There is another question from me now, is WebHDFSClient depreciated? If so, which package/library should I turn to if I want to write an uploading program from local to regular HDFS?

Thanks!
Jul 11, 2016 at 4:29 PM
Edited Jul 11, 2016 at 4:30 PM
Hi,

I had the same issue. I had to modify CreateFile(string localFile, string remotePath) in WebHDFSClient.cs
    public async Task<string> CreateFile(string localFile, string remotePath, TimeSpan? timeout = null)
    {
      HttpClient hc = this.CreateHTTPClient(true);

      // Set timeout.
      if (timeout.HasValue)
      {
        hc.Timeout = timeout.Value;
      }

     ...
And call it like:
client.CreateFile(source, hdfsLocalPath, TimeSpan.FromMinutes(60));