Can I configure timeout for WebHDFSClient?

Jul 8, 2016 at 8:06 PM
Edited Jul 8, 2016 at 8:07 PM
I am new to .NET Hadoop, I am trying to write a program to upload my local file to HDFS using WebHDFSClient. But it seems the default timeout is 60 seconds and this cannot work when it comes to a big file which takes minutes or hours to finish uploading. It always throw exceptions indicating the task is canceled. So can anyone point me that if I can specify the timeout for WebHDFSClient? There is a similar discussion but it is not about C# library:

There is another question from me now, is WebHDFSClient depreciated? If so, which package/library should I turn to if I want to write an uploading program from local to regular HDFS?

Jul 11, 2016 at 3:29 PM
Edited Jul 11, 2016 at 3:30 PM

I had the same issue. I had to modify CreateFile(string localFile, string remotePath) in WebHDFSClient.cs
    public async Task<string> CreateFile(string localFile, string remotePath, TimeSpan? timeout = null)
      HttpClient hc = this.CreateHTTPClient(true);

      // Set timeout.
      if (timeout.HasValue)
        hc.Timeout = timeout.Value;

And call it like:
client.CreateFile(source, hdfsLocalPath, TimeSpan.FromMinutes(60));