Azure webjob stopped with Unhandled Exception: 503 Server unavailable

I have a WebJob with a QueueTrigger. I terminate the process function with try & catch, but it seems that the azure queue is throwing an unhandled exception that I cannot catch in my code. how can i automatically restart webjob? how can i add retry policy to webjob sdk?

this is the exception I see in the webjob log (which shows that the error occurs in the sjk webjob timer):

04/18/2015 11:10:52 > 2d2f34: ERR ] 
[04/18/2015 11:10:52 > 2d2f34: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (503) Server Unavailable. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.<ExistsImpl>b__1d(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    --- End of inner exception stack trace ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar)
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ExecuteAsync>d__4.MoveNext()
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
[04/18/2015 11:10:52 > 2d2f34: ERR ] --- End of stack trace from previous location where exception was thrown ---
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[04/18/2015 11:10:52 > 2d2f34: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[04/18/2015 11:10:52 > 2d2f34: SYS INFO] Status changed to Failed
[04/18/2015 11:10:52 > 2d2f34: SYS ERR ] Job failed due to exit code -532462766

      

this is my code:

public static async Task  ManualTrigger([QueueTrigger(ConstantVars.TargetjobQueue)] TargetingJob job, TextWriter log)
{
    try
    {
        AddTextWriterToAppender(log);

        _logger.DebugFormat("Starting job");
        _logger.DebugFormat("Job details: {0}", job);

        await ProcessJob(job).ConfigureAwait(false);

        _logger.Debug("Finished job");

        RemoveTextWriterFromAppender();
    }
    catch (Exception e)
    {
        _logger.Error(string.Format("Unhandled exception was caught in ManualTrigger.\n jobParams={0}",job.ToString()),e);              
    }
}

      

+3


source to share


1 answer


As you have defined, the storage exception happens outside of your job function, so you cannot catch it. However, the webjobs SDK will handle failures and automatically retry failed calls based on its own internal randomized exponential backoff strategy. You can control the number of retries set by JobHostConfiguration.Queues.MaxDequeueCount at startup.



+1


source







All Articles