In Azure Batch, customers running long tasks on Linux (Ubuntu) compute nodes sometimes observe that a task is interrupted and automatically re-run from the beginning on a recurring basis, often once per day, around the same time, even though no one cancelled or resubmitted the task, and the application itself never failed. If your task takes longer than a few hours, this can mean it never reaches completion.
This article explains why this happens and how to prevent it, so your long-running workloads complete reliably.
What you might observe
- A task restarts from the beginning rather than failing, it is re-queued, not errored.
- The restarts recur daily within a narrow time window (for example, some time between 06:00–07:00 UTC), and happen on different nodes a few minutes apart.
- The affected node briefly becomes unavailable and then rejoins the pool healthy, with no reboot or resize on your side.
- It is not caused by your application (the same task completes successfully when it isn’t interrupted), and it is not a Spot/Low-Priority eviction (it also happens on Dedicated nodes).
Why it happens
Azure Batch Ubuntu Marketplace images ship with the operating system’s daily automatic update job (unattended-upgrades) enabled by default, just like a standard Ubuntu installation. Once a day, this job checks for and installs OS package updates.
When that update includes a core system component such as systemd, the operating system restarts its running services as part of applying the update. The Batch node agent is the process that runs your tasks and maintains their state. Because it runs as a system service, it is restarted along with the others.
From Batch’s perspective, when the node agent restarts while a task is running, the task’s lease is considered lost. Following its normal design, Batch then automatically re-queues the task and runs it again from the start on a healthy node. That is the restart you observe.
Key point: This is not a Batch platform failure or a bug in your workload. It is a side effect of a routine, OS-level maintenance job that ships enabled in the base Ubuntu image. The timing is staggered across nodes because the update timer includes a random per-node delay.
How to confirm it on a node
If you have access to an affected node (or can reproduce the issue on a test pool), you can confirm the correlation directly from the OS.
Check the daily update timer and its schedule and note the OnCalendar time and the randomized delay that staggers nodes
systemctl list-timers ‘apt-daily*’ –all
systemctl cat apt-daily-upgrade.timer
Check what the automatic update actually installed during the window (look for systemd or libc)
grep -A3 “$(date +%Y-%m-%d)” /var/log/apt/history.log
Correlate the update run with the node agent restart
journalctl -u azbatch-agent -u azbatch-controller –since “06:00” –until “07:10”
You will see the unattended-upgrade run immediately followed by the node agent services stopping and starting again – a clean, orchestrated stop, not a crash.
Tip: Batch nodes don’t keep logs across reboots by default. To retain them, set Storage=persistent in /etc/systemd/journald.conf and restart systemd-journald before reproducing.
How to prevent it
The reliable fix is to prevent the OS automatic-update job from restarting the node agent while your tasks are running. There are two supported ways.
Option 1 — Disable the auto-update timers via a pool start task (recommended)
Add a start task to your pool that runs as an administrator/root user before your tasks are scheduled. This disables the daily update timers on every node as it joins the pool
sudo systemctl disable –now apt-daily.timer apt-daily-upgrade.timer
Because a start task runs automatically on every node, including new nodes added by autoscale or resize, this consistently protects the whole pool.
Option 2 — Bake the setting into a custom image
If you manage your own image, apply the same change and capture it in an Azure Compute Gallery image, then create your pool from that image. See Use the Azure Compute Gallery to create a custom image pool.
Important considerations
Security: Disabling automatic OS updates means your nodes will not receive OS security patches automatically. If you use long-lived pools, plan to recreate your pools or refresh your custom image periodically, or apply updates during a controlled maintenance window, so your nodes stay current.
For very long tasks: As an additional safeguard against any interruption, OS updates, hardware maintenance, or (on Spot nodes) evictions, consider implementing checkpointing in your application so a re-run resumes from the last saved point instead of starting over.
Summary
If long-running tasks on your Ubuntu Batch pools restart on a daily schedule, the most common cause is the operating system’s built-in daily update job restarting the Batch node agent mid-task. Disabling the OS auto-update timers with a start task (or in a custom image) prevents the restarts and lets your long-running tasks complete while keeping your nodes patched through a controlled process.