Chromebooks abstract out a lot of complexity from the end users and have made it easy for someone new to use it. ChromeOS AutoUpdate infrastructure does the same for IT Admins, who are responsible for managing OS updates to the ChromeOS devices.
The focus on security, speed and usability has made ChromeOS to be one of the most frequently updated Operating Systems in the world. The ChromeOS team releases one new major OS release every 4 weeks and attempts to do 2 or more minor releases within those 4 weeks. IT Admins who understand how ChromeOS AutoUpdates work can play a big role in how these devices are perceived and adopted in their organizations.
Learning about ChromeOS AutoUpdates
To “Manage” ChromeOS devices centrally, its required that you use ChromeOS Enterprise Licenses to explicitly add the Chromebooks to the “domain” for remote management. If you are unfamiliar with this, please take a moment to read more about Managing ChromeOS devices here.
You would also need to know what “Channels” are available and why certain channels are better than others for specific use cases. I would recommend you to review this post if you are unfamiliar with it.
ChromeOS AutoUpdates challenges
To make my recommendations, I’ll use three concrete challenges IT Admins see in their network. Each of these challenges have crippling impact on organizations, and its something most experienced IT Admins know how to avoid.
Issue 1: Sudden breakage of key feature
Challenge: Imagine an IT Admin responsible for 10,000 Chromebooks spread across 20 different buildings, in 5 cities and 3 continents who suddenly gets notified that the latest version of ChromeOS is not compatible with their primary app. That can quickly escalate to to the CEO/CTO resulting in hours of lost time explaining why the ChromeOS devices broke at the same time.
- ChromeOS teams have multiple channels available for customers to use. Stable is the default channel which most folks use, but there are also Beta, Dev and Canary channels available.
- We recommend that you keep 5% of your ChromeOS fleet on Beta Channel using the “Release channel” setting in your admin policy list. If you have 1000 ChromeOS devices in your fleet, consider moving approximately 50 of your most technical users to Beta channel and ask them to give you a heads-up on potential issues they notice. These users may feel proud to be considered “trusted” or “technical” and their feedback can potentially give you up to 4 weeks of early heads-up to find mitigation workarounds.
- If you find an issue, you can use policy controls to stop AutoUpdates on the fleet (by setting “Allow updates” to “Block updates”), or limit the devices to go to a certain OS version (using “Restrict Google Chrome version to at most”) which you believe is not impacted.
Issue 2: Network bandwidth consumed by AutoUpdates
Challenge: If you have a 30 Chromebooks in a class all trying to download a 3GB file at the same exact time, imagine what the Network Admin may have to deal with when rest of the organization complains that rest of the websites are slow to load. Imaging now doing this with 100s or 1000s of devices. This is a very common problem, and one which is easy to solve.
- Scattering: Use “Scatter updates” to avoid having all device update at the same exact time. You can control how much the devices should be scattered. I recommend keeping this within 7 days, but you can scatter this over 14 days if you would like. The duration you pick would be based on how much bandwidth you have.
- Rollout planning: Google now provides Advanced rollout scheduling tool to allow you to control how the rollouts should happen.
- HTTP Caching: While Google tries to use HTTPS for most of its communication, it made some exceptions to the way how it communicates with update servers to allow customers to setup HTTP Proxy servers which can provide caching capability. If you have an in-line or out-of-band proxy server in your network, you can configure the Chromebooks to use them for updates. By caching the payloads on the Proxy server, you could significantly reduce the egress bandwidth your organization may have to pay for if you are being charged by bandwidth. But more importantly, it will ensure that your network will continue to be usable even if most devices try to update within a short period of time.
Note that you have to turn on “Use HTTP for update downloads” in the AutoUpdate settings to use this feature.
- P2P caching: If you have similar devices on the network, and if you have Multicast turned on, you would be pleasantly surprised that the update binaries would be shared between the devices without taxing your internet bandwidth. However, note that Multicast is not usually turned on in Enterprise networks, and this is why most devices cannot today take benefit of P2P updates.
Issue 3: Always-on Chrome Devices are not updating
Challenge: Some customers are using Chromeboxes for signage and kiosk use cases. These devices are never powered off. Since the Autoupdates happen silently in the background and generally wait for a reboot for the OS to switch to a new version, its not uncommon that devices which never reboot get stuck on very old versions for a long time
Solution: You can configure how long a device should be up in Kiosk mode before its rebooted. You can read more about this setting here. You can also schedule a Daily/Weekly/Monthly reboot schedule as well irrespective of whether the device is Kiosk or not.
ChromeOS was designed with simplicity and security in mind but understand how they work is key to being successful at running a large Chromebook fleet. The recommendations above are some of the most basic recommendations on how to manage AutoUpdates, and I expect to update this page with more challenges and solutions over time. If you have had update challenges and want to to see something added here please leave me a comment below.