One problem less - Fully Automatic Windows Server Patching with Ansible

Are you still patching manually?
Do you still manually patch your virtual Windows servers? With complete automation, you save time and effort here and also increase security. Fully automated server patching with ansible…
Sip a mojito on the beach, while the systems patch. You get a notification when all is done.
opensight.ch – roman hüsler

Fully automated server patching with ansible

• “Are there new security updates for my server VM’s?”
• “Did the backup run successfully last night?”
• “Does the VM have existing snapshots?”
• “Does the VM have enough free disk space?”
• “Can I start patching the VMs at 21:00 in the evening?”

You can either find out the answers to these questions manually on your infrastructure or automate this process from A-Z using Ansible. The patching process often involves necessary checks and precautions on all peripheral systems (backup, hypervisor snapshots, monitoring). In this blog post we would like to show how the open source tool “Ansible” can help with automation.

Before we get started – you can also take a look at all the playbooks and modules mentioned in our corresponding open source repository on Github.
opensight.ch – roman hüsler

About Ansible

Ansible is an automation tool (open source) that combines functions such as software distribution, ad hoc command execution and configuration management.

The Ansible system uses files in YAML format (playbooks) to describe the desired status of systems. The Ansible “master server” then connects to the “slave servers” (or WinRM in the case of Windows) via SSH in order to carry out the necessary configurations. There are numerous out-of-the-box “Ansible modules” with which you can carry out all sorts of system configurations. Of course, you can also develop your own modules – we use exactly those for checks on peripheral systems such as Hyper-V hosts.

Gitops / Infrastructure as Code
If you combine Ansible (or its YAML files) with the functionalities of GIT, you have the option of versioning the configuration of your infrastructure. Keyword “Infrastructure as Code (IaC)”, “Gitops”
opensight.ch – roman hüsler

Web GUI - AWX Installation

Today, Ansible is one of the most established open source orchestration tools. It is based on the Python programming language and is operated via the command line. However, there are also graphic interfaces that make operation easier, offer a better overview and also provide scheduling functions. Red Hat’s “Ansible Tower” is the most well-known graphical user interface – however, we’re using the open source version of it here: “Ansible AWX”.

In our case, we used an existing Kubernetes cluster to set up AWX – but of course there are other options. The “AWX Operator” is recommended for deploying AWX version 20 on Kubernetes. This creates a Custom Resource Definition (CRD) “AWX” on your Kubernetes cluster, which manages the AWX deployment. The database runs on PostgreSQL. We cannot go into detail about the installation at this point, but the documentation can be found here.

Configuration of Windows Slaves

This blog post focuses in particular on patching Windows server systems with Ansible. However, you can also patch Linux systems in the same way (in a slightly different way).
opensight.ch – roman hüsler

For Ansible to access a Windows Slave Server VM to be patched, the Windows Server must have at least PowerShell version 3.0 and .NET Framework 4.0 installed. After that, the WinRM service needs to be configured for Ansible. The WinRM service on the target systems should be configured correctly so that communication can take place securely and encrypted. We recommend our blog post “WinRM – Remote administration of Windows servers“.

Creating Ansible Playbooks

Now let’s prepare some playbooks that help us achieving our goal “Fully automated server patching with ansible”. There are some playbooks in the GIT repository that will be useful when patching Windows servers:

• Playbook – Check Windows Updates
  Performs a search for available Windows updates on the clients
• Playbook – Check Disk Space
  Checks whether the c: drive has at least 20GB of free space
• Playbook – Check VM Snapshots
  Checks if the VM has active snapshots on the virtualization server

A sequence of “tasks” is defined in each playbook, which are to be carried out on the defined systems in sequence. If one of these tasks fails, execution is aborted.

Playbook - Check Windows Updates

Now it’s time to test the system. We use a playbook (check-updates.yml) for this, which lists the available updates on all Windows servers.

---

- name: windows server patching
  hosts: all
  tasks:

    - name: Check for missing updates
      win_updates:
        state: searched
      register: update_results

    - name: report update results
      debug:
        msg: |
          {% for k in update_results.updates %}
          {{ k.title }}
          {% endfor %}

We integrate the playbook into Ansible AWX as a “Job Template”. After its execution we see (image – lines in green) that a Windows system has pending updates.

Playbook - Check Disk Space

Similarly, we define another job template that checks the available storage space on the c: drive. If less than 20GB is available, this test is said to fail.

---

- name: check free disk space
  hosts: all
  tasks:
    - name: 'Check free space in C:'
      win_shell: 'write-host ([math]::Round((Get-PSDrive C | Select-Object Free).free / 1024 / 1024 / 1024,2))'
      register: freespace

    - name: report free disk space
      debug:
        msg: |
          "{{ freespace.stdout }} GB"

    - name: fail if there is not enough space
      fail:
        msg: "VM {{inventory_hostname}} has not enough space"
      when: "{{freespace.stdout | int < 20}}"

Playbook - Check VM Snapshots

In our case we want the patching to start only when there are no active snapshots on the virtual machines (just as an example to see what’s possible).

In the GIT repository we have a custom module for Ansible, which can check whether a Hyper-V VM currently has a snapshot. So let’s test:

• We create a snapshot of the VM bnetprtg01
• In the Ansible Inventory, we add the variable “hyperv_vmname=bnetprtg01” to the host bnetprtg01, which is necessary for the module / playbook. (see documentation)
• We add the necessary variable “module_hyperv_host: 192.168.1.2” to the Ansible inventory, which points to the Hyper-V host. (see documentation)
• Then we run the playbook from the git repository (hyperv-check-nospanshot.yml)

As hoped, the playbook for the VM bnetprtg01 fails because it has an active snapshot. Of course, the same playbook could be modified slightly to create a snapshot before patching and delete it after patching.

Take a look at the GIT repository, it also contains playbooks for checking the monitoring status of a system in PRTG, pausing or resuming the monitoring.
opensight.ch – roman hüsler

We will not describe other playbooks at this point. But of course it is also possible to check or change the snapshots analogously to a VMware-based infrastructure. Or to check the backup on a Veeam server, etc.

Grande Finale - Fully Automated Patching Process

With a “workflow job template” we now link all previously defined playbooks (“job templates”) in Ansible AWX to a complete automated process.

• Check disk space
• Check snapshots
• Looking for updates
• Install updates and restart

We define a job schedule so that a fully automatic Windows Server patching is carried out once a month, at 02:00 in the morning. In a real patching scenario, it would be best to synchronize the patching time with “Microsoft Patch Tuesday”.

Reporting and Notifcations

With regard to reporting, for example, you have the option of setting up an e-mail notification.
The formatting of the e-mail can be adjusted slightly. However, there are of course other and more advanced options. As an example, you could stream all the logs into an Elasticsearch cluster using Logstash, which we may look at in a future blog post.

Wrapping up

Ansible has rightly risen to the premier league of the most popular orchestration and automation tools. Operation is intuitive and of course you can also use it to automate processes other than “Fully automated server patching with ansible”.

Did you find the article helpful or are there open questions? Take our quick survey - tell us how to improve.