Thursday 14th June saw the latest London VMUG take place at Tech UK, London, with the User Group marking it’s third outing for 2018 in just six months! Rarely does any event see such heavy hitters as Duncan Epping, Frank Denneman, and Niels Hagoort in one place, but today, we got to see all three in attendance. Add to that line-up further awesomeness in the form of vCommunity member, Chris Porter, and this was one truly great London VMUG indeed! I’ve been to a number of VMUGs around the UK, however, this was to be my first time joining the London gang.
Page 2 of 3
By design, there are certain virtual machines and/or appliances within vSphere which are protected to prevent editing (this can include NSX Controllers, Edges, Logical Routers, etc.) In a live/production environment, you’d not normally care about editing these appliances, however, in a lab environment (especially one where resource is tight), reducing memory and/or CPU allocation can help a lot. As such, this article will cover the process of removing the lock on protected VM in vSphere, in order to enable editing.
The scenario: a customer needs to reduce the resource allocation of an NSX Controller, however, due to the VM in question being protected/locked, editing the VM’s resources is not possible via UI or PowerCLI.
The process of removing this lock is quick and easy, however, we first need to identify the virtual machine’s Managed Object Reference (moRef ID). Please note, VMware do not support or recommend this procedure in any way. As such, this procedure should not be implemented in a production environment.
In Part 1 of this series we covered the simple installation and configuration of VMware vRealize Log Insight. In Part 2 we will cover how we can further configure and customise Log Insight via Content Packs in order to leverage further logging capabilities.
As mentioned in Part 1, one of the caveats of utilising this ‘free’ version of Log Insight (or more aptly, the 25 OSI license available to all vCenter Server licensees), is the ability to use VMware-only Content Packs. This is far from a bad thing and, as a result, enables us to integrate with other VMware products including NSX, Horizon, SRM, etc. In this article we will focus on the former product.
If, like most of us, you forward vCenter and ESXi host Syslog data to centralised Syslog targets (and if you don’t, then I’d advise you do), then you’ll be pleased to hear that (as long as you have a valid vCenter Server license) you’ll be able to utilise the power of VMware vRealize Log Insight to interrogate this data.
This article will be the first in a two part VMware vRealize Log Insight series, the first of which will detail the simple installation and configuration process, with the second article focusing on advanced configuration and integration with VMware NSX via vRealize Log Insight Content Packs (vRealize Log Insight add-ins enabling further integration with both VMware and 3rd party products).
If you’ve somehow managed to miss these brilliant (and free) VMware NSX guides, then worry not, as here are the links in all their glory. I cannot praise these books enough. Simply brilliant (and free!)
VMware NSX Micro-segmentation Day 1, by Wade Holmes
In Day 1, Wade Holmes details the migration away from a perimeter-orientated approach, to that of a micro-segmented architecture. VMware NSX enables organisations to utilise enhanced security functionality, whilst visualising traffic within the software-defined data centre.
VMware NSX Micro-segmentation Day 2, by Geoff Wilmington
In Day 2, Geoff Wilmington complements the first guide by delving deeper into micro-segmentation, and details the process of both building and planning an architecture best suited to your applications. Also touched on are the additional tools such as VMware Log Insight, Application Rule Manager, and vRealize Network Insight.
From a personal point of view, the process of planning the migration of applications into NSX was a little daunting during my own implementation, and this guide has been simply invaluable.
Operationalizing VMware NSX, by Kevin Lees
In Operationalizing VMware NSX, Kevin Lees discusses how best to bring VMware NSX into ‘business as usual’. Both monitoring and troubleshooting are covered, and insights into team structures and cultures, team roles and responsibilities, etc., are provided. Unlike the ‘how-to’ style of the first two books, this third guide provides a fantastic insight into how NSX can be brought into service.
Automating NSX for vSphere with PowerNSX, by Anthony Burke
Lastly, Automating NSX for vSphere with PowerNSX by Anthony Burke will be a firm favourite for all PowerShell fans wanting to get down and dirty with NSX.
Following on from my previous posts (What’s New in vSphere 6.5 and VMware VCSA 6.5: Installation & Configuration), a major area for discussion (and excitement) is the VMware Migration Assistant which, should you wish, is able to easily migrate you away from the Windows-based vCenter Server to the Linux-based vCenter Server Appliance (VCSA).
There are pros and cons to the vCenter appliance of course, as well as a healthy number of supporters in each camp, but if you fancy shaving some licensing costs (Windows Server and SQL Server), would like to enjoy a faster vSphere experience (since 6.0), or would just like to be able to take a quick backup of vCenter without having to either snapshot both Windows and SQL Servers elements, or by utilising your backup product of choice to take a full image of your environment, you might just want to take VCSA for a spin.
I’m a fan of upgrades that ‘just work’, but rarely do they run without a few unforeseens jumping out at you. Reading the VMware Upgrading VMware vCNS 5.5.x to NSX 6.2.x (2144620), I was surprised to see just five upgrade areas. Five? Really?? As this is a business critical system (and one with the potential of being able to turn a long day into an even longer day were things to go awry), I was a little sceptical, however, the vCNS to NSX upgrade process really is that easy.
VMware recommend the below implementation path when upgrading to NSX from vCNS, and if you’re not utilising any advanced features such as the vCNS Edges, you can cut this process down to just the first three steps.
- vCNS/NSX Manager
- Host Clusters and Virtual switches/wires
- vShield App (replaced by NSX Distributed Firewall)
- vShield Edge
- vShield Endpoint
Stick with me, I know you think I’m lying…
So, a requirement exists whereby I need to replace a VMware vCNS 5.5.4 environment with VMware NSX 6.2.5 due to the former going end-of-life in Q4 2016. As I see it, I have two options; a) install NSX and migrate the vCNS workload to the new compute hardware, or b) upgrade vCNS in-place. As there aren’t any spare hosts lying around, the option will see us progressing with the in-place upgrade.
Note, configuration of NSX, as well as integration with AD Security Groups, will be covered in a future post.
Okay, so there are some prerequisites (when would there not be?) Before initiating the upgrade process, you will need to ensure the below checklist has been completed:
- Physical network must be configured with a minimum MTU of 1600 due to the VXLAN overlay.
- As the NSX vSwitch is based upon vSphere Distributed Switches (vDS), if you’re currently running standard virtual switches, you’ll need to migrate to vDS first.
- Your backups have run successfully
- Snapshots of both vCenter and vCNS Manager have been taken
- vCNS Manager – e1000 network adapter replaced with a VMXNET3 adapter
- vCNS Manager – configured with at least 16GB RAM
- vCNS Manager – Tech Support Bundle created
- Download the relevant vCNS to NSX Upgrade Bundle
Upgrade vCNS 5.5.4 to NSX 6.2.5
7. Following the reboot, browse to the previous vCNS Manager FQDN (https://server_name.domain.local), and you will be presented with the new NSX Manager. Note, the default admin credentials will have changed as part of the upgrade process:
- Username – admin
- Password – default
13. After logging in to the vSphere Web Client as email@example.com (we’ll configure NSX users and groups via Active Directory in a later post), you’ll now be able to see the new Networking & Security tab.
At this point we will need to apply licensing, upgrade the ESXi host VIBs, and upgrade the vCNS Firewall to the new NSX Distributed Firewall. Until this takes place, any/all firewall amendments will not be seen by the ESXi hosts.
1. Using the vSphere Web Client, browse to Administration > Licensing > Licenses, click Add (+).
1. Browse to Networking & Security > Installation > Host Preparation.
5. After the migration has finished, browse to Networking & Security > Service Definitions, and remove the now legacy vShield-App-Service.
6. If you have any Edges in play, simply browse to NSX Edges, right-click the Edge in question, and choose Upgrade Version.
This concludes the upgrade of VMware vCloud Networking & Security 5.5.4 to VMware NSX 6.2.5. In a future post, we will cover the configuration of NSX itself, as well as the management of NSX via AD Groups.
A great new range of informational overviews is available via the VMware Product Walkthroughs website. Covering a range of product overviews (from vSphere 6.5, vRealize Network Insight, and more), to the specifics of vSphere 6.5 Encrypted vMotion, NSX VXLAN Configuration, Virtual SAN Fault Domains. Great on so many levels, enabling us to up-skill and dry-run new products, demonstrate solutions to management and technical teams, etc.
Visit the parent website at https://featurewalkthrough.vmware.com.
Good job VMware.
A reoccurring issue this one, and usually due to a failed backup. In my case, this was due to a failure of a Veeam Backup & Replication disk backup job which had, effectively, failed to remove it’s delta disks following a backup run. As a result, a number of virtual machines reported disk consolidation alerts and, due to the locked vmdks, I was unable to consolidate the snapshots or Storage vMotion the VM to a different datastore. A larger and slightly more pressing concern that arose (due to the size and amount of delta disks being held) meant the underlying datastore had blown it’s capacity, taking a number of VMs offline.
So, how do we identify a) the locked file, b) the source of the lock, and c) resolve the locked vmdks and consolidate the disks?
Identify the Locked File
As a first step, we’ll need to check the hostd.log to try and identify what is happening during the above tasks. To do this, SSH to the ESXi host hosting the VM in question, and launch the hostd.log.
tail -f /var/log/hostd.log
While the log is being displayed, jump back to either the vSphere Client for Windows (C#) or vSphere Web Client and re-run a snapshot consolidation (Virtual Machine > Snapshot > Consolidate). Keep an eye on the hostd.log output while the snapshot consolidation task attempts to run, as any/all file lock errors will be displayed. In my instance, the file-lock error detailed in the Storage vMotion screenshot above is confirmed via the hostd.log output (below), and clearly shows the locked disk in question.
Identify the Source of the Locked File
Next, we need to identify which ESXi host is holding the lock on the vmdk by using vmkfstools.
vmkfstools -D /vmfs/volumes/volume-name/vm-name/locked-vm-disk-name.vmdk
We are specifically interested in the ‘RO Owner’, which (in the below example) shows both the lock itself and the MAC address of the offending ESXi host (in this example, ending ‘f1:64:09’).
The MAC address shown in the above output can be used to identify the ESXi host via vSphere.
Resolve the Locked VMDKs and Consolidate the Disks
Now the host has been identified, place in Maintenance Mode and restart the Management Agent/host daemon service (hostd) via the below command.
Following a successful restart of the hostd service, re-run the snapshot consolidation. This should now complete without any further errors and, once complete, any underlying datastore capacity issues (such as in my case) should be cleared.
For more information, an official VMware KB is available by clicking here.
Following the general release of VMware vSphere 6.5 last month (see my What’s New in VMware vSphere 6.5 post), I’ll be covering a number of technical run-throughs already in discussion throughout the virtual infrastructure community.
We’ll be starting with a fresh installation of the new and highly improved vCenter Server Appliance (VCSA), followed by a migration from the Windows-based vCenter Server 6.0; the latter task made all the easier thanks to the vSphere Migration Assistant. More on this to come. Lastly, I’ll be looking at a fresh installation of the Windows-based product, however, the experience throughout all of these installation/migration scenarios has been vastly improved.