Page 5 of 6

VMware NSX Presentation

2017, The Final Quarter

It’s been a busy few weeks (when does ‘busy’ stop being ‘busy’ and just become ‘BAU’?), and with the final quarter upon us, I’m working to complete the last of our projects and implementations, and there aren’t many on my list bigger than a major data centre migration.

One item from the list I’m excited about is our in-house training. Compared with other projects, technical designs, or R&D, internal training can sometimes be seen as a secondary concern, however, rather than simply handing over a solution to an operational support team, I’m a huge fan of getting every member of the team around a table to discuss, challenge, and question the solution, the designs, and the technology. Specifically, myself and colleagues within our Technical Operations team (made up of both Infrastructure and Network Architects) will regularly provide internal training and/or overview sessions to both business owners and technical teams, as well as deep dives into the technologies we either have in development or the designs and implementations we are transitioning into live service.

At this time of year, it’s nice to step back and try not to take things for granted. It’s a real privilege to be able to work with such great partners, technologies (VMware NSX, Horizon, Pure Storage), our colleagues, and being part of a team that’s so passionate about the solutions we design and deploy; ultimately enabling the business to support both our users and members. Thanks to such projects and technologies we have been able to enhance security and automation within the SDDC, provide micro segmentation of critical workloads, and deliver anything services and applications wherever the’re located.

VMware NSX Presentation

VMUG USERCON

UK VMUG USERCON 2017

VMUG USERCON

It’s that time of year folks, with the United Kingdom VMUG UserCon 2017 taking place on Thursday 16th November at the National Conference Centre, Birmingham.

This year will see Vice President & Chief Technology Officer, EMEA at VMware, Joe Baguley presenting the opening keynote, followed by an astounding list of technology breakout sessions and speakers, as well as the much coveted VMware Hands On Labs. This truly is an event not to be missed!

Having designed and deployed VMware NSX in 2017, I’m personally looking forward to the NSX sessions, as well as those covering the recent integration of VMware Cloud and AWS, and I look forward to catching up with the old faces, as well as meeting any new ones.

For information on this year’s UK VMUG UserCon, and to follow the agenda as it’s released, simply visit www.vmug.com, or to register click here.

 

VMware Horizon Logo

Customising the VMware Horizon Web Portal – Part 1

Following a recent VMware Horizon 7 upgrade, we had a few issues whereby users were unable to download the VMware Horizon Client via the Web Portal. Specifically, clicking the Install VMware Horizon Client link simply resulted in a 404 error. So, where had the installation media gone?

On closer inspection, it appears VMware Horizon 7 handles this configuration slightly differently than in previous versions, but the issue can be easily remedied.

Customising the VMware Horizon Web Portal
Installing the VMware Horizon Client via the Web Portal
Customising the VMware Horizon Web Portal
Attempts to download the VMware Horizon Client result in a 404 error

1. First of all, we’ll need to download all clients relevant to your environment (Windows, Linux, Mac, iOS, Android, etc.) via www.vmware.com/go/viewclients. These will need to be saved to C:\Program Files\VMware\VMware View\Server\broker\webapps\downloads on each of your Connection servers.Customising the VMware Horizon Web Portal

2. The URL utilised in the Web Portal is defined in the portal-links-html-access.properties configuration file (available at C:\ProgramData\VMware\VDM\portal\). Amend each of the links (or rather the filenames) accordingly to your required platforms:

32-bit Windows installer:
link.win32=https://Server-FQDN/downloads/VMware-Horizon-Client-x86-build#.exe

64-bit Windows installer:
link.win64=https://Server-FQDN/downloads/VMware-Horizon-Client-x86_64-build#.exe

Windows Phone installer:
link.winmobile=https://Server-FQDN/downloads/VMware-Horizon-Client-build#.appx

32-bit Linux installer:
link.linux32=https://Server-FQDN/downloads/VMware-Horizon-Client-build#.x86.bundle

64-bit Linux installer:
link.linux64=https://Server-FQDN/downloads/VMware-Horizon-Client-build#.x64.bundle

Mac OS X installer:
link.mac=https://Server-FQDN/downloads/VMware-Horizon-Client-build#.dmg

iOS installer:
link.ios=https://Server-FQDN/downloads/VMware-Horizon-Client-iPhoneOS-build#.ipa

Android installer:
link.android=https://Server-FQDN/downloads/VMware-Horizon-Client-AndroidOS-build#.apk

Chrome OS installer:
link.chromeos=https://Server-FQDN/downloads/VMware-Horizon-Client-ChromeOS-build#.apk

Following the amendments, your portal-links-html-access.properties file should resemble something like the below. Please note, I mention ‘https://ConnectionServerFQDN/ below. In my instance, and as this utilises multiple this is the load-balanced address. When complete, save and close.

# Configure whether download page is accessible
enable.download=true

# Configure whether web client page is accessible
enable.webclient=true

# Configure the download page's URL address
link.download=https://www.vmware.com/go/viewclients

# Configure the help page's URL address
link.help=https://www.vmware.com/support/viewclients/doc/viewclients_pubs.html

# Links of view clients installers on different platforms
link.win32=https://ConnectionServerFQDN/downloads/VMware-Horizon-Client-4.6.1-6748947.exe
link.win64=https://ConnectionServerFQDN/downloads/VMware-Horizon-Client-4.6.1-6748947.exe
link.linux32=https://ConnectionServerFQDN/downloads/VMware-Horizon-Client-4.6.0-6617224.x86.bundle
link.linux64=https://ConnectionServerFQDN/downloads/VMware-Horizon-Client-4.6.0-6617224.x64.bundle
link.mac=https://ConnectionServerFQDN/downloads/VMware-Horizon-Client-4.6.0-6607320.dmg

3. Lastly, restart the VMware Horizon View Web Component service.Customising the VMware Horizon Web Portal

4. Revisit the Web Portal and all should now be well.Customising the VMware Horizon Web Portal

VMware NSX Guides

VMware NSX Guides

VMware NSX Guides

If you’ve somehow managed to miss these brilliant (and free) VMware NSX guides, then worry not, as here are the links in all their glory. I cannot praise these books enough. Simply brilliant (and free!)

VMware NSX Micro-segmentation Day 1, by Wade Holmes

In Day 1, Wade Holmes details the migration away from a perimeter-orientated approach, to that of a micro-segmented architecture. VMware NSX enables organisations to utilise enhanced security functionality, whilst visualising traffic within the software-defined data centre.

VMware NSX Micro-segmentation Day 1, by Wade HolmesVMware NSX Micro-segmentation Day 1, by Wade Holmes

VMware NSX Micro-segmentation Day 2, by Geoff Wilmington

In Day 2, Geoff Wilmington complements the first guide by delving deeper into micro-segmentation, and details the process of both building and planning an architecture best suited to your applications. Also touched on are the additional tools such as VMware Log Insight, Application Rule Manager, and vRealize Network Insight.

From a personal point of view, the process of planning the migration of applications into NSX was a little daunting during my own implementation, and this guide has been simply invaluable.

VMware NSX Micro-segmentation Day 2, by Geoff WilmingtonVMware NSX Micro-segmentation Day 2, by Geoff Wilmington

Operationalizing VMware NSX, by Kevin Lees

In Operationalizing VMware NSX, Kevin Lees discusses how best to bring VMware NSX into ‘business as usual’. Both monitoring and troubleshooting are covered, and insights into team structures and cultures, team roles and responsibilities, etc., are provided. Unlike the ‘how-to’ style of the first two books, this third guide provides a fantastic insight into how NSX can be brought into service.

Operationalizing VMware NSX, by Kevin LeesOperationalizing VMware NSX, by Kevin Lees

Automating NSX for vSphere with PowerNSX, by Anthony Burke

Lastly, Automating NSX for vSphere with PowerNSX by Anthony Burke will be a firm favourite for all PowerShell fans wanting to get down and dirty with NSX.

Automating NSX for vSphere with PowerNSX, by Anthony BurkeAutomating NSX for vSphere with PowerNSX, by Anthony Burke

vSphere vCenter Server Migration Featured

VMware vSphere 6.5: Migration from Windows vCenter to vCenter Server Appliance

Following on from my previous posts (What’s New in vSphere 6.5 and VMware VCSA 6.5: Installation & Configuration), a major area for discussion (and excitement) is the VMware Migration Assistant which, should you wish, is able to easily migrate you away from the Windows-based vCenter Server to the Linux-based vCenter Server Appliance (VCSA).

There are pros and cons to the vCenter appliance of course, as well as a healthy number of supporters in each camp, but if you fancy shaving some licensing costs (Windows Server and SQL Server), would like to enjoy a faster vSphere experience (since 6.0), or would just like to be able to take a quick backup of vCenter without having to either snapshot both Windows and SQL Servers elements, or by utilising your backup product of choice to take a full image of your environment, you might just want to take VCSA for a spin.

Continue reading → VMware vSphere 6.5: Migration from Windows vCenter to vCenter Server Appliance

VMware vCNS 5.5.4 to NSX 6.2.5 Upgrade

VMware vCNS to NSX Upgrade

I’m a fan of upgrades that ‘just work’, but rarely do they run without a few unforeseens jumping out at you. Reading the VMware Upgrading VMware vCNS 5.5.x to NSX 6.2.x (2144620), I was surprised to see just five upgrade areas. Five? Really?? As this is a business critical system (and one with the potential of being able to turn a long day into an even longer day were things to go awry), I was a little sceptical, however, the vCNS to NSX upgrade process really is that easy.

VMware recommend the below implementation path when upgrading to NSX from vCNS, and if you’re not utilising any advanced features such as the vCNS Edges, you can cut this process down to just the first three steps.

  1. vCNS/NSX Manager
  2. Host Clusters and Virtual switches/wires
  3. vShield App (replaced by NSX Distributed Firewall)
  4. vShield Edge
  5. vShield Endpoint

Stick with me, I know you think I’m lying…

Scenario

So, a requirement exists whereby I need to replace a VMware vCNS 5.5.4 environment with VMware NSX 6.2.5 due to the former going end-of-life in Q4 2016. As I see it, I have two options; a) install NSX and migrate the vCNS workload to the new compute hardware, or b) upgrade vCNS in-place. As there aren’t any spare hosts lying around, the option will see us progressing with the in-place upgrade.

Note, configuration of NSX, as well as integration with AD Security Groups, will be covered in a future post.

Prerequisites

Okay, so there are some prerequisites (when would there not be?) Before initiating the upgrade process, you will need to ensure the below checklist has been completed:

  1. Physical network must be configured with a minimum MTU of 1600 due to the VXLAN overlay.
  2. As the NSX vSwitch is based upon vSphere Distributed Switches (vDS), if you’re currently running standard virtual switches, you’ll need to migrate to vDS first.
  3. Your backups have run successfully
  4. Snapshots of both vCenter and vCNS Manager have been taken
  5. vCNS Manager – e1000 network adapter replaced with a VMXNET3 adapter
  6. vCNS Manager – configured with at least 16GB RAM
  7. vCNS Manager – Tech Support Bundle created
  8. Download the relevant vCNS to NSX Upgrade Bundle

Upgrade vCNS 5.5.4 to NSX 6.2.5

1. First of all, we will need to download the Upgrade Bundle from VMware. Login to your MyVMware account and download.

2. Next, log into vCNS Manager, and browse to Settings & Reports > Updates.

3. Click Upload Upgrade Bundle, and upload the bundle we downloaded in Step 1.

4. Once uploaded, review the version details, and click Install.

5. When requested, click Confirm Install, and monitor the progress as per the below screenshots.

6. Monitor the reboot process via the appliance’s console and, once complete, we can proceed.

7. Following the reboot, browse to the previous vCNS Manager FQDN (https://server_name.domain.local), and you will be presented with the new NSX Manager. Note, the default admin credentials will have changed as part of the upgrade process:

  • Username – admin
  • Password – default

8. Login using the new credentials and ensure the NSX Management Service is running before proceeding. Note, this is a lab environment, hence the 4GB RAM.

9. Browse to Manage > NSX Management Service. In the Lookup Server URL section, configure by clicking Edit.

10. For this lab environment, I am configuring the lookup service to utilise vSphere SSO which, in this instance, integrates with my vCenter Server.

11. When prompted, accept the SSL certificate.

12. Ensure the Status for both Lookup Server URL and vCenter Server shows Connected.

13. After logging in to the vSphere Web Client as administrator@vsphere.local (we’ll configure NSX users and groups via Active Directory in a later post), you’ll now be able to see the new Networking & Security tab.

14. As this procedure details the upgrade process of vCNS to NSX, browsing to Networking & Security > Firewall, you will happily see that all vCNS Firewall rules have been retained.

At this point we will need to apply licensing, upgrade the ESXi host VIBs, and upgrade the vCNS Firewall to the new NSX Distributed Firewall. Until this takes place, any/all firewall amendments will not be seen by the ESXi hosts.

Licensing

1. Using the vSphere Web Client, browse to Administration > Licensing > Licenses, click Add (+).

2. When prompted, enter your license keys, and click Next. 

3. Confirm your license key information, amend the names where required, click Next.

4. Review your license information and click Finish.

5. Browse to Administration > Licenses > Assets > Solutions, and assign the new license by clicking the Assign icon.

6. Select the newly added license, and click OK.

Host Preparation

1. Browse to Networking & Security > Installation > Host Preparation.

2. Select the cluster you wish to upgrade, and click Actions > Upgrade.

3. As part of the upgrade process, note the below tasks as hosts and VMs are reconfigured.

4. Once the Host Preparation is complete, you will be requested to finalise the upgrade from vShield App Firewall to NSX Distributed Firewall. When prompted, click Upgrade.

5. After the migration has finished, browse to Networking & Security > Service Definitions, and remove the now legacy vShield-App-Service.

6. If you have any Edges in play, simply browse to NSX Edges, right-click the Edge in question, and choose Upgrade Version.

This concludes the upgrade of VMware vCloud Networking & Security 5.5.4 to VMware NSX 6.2.5. In a future post, we will cover the configuration of NSX itself, as well as the management of NSX via AD Groups.

VMware Product Walkthroughs

VMware Product Walkthroughs

VMware Product Walkthroughs

A great new range of informational overviews is available via the VMware Product Walkthroughs website. Covering a range of product overviews (from vSphere 6.5vRealize Network Insight, and more), to the specifics of vSphere 6.5 Encrypted vMotionNSX VXLAN Configuration, Virtual SAN Fault Domains. Great on so many levels, enabling us to up-skill and dry-run new products, demonstrate solutions to management and technical teams, etc.

Visit the parent website at https://featurewalkthrough.vmware.com.

Good job VMware.

VMware vSphere: Locked Disks, Snapshot Consolidation Errors, and ‘msg.fileio.lock’

A reoccurring issue this one, and usually due to a failed backup. In my case, this was due to a failure of a Veeam Backup & Replication disk backup job which had, effectively, failed to remove it’s delta disks following a backup run. As a result, a number of virtual machines reported disk consolidation alerts and, due to the locked vmdks, I was unable to consolidate the snapshots or Storage vMotion the VM to a different datastore. A larger and slightly more pressing concern that arose (due to the size and amount of delta disks being held) meant the underlying datastore had blown it’s capacity, taking a number of VMs offline.

So, how do we identify a) the locked file, b) the source of the lock, and c) resolve the locked vmdks and consolidate the disks?

snapshot_consolidation_disklocked_01
Disk consolidation required.
snapshot_consolidation_disklocked_02
Manual attempts at consolidating snapshots fail with either DISKLOCKED errors…
...and/or 'msg.fileio.lock' errors.
…and/or ‘msg.fileio.lock’ errors.
snapshot_consolidation_disklocked_03
Storage vMotion attempts fail, identifying the locked file.

Identify the Locked File

As a first step, we’ll need to check the hostd.log to try and identify what is happening during the above tasks. To do this, SSH to the ESXi host hosting the VM in question, and launch the hostd.log.

tail -f /var/log/hostd.log

While the log is being displayed, jump back to either the vSphere Client for Windows (C#) or vSphere Web Client and re-run a snapshot consolidation (Virtual Machine > Snapshot > Consolidate). Keep an eye on the hostd.log output while the snapshot consolidation task attempts to run, as any/all file lock errors will be displayed. In my instance, the file-lock error detailed in the Storage vMotion screenshot above is confirmed via the hostd.log output (below), and clearly shows the locked disk in question.

snapshot_consolidation_disklocked_06
File lock errors, detailed via the hostd.log, should be fairly easy to identify, and will enable you to identify the locked vmdk.

Identify the Source of the Locked File

Next, we need to identify which ESXi host is holding the lock on the vmdk by using vmkfstools.

vmkfstools -D /vmfs/volumes/volume-name/vm-name/locked-vm-disk-name.vmdk

We are specifically interested in the ‘RO Owner’, which (in the below example) shows both the lock itself and the MAC address of the offending ESXi host (in this example, ending ‘f1:64:09’).

snapshot_consolidation_disklocked_04

The MAC address shown in the above output can be used to identify the ESXi host via vSphere.

snapshot_consolidation_disklocked_05

Resolve the Locked VMDKs and Consolidate the Disks

Now the host has been identified, place in Maintenance Mode and restart the Management Agent/host daemon service (hostd) via the below command.

/etc/init.d/hostd restart

snapshot_consolidation_disklocked_06

Following a successful restart of the hostd service, re-run the snapshot consolidation. This should now complete without any further errors and, once complete, any underlying datastore capacity issues (such as in my case) should be cleared.

snapshot_consolidation_disklocked_07

For more information, an official VMware KB is available by clicking here.

Configure Azure Internal Load Balancer (ILB) Listener for SQL Always On Availability Groups

Some time towards the end of Q4 2015, I was tasked with re-designing an Azure environment which had initially be commissioned to host a custom web solution by an outsourced team. The original design lacked any real resilience or high availability, with the solution itself being, for the most part, a collection of single instance servers. For example, SQL offered no HA via clustering or AAGs; no load-balancing had been implemented on the frontends, etc.

After a number of website outages due to server failures and/or high traffic volumes resulting in application crawl, the organisation approached my team with a request of redesigning the infrastructure.

My design was simple; I would continue to utilise Azure’s Infrastructure as a Service (IaaS) for the medium-term, deploy multiple, load-balanced IIS front-ends (which would eventually host the Sitecore Content Delivery nodes), and a simple, two-node SQL Always-On Availability Group (AAG) back-end with witness server.

The first challenge was Azure itself. Cast your minds back twelve months ago, and the Azure platform was experiencing a storm of changes on nearly a daily basis. Throughout the official Microsoft Azure (Implementing Microsoft Azure Infrastructure Solutions: 20533C) course (which I sat just a month previous to this project), a continuous theme of ‘this is the old portal, and this is the new’ kept recurring and, after even after returning from the course, a number of items had already  been added, relocated, or simply removed. You can see the challenge.

Documentation in regards to how Azure interacts with other Microsoft products was another hurdle. You might say, Azure was starting to fight me all the way…

One area which boasted no documentation at the time, and even had the Microsoft Azure team scratching their heads, was the inability for SQL AAG nodes to communicate with their AAG Listener ‘out of the box’. All nodes were located in the same geographical Azure data center and all were housed within the same Cloud Service. All should be working right? Wrong.

After much communication back and forth with Microsoft, they eventually advised that an Internal Load Balancer was required. Nice, I was now back on track. After putting a simple PowerShell script together, as well as a re-configuration on all SQL AAG replica nodes, communication was up, and the Listener was in the game.

I’ll be honest, I’ve configured countless Listeners in our on-prem environment in the last 12 months alone, and the additional work required to get this working in Azure was a bit of a frustration. So, below is the exact procedure I used, fully documented, and now stored in our IT documentation library.

Configure Azure Internal Load Balancer (ILB) Listener for SQL Always On Availability Groups

1. Launch Windows PowerShell

2. Login to Azure using the appropriate credentials by using the below cmdlet.

# Login to Azure
Add-AzureAccount

3. View all available Subscriptions.

# View all available subscriptions
Get-AzureSubscription

4. Select the relevant Subscription.

# Select the relevant subscription
Select-AzureSubscription Suscription_Name

5. Define variables and create Internal Load Balancer using the below script. Note, this can take around 10 minutes to complete.

# Define variables for new Azure Internal Load Balancer
$ServiceName = "Cloud_Service_Name" # Cloud Service containing AAG nodes
$AGNodes = "SQL_Node_A","SQL_Node_B" # AAG nodes, all hold replica databases
$SubnetName = "Subnet_Name" # Subnet name
$ILBStaticIP = "X.X.X.X" # IP assigned to AAG Listener
$ILBName = "ILB_Name" # ILB name you wish to assign

# Create Azure Internal Load Balancer
Add-AzureInternalLoadBalancer -InternalLoadBalancerName $ILBName -SubnetName $SubnetName -ServiceName $ServiceName -StaticVNetIPAddress $ILBStaticIP

# Configure a load balanced endpoint for each AAG node in $AGNodes using ILB
ForEach ($node in $AGNodes)
{
    Get-AzureVM -ServiceName $ServiceName -Name $node | Add-AzureEndpoint -Name "ListenerEndpoint" -LBSetName "ListenerEndpointLB" -Protocol tcp -LocalPort 1433 -PublicPort 1433 -ProbePort 59999 -ProbeProtocol tcp -ProbeIntervalInSeconds 10 -InternalLoadBalancerName $ILBName -DirectServerReturn $true | Update-AzureVM

}

6. Once the above script has completed, the below output should be displayed.azure_sql_ilb_01-1

7. Next, launch the Azure portal and browse to each of the SQL AAG nodes. You’ll now be able to see that each node is included within the new load-balanced endpoint and Internal Load Balancer will have been allocated to each AAG node.azure_sql_ilb_02

8. Finally, on each of the SQL AAG nodes, launch an elevated Windows PowerShell window, and run the below script.

# Define variables
$ClusterNetworkName = "Cluster_Network_Name" # Cluster Network Name
$IPResourceName = "IP_Address_Resource_Name" # IP Address resource name
$ILBIP = "X.X.X.X" # ILB IP Address (AAG Listener IP)

Import-Module FailoverClusters

Get-ClusterResource $IPResourceName | Set-ClusterParameter -Multiple @{"Address"="$ILBIP";"ProbePort"="59999";"SubnetMask"="255.255.255.255";"Network"="$ClusterNetworkName";"EnableDhcp"=0}

9. Once complete, ensure the AAG Listener port is still set to 1433 via SQL Server Management Studio. The port seemed to have dropped for me following the above script. If this is the case, simply reset to 1433.azure_sql_ilb_03

10. After a few minutes, launch SQL Server Management Studio and test Listener connectivity by attempting to connect to the Listener name from both SQL nodes.

For more information regarding SQL AAGs, Azure ILBs, etc., a Microsoft KB is now available at https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-classic-ps-sql-int-listener.

VMware vCenter Server Appliance 6.5: Installation & Configuration

Following the general release of VMware vSphere 6.5 last month (see my What’s New in VMware vSphere 6.5 post), I’ll be covering a number of technical run-throughs already in discussion throughout the virtual infrastructure community.

We’ll be starting with a fresh installation of the new and highly improved vCenter Server Appliance (VCSA), followed by a migration from the Windows-based vCenter Server 6.0; the latter task made all the easier thanks to the vSphere Migration Assistant. More on this to come. Lastly, I’ll be looking at a fresh installation of the Windows-based product, however, the experience throughout all of these installation/migration scenarios has been vastly improved.

Continue reading → VMware vCenter Server Appliance 6.5: Installation & Configuration