Page 2 of 3

vRealize Log Insight Logo

VMware vRealize Log Insight: Part 2 – VMware NSX Integration via Content Packs

Reading Time: 4 minutes

In Part 1 of this series we covered the simple installation and configuration of VMware vRealize Log Insight. In Part 2 we will cover how we can further configure and customise Log Insight via Content Packs in order to leverage further logging capabilities.

As mentioned in Part 1, one of the caveats of utilising this ‘free’ version of Log Insight (or more aptly, the 25 OSI license available to all vCenter Server licensees), is the ability to use VMware-only Content Packs. This is far from a bad thing and, as a result, enables us to integrate with other VMware products including NSX, Horizon, SRM, etc. In this article we will focus on the former product.

Continue reading → VMware vRealize Log Insight: Part 2 – VMware NSX Integration via Content Packs

vRealize Log Insight Logo

VMware vRealize Log Insight: Part 1 – Install & Configuration

Reading Time: 5 minutes

If, like most of us, you forward vCenter and ESXi host Syslog data to centralised Syslog targets (and if you don’t, then I’d advise you do), then you’ll be pleased to hear that (as long as you have a valid vCenter Server license) you’ll be able to utilise the power of VMware vRealize Log Insight to interrogate this data.

This article will be the first in a two part VMware vRealize Log Insight series, the first of which will detail the simple installation and configuration process, with the second article focusing on advanced configuration and integration with VMware NSX via vRealize Log Insight Content Packs (vRealize Log Insight add-ins enabling further integration with both VMware and 3rd party products).

Continue reading → VMware vRealize Log Insight: Part 1 – Install & Configuration

VMware NSX Presentation

2017, The Final Quarter

Reading Time: 2 minutes

It’s been a busy few weeks (when does ‘busy’ stop being ‘busy’ and just become ‘BAU’?), and with the final quarter upon us, I’m working to complete the last of our projects and implementations, and there aren’t many on my list bigger than a major data centre migration.

One item from the list I’m excited about is our in-house training. Compared with other projects, technical designs, or R&D, internal training can sometimes be seen as a secondary concern, however, rather than simply handing over a solution to an operational support team, I’m a huge fan of getting every member of the team around a table to discuss, challenge, and question the solution, the designs, and the technology. Specifically, myself and colleagues within our Technical Operations team (made up of both Infrastructure and Network Architects) will regularly provide internal training and/or overview sessions to both business owners and technical teams, as well as deep dives into the technologies we either have in development or the designs and implementations we are transitioning into live service.

At this time of year, it’s nice to step back and try not to take things for granted. It’s a real privilege to be able to work with such great partners, technologies (VMware NSX, Horizon, Pure Storage), our colleagues, and being part of a team that’s so passionate about the solutions we design and deploy; ultimately enabling the business to support both our users and members. Thanks to such projects and technologies we have been able to enhance security and automation within the SDDC, provide micro segmentation of critical workloads, and deliver anything services and applications wherever the’re located.

VMware NSX Presentation

VMware NSX Guides

VMware NSX Guides

Reading Time: 2 minutes

VMware NSX Guides

If you’ve somehow managed to miss these brilliant (and free) VMware NSX guides, then worry not, as here are the links in all their glory. I cannot praise these books enough. Simply brilliant (and free!)

VMware NSX Micro-segmentation Day 1, by Wade Holmes

In Day 1, Wade Holmes details the migration away from a perimeter-orientated approach, to that of a micro-segmented architecture. VMware NSX enables organisations to utilise enhanced security functionality, whilst visualising traffic within the software-defined data centre.

VMware NSX Micro-segmentation Day 1, by Wade HolmesVMware NSX Micro-segmentation Day 1, by Wade Holmes

VMware NSX Micro-segmentation Day 2, by Geoff Wilmington

In Day 2, Geoff Wilmington complements the first guide by delving deeper into micro-segmentation, and details the process of both building and planning an architecture best suited to your applications. Also touched on are the additional tools such as VMware Log Insight, Application Rule Manager, and vRealize Network Insight.

From a personal point of view, the process of planning the migration of applications into NSX was a little daunting during my own implementation, and this guide has been simply invaluable.

VMware NSX Micro-segmentation Day 2, by Geoff WilmingtonVMware NSX Micro-segmentation Day 2, by Geoff Wilmington

Operationalizing VMware NSX, by Kevin Lees

In Operationalizing VMware NSX, Kevin Lees discusses how best to bring VMware NSX into ‘business as usual’. Both monitoring and troubleshooting are covered, and insights into team structures and cultures, team roles and responsibilities, etc., are provided. Unlike the ‘how-to’ style of the first two books, this third guide provides a fantastic insight into how NSX can be brought into service.

Operationalizing VMware NSX, by Kevin LeesOperationalizing VMware NSX, by Kevin Lees

Automating NSX for vSphere with PowerNSX, by Anthony Burke

Lastly, Automating NSX for vSphere with PowerNSX by Anthony Burke will be a firm favourite for all PowerShell fans wanting to get down and dirty with NSX.

Automating NSX for vSphere with PowerNSX, by Anthony BurkeAutomating NSX for vSphere with PowerNSX, by Anthony Burke

vSphere vCenter Server Migration Featured

VMware vSphere 6.5: Migration from Windows vCenter to vCenter Server Appliance

Reading Time: 5 minutes

Following on from my previous posts (What’s New in vSphere 6.5 and VMware VCSA 6.5: Installation & Configuration), a major area for discussion (and excitement) is the VMware Migration Assistant which, should you wish, is able to easily migrate you away from the Windows-based vCenter Server to the Linux-based vCenter Server Appliance (VCSA).

There are pros and cons to the vCenter appliance of course, as well as a healthy number of supporters in each camp, but if you fancy shaving some licensing costs (Windows Server and SQL Server), would like to enjoy a faster vSphere experience (since 6.0), or would just like to be able to take a quick backup of vCenter without having to either snapshot both Windows and SQL Servers elements, or by utilising your backup product of choice to take a full image of your environment, you might just want to take VCSA for a spin.

Continue reading → VMware vSphere 6.5: Migration from Windows vCenter to vCenter Server Appliance

VMware vCNS 5.5.4 to NSX 6.2.5 Upgrade

Reading Time: 6 minutes

VMware vCNS to NSX Upgrade

I’m a fan of upgrades that ‘just work’, but rarely do they run without a few unforeseens jumping out at you. Reading the VMware Upgrading VMware vCNS 5.5.x to NSX 6.2.x (2144620), I was surprised to see just five upgrade areas. Five? Really?? As this is a business critical system (and one with the potential of being able to turn a long day into an even longer day were things to go awry), I was a little sceptical, however, the vCNS to NSX upgrade process really is that easy.

VMware recommend the below implementation path when upgrading to NSX from vCNS, and if you’re not utilising any advanced features such as the vCNS Edges, you can cut this process down to just the first three steps.

  1. vCNS/NSX Manager
  2. Host Clusters and Virtual switches/wires
  3. vShield App (replaced by NSX Distributed Firewall)
  4. vShield Edge
  5. vShield Endpoint

Stick with me, I know you think I’m lying…

Scenario

So, a requirement exists whereby I need to replace a VMware vCNS 5.5.4 environment with VMware NSX 6.2.5 due to the former going end-of-life in Q4 2016. As I see it, I have two options; a) install NSX and migrate the vCNS workload to the new compute hardware, or b) upgrade vCNS in-place. As there aren’t any spare hosts lying around, the option will see us progressing with the in-place upgrade.

Note, configuration of NSX, as well as integration with AD Security Groups, will be covered in a future post.

Prerequisites

Okay, so there are some prerequisites (when would there not be?) Before initiating the upgrade process, you will need to ensure the below checklist has been completed:

  1. Physical network must be configured with a minimum MTU of 1600 due to the VXLAN overlay.
  2. As the NSX vSwitch is based upon vSphere Distributed Switches (vDS), if you’re currently running standard virtual switches, you’ll need to migrate to vDS first.
  3. Your backups have run successfully
  4. Snapshots of both vCenter and vCNS Manager have been taken
  5. vCNS Manager – e1000 network adapter replaced with a VMXNET3 adapter
  6. vCNS Manager – configured with at least 16GB RAM
  7. vCNS Manager – Tech Support Bundle created
  8. Download the relevant vCNS to NSX Upgrade Bundle

Upgrade vCNS 5.5.4 to NSX 6.2.5

1. First of all, we will need to download the Upgrade Bundle from VMware. Login to your MyVMware account and download.

2. Next, log into vCNS Manager, and browse to Settings & Reports > Updates.

3. Click Upload Upgrade Bundle, and upload the bundle we downloaded in Step 1.

4. Once uploaded, review the version details, and click Install.

5. When requested, click Confirm Install, and monitor the progress as per the below screenshots.

6. Monitor the reboot process via the appliance’s console and, once complete, we can proceed.

7. Following the reboot, browse to the previous vCNS Manager FQDN (https://server_name.domain.local), and you will be presented with the new NSX Manager. Note, the default admin credentials will have changed as part of the upgrade process:

  • Username – admin
  • Password – default

8. Login using the new credentials and ensure the NSX Management Service is running before proceeding. Note, this is a lab environment, hence the 4GB RAM.

9. Browse to Manage > NSX Management Service. In the Lookup Server URL section, configure by clicking Edit.

10. For this lab environment, I am configuring the lookup service to utilise vSphere SSO which, in this instance, integrates with my vCenter Server.

11. When prompted, accept the SSL certificate.

12. Ensure the Status for both Lookup Server URL and vCenter Server shows Connected.

13. After logging in to the vSphere Web Client as administrator@vsphere.local (we’ll configure NSX users and groups via Active Directory in a later post), you’ll now be able to see the new Networking & Security tab.

14. As this procedure details the upgrade process of vCNS to NSX, browsing to Networking & Security > Firewall, you will happily see that all vCNS Firewall rules have been retained.

At this point we will need to apply licensing, upgrade the ESXi host VIBs, and upgrade the vCNS Firewall to the new NSX Distributed Firewall. Until this takes place, any/all firewall amendments will not be seen by the ESXi hosts.

Licensing

1. Using the vSphere Web Client, browse to Administration > Licensing > Licenses, click Add (+).

2. When prompted, enter your license keys, and click Next. 

3. Confirm your license key information, amend the names where required, click Next.

4. Review your license information and click Finish.

5. Browse to Administration > Licenses > Assets > Solutions, and assign the new license by clicking the Assign icon.

6. Select the newly added license, and click OK.

Host Preparation

1. Browse to Networking & Security > Installation > Host Preparation.

2. Select the cluster you wish to upgrade, and click Actions > Upgrade.

3. As part of the upgrade process, note the below tasks as hosts and VMs are reconfigured.

4. Once the Host Preparation is complete, you will be requested to finalise the upgrade from vShield App Firewall to NSX Distributed Firewall. When prompted, click Upgrade.

5. After the migration has finished, browse to Networking & Security > Service Definitions, and remove the now legacy vShield-App-Service.

6. If you have any Edges in play, simply browse to NSX Edges, right-click the Edge in question, and choose Upgrade Version.

This concludes the upgrade of VMware vCloud Networking & Security 5.5.4 to VMware NSX 6.2.5. In a future post, we will cover the configuration of NSX itself, as well as the management of NSX via AD Groups.

VMware Product Walkthroughs

VMware Product Walkthroughs

Reading Time: < 1 minute

VMware Product Walkthroughs

A great new range of informational overviews is available via the VMware Product Walkthroughs website. Covering a range of product overviews (from vSphere 6.5vRealize Network Insight, and more), to the specifics of vSphere 6.5 Encrypted vMotionNSX VXLAN Configuration, Virtual SAN Fault Domains. Great on so many levels, enabling us to up-skill and dry-run new products, demonstrate solutions to management and technical teams, etc.

Visit the parent website at https://featurewalkthrough.vmware.com.

Good job VMware.

VMware vSphere: Locked Disks, Snapshot Consolidation Errors, and ‘msg.fileio.lock’

Reading Time: 3 minutes

A reoccurring issue this one, and usually due to a failed backup. In my case, this was due to a failure of a Veeam Backup & Replication disk backup job which had, effectively, failed to remove it’s delta disks following a backup run. As a result, a number of virtual machines reported disk consolidation alerts and, due to the locked vmdks, I was unable to consolidate the snapshots or Storage vMotion the VM to a different datastore. A larger and slightly more pressing concern that arose (due to the size and amount of delta disks being held) meant the underlying datastore had blown it’s capacity, taking a number of VMs offline.

So, how do we identify a) the locked file, b) the source of the lock, and c) resolve the locked vmdks and consolidate the disks?

snapshot_consolidation_disklocked_01
Disk consolidation required.
snapshot_consolidation_disklocked_02
Manual attempts at consolidating snapshots fail with either DISKLOCKED errors…
...and/or 'msg.fileio.lock' errors.
…and/or ‘msg.fileio.lock’ errors.
snapshot_consolidation_disklocked_03
Storage vMotion attempts fail, identifying the locked file.

Identify the Locked File

As a first step, we’ll need to check the hostd.log to try and identify what is happening during the above tasks. To do this, SSH to the ESXi host hosting the VM in question, and launch the hostd.log.

tail -f /var/log/hostd.log

While the log is being displayed, jump back to either the vSphere Client for Windows (C#) or vSphere Web Client and re-run a snapshot consolidation (Virtual Machine > Snapshot > Consolidate). Keep an eye on the hostd.log output while the snapshot consolidation task attempts to run, as any/all file lock errors will be displayed. In my instance, the file-lock error detailed in the Storage vMotion screenshot above is confirmed via the hostd.log output (below), and clearly shows the locked disk in question.

snapshot_consolidation_disklocked_06
File lock errors, detailed via the hostd.log, should be fairly easy to identify, and will enable you to identify the locked vmdk.

Identify the Source of the Locked File

Next, we need to identify which ESXi host is holding the lock on the vmdk by using vmkfstools.

vmkfstools -D /vmfs/volumes/volume-name/vm-name/locked-vm-disk-name.vmdk

We are specifically interested in the ‘RO Owner’, which (in the below example) shows both the lock itself and the MAC address of the offending ESXi host (in this example, ending ‘f1:64:09’).

snapshot_consolidation_disklocked_04

The MAC address shown in the above output can be used to identify the ESXi host via vSphere.

snapshot_consolidation_disklocked_05

Resolve the Locked VMDKs and Consolidate the Disks

Now the host has been identified, place in Maintenance Mode and restart the Management Agent/host daemon service (hostd) via the below command.

/etc/init.d/hostd restart

snapshot_consolidation_disklocked_06

Following a successful restart of the hostd service, re-run the snapshot consolidation. This should now complete without any further errors and, once complete, any underlying datastore capacity issues (such as in my case) should be cleared.

snapshot_consolidation_disklocked_07

For more information, an official VMware KB is available by clicking here.

VMware vCenter Server Appliance 6.5: Installation & Configuration

Reading Time: 4 minutes

Following the general release of VMware vSphere 6.5 last month (see my What’s New in VMware vSphere 6.5 post), I’ll be covering a number of technical run-throughs already in discussion throughout the virtual infrastructure community.

We’ll be starting with a fresh installation of the new and highly improved vCenter Server Appliance (VCSA), followed by a migration from the Windows-based vCenter Server 6.0; the latter task made all the easier thanks to the vSphere Migration Assistant. More on this to come. Lastly, I’ll be looking at a fresh installation of the Windows-based product, however, the experience throughout all of these installation/migration scenarios has been vastly improved.

Continue reading → VMware vCenter Server Appliance 6.5: Installation & Configuration