TechOnTip Weblog

Run book for Technocrats

Archive for July, 2011

ESXi Network Card Driver Installation Steps

Posted by Brajesh Panda on July 30, 2011

Broadcom & Intel Network Driver Installation Guide

  • 1st we have to determining firmware & driver version of network cards in ESXi
  • Use “esxcfg-nics –l” to determine Driver names of each network card
  • Then use “ethtool –I <NICName> to determine firmware & driver version
  • Below screenshot each Broadcom Adapter is using out of box driver bnx2 2.0.7d-3vmw

  • Below screenshot each Intel Adapter is using out of box driver ixgbe 2.0.38.2.5.1-1vmw-RSCpsfix-NAPI


For Driver update – download certified driver CDs from VMware Site!

For my Intel 10Gb I have downloaded from below URL;

Intel 10GB- 82599 controller: http://downloads.vmware.com/d/details/esx4_intel_ixgbe_dt/ZCV0YmRqKipiZHdldA==#drivers_tools

In ESXi drivers for network card or other attached devices can be updated in different ways;

  1. At the time of ESXi Installation
  2. After ESXi Installation or In a running ESXi installation
  • Using vSphere CLI:
    • We have to put the host in maintenance mode & use “vihostupdate <conn_options> –install –bundle <offline-bundle>.zip”
    • Reboot the server to driver modules to get loaded
  • Using ESXUpdate
    • Mount Driver CDROM or copy driver files to any Datastore & run esxupdate command as below screenshot
    • Reboot the server to driver modules to get loaded


  • Verify Drivers after Server reboot
  • Now here you can see updated drivers for Broadcom & Intel Adapaters
  • Below screenshot each Broadcom Adapter is using out of box driver bnx2 2.0.22f.v41.2
  • Below screenshot each Intel Adapter is using out of box driver ixgbe 6.2.14 bc 5.2.4 NCSI 2.0.11


  • Verify installed Software/Bulletins


  • In case you like to uninstall any driver use below command from VMware CLI
    • Vihostupdate.pl –server <serverIP> –remove –bulletin < bulletinname>

Posted in VMware | Tagged: , | Leave a Comment »

Dell Open Manage Server Administrator for ESXi 4.1

Posted by Brajesh Panda on July 27, 2011

What are the Methods?

There are two methods thru which we can manage Dell Server Hardware where ESXi is installed.

  • Using Dell Open Manage Server Administrator from Individual ESXi Servers
  • Using Dell Open Manage Server Administrator Plug-in for vCenter. (Advance Functionality)

How stuff works?

  • Both of these methods need Dell Open Manage Server Administrator Agent to be installed in the ESXi server. Generally this Agent doesn’t have any web interface, so we need software to pull out information from this Agent. This Agent provides different hardware management information from ESXi CIM (common interface model). Having said that after agent installation we have to make sure UserVars: CIMoemProviderEnabled
    advance setting is set to 1 in each agent installed server. For advance monitoring we may like to enable SNMP Traps which will be used by management applications.
  • Hence in case of 1st Method we need Dell Open Manage Server Administrator Manage Node software to be installed in any windows server. We will access web interface of Manage Node software & then connect to each ESXi host (where above agent is running) to see its hardware information. Just to note like normal Windows or Linux open server administrator application, we will not able to update any firmware from this interface.
  • In case of 2nd Method we have to deploy Dell vCenter Plug-in Virtual Appliance (OVF) to vCenter infrastructure. Then we have to register the required vCenter Servers in the appliance with other configuration. Using this tool we will able to update the server hardware firmware, because apart from agent, it can talk to the system using iDRAC interface using it’s credentials. May be due to this Dell is selling this product as a licensed product, you have to buy it. But I think they should give this product as free & if anybody needs support for this product they should only sell support package.

In this post I will demonstrate about Dell Open Manage Server Administrator Agent & Manage Node Installation & Configuration. I will talk about the 2nd vCenter Plug-in method in another post!

My Environment:

  • Server:        Dell Server PowerEdge R710
  • Hypervisor:    VMware ESXi, 4.1.0, 348481

Required Software:

  • Agent:        OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
  • Manage Node:     OM-SrvAdmin-Dell-Web-WIN-6.5.0-2247_A01.10.exe
  • Installation Tool: VMware-vSphere-CLI-4.1.0-254719

Agent Installation & Configuration Process!

  • Before we start agent installation make sure the ESXi node is in maintenance mode. To put the node into maintain ace mode either we have to shut down all virtual machines or move them to another node.
  • Open vSphere CLI & change the directory to Bin folder which contains a lot management Perl scripts
  • Use below command to install the “OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip” agent file in the server

    “vihostupdate.pl –server <ServerName/IP> –username root –password <password> -i -b “Path to the Agent file”

  • Here is the screenshot of command;

  • While it is updating you can see the task in recent task window of vSphere Client

  • After Agent Installation Reboot the Host to get the changes to be effective.
  • If you like to verify if it this agent installed or not you can use vihostupdate.pl –server <ServerName/IP> –username root –password <password> –query
  • If you like to remove this agent you can use vihostupdate.pl –server <ServerName/IP> –remove –buletin <Name>
  • Now we have to make sure UserVars: CIMoemProviderEnabled
    advance setting is set to 1
  • Select the ESX Server – Configuration Tab – Advance Setting – On right hand side make sure UserVars: CIMoemProviderEnabled is set to 1. If it is set to 0, change the value & reboot the server.
  • Reboot the ESXi server to get this setting effective

Dell Open Manager Manage Node Installation & Configuration Process!

As I mentioned earlier you need Dell Open Manager Manage Node to connect to above installed agents to do server hardware administration & monitoring. This installation is pretty straight forward like next-next &done. Use OM-SrvAdmin-Dell-Web-WIN-6.5.0-2247_A01.10.exe file to install the same. You may like to install in any windows server; in my lab environment I prefer vCenter server.

After Installation you can start the application by double clicking Dell Server Administrator inside the same server or accessing https://<servername&gt; :1311/servlet/OMSALogin?msgStatus=null remotely

Here is the screenshot how it looks;

  • In Hostname/Ip Address field type ESXi Server Name or IP
  • And in Username & Password field corresponding username & password. In lab I have tried with ESXi “root” username & corresponding password
  • If you don’t have trusted certificate installed you may like to select “Ignore certificate warnings”
  • After we successfully authenticate should able to see different hardware & firmware details like normal dell open manage server administrator. Here is the screenshot!

  • Here is a screenshot you can see my PERC H200 Integrated RAID card firmware has been outdated.

  • Here is another screenshot about my network cards

Posted in Dell OEM, VMware | Tagged: | Leave a Comment »

VMware Data Recovery – Installation & Configuration

Posted by Brajesh Panda on July 25, 2011

To use VMware Data Recovery technology we have to install Data Recovery Plug-in & Appliance. Data Recovery Plug-in for vSphere Client will help us to connect to the appliance & do necessary configuration to take backup job. If you are interested to know how it works etc…I have another post regarding VMware data recovery; you may like to check that out!

Installing VMware Data Recovery Client Plug-In

  • VMware Data Recovery Plug-In verification; vSphere Client -> Plug-In -> Manage Plug-In
  • Download VMware Data Recovery CD from & install client plug-in


  • Close your VI client & re-open for plug-in registration. Now validate the Plug-in section again.


  • Client Plug-in will connect to Data Recovery appliance on port 22024, so in case any firewall; make sure you open the ports

Installing VMware Data Recovery Appliance

  • To Deploy Data Recovery Appliance; In vi client on file menu click deploy OVF template, browse data recovery CD & select VMwareDataRecovery_OVF10.ovf file & click next



  • On OVF template details page you can read product information;
  • In Name & Location Window define Virtual Machine name
  • In Data Store windows select right data store where you like to keep this virtual machine.
  • Select correct Disk format; Seems like Thick Provisioning will take 5GB of space


  • In Properties window select Time Zone & Click next to start import process
  • After Import gets completed you will see this new virtual appliance like other virtual machines


Configuring VMware Data Recovery Appliance

  • Switch on the virtual machine; As it is a Linux appliance, Linux Boot Loader Process will started & at the end console will look like below. I have a test DHCP in my lab, so it picked up IP address from the same.


  • Select Login & press enter to login; you will be challenged with authentication prompt. If this the first time you are logging to the appliance use factory default username & password (root & vmw@re)
  • You can change default password using “passwd” UNIX command.
  • Access the Appliance URL (displayed in the console screen)


  • Type User Name as root & it’s password; Here is the initial screenshot


  • Click on Network Tab then Address; Type necessary information & click on save settings.


  • Tried to install VMware Tools but seems like it is already installed 😉


Configuring VMware Data Recovery Appliance to Start Data Recovery

  • Shutdown the virtual machine.
  • From Virtual Machine settings add a Virtual Hard Disk to the Virtual Machine. We will use this device as a backup destination.
  • Open VI Client, Home & then Solutions and Appliances. You will find VMware Data Recovery appliance;


  • Click on the VMware Data Recovery Appliance, it will open up below screen.


  • Type IP Address or Name of the Appliance & Click connect!
  • It will challenge you for correct vCenter authentication. Appliance uses this user account password to connect to vCenter to perform backup & restore jobs etc. So this ID must have administrator right to the vCenter Infrastructure.


  • It will start Getting Started Wizard to configure different backup options
  • In credentials window re-type correct authentication details & click Next


  • It will show you to choose the backup destination for the appliance. If you have already added a VMDK (virtual drive) to the appliance it will show up over here. You can also add a network share (CIFS) from this window.


  • Select the volume where you like to store data & next.
  • Now observe new changes related to backup & restore for each virtual machine.


  • If you like to change any configuration you can do it from configuration tab


  • To use the configured backup destination to store data; You have to Format the Device & Mount the Device to the appliance. For the same click the device name (like /dev/sdb in above snapshot) & click Format. It will warn you for data lost. If you are configuring a brand new device click okay. But if you are connecting to a old de-duplication store click Cancel & Just mount the same from mount option.


 

  • After you click okay, it will format the device – assign SCSI device number & mount the same.


  • If you select the backup device it will show device details at bottom of the screen. As it is a new store duplication size etc are empty.


Creating Backup Job

  • Under Home – Solutions and Applications – Select VMware Data Recovery
  • In Getting Started Tab click on Create a Backup Job

     


 

  • Provide a job name & click next


  • In Backup Job: Virtual Machine window select the virtual machines you would like to backup using this configuration. Either you can select a individual machine or multiple machines or any top level of objects from the vCenter. For demonstration I have selected one machine (Windows 2008 R2 + Exchange 2010) server.






While a job is running you can see Original virtual machine hard disk has been hot added to the virtual appliance & it is copying data to destination.


In the same time when backup is running, you can browse the virtual machine folder & see the snapshot file.


After completion of backup you will able to see backup success or failure logs in the reports section.


Here are some impressive results from this tool in my lab;

All of my 5 test virtual machines are based on Windows 2008 R2 operating system & having some specific roles. You can see below screen for the total disk space usage in the 4th colum from the left.


Here is the screenshot of my backup destination data store;


Let’s check how many restore points I have for these virtual machines; Remember this is my lab – there are not much data is getting changed!


Posted in Deduplication, vDR, VMware | Tagged: | 26 Comments »

Understanding VMware Data Recovery

Posted by Brajesh Panda on July 25, 2011

Although I have used many backup products for VMware ESX infrastructure, I never used VMware Data Recovery appliance. So last week I was researching on the same. Here are my notes…may help you to understand the product architecture, issues, limitations etc. Well while I am researching on this product new version (Data Recovery 2.0) has been released by VMware. Here is the white paper for the Data Recovery 2.0. There are some interesting features has been included. When I will get a chance to work on DR 2.0, will post my understanding in this series! Not to duplicate the work – I am going to give you some URLs or copy paste some contains 😉

VMware’s Backup and Recovery product (From VMware Uptime Blog)

One of the many capabilities introduced in VMware vSphere 4 is VMware Data Recovery (VDR), a virtual machine backup and recovery product.  Market research and customer feedback showed that many people wanted an integrated option for protecting virtual machines in a VMware environment.  Further analysis showed that this was more eminent for VMware customers that had (or plan to have) fewer than 100 virtual machines in their environment and where IT responsibilities (including VMware) were shared among 2-3 IT administrators (as opposed to having a dedicated VMware administrator on-staff).

VMware has been helping customers address their backup challenges in two ways:  making significant investments in the vStorage APIs for Data Protection that third-party backup tools use to integrate their backup/recovery products with vSphere, and in providing an integrated option optimized for vSphere customers with smaller environments.  VDR is built using the vStorage APIs for Data Protection and incorporates a user interface, policy engine and data duplication – see the diagram below on how it all fits together.  I’ll cover these blocks in a series of blogs but I wanted to start out by discussing Data Deduplication (de-dupe).

Given that we had a made a decision to only use disks as the destination for the VDR backups, we had to look for a solution that offered disk storage savings – and this is where de-dupe comes in.  In a nutshell, de-dupe avoids the same data to be stored twice – and de-dupe is HOT – just check out the mergers and acquisitions news!

What VMware decided to implement for VDR de-dupe is (take a deep breath) – block based in-line destination deduplication.  Deconstructing it means the following:

    1. We discover data commonality at the disk block level as oppose to the file level.

    2. It is done as we stream the backup data to the destination disk as opposed to a post-backup process.

    3. The actual de-dupe process occurs as we store the data on the destination disk as opposed to when we are scanning the source VM’s virtual disks prior to the backup.

When it comes to deduplication, there are different techniques and hash algorithms used to accomplish the result.  I am not going to get into a theoretical discussion of the pros and cons of the various types of de-dupe technologies available and which approach provides the best disk savings.  I personally think that it totally depends on the customers’ IT environment constraints and their overall business goals plus a lot of the storage savings is going to be data driven anyway (the more data commonality there is, the better the de-dupe rate).  We chose this de-dupe architecture because it fit best with what we were trying to achieve with VDR and what the vSphere platform provided to us.  What were these reasons?  Stay tuned to this space……

VMware Data Recovery Taking Advantage of vSphere 4 (From VMware Uptime Blog)

I wanted to explain in more detail why we chose the type of de-dupe that we did.  As I had mentioned in my previous post, we chose to implement block based in-line destination deduplication for VMware Data Recovery (VDR).  There are a few reasons for this, two of which are due to enhancements in the VMware vSphere 4 platforms itself.

1) Change block tracking:  Any new VM provisioned on vSphere will use virtual hardware version 7 (you can also upgrade your existing VM version 4 to version 7).  With VM version 7, the VMkernel tracks the changed blocks of the VM’s virtual disks.  (By the way, this the same change block tracking functionality that enhances Storage VMotion in vSphere 4). So, instead of having to scan the VM’s virtual disks to determine which blocks have changed every time a backup occurs, VDR just makes an API call to the VMkernel and gets this information “for free”.

Thus, VDR is able to dramatically cut down the amount of time and CPU cycles to calculate the changed blocks on a virtual disk.  In addition, change block tracking also helps on the restore side of the equation.  For example, if you wanted to restore yesterday’s VM image, VDR will make the reverse change block API call and will just transfer the changed blocks from yesterday’s backup to revert the VM to its previous state.  So, given that there is a lot of intelligence in the platform about virtual disk blocks, block based de-dupe seemed like a natural direction for VDR to take.

2) Hot add disk:  VDR can “hot add” virtual disk snapshots directly to the VDR virtual appliance.  This is accomplished by leveraging capabilities of the vSphere storage stack.  This means that VDR can bypass the LAN and stream the data from the snapshots directly to de-dupe destination disk.  In addition to reducing load on the LAN and effectively eliminating the need to block out other LAN traffic during the backup window, the streaming of data to the destination de-dupe disk on the Data Recovery appliance will be considerably faster.

Note that there are three caveats to enabling hot add disk with VDR:

a.       The source virtual disks need to be on shared storage

b.       The ESX host where the VDR appliance is running needs to have visibility to this shared storage

c.       You will need a vSphere edition that includes Hot Add as a feature

The knock against destination (or target) based de-dupe is the fact that it consumes precious network bandwidth with the unnecessary transfer of data that will be discarded as part of the de-dupe process.  However, given that VDR only transfers changed blocks and can transfer these blocks off-LAN, the concern did not apply and thus we felt comfortable with a destination based de-dupe architecture.

So does this mean that unless you have both change block tracking and hot add disk features enabled in vSphere 4, VDR and its de-dupe capability is useless to you?  Absolutely not!  All data that is protected by VDR will be de-duped, so you will enjoy the storage savings independent of what VM version is being backed up or what vSphere edition you are have installed.  What change block tracking and hot add disk adds is additional efficiency and performance gains that will allow even more data to be protected in an ever shrinking backup window.

If you like to know more about Changed Block Tracking; here are some good reads…

http://www.yellow-bricks.com/2009/12/21/changed-block-tracking/ (read comment section for Eric’s finding about block size & how it works)

http://itknowledgeexchange.techtarget.com/virtualization-pro/what-is-changed-block-tracking-in-vsphere/

http://www.punchingclouds.com/?tag=vmkernel

There are bunch of features like Data Integrity check, Retention policy etc are not exposed to outside for reconfiguration. However if you want to change the behavior of the appliance you may like to do some tuning inside the appliance. You can create a “datarecovery.ini” use the same. Here is a VMware KB article talks about the same; “VMware Data Recovery datarecovery.ini options”

Data recovery appliance runs automated Integrity checks for the backup destination store. If due to any reason restore point of a virtual machine get corrupted, it locks the store & it will not execute backup jobs. Till 1.2.1 version there was no automated notification around the same. Every time you have to check manually report or restore section for this kind of error.

If you are facing any issues related to this; below URLs may help you to fix the same.

Remove Lock files on backup destination stores:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1018060

Run Manual Integrity check:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029296

How to perform manual integrity check when you have a damaged restore point?

http://www.vladan.fr/how-to-perform-a-manual-integrity-check-in-vdr-when-you-have-damaged-restore-point/http://communities.vmware.com/thread/225510

Some final Notes:

Requirement to Implement/Use VMware Data Recovery

  • VDR is available only for Essentials Plus, Advanced, Enterprise and Enterprise Plus
  • vDR Appliance should have access to all shared datastores
  • Virtual Machines must be version 7

Feature Limitations of VMware Data Recovery

  • Limited to 100 virtual machine
  • Per virtual machine once a day backup
  • VDR will not backup virtual machines with Fault Tolerance enabled
  • VDR does not support VC Linked mode
  • Virtual Machines with hardware version lower than 7 will take longer to be backed up.
  • The deduplication feature cannot be disabled so all backups done by VDR are deduplicated
  • Not compatible with VI3 hosts. VDR requires the presence of a VMware vCenter Server 4
  • VDR will not backup your VM if it is stored in a RDM not in virtual compatibility mode
  • Doesn’t support IPv6
  • Only Two Deduplication stores per appliance
  • 1TB backup destination store limitation
  • Maximum 8 concurrent jobs (backup or restore)
  • No Backup Server level GUI or Command line tool to do administration
  • No Powershell support
  • No Email Reporting
  • No Replication to another backup store
  • No support for tape/external media

Apart from these feature limitations, I like this product very much. It perfectly does what it promises. Check out my installation & configuration guide with lab results over here.

Posted in VMware | Tagged: | 6 Comments »

Symantec BackupExec 2010 R3

Posted by Brajesh Panda on July 22, 2011

Symantec has been released Backup Exec 2010 R3 version.

Release Notes: http://www.symantec.com/connect/blogs/backup-exec-2010-r3-released-today

Webcast: https://www4.symantec.com/Vrt/offer?a_id=114713

Trial: https://www4.symantec.com/Vrt/offer?a_id=93355

Posted in BackupExec 2010, BackupExec2010R3 | 1 Comment »

Pause and Wait Implementation in command line batch Files

Posted by Brajesh Panda on July 15, 2011

If you like to suspends processing of a batch program and displays the message: Press any key to continue….

PAUSE
If you like to start execution of next command after sometime; like 2mins (120secs)

@choice /T 120 /C y /CS /D y

Posted in Uncategorized | Tagged: | Leave a Comment »

Registry Key to Auto Admin Login

Posted by Brajesh Panda on July 15, 2011

REG ADD “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon” /v AutoAdminLogon /t REG_SZ /d “1” /f
REG ADD “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon” /v DefaultUserName /t REG_SZ /d “Administrator” /f
REG ADD “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon” /v DefaultPassword /t REG_SZ /d “TypePasswordHere” /f

Posted in Uncategorized | Tagged: | Leave a Comment »

Registry Key to Change Windows Update Options

Posted by Brajesh Panda on July 15, 2011

Configure Windows Update to Download Updates & let me choose what to install
REG ADD “HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update” /v AUOptions /t REG_DWORD /d 3 /f

2 – Notify for download and notify for install
3 – Auto download and notify for install
4 – Auto download and schedule the install

Posted in Uncategorized | Tagged: | Leave a Comment »

Uncheck Network Protocols from Network Properties Using Commandline

Posted by Brajesh Panda on July 15, 2011

Unchecking QoS Check Mark in Local Area Connection Applet
nvspbind.exe /d “Local Area Connection” ms_pacer

Unchecking TCPIPv6 Check Mark in Local Area Connection Applet

nvspbind.exe /d “Local Area Connection” ms_tcpip6

Unchecking “Link-Layer Topology Discovery Mapper I/O Driver” Check Mark in Local Area Connection Applet nvspbind.exe /d “Local Area Connection” ms_lltdio

Unchecking QoS Check Mark in Local Area Connection Applet
nvspbind.exe /d “Local Area Connection” ms_rspndr

You have to download nvspbind.exe from MDN portal. http://archive.msdn.microsoft.com/nvspbind

When invoking NVSPBIND, if no NIC is defined the specified operation is performed on all NICs.
If no protocol is included, the specified operationis performed on all protocols.
If no options are specified, the default actionis to enumerate all bindings for the specified NIC.
For more information on using NVSPBIND, see the readme.doc included in the download package.

Release Notes: http://social.technet.microsoft.com/wiki/contents/articles/hyper-v-tools-nvspbind.aspx

Posted in Uncategorized | Tagged: | Leave a Comment »

Registry Key to Disable User Access Control

Posted by Brajesh Panda on July 15, 2011

Disable UAC
REG ADD “HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System” /v EnableLUA /t REG_DWORD /d 0 /f

Posted in Uncategorized | Tagged: | 2 Comments »

Registry Key to Lock Task Bar

Posted by Brajesh Panda on July 15, 2011

Lock Task Bar from movement
Reg add HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v TaskbarSizeMove /t REG_DWORD /d 0 /f

Posted in Uncategorized | Tagged: | Leave a Comment »

Registry Key to Disable Internet Explorer Enhenced Security

Posted by Brajesh Panda on July 15, 2011

Disable IE ESC – For Admins
REG ADD “HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}” /v IsInstalled /t REG_DWORD /d 00000000 /f

Disable IE ESC – For All Users
REG ADD “HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}” /v IsInstalled /t REG_DWORD /d 00000000 /f

GUI: Servermanager – IE ESC – Switch Off

Posted in Uncategorized | Tagged: | 1 Comment »

Registry Key to Enable RDP

Posted by Brajesh Panda on July 15, 2011

Enable Remote Desktop Services
REG ADD “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server” /v fDenyTSConnections /t REG_DWORD /d 0 /f

Enable Remote Desktop from any version of RDP client
REG ADD “HKLM\SYSTEM\ControlSet001\Control\Terminal Server\WinStations\RDP-Tcp” /v UserAuthentication /t REG_DWORD /d 1 /f

Posted in Uncategorized | Tagged: | Leave a Comment »

 
%d bloggers like this: