TechOnTip Weblog

Run book for Technocrats

Archive for the ‘EMC Storage’ Category

VNX, Cellerra, Clarrion

Storage Migration Steps/Blueprint

Posted by Brajesh Panda on June 3, 2017

Discover, Discover & Discover

This is THE key step for migration

  1. Most of the storage systems can produce some level reports for discovery.
  2. Current System Spec, Features
  3. Used Features, if they can be moved or need to be rebuilt in the new one. Like Deduplication, pretty much dependent on storage. You can just simply move to another type of array.
  4. Configuration, registered hosts, their way of registration, attached LUN/volume info, zoning, HBA wwn/iqn details, access details, how volumes are mounted or used by the targeted system.
  5. Raid group/volume/luns/filesystem details, what features are enabled on them
  6. Extended/merged meta volumes if any
  7. Network config details for iSCSI or NFS related targets. Not only basic info but deep info like MTUs, LACP, mode of switch ports, VLANs, isolations, security configs etc.
  8. Client-side multipath software, initiator, driver versions, compatibility with the new system or any upgrade required.
  9. Which software is using the client-side mounts, if anything configured at application level like deduplication, encryption, compression etc.
  10. Client-side environment ownership, server owner, application owner. Talk to them find out criticality of application, who use that volume, number of users, how long outage can be available in the worst case.
  11. Any san level backup tool, cross-site replication tool etc.
  12. Another step I followed in past from project to project is to clean up necessary environment, you may face roadblocks from Application owners because you asking them to clean up their bad job. But it helps to fast-track the process, over the period people dump unnecessary data in volumes, like installable, old backup files etc.
  13. Any OEW support contract, how quickly they respond & what is included in the support.Β  You may need it to save life πŸ˜‰

Design, Configure, Trial, Acceptance Testing & Scheduling Plan

  1. Assuming you already have up and running the new system.
    1. If not that’s another animal. If you are consulting to buy a new box, sizing is the key concept you have to look at. And you may like to consider new technologies for better performance or easy technologies to optimize cost and implementation. Like moving from Fiber to iSCSI or traditional SAS/Fiber disk to SSD, 1gbps to 10gbps, getting rid of old brocade switches, introducing infiband ;-). While proposing or buying a new system also look for features and process how to move data from old system to new.
    2. If it is a new system, rack and stack, configurations are another project.
  2. Design data migration solutions
    1. Host registration, multipathing, software if any
    2. Storage to Storage Data Moves by LUN Mirroring/SAN Copy etc. How sans can talk to each other. Is there any native tool available or you need a 3rd party.
      1. No need of downtime
    3. Data migration inside the server, like moving the database from old mounted volume to new or moving virtual infra to new datastores.
      1. May or may not need downtime, depending on application
    4. Plan for reversal procedure, restore procedures
    5. Decommission old LUN/volume by taking out access. Try to avoid quick removal of volumes.
  3. Try out storage migration processes, document and fine tune.
  4. After migration check, if there is any activity in the old system.
  5. Use Dev/Test servers for operational acceptance testing.
  6. Be careful of clustered servers, sometimes they don’t like SAN copy stuff, but you can get around them.
  7. Map out migration steps for each client/applications volumes/luns to new system & success criteria. This is the Operational Run book, which can be used by the operation team to run the show.
  8. Schedule time with application owners and run them thru your plan. Find out when can be done (timings etc) and schedule them into migration calendar, assign application administrators, testers. This will be your project schedule & tracking card.
  9. Assign schedules and run book steps to respecting operation engineers, application engineers, testers. This will be your work breakdown structure to hold responsibilities.

Actual Project execution

  1. Implement necessary change procedure &
  2. Notify each party before starting the job
  3. Work with application owners to achieve the goal
  4. Make sure data backup process moved to the new system.
  5. Sign off



Posted in EMC Storage, Mix & Match | Leave a Comment »

EMC VNX: Volume is unreachable by server_2, Message ID: 13690601492

Posted by Brajesh Panda on December 6, 2012

I was running out of space in NFS stores, which are assigned to VMware farm. I created two LUNs added to ~filestorage Storage Group.

To get them detected by the NAS Pool quickly I ran Storage Scan from Unisphere; ended up with below error;

Discovering storage on corp-cstst-01 (may take several minutes)
d7 d10 d8 d9 d11 d12 d13 d15 d14: volume is unreachable by

Message ID: 13690601492

Also tried to scan from “nasadmin” shell using “server_devconfig server_2 -c -s -a” but ended up nearly same kind of error. Which I forget to noted down. πŸ˜‰

Later on while troubleshooting found all disks (LUNs) assigned to ~filestorage Storage Group are only attached to one data mover server i.e. Server_2. As my two new luns are not showing I removed them & did couple of scanning but no luck.

After working with EMC Engineering Support found this is a bug in my existing NAS code; which has been fixed in 7.0.54.x higher. EMC engineer referred to EMC Primus article emc295151 but never gave me a copy though and told me to stay away from that debug page ;-(

Well he logged me into VNX Debug URL (https://StorageProcessorIP/debug) & then we ran “Force full poll” in both storage processor and after completion storage processor scan in nasadmin shell. It helped to rebind those LUNs to both data mover servers.

Ten added my two new luns & checked status with “nas_disk -l“, they are still unassigned to server_3.
After running a storage scan they automatically assigned to server_3. And I am able to create new NFS stores.

Posted in EMC Storage | 2 Comments »

EMC VNX Performance Analysis – 1

Posted by Brajesh Panda on September 5, 2012

  • You must have Navisphere/Unisphere Analyzer Enabler installed on the storage array
  • Analyzer Enabler (license) will help to generate readable statistics log files, called as NAR files.
  • Without this license it will generate encrypted logs (NAZ files) and you may need help of EMC support analyze the same.

  • Make sure you start “Statistics Logging” from system properties

Read the rest of this entry »

Posted in EMC Storage, EMC VNX | 2 Comments »

How to retrieve EMC Powerpath Keys?

Posted by Brajesh Panda on March 19, 2012

Use β€œpowermt check_registration” command on the existing node where Powerpath is installed & registered with a key.

Posted in EMC Storage | Leave a Comment »

EMC Avamar Vs EMC Data Domain

Posted by Brajesh Panda on November 21, 2011

Here is a nice article. Kudos to the author.

Posted in EMC Storage | Tagged: | Leave a Comment »

The Celerra Management Service is not responding for task Query Data Movers All

Posted by Brajesh Panda on October 6, 2011

Model: VNX 5300

Unisphere: V1.

Control Station: Linux release 3.0

Nas Version: 7.0.14-0

Symptom: While browsing File Hardware details like Data Movers, File Network Details etc using Unisphere it didn’t render details & show a warning like “The Celerra Management Service is not responding for task Query Data Movers All

Data Movers looks good from ssh

Ran Health Check & Seems everything is good.

# /nas/bin/nas_checkup

Checked HTTP Daemons in Control Station

Checked NAS Daemons in Control Station

Restarted HTTPD daemon thinking Unisphere might not able to pull in details about data movers

Now Management Service Error gone – New Java webserver error popped up

Again Stopped & Started HTTPD daemon & it comes back to old state

So what is this error? Let’s reboot the control station & see how it is going πŸ˜‰

SSH as Root & reboot

It comes back to good state

However is there a service/daemon which can be recycled to fix this issue without rebooting control station? I will update this thread when I will get to know that πŸ˜‰

Spoke to EMC support, support guy told me – APL daemon was crashing which provides information to unishpere GUI window. Recommended way to restart the control station just to refresh all daemons. Just to note there will be no production downtime due to CS reboot.

Posted in EMC Storage, VNX5300 | Tagged: , | 3 Comments »

%d bloggers like this: