XtremIO and AIX Jam Session – Journey into ODM and Powerpath

Well, I did it.  Took the Summer off again from blogging.  I’ve been having fun in my Jeep, catching some waves at the beach and just trying to get the energy levels flowing for the Fall.  So what’s new?  Well for one, Varrow was acquired by Sirius Computer Solutions so I am still with them and continue to do what I do best.  Bring the world of storage to you, my readers of technology.

Recently I have been tasked to help out on several IBM AIX hosts attaching to XtremIO.  I don’t always admit it, but AIX used to be my Jam back in the day.  So I was quite excited when someone called me up and said they were having trouble with AIX seeing some LUNs from XtremIO so I began to help.  Essentially there are some steps you need to take in AIX to make everything look right and happy before you can start using LVM and carving out storage for applications.  Now, I’m not gonna get into all of that here, but I wanted to give some basics on updating EMC ODM Definitions in AIX, and installing EMC Powerpath.  Now this isn’t a be-all to end-all guide, but it should get you up and going pretty quick.  I will defer to the “AIX Host Connectivity Guide” and “Powerpath for AIX installation guide” for a complete and concise way of installing things, but I wanted to type these steps up for future use.  So here we go.

Technical Summary (EMC software installation):

We will start by insuring XtremIO is properly zoned, configured and presenting disks to the AIX hosts according to the Host Connectivity Guide for AIX from EMC. After this is complete, we will need to install storage definition files for AIX called “ODM”s. After the ODM installation is complete, we will install powerpath and reboot the host.

AIX EMC ODM Installation:
· Grab EMC ODM Defs from ftp.emc.com
· cd /pub/elab/aix/ODM_DEFINITIONS
· get EMC.AIX.
· gunzip EMC.AIX.
· tar -xvf EMC.AIX.
· rm .toc
· inutoc .
· installp -ad . EMC.XtremIO.aix.rte
· installp -ad . EMC.XtremIO.fcp.rte

Powerpath Install:
· Grab Current Powerpath from support.emc.com. Search for powerpath aix under downloads. For AIX 6.1-7.1 as of August, 2015 the most current is EMCPower.AIX.6.0.GA.b159
· gunzip EMCPower.AIX.6.0.GA.b159.tar.gz *or current version
· tar -xvf EMCPower.AIX.6.0.GA.b159.tar
· inutoc .
· installp -agXd . EMCpower
· powermt version *verifies version.
· Once installed, reboot host

Powerpath Registration:
If you are upgrading from an earlier version of PowerPath, you do not need to reregister. PowerPath will use your existing license key if necessary, converting the old 12-character license key to a 24-character key.
1. To register the PowerPath software, type:
· emcpreg -install
· follow prompts
At the end of the registration, you should see an output that says “1 Key(s) successfully registered”

Powerpath Basic Commands:
powermt version *displays current version
powermt –h *displays help for command
powermt display *displays HBA info
powermt display dev=all| more *displays device/lun information.

Here is an example output of powermt display dev=all. Note “Pseudo Name”, this is the name you will use when carving out storage in AIX LVM.

Pseudo name=hdiskpower0
Logical device ID=514F0C5430600006 [CUSTOMERPOC_1]
state=alive; policy=ADaptive; queued-IOs=0
————— Host ————— – Stor – — I/O Path — — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
—– Host Bus Adapters ——— —— I/O Paths —– —— Stats ——
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
0 fscsi2 optimal 10 0 – 0 0
1 fscsi3 optimal 10 0 – 0 0

So there you have it!  Hope this helps someone.


Posted in AIX, EMC, EMC Elect, powerpath, XtremIO | 2 Comments

EMCWorld 2015 – VMAX3 Replication and SRDF Updates

Last Thursday I was able to attend my last session of the conference by Mike Bresnahan and Paul Martin who live across the pond in Cork, Ireland titled: “VMAX3: The replication Revolution”  Paul is @rawstorage on twitter and also an EMCElect.  I had various conversations with him throughout the week on the new VMAX3 and it was great talking to him.

The presentation began with Mike Bresnahan talking about the new SNAPVX structure, its limits and feeds and speeds.  SNAPVX is the replacement for old timefinder/bcv, etc commands and it is a LOT better.  Gone are the days of setting up consistency groups, creating a lun for every clone, etc.  This has a great effect on space efficiency and obviously management overhead.

Here are some bullet points:

  • Targetless Snapshots conserve resources
  • Track sharing
  • Target lun only required for host access (linked-target)
  • Ease of Use
  • Storage group operations enable protection of entire applications
  • Snapshots identified by user-defined name
  • Automatic expiration
  • Create up to 1024 unique environments from a single snapshot
  • Snapshots and linking targets
  • Snapshots are targetless
  • Target lun linked to snapshot to present point-in-time to a host
  • Standard thin device
  • Nocopy by default
  • Link multiple targets to a snapshot
  • SRDF interoporablity
  • Cascading sessions example
  • Unlimited cascading
  • Each linked target lun can have 256 snapshots
  • Each snapshot can have multiple linked targets
  • Break the cascade at any point
  • Storage groups control all snaps.
  • Storage group operations
  • Natively consistent
  • Automatic snapshot expiration
  • TTL optional setting
  • SRPs could be used for source/targets  but Hypermax has a SRP reserved capacity setting
  • When SRP reached 90% full only host writes can consume the last 10%
  • Alerts available to warn of approaching the limit.

In addition to the above Mike talked about cascading snapshots, and the relink functionality.  Relinking a differential copy of a SNAPVX snapshot does take a separate lun, but it opens up the functionalities of being able to mount said snapshot to another host, or use it to archive off things…or ultimately restore back to the original lun if for some reason you do something really bad during testing and have to revert back to an older snap 🙂

After the work on SNAPVX, Paul began to talk about SRDF functionality.  I was a little bit dissapointed that they didn’t really cover the new “Active/Active” replication, but I understand why they did because it was just announced a couple days before. Here are a few bullets from the SRDF portion of the presentation:

  • SRDF Simplified Setup w/ Unisphere
  • Uses storage groups
  • You can set modes in SRDF
  •  SRDF Star still requires a consistency group and is CLI only.

All in all this was a great session and it was very refreshing to see all of the updates that the VMAX3 brings to the table.  I am very much looking forward to seeing what is next in this space and how things pan out with the VMAX3 with our customers!


Posted in Conferences, EMC, EMC Elect, VMAX | Leave a comment

EMCWorld 2015 – VMAX3 Performance!

The past couple days have been awesome here at EMCWorld 2015.  I wanted to share with you information about VMAX3 Performance through a couple sessions I attended plus some conversations with several EMC engineers.

Here are some Highlights:

No more CPU bind on FA’s.

In previous VMAX revisions you had to be careful with CPU bind on traffic that went through ports.  This was difficult to monitor and you had to be very careful with balance / worry about fan out ratio, etc. Now with the VMAX3, you can have the power of all cores of the CPU to a single port, and the VMAX3 has a great way of managing all of this under the covers.  Minimal port count is about 4, but you’ll want to balance across directors.

No need to use 15K drives anymore

I’ll talk to this a bit later but 15K’s are now out in favor of 2 tier configurations. 10K is ½ the cost with similar performance to 15K.  120 IOPS, 10-12 msec response

Here are some bullet points from the performance session:

  • Flash 3000+ IOPS <1 – ~1 msec response time
  • Configure all 2.5” drives for best performance and density
  • 2.5” DAE supports 120 Drives,
  • virtually every system will have flash and 2.5” DAE
  • 2.5” 10K offers 14,400 IOPS per DAE at 10-12 msec
  • whereas 3.5” 7200 offers 3,000 IOPS per DAE at 17-20 msec
  • 7200 RPM only available in 3.5” form factor
  • there are use cases for 7200 RPM drives but not for general use stuff so virtually all VMAX3’s shipping won’t have any 7200 RPM drives in them.
  • Three supported raid types between all disks
    • Raid 1
    • Raid 5
    • Raid 6

Virtually all configurations will be a single SRP (Storage Resource Pool) and a 2 tier configuration using only FLASH and 10K RPM drives. They do this to not only provide the best performance, but also is more simple to manage and provides the best Skew performance.

Skew Performance:

  • 85/15 is typical of most workloads
  • skew can vary during a business cycle and over life of an array
  • skew and traditional tiering
  • typical vmax 1st gen drive mix is 3/27/70
    • 3% flash 27% 10K fc 70% 7200 sata
    • Provided OK performance but when not monitored you would run into performance issues faster.
  • Typical VMAX3 drive mix is a lot heavier on EFD with no SATA/7200rpm drives

With the VMAX3 and a two tiered array, flash and 1.2 will do most workloads and Skew performance is effected less.

Monitor the system with unisphere – If you are NOT monitoring, you will have no idea what your array is doing

VMAX3 Defaults work!

Start with optimized SLO if you’re not sure, uses 10K first then EFD, 7.2 as a last resort

EMC will always configure a single SRP but you can order one with multiple if you have regulatory reasons

How do I know which FAs to plus servers in…

  • VMAX3 is a lot easier
  • All cores in the emulation can service 1 port
  • Greatly reduces port sharing contention
  • Burst handling is much better in VMAX3
  • Two ports across two directors for OLTP for starters
  • Suitable for large percentage of workloads
  • May want 4 ports with 2 each on dual fabrics for secillency
  • Four ports for DSS/large block workloads
  • Provides maximum port level aggregate bandwidth
  • Linear scaling as ports are added

 Highlights on Lun creation

  • No more meta volumes
  • 16TB luns
  • Single VMAX3 volume outperforms vmax metas by a lot
  • Can do 10’s of thousands of IOs
  • Offers other advantages
  • Ease of provisioning
  • Reduced lun count
  • Maintaining Balance
  • Plan ahead

That is the highlights from the airplane!  More good stuff coming up in the next week or so 🙂


Posted in EMC, EMC Elect, VMAX | Leave a comment

EMCWorld 2015 – EMC Elect Session with John Roese – EMC CTO

I wanted to get this blog out to give some really interesting information that happened yesterday while the EMCElect had a chance to have a social conversation with John Roese, Global CTO of EMC Corp.

John started on giving his background as CTO of several tech companies in his career history.  It was really interesting to hear his background and also that he is a Chairman for Cloud Foundry as well.

Once we got into the deep of Q/A we really started to talk about security and how it plays in IT in general.  We talked about the transformation that IT is taking now which was also a theme of the week here at EMCworld.

A lot of data needs to be secure and traditional file based systems, are not adequate especially around analytics.  John talked about secure autoclaves that provide security around analyzing data queries to people that need it.  This way people are only working with the data that they have for the specific query or work set.

We also talked about how a lot of the new security / big data stuff is very PKI intensive which hurts performance during analytics.  This is a very heavyweight model with cloud security and now some people are going back to things that are more simple like Kerberos, etc.

Then we talked about Genomics data and this is where things got really interesting.  He talked about the Worlds center of Genomics – common place where people access central genomics – what they discovered is that it hasn’t become clear on how to become that center.  They originally started putting everything in google cloud engine, and then they said woops, that info is really personal.

What could the evil doers of the world do with this information if for some reason it was made public?  Well DNA could be sequenced and planted at a crimescene…through DNA fabrication and basically be framed for a crime.  We talked about how DNA fabrication equipment is fairly inexpensive as well.   So there you have it Blacklist or CSI fans, I bet you see this in a future episode!

We talked then about Twitter data and how queries can be run to see what kind of trends are going on in the world and actually map it out.  2 places in the world where all twitter day is aggegrated.  MapD and MIT.

Someone also asked about Data Governance – Value of data conversation and research behind it is awesome.  Insurance companies are trying to place value on data and write policies on this but they really don’t know what it the value is.  Steve Todd at EMC has written about this on his blog and guess what?  He’s also an EMCElect 🙂 You can check out his blog here: http://stevetodd.typepad.com/

So lastly, we talked about how when datasets get bigger, metadata matters. The problem behind this is that in an exoscale environment (100’s of PB’s) is that you can’t build out something that is posix compliant, so he was talking about how people are starting to work or experiment with ad-hoc filesystems that could be created dynamically as you work with that data to fulfill these needs.

We concluded by talking about GINA which is a program at EMC that sparks innovation.  Anyone that sends data to gina@emc.com will be indexed.  I didn’t quite understand exactly what GINA does, but you can read about innovation and the program here: https://www.emc.com/corporate/sustainability/workforce/innovation.htm

That was about it!  It was a very good conversation and I look forward to doing something like this in the future!


Posted in Conferences, EMC, EMC Elect | Leave a comment

EMCWorld 2015 – Day 1 – May 4th, 2015

First of all, may the fourth be with you… #Starwarsday!


OK, now that we got that out of the way, EMCworld on Monday was a blast.  There were several Starwars themed things spread across the conference today, and even Mr. Spock was spotted.  The main take away from Monday’s events and also noticing as a theme across what EMC is messaging is that we are in a constant shift into information about everything we do.  Analytics, data, etc are becoming increasingly important to businesses and your IT department is going to have a larger roll of this in the future.  I see this as an opportunity for IT organizations to be involved and to talk to others in the business as storage and other aspects are being used to meet business objectives and goals.  I.E. the CTO probably will need to drive this throughout the organization has businesses shift.

First Keynote of the day had a ton of information, for a blow-by-blow of everything that happened, check out my Coworker Andrew Miller’s blog at: Thinkmeta.net Live EMCworld Keynote. He does a good job giving you all the stuff that happened during the keynote.  Here are some of the highlights:

89% of the fortune 500 companies in 1955 don’t exist today.  Key thing here is adapt or die.  EMC believes that the current shift in technology is more disruptive.  Things like Uber, WICK, Wearables, etc are changing the way we think and do things.  Very cool.

Hardware wise, EMC annouced the VXRack  this is an all-in-one box like a Vblock – designed and supported by EMC using EMC and VMware technologies.  The VXrack can be built in either a storage or compute intensive configurations.

EMC talks about enhancements around XtremIO 4.0.  Recoverpoint will be more integrated (more details on that later), EMC has 30% marketshare in the AFA space which is 10% more than anyone else.

EMC also has a ton of information about VMAX3 now.  They demoed a 5th grader provisioning a lun.  The theme of ‘not complex storage’ is echoed in the sessions I also attended today.

That’s about it for highlights on the Keynote.

Sessions attended:

I attended two formal sessions and also a private session in the EMCElect lounge.

VMAX3: Unleash the Power of Service Level Objectives with John Madden @ EMC.

This was a great session covering and confirming what we already knew in our lab.  The best part of the session was the interaction with the audience and John answering questions.  The theme of the session is that we need to shift our focus from a capacity based thinking into a service oriented thinking.  For large IT shops with a ton of Silos, I see this as a challenge for a lot of people.  Basically starting out you want to put everything in the default service level objective, and then move it over to appropriate workload-based tiers when you have the data to prove it out.

Key things to keep in mind:

  • Target is defined as response time objective emulating a disk technology.
  •  Amount of a particular SLO depends on the mix of those disk technologies.  Real time utilization of resources include in head room provisioning.

More VMAX stuff will be coming.  I also noticed a lack of VNX sessions at the conference this year…

VMAX3 Performance: Embedded NAS

The key to this session was that VNX File Gateway is now Software Defined into the VMAX3’s Hypermax hypervisor.  The session covered Embedded NAS performance monitoring and management, and the ways in which it is like or dislike the old VNX gateway boxes.  I will be attending a couple more Embedded NAS sessions this week to really wrap my head around things.

Private Brocade Session (EMCelect):

Several of us sat down and had a roundtable discussion on the future converged (FC and Network) equipment that Brocade has.  It was a very technical session talking with a Sr. SME from Brocade and also the Director of Product Management Truis Myklebust.  Since I have a small Brocade past (I really focus on Cisco lately) it was really cool to see what is new and what Brocade is doing in this space.

Solutions Pavilion:

I got to walk around the solutions pavilion and talk to some folks around what is new in the EMC datacenter.  EMC has a DC setup in the center of the pavilion running live versions of a ton of the products.  I will be doing a short video/interview of some things tomorrow!

Varrow/Sirius Customer Event:

Finally to wrap up the day, I got to hang out with my favorite people.  Coworkers and our customers.  It was also really good to talk to my new coworkers at Sirius and ask them questions about the company, etc.

More updates coming tomorrow!




Posted in EMC, EMC Elect, varrow, VMAX | Tagged , , | Leave a comment

VMAXHumpday, April 29th, 2015 – EMCWorld 2015 and other good stuff.

Hey readers!

In my last VMAXHumpday, I talked about my presentation about VMAX3 with Sean Cummin’s at EMC.  It was a really awesome presentation, but unfortunately my recording was REALLY bad.  I hope to do another overview/demo now that I bought myself a real blogging microphone.

I’m really excited about next week.  I will be attending what will be my…I believe 5th EMCWorld. I always get a lot out of these conferences and energy that they provide.  Next week will be no exception to this.  It’s always cool to see what is out there that is new, and also gives me a bit of a refresher on what is out there and what is coming in the future so that I can prepare!

I plan on talking to a few people about VMAX3 while I am there so that we can be prepared to deliver these boxes to our customers.  As I’ve talked about this before, we have had the VMAX3 in our lab as an early Beta for almost close to a year now.  It’s been really cool to have the tech in the lab and be able to work with it.

In edition to VMAX3, I plan on attending some sessions hopefully on EMC Viper, and also some of the EMCElect special sessions.

Really looking forward to this year’s event.  If you’re around and want to hang out and talk tech, just reach out to me on twitter and I’ll be glad to at least say hi!

Hope everyone is having a great April.  More blogging next week!


Posted in Conferences, EMC, EMC Elect, VMAX Hump Day | Leave a comment

VMAXHumpday, March 11, 2015 – Varrow Madness and VMAX3!

VMAX_HumpDayHey readers!  Just wanted to release a quick note that next week March 19th, I will be doing a presentation on VMAX3 with Sean Cummins from EMC at the annual Varrow Madness being held this year in Greensboro, NC.

Sean Cummins, has been working on VMAX/Symmetrix for a long time, and it is exciting to be able to present with him.  He’s done some really cool stuff around dockerizing certain elements around SYMCLI and also has a great post on VMAX FAST VP best practices.  We should have a great session.  He can be found on twitter as @scummins and his blog is http://blog.scummins.com so check it out!

If you want to come check out the VMAX3 and see a live demo of it, it’s not too late to register.  Head on over to http://varrowmadness.com and check it out.

Here is a session blurb:
EMC’s next generation VMAX3: Architecture and Live Demo

EMC’s VMAX family of Storage Arrays has a decades-long history of reliability, servicing the world’s most critical applications.  Modern versions of the VMAX continue this legacy, and build upon it with features like a data services hypervisor and service level objective (SLO) provisioning. This session will cover the architectural differences of the new VMAX3, and new features that make VMAX3 most powerful, trusted, and agile Symmetrix yet.  Both EMC and Varrow will be doing a live demonstration of these features and technical deep dive on common storage tasks using the latest management tools.

There are also a ton of other sessions scheduled for the day.  Some of the highlights will be centered around Mobility, Cloud and Security.  Make sure you follow #VM15 for live tweets from the event.

Some of the top vendors from around the world will be attending along with some of the best customers too!

Hope to see you there!


Posted in Conferences, EMC, EMC Elect, VMAX, VMAX Hump Day | Leave a comment

On EMC Elect, Snow Days and Mr. Spock


347028-graphic-EMC Elect 2015-hires.jpgThis week started absolutely fantastic. I was returning from a great weekend in Wilmington, NC excited to know that I was now able to say that I was EMC Elect for 2015! This is my first time being a part of EMC Elect and I am excited, and honored to be a part of it. Most of the past year I have been blogging about EMC’s VMAX, and other topics. I have been active on twitter, LinkedIn and EMC community forums as well. In short, I like sharing things about technology as it pertains to storage, and the datacenter as whole. You can read about the program here: https://community.emc.com/community/connect/emc_elect  I will continue to share with the community in 2015 and hope to add more stuff soon!  Hint: I’ve been meddling with a VMAX3 in our labs and am doing a presentation at Varrow Madness next month on it!

After enjoying and celebrating my Elect status, I sat down on Monday, and started typing up my blog entries and well to be perfectly honest, I had a little bit of writers block. Monday turned into Tuesday, Tuesday turned into a snow Day, Wednesday seemed promising, but Thursday even more snow. Now granted, I grew up in Michigan and can handle the snow. I love driving in it with my Jeep, but it just makes everything all weird in North Carolina. You see, there is this certain ‘bread and milk’ craziness that goes on when there is even an inkling of snow coming to the area. Neighbors freaking out before the storm warning me that Walmart is packed and I should run out and get my bread and milk before the lines get long…and Neighbors freaking out after the storm frantically shoveling their driveways after the storm. If they would only wait until about 24 hours, they would discover that snow really doesn’t stick around all that much. My driveway is perfectly clear…and I don’t own a shovel. Oh and bread and milk? Really? That’s what you’re thinking about? Anyways, I digress.

color_nimoy_headshotAnd then Friday hit. As everyone was worried about what color a dress was (crazy internet thing) I learned of Leonard Nemoy’s passing at 83 years. I grew up with Leonard Nimoy from episodes of Star Trek, Awesome Star Trek movies about transparent aluminum and Khan. His voice roles in Transformers as Galvatron and Sentinel Prime catered to my inner child’s dreams of discovering how awesome the transformers were. His work on the Fringe TV series was pretty nerdy too. You could say that Leonard Nimoy embodied the inner nerd in most of us and I thank him for that. His final tweet a few days ago was : “A life is like a garden. Perfect moments can be had, but not preserved, except in memory. LLAP” Thank you Mr. Spock. Here is a toast of Romulan Ale to you. Live long and Prosper.


Posted in EMC, EMC Elect, Greater Good, random, varrow, VMAX | Leave a comment

VMAXHumpday, 12/17/2014 – VMAX3 Unisphere Overview, a First Look


Hello!  Welcome to my semi-regular-weekly blog about VMAX.  Today I wanted to do a quick overview of my initial impressions of the new Unisphere and VMAX3.  I decided to create a video today.  I basically spent 10 minutes on the VMAX3 before looking at this and recording, to excuse my over-use of transitional words such as ‘Umm’ and ‘stuff like that’.

Also, if you need a good refresher on why Enterprise Class Storage is still very relevant, you should go over to @thejasonnash ‘s blog @ http://jasonnash.com/2014/10/14/the-vmax3-why-enterprise-class-is-still-very-relevant/  I briefly reference it in the video.



Posted in EMC, VMAX, VMAX Hump Day | 2 Comments

VMAXHumpday, 12/10/2014 – How to Expand a Striped Meta in Unisphere Online!


Hump Day!!! Whaaaaaat?

I originally took the summer off in what was my regular VMAX humpday posts…and then a little more time from blogging due to general life getting in the way…well no excuses, I should probably remedy that right now.

Recently I have been involved in several large scale implementations of VMAX’s for customers and a common theme that comes up to those that are new to VMAX is operational tasks that happen after I leave the site and customers actually have to manage an array.  One of those most common questions I get is: “Well, I’m glad you set this all up with the exact sizing that I need, but what happens if I want to expand a lun?”  or “In VNX, I just have to right click on a LUN and expand the size, is there an option to do this in VMAX?”  Well the short answer is that you can expand a LUN online, however the process to do so is not as easy as it is in VNX.  Below I will explain how to do this, break down why we do this in this way and hopefully make it a little less complicated.  I will demonstrate how to do this in Unisphere.

Before starting anything, I like to break down the components to give a little background and terminology.  Here are some terms that I will mention in this post:

Meta Member – A virtual volume that is no larger than 240GB and used / striped or concatenated together to form a Striped Meta.

Striped Meta – Meta volumes are used in VMAX to make volumes larger than 240GB’s and/or address performance considerations.  Normally striped metas are formed in 4,8 or 16 members to meet size or performance needs.  My general rule of thumb (yours may vary!) is to form Meta Luns starting around 100-150GB at 4 members, and as I reach 1TB, I start expanding them out to 8 members and beyond.  I can go into the reasoning in more detail, but it will go well beyond this blog post.

BCV-TDEV – BCVs can be detached from the active data storage at a point in time and mounted on non-critical servers to facilitate offline backup or parallel processing. Once offline processes are completed, these BCVs can be either:

  • discarded
  • re-attached (re-synchronized) to the production data again
  • used as a source to recover the production data

Why is this important to expanding a LUN?  Well a BCV is used to make a mirror of the production data while the LUN is expanded and re-striped and then mirrored back to the newly formed LUN.  This is why VMAX is cool…it protects and re-stripes.  It is also why the process takes a while to accomplish.

Alright, now that we have some terminology out of the way, let’s get to it!!

Continue reading

Posted in EMC, varrow, VMAX, VMAX Hump Day | 2 Comments