The first thing I would like to mention is a HUGE shoutout to our awesome Varrow customers that I got to spend time with Tuesday night at Sushi Samba and subsequent other places on Las Vegas Blvd.  We had a great time eating sushi, and cooking Kobe Beef slices on a huge rock.  It was a nice time.  Here are some shots I took:

.

So Wednesday I had a chance to hit up the blogger’s lounge and invade the EMCElect Space.  I had a great conversation with @colinmcnamara and @mjbrender around what it takes to be EMC Elect and why I was rummaging through their Polo shirts (I ninja skilled my way in to retrieve these for our very own @thejasonnash and @virtualtacit).

I learned a bit more about EMC Elect and how it would be really cool to be a member.  If you would like to learn more and nominate someone (or yourself!) go to: https://docs.google.com/forms/d/1Msk1obeUw1v-VwvRDEo3m3wOVGLZ0IvTlYEErnWhewQ/viewform

You have until November to submit.  It’s really cool.

I got some really sweet Legos from the VCE booth.  ”No Assembly Required” at least for the VBLOCK part.

I also attended VMAX best practices and the VMAX birds of a feather session…  That I will have to save for another post!

I did however witness my company Varrow winning…for the 5 consecutive year the EMC Services Quality Award.

Pictured are a bunch of folks from Varrow and I managed to sneak a picture.  Here is the banner of the recipients.  (BTW, 1 7 year, 2 six year and 3 five year award winners).

I managed to talk with some VMAX folks during a session about the new VCOPs plugin’s for VMware, and have a deep dive discussion around FAST technologies and how they are used in customer environments.  It’s real nice to talk with engineers that actually work on the products day to day and less marketing stuff (no offense to my marketing friends!!! I’m just a geek).

…and finally.  There is this superhero theme going on.  Still not sure about that but managed to get a shot with some X-men.

This is my first time in a few years since I presented in Boston back when I was an EMC Customer.  Lots of things have changed and lots of things remain the same.

First, it still is a very cloudy cloudy world.  What surprised me is that there doesn’t happen to be clouds hanging all over the place as in previous years, however I would say that the buzzword is still there and still flows through presentations and key notes freely.  The one thing to note though is that “Cloud” is a bit more defined than years of past and perhaps we’ve gone beyond buzz to something that is actually quantifiable.  Which makes people sound WAY more intelligent than back in 2009.

I also am enjoying Las Vegas and what it has to offer.  This place is a bit crazy, smoky and outright strange sometimes but with a bunch of IT professionals, 174,480 cups of coffee served [from the Jeremy Burton's @jburton keynote] it is bound to be a great time.

Looking forward to the solutions pavilion, networking with fellow EMC fans and partner awards tonight!!!

As I was waiting for the tow truck I was thinking to myself, “Self, how could I have prevented this?” One I could have listened to my wife when she said, “Hey the car was sounding weird, why don’t you take it to the mechanic before you start driving it?” but of course being a warm blooded male of the human race, I kindly dismissed the idea and kept driving…until I ended up in the current situation I was in.As the tow truck driver arrived to snatch me away from being defeated by the evil Joker, I advised him that I would like to be towed to my local mechanic. See, I trust this mechanic. He’s not out to make a quick dollar, or to turn around and tell me I’ve got something major going on, buy that car and then sell it again to make a quick profit (this has happened to a couple of my friends). You need to trust the people you are working with, make sure they have the qualifications and continue to use them and support them.
So what the heck does this have to do with Storage or IT for that matter? Well a lot more than you can imagine. Here are a few examples.

• Certifications and Qualifications – Both in mechanics and in IT you really should know what you are talking about. Yah, you might be the best person on the block fixing Microsoft SQL clusters or installing EMC VMAX arrays but if you don’t have certifications or the qualifications, how does one know that you are qualified to do the job they are paying you to do? Yes a long and comprehensive curriculum vitae will help in this scenario, but industry standard certifications such as ASE’s comprehensive lists for mechanics or EMC, Cisco and VMware certifications give a standard of which can be met and/or exceeded. Combine both certifications and qualifications and now you are someone I want to do business with.
• Continuing Education – Be a master of your trade but become diverse in other things. Invest in yourself and your ‘Brand’. Find a company that provides awesome training videos such as Trainsignal. Also take time to learn something different! It’s amazing to me that more people don’t branch out from their normal everyday routines and learn something new. Stuff like just learning Java for fun, or picking up a new book on how to deal with difficult people can be extremely beneficial. So if you’re a UNIX / Storage guy like me, maybe it’s time to put away the UNIX books and pick up a book on public speaking for example. You are your greatest promoter and brand. Invest in yourself for long term success.
• Ongoing Maintenance – If I had listened to my wife and been a little more pro-active with the maintenance on my vehicle I would have probably not had to pay the tow truck driver (although I did enjoy my back country ride with the tow-truck driver…perhaps another blog posting is in order on that one). As Varrow, we provide ongoing vCheck services for our clients which tell them what they should and shouldn’t do with their VMware, EMC, Backup and networking environments. If they choose to ignore our advice, that is fine but they need to understand the implications of putting off a specific vendor patch, not tuning a storage array properly or not adhering to best practices. When you buy a new car, you change the oil in your car right? Let’s start there and continue having that mentality in our IT Datacenters. If you have to take a downtime then let’s get it scheduled! Planned and expected downtime is always better than getting that page at 3am on a Saturday morning…during a holiday weekend.
• Trust – Build a level of trust with whoever is working with you. Be it your local mechanic for your car or the ongoing relationship with a VAR that actually provides value. Are we always 100% right all the time? No. But that is what the relationship is for. You know in a time of need we will be there and if it is wrong, we will try to make it right.

In conclusion, things like certifications, qualifications, education, maintenance and trust are all important for overall IT strategy. I’m sure there is a ton more but this is what I was thinking about when I broke down on the highway last week. I guess you could say I’m a mechanic of storage. …and yes I should have listened to my wife. @sangeek

Then, I thought to myself, “Self”…”How come Farmers have all the fun?”… ”Us IT Folks have fun too!”.

So whilst the inspiration for this next video comes from eating way too much Super Bowl Chili with bacon in it…I present to you, in my own words and possibly other geeks like me, “God Made an IT Worker“.  @sangeek

Transcript to follow:

And in the 1930’s and 1940’s, God looked down on the earth and said I need someone who understands all of these numbers and specifications….but needed someone to automate and oversee these great computer creations.

So God made an IT worker…

God said I need somebody willing to stay past five o’clock, work all night patching windows servers, cable up three new servers, and then order pizza online for his family because he’s going to be pulling an all nighter again.

So God made an IT worker…

I need somebody with a brilliant brain strong enough to figure out how to make things fast enough for DBA’s yet his demeanor needs to be willing enough to explain to his mother, for the eighth time how to print in duplex mode, or remove a virus from his brother’s computer.

So God made an IT worker…

God said I need somebody willing to sit up all night with the newbie employee while it’s his first time installing new software, teach him the right way to do things, watch the newbie fail, then dry his eyes and say maybe next week.  I need somebody who can P2V a windows 2000 server, who can do a bare metal restore of 12 year old hardware, diagnose the most obscure bluescreen of death and finish up his 40 hour week by Tuesday under the dim hue of his 27” monitor documenting the whole process, and then paining from carpel tunnel, will put in another 72 hours.

So God made an IT worker…

God had to have somebody willing to learn the newest technology, buzzwords and ride the information super highway before cloud computing came about, yet stop in the middle of it all to race over to the next cubicle to comfort his coworker when he sees him yelling at the monitor “PC Load letter What Does that mean?”

So God made an IT worker…

God said I need somebody strong enough to brush off people telling him he’s a nerd or geek, yet gentle enough to show those same people, in a calming and reassuring voice, “I think your caps lock key is on”.  It had to be somebody who’d read various documentation obviously translated from another language, be able to install a power supply with the power feed still on, replace hardware, restore lost files, patch, backout, patch again, Control, Alt, Delete and when he gets home, fix his families iPhones by laying them in a bed of Rice.  Somebody who would bring his family together with the bonds of various electronic devices, teach them the value of game time, laugh and then with a tear in his eye be proud when his son comes into his home office, looks over that 27” monitor and says, “Dad I want to be a nerd too when I grow up.”

While I’ve been a little silent on the scene since my presentation at EMC World 2010 and this year’s Varrow Madness I can tell you it is not from lack of interest in presenting!  I’m never very good at tooting my own horn, but hopefully this years session is a blast.

Here is my abstract:

Session: #2239 Deciphering the Mystical World of Storage Performance

Abstract:  An introduction to SAN and NAS attached volume types within VMware. The purpose of this presentation is to help VMware and Storage administrators that are new to the virtualization scene to appreciate the implications of RAID types, drive performance, throughput and bandwidth in an easy to understand and fun way.

This session will break down the differences between Network (iSCSI, NFS), and Fiber Channel storage but not be vendor specific. In addition the usage of certain drive types, what they are good for and what they are not good for will be discussed. Example production environments will be shown to include VM and Storage as a Service, archival, and application delivery.

Cool huh?  I hope it will be.  I don’t want to get crazy on specific vendors feeds and speeds, how much cache they have or that twenty thousand EFD’s in an array beats Michael Schumacher and Barry Sanders in a game of horseshoes. I want to get down and dirty into the science of it all.  As my Physics teacher once told me Physics is Phun!

If you could please vote for me, I would really appreciate it.

You can clicky here

Also while you’re at it, please consider voting for my other co-worker colleagues and here are their session abstracts:

Joe Kelly (blog:  http://blog.virtualtacit.com/) and Michael Bailess from American National Bank are submitting:

Session: #1883 – Deploying an Active/Active Datacenter with SRM 5

Abstract:  In this session we will discuss the challenges that face a single physical datacenter as well as how these challenges can be resolved with SRM 5. We will explain the design that we deployed at American National Bank which did not include more expensive technologies that would have been needed to run a stretched cluster environment. American National Bank has implemented their own private cloud that is not linked to any physical site. This solution saved American National Bank from long late night maintenance windows to correct facility issues, spread the load between to geographic regions, and created a DR plan that can be fully tested daily.

Tom Cornwell (blog: http://blog.piratesjade.com/) submitted:

Session: #2848 Cloud Continuity: How Does the Cloud Fit into Your Business Continuity Plan?

Abstract:  Today cloud is becoming ubiquitous. However, it can be utilized in many different was. Whether you utilize cloud for your production environments or not, cloud can play a part in your business continuity plan. This session describes different business continuity strategies utilizing cloud. Topics include: cloud as a source, cloud as a target, and cloud to cloud DR. In addition, we will discuss mixed strategies and different types of cloud implementations including Platform-as-a-Service, Infrastructure-as-a-Service, Storage-as-a-Service, and DR-as-a-Service.

Jason Nash (blog:  http://jasonnash.com) submitted four sessions focused on networking:

Session: #2181 – Cisco Nexus 1000v: Architecture, Deployment, and Management

Abstract:  This session will walk attendees through the architecture, deployment, and management of the Cisco Nexus 1000v virtual distributed switch. The information in this session is based on experience with numerous production deployments of the Nexus 1000v and the audience will benefit from many lessons learned from the field. The session begins with a high-level overview of the 1000v’s design and components followed by several considerations and preparations that anyone looking to deploy this new distributed switch should consider. This includes items such as network design, existing network infrastructure, and overall integration in to the organization’s IT processes. Next, the focus will shift to technical details of the configuration process and several examples will be shown covering many of the common scenarios seen in the field. Day-to-day operations and management of the switch will be covered in detail and include routine items such as adds, moves, and changes, as well as upgrades and maintenance. Finally, a section on troubleshooting processes and information will give attendees the tools they need to support their new virtual switch.

Session:  #2197 - A Deep Dive on Virtual Distributed Switching & Cisco Nexus 1000v

Abstract:  This session will provide an in-depth look at the distributed virtual switching technologies available in VMware vSphere. The discussion will start with an overview of both the integrated Distributed Virtual Switch as well as Cisco’s Nexus 1000v. We will compare and contrast the options highlighting features, functionality, management, complexity, and operational considerations. Each available option provides its own set of features, functions, challenges, and design and deployment considerations. During the session the components, design and implementation considerations, as well as troubleshooting recommendations will be covered in depth. Attendees should expect to walk away with the knowledge they need to decide which of these technologies fit the requirements for their environment as well as the understanding to deploy them.

Session:  #2207 - vSphere Distributed Switch – Technical Deep Dive

Abstract:  While the vSphere Distributed Switch (vDS) has been around since vSphere 4, vSphere 4.1 and 5.0 have added a number of enhancements. This session will provide a technical deep dive in to the vSphere Distributed Switch. This includes design and deployment considerations, configuration, migration steps, tuning, and troubleshooting. Special attention will be paid to migrating an existing production environment from the standard vSwitch to the vDS with no or very minimal disruption. Extended features such as Network I/O Control (NIOC), Network Resource Pools, and Load-Based Teaming (LBT) will be discussed in depth with use cases and recommendations given. Finally, methods and tools for troubleshooting network connectivity and performance problems will also be highlighted. The inclusion of accessing a live lab environment will make for a very interactive session.

Session:  #2463 - vSphere Physical Connectivity – Deep Dive & Best Practices

Abstract:  This session will provide an in-depth look at your options for physically connecting vSphere hosts to the network. The discussion will center around common question areas that come up during knowledge workshops and customer design sessions. Throughout the session videos and animations will be used to help attendees easily see the expected result from many of these configuration options. The presentation will focus heavily on the different hashing types and traffic control, especially the more advanced options such as Load-Based Teaming and Network I/O Control. Other areas of focus include physical separation of traffic, networks of differing security requirements such as DMZs, and suggested NIC configurations for both 1Gb and 10Gb environments. Finally, recommendations for physical switch configurations will also be covered. Throughout the session best practices, recommendations, and lessons learned from many production deployments will be shared. Attendees should expect to walk away with a deep understanding of the physical connectivity options available with vSphere, how they can be utilized in their environment, and the best methods for deploying them.

Thank you for your support!  We greatly appreciate it. Oh and yes, I’m a Detroit Lions and F1 Fan.

As my coworker, friend and colleague Joe Kelly @virtualtacit says:

It’s not all pancakes and banana sandwiches!

Why is this? Well because even the best algorithms in the world can’t account for every single application in the world. They get very close, and in the case of EMC Fast Cache on VNX, it does very very well in virtual environments, certain database environments and a ton of other success stories but you have to make sure that you just don’t turn it on because you have it for everything which I know is very tempting. I’ve done it.

But SANGEEK? Come on? I bought these really fancy EFD drives and I want to see them work!!!

Don’t worry. You’re EFD drives will work. Very well. But you need to look at everything you put on your array and determine if it is appropriately being used. By making educated decisions on which luns to enable FAST Cache, your whole array will perform better, more efficiently, and most importantly make your boss very very happy.

Some examples of where you will most likely see the best performance gains:

For example, if Joe is trying to buy a part for his car to fix his air conditioning online and discovers that he needs another part to fix the air conditioning system than originally searching for, there is a high probability that all of these part numbers are going to be somewhere close to the tablespace or hard drive that the original part number was on.

Some examples in which you should consider turning FAST Cache off:

If you could take one thing away from this post, and you’ve made it this far. Know where your data is (locality of reference) and try to gather as much information about the size in which things are being written and read from the array.

Please email me or comment if you have any specific questions or ideas to bounce off of me.

…and remember. It’s not all pancakes and banana sandwiches in the storage world.

@sangeek

After looking at FAST VP with our customer for almost a year now, a multitude of conversations with the best people at EMC and probably a few conversations w/ VMware our customer who prefers to remain nameless would like to share his ‘formulas’ with the greater good. You know who you are and next time I see you I owe both you and your boss a beer!

In short, vSphere + Dead Space reclamation is wierd. How VMAX and FAST VP presents it is also very difficult to report and see what is going on. This is why I call it Zombie Space Reclamation because you don’t really know if it is dead!

FAST VP (on Enginuity 5875.198 or higher) will work with VMware 5.x datastores that are defined as a tdev and not fully allocated. It will release space from the anchored tier when it is moved to another tier, but one must understand what values DO and DO NOT change.

1) Current Subscription percent value will NOT change in the anchored tier.
2) Allocated capacity will change in the anchored tier.
3) Total written capacity will NOT match Allocated capacity on the anchored tier.
4) Allocated capacity will be correct on the destination tier.

To correctly determine if FAST VP is working properly and if you have space available in the anchored tier (if you don’t want to oversubscribe), one can use the following formulas:

A person could use this formula to validate FAST VP is releasing space from the anchored tier, if Allocated Capacity equals what the VMAX is saying.
(Capacity of Anchored tier * Current Subscription % of Anchored tier) – Allocated Capacity of Destination tier related to devices using FAST VP and not anchored in the destination tier = Allocated Capacity of Anchored tier.

Capacity of Anchored tier – Allocated Capacity of Anchored tier = Available space on Anchored tier

And here is the biggest issue that we still see going on… in regards to VMAX +VMware ESXi and Storage Vmotion…

A dead space reclamation issue is only related to Storage vMotion when a virtual disk is deleted or migrated to a different datastore…and not releasing it’s space.

This is why I like to call it Zombie Space Reclamation…because you think the space is gone and possibly dead…but it still is there existing on a TDEV someplace and not being reclaimed.

Perhaps there should be a different way vMotion should be working?  Or how it reports back to the VMAX that it should be Zeros?  Time will tell.

**Disclaimer #1 VMAX FAST VP is still very much working well in customer’s environment but the strangeness in regards to how it behaves with VMware is an interesting topic.

**Disclaimer #2  This is not VNX FAST VP.

Before I get into a few details, here are some highlights from last year that I like to brag about…in somewhat chronological order both personal and professional and what I feel like changed the direction my life took.

The most important thing is “Making a list and Checking it Twice”.  Normally reserved for Santa Claus, I started making personal and professional goals for myself, writing it on a sheet of paper and posting it on the wall before entering my bathroom.  I enter my bathroom quite a lot during the day to brush my teeth, take a shower, change my clothes and even to do a little light reading from time to time.  All bathroom humor aside, it is important that this list is at least visible to trigger your self conscious.  You know what is on this list.  Sometimes I even stop to read it, but for the most part it is there as a reminder of things I want to do, things I must do and maybe even ‘things that I might not get but are fun to see on a list’.  For your information, “Winning the lottery and buying a small Caribbean island to live out the rest of my life as Jack Sparrow would do” is still on this year’s list…  Arrrrrrrrgggghhhhh!!!

Some other things from this list (things that I actually did accomplish):

• Get a new job.
• Put a lift kit on my Jeep and go off roading more often.
• Go Camping at least 4 times.
• Go to Colombia and spend time with family more.
• Put wood floors in the house.
• Build my Wife’s Shoe Closet.
• Blog More.

There were a few more…but you get the idea.  The more things you put on the list, the more things you actually do, thus you have a more positive way to think about your life and all of that mumbo jumbo.

So yah, the new job thing.  Varrow found me.  And boy have I been busy.  Highlights since last Varrow Madness.

• Learned more VMAX than I thought I could learn
• Challenged myself and got myself into situations that pushed me out of my comfort zone enough that I had to ask for help….on several occasions.
• Built an ESXi 5 VMware environment in my house with my own storage array, several VMware guests and all sorts of geeky goodies.
• Learned some Isilon.
• Learned more VNX.
• Presented on VMAX.
• Finally got a couple certifications.
• Brought a Keg of IPA to a yearly kickoff/retreat in the mountains and had a blast with Co-Workers.
• Hosted my first #vBeers in Raleigh.

Those are kinda the highlights, but it has been an interesting year.

Here are a couple embedded highlights from my year:

Varrow Madness Flash Mob:

Oh and it you’ve made it this far.  Thank you for reading…and here is a taste of how I will live my life when I when the lottery:

Thanks!

@sangeek