EMC Avamar v5.0 Storage Based Backup Implementation
Closed     Case # 10048     Affiliated Job:  New Trier Township District 2031
Opened:  Tuesday, February 1, 2011     Closed:  Monday, February 28, 2011
Total Hit Count:  52633     Last Hit:  Tuesday, July 23, 2024 4:46:50 PM
Unique Hit Count:  11440     Last Unique Hit:  Tuesday, July 23, 2024 4:46:50 PM
Case Type(s):  Server, Vendor Support
Case Notes(s):  All cases are posted for review purposes only. Any implementations should be performed at your own risk.

After discussion of the future of our backup strategy utilizing Backup Exec; we finally decided to move forward with an EMC Avamar solution. EMC performed an assessment of our network and determined we would be best fit with an Avamar "DS7" grid consisting of a Utility node, Spare node and 5 Data nodes. Under a new platform of Avamar v5.0, each data node includes 3.3 TBs of available storage space.

Since we have nearly virtualized our entire server environment in our array of 5 host servers running the latest and greatest ESX 4.1 vSphere (which we have since learned of EMCs plans to do away with ESX and shift to ESXi for version 5) along with the storage of our infrastructure on an EMC CX-120 - we are good to move forward with a storage based backup solution.

We hired the help of a certified partner of EMC to perform the major install; however, the partner consulting company encountered a great number of difficulties with the hardware provisioned and after a week of troubleshooting, ended up replacing two nodes along with deploying EMC teches to return the grid to a fresh from scratch state.

Therefore, in our planned week worth of time we were only provisioned two actual days, "Day 1" for install and "Day 2" to briefly go over the high level details & job configuration. So below are lessons learned through work I have had to perform to troubleshoot and self-educate various implementations either left uncompleted or missed during the "install."

The consultant performed the install on the nodes individually, then through the "wizard" brought the grid online into a collective state; set the Avamar utility node to trust the root certificate of the vCenter server, setup the initial Avamar proxy (which captures the image level backups of a virtualized machine); imported the VM server environment into Avamar; created the requested datasets, schedules, retention policies; explained the various agents to be installed for File based (rather than image based backups which permits file exclusions, restore permissions with files etc.), SQL, Exchange; LDAP authentication into our A.D. infrastructure and SMTP notifications of "Call Home" to EMC as well as sending us server notifications.

What the consultant failed to explain or setup were a couple items we then figured out on our own. When he finished (on "Day 2") we began an "On Demand" on a couple backups to begin while keeping all others in a disabled state. And before the consultant left we had arranged a return date for him to explain the "how to" restore, review the backed up jobs (success/failure), Avamar reporting, change root/admin passwords, shutting down the grid, backing up of external NAS resources, setup of additional Avamar proxy servers (each proxy performs a single thread - to enhance backup time EMC suggests a proxy per VM host), review of the "System State" File Agent backups on Windows 2003 (Which we discovered were NTBACKUP bkf files being placed on the C volume and in some cases were consuming what little space was left)

To address the above, a great majority had to be performed on our own - simply because our consultant had already spent the 5 days budgeted on our contract building the original units only to find the failed hardware components and therefore had to spend an additional two days to reconfigure the two new nodes with the original 5 other nodes all of which were returned to factory settings after repair. Because of this, the consultant neglected to return on the date arranged and instead the questions were briefly explained through email rather than demonstrated.

Action(s) Performed:
Total Action(s): 4
Action # Recorded Date Type Hit(s) User Expand Details
10165 2/28/2011 2:37:07 PM Server 3906 contact@danieljchu.com Image Level File Level Restores Some of the lessons learned, most i  More ...
10166 2/28/2011 2:36:07 PM Server 3587 contact@danieljchu.com Add Additional Avamar Image Backup Proxies The creation of addition  More ...
10168 2/28/2011 2:35:07 PM Server 3395 contact@danieljchu.com Image Level Backup Failures Image level backups on VMs that have a   More ...
10167 2/28/2011 2:34:07 PM Server 3702 contact@danieljchu.com Backup Performance A couple things to mention, when performing back  More ...

Profile IMG: Footer Left Profile IMG: Footer Right