EMC Avamar Tape-Out Option Closed     Case # 10051     Affiliated Job:  New Trier Township District 2031 Opened:  Tuesday, March 1, 2011     Closed:  Friday, March 25, 2011 Total Hit Count:  50446     Last Hit:  Friday, April 19, 2024 2:23:21 PM Unique Hit Count:  10745     Last Unique Hit:  Friday, April 19, 2024 2:23:21 PM Case Type(s):  Server Case Notes(s):  All cases are posted for review purposes only. Any implementations should be performed at your own risk. Project: Avamar Tape Out Option It's only been a couple weeks since EMC implemented our Avamar to Networker/Backup to Tape option... In our environment we decided to archive within Avamar 30 dailies, 3 monthlies and archive the month ends out to tape for a more extended period of retention while not reducing our storage within Avamar. There are a couple variations to select from for the Tape out Option. -    ATO "Avamar Tape-Out:" Re-Hydrate the data to a server attached to a tape library with Networker installed, Avamar pushes the data to the server and then calls Networker to archive the data to tape -    ADT "Avamar Data to Tape(?):" Variations of how to take data directly out of Avamar and store this already data duped data out to tape. The first option is what we selected, I am explained that the other option "ADT" is a much more involved process of restoring data back. The "ATO" option, which we have now used for a full year, has worked extremely well. The initial setup, confusing and incomplete - we have worked to fit our needs and customized based on our environment. The biggest variance between these options - ATO is very portable to restore the data and very quick; I am explained ADT pushes data back into Avamar, is more involved and time consuming to restore data. On the flip side, ADT requires less tapes since the data is data duped; however, unlike what we are doing, ATO offers the ability to perform incrementals rather than fulls; however, since we perform ATO on VM image level backups, SQL databases and only once a month, full ATO works best for us. SQL or Exchange agent level backups - they are re-hydrated and then stored to tape. Originally we were explained by EMC that to restore from tape a Avamar client based backup of SQL or Exchange, would require the assistance of EMC; however, we later proved this incorrect. On several test restores, I have taken the f-0 etc files and restored them manually - they appear to be like SQL dump files - throw on a .bak (not required, but then SQL sees them while browsing) and SQL will restore them right back into the environment. We have not tested nor had the need to restore an Exchange database; however, I have had direct confirmation that this as well may be restored without the need to involve EMC. So I wanted to document some of the process I have learned while troubleshooting various issues we have encountered during our initial implementation. The consultant from EMC created the overall process - the server used for ATO was an old system we beefed up with large cheap storage and our previous tape library. Originally, we were setup with a single "batch" that called each of the groups; however, we found, for our situation calling each one individually over a span of 7 days worked out best. If the next group is called during a maintenance window - the group would fail - we were explained this was because some data is written to the Avamar grid; therefore, breaking the groups out and calling each right after a maintenance window over a span of 7 days proved to work best for our retention plan. Action(s) Performed: Total Action(s): 3 Action # Recorded Date Type Hit(s) User Expand Details 10178 3/25/2011 10:59:02 AM Server 3695 contact@danieljchu.com Utility Node: From a file level configuration view on our Utility N  Collapse ... Last Hit: Friday, April 19, 2024 11:48:23 AM Utility Node: From a file level configuration view on our Utility Node our setup is as follows: -    /usr/local/avamar/etc/ -    atoclient.cfg    o    This possess the group configurations i.e. what servers are in a group, file/path exclusions, path on the Networker server to restore to -    atobatch-monthend.cfg    o    The month end is scheduled in Cron and calls each of the below .cfg files, each of the below files include the specific parameters i.e. the type of restore i.e. VM image, file, sql or exchange agent level restores, the specific group that should be called to restore, -last or -first to restore, among other configurations with regard to how to process the group o    [Update]: We found running together as a single process would at times fail. If a group is initiated during a maintenance window, information cannot be written to the Avamar grid and therefore the process fails; therefore, we have since broken out the below to run over a span of 7 days o    atobatch-fileserver.cfg o    atobatch-database.cfg o    atobatch-vmimage.cfg o    atobatch-exchange.cfg -    I have manipulated these files directly while troubleshooting with a tech from EMC and have found configuring already existing details is much easier in these files directly rather than through the ATO GUI. From ATO: -    Configuration Manager    o    Create/Maintain Groups, Groups are both the set of servers grouped together as well as basic settings about what exclusions and time sets to grab out of Avamar o    So you want to group together commonalities - i.e. a group should be database only or exchange only or virtual machine image only servers... also the group should be the same as to the exclusions set i.e. fileserver01 group may be made up of 3 servers all of which will exclude the C drive while fileserver02 group maybe another 3 servers which exclude the P drive... etc... o    If a group consists of only 1 server which you plan to replace with another - be sure to first add the new server before removing the only server in the group... otherwise the group will be removed once all servers exit the group (so we are told; however, in atoclient.cfg appears to continue to possess the prior groups even after all servers are removed - but the settings of exclusions and so forth are lost when the last server is removed) -    Batch Manager    o    The first menu displays the logs and color codes what jobs have run and their status - white is running, red has failed/cancelled, green was successful; this area also lets you cancel a running job o    The "m" mode, this menu allows you to schedule new jobs into "Cron", execute a job and create/modify/delete existing jobs - the jobs are the cfg files above and here you can change their configuration; however, I think from the file level it is a little more flexible Command Line: -    Define a job on a single group member    o    ato -gid fileserver -select -client server.domain.com -last -    Execute the defined job above    o    ato -gid fileserver -tapeout -    Execute the defined job above in a debug mode pushing runtime to a file log.txt    o    ato -gid fileserver -d -d -d -d -d -tapeout > log.txt 2>&1 -    Execute the fileserver group only initiating the Networker tape out options defined by the group, not the restore also in debug mode pushing runtime to a file log.txt    o    ato -gid fileserver -d -d -d -d -d -tapeout -tapeonly > log.txt 2>&1 10177 3/25/2011 10:59:02 AM Server 3898 contact@danieljchu.com Backup Client: Some of our servers being backed up to the grid - on  More ... 10176 3/25/2011 10:59:02 AM Server 3568 contact@danieljchu.com Networker Server: We kept receiving errors with regard to the SSL c  More ... Outcome: [Contributed: 2/21/2012] In conclusion, I want to stress how happy we have been both with Avamar and the ATO option. We have been fortunate, over the past year, to have only required data restored out of Avamar directly; however, we have performed trial restores off tape and the process has been quite simple. Restoring SQL, files and helping to keep storage in our Avamar grid just below the 80% threshold - the ATO option has helped us retain a proper retention policy.