EVault .CAT issues and things that aren't mentioned in regards to Exchange 2010

The company that I work for have used LiveVault for it's backup needs and even when they were still part of Iron Mountain, it worked only OK most of the time. LiveVault was sold off twice with the final destination being Autonomy and the second part of that sale being that of Autonomy to HP........not that I have a problem with HP but I worry when HP feels like buying companies and entering a market segment such as SaaS (Software as a Service) back up just because they felt like it and had the resources to do so. Beyond the skepticism of HP entering the SaaS game, if you have not used LiveVault's web UI, let me sum it up in a few words: it sucks huge jungle-y d!#k; and it has not even changed at all since the buy-outs. The WebUI alone is enough to drive you away from the product (starting with your sanity) and the fact that it hasn't been improved upon is disturbing, and add to that the less than stellar customer service response and fixes and all the breakages it has.....it's time to change.

After checking out and speaking to a bunch of vendors, we settled on EVault as our new backup vendor since it really does offer very similar features to the competition but we liked the feature set currently available as well as what's in the pipeline from them. One of the requirements that we had that many vendors could not accommodate was that we perform hourly backups on some of our servers and EVault was one of the few who said that they could. After testing various vendors we settled on them.

Before I go on, I came across RestartIT, who as it turns out is a re-seller/vendor of other company's backup technology and do not offer their own and because they did not mention this fact; at one point I was testing RestartIT side by side with EVault and realized that it was really EVault re-branded as their own and I was now running trials of EVault and EVault.... weak. It's not that they are a re-seller that I have an issue with, it's the lack of transparency of the fact that bothered me the most; and in comparison to dealing with EVault directly for customer service and what RestartIT provides for customer service, hands down EVault was a better choice. The worst was when I called them out directly in a conference call with RestartIT that they were re-branding EVault, they kept trying to gloss over the point I made like I didn't even say it.

With any new technology being purchased, what we all hope as a consumer is that it works as advertised but when it comes to the enterprise arena of technology, that hope of it working as told seems to be far more optional than consumer technology. One of the first hurdles that we had to clear during the evaluation stage was the hourly backup requirement; just because it can be done doesn't mean it can be done well. All of the vendors we tried choked on this requirement, there simply is far too much data to process in the hourly backups and to be fair; LiveVault did this one thing well when it was working. Since we need that granularity of a hourly window we had to keep trying various fixes and workarounds and this here is why we chose EVault, they truly worked with us to solve this problem. There are many 'issues' that 'must use workarounds' which must be setup properly for the backup software to work properly, especially with an Exchange 2010 server that sadly is undocumented.

EVault can't use IPv6 on Exchange 2010 at this time.....

I had intermittent Exchange 2010 backup issues and after many hours with EVault support who were just awesome to deal with, finally came to a fix. The main thing was that the backups were inconsistent and would fail to a stoppage due to catalog files that did not sync properly and one of the unmentioned things is that IPv6 is not supported for Exchange 2010 backups. This means disabling IPv6 which I already do for internal corporate servers but it has to go beyond that, it has to go as far as creating a new DWORD value in the Exchange server's registry to stop IPv6 functions from even starting so that the IPv6 components aren't detected by the EVault agent.

There is a Microsoft KB regarding this topic that is an excellent source and here are the highlights of what you need to know;

Run command regedit

and browse down to HKLM\System\CurrentControlSet\Services\TcpIp6\Parameters

There create a 32-bit DWORD value called DisabledComponents and assign the value of ffffffff (yup, that's 8 f's) and restart the server for the new registry value to be read by the server.

Now that the IPv6 mess is sorted out, the vault has to re-sync and re-download the catalog files and the catalog files is the magic sauce that makes the EVault WebUI really snappy by having localized indexes to backups.

Collision during synchronization...

Since we run hourly backups, what can't happen during a synchronization is for a backup to run. One would think a fault tolerant system would know a synchronization is occurring and would not run the scheduled backup until the synchronization completes. Well, this is only partly true. When we first tested EVault, we had errors from deferring, which is exactly what it sounds like. The scheduled backup will defer based on bandwidth allotment as well as other resources and if another backup or synchronization event is running at that time and will defer a schedule until ideal resources/time/bandwidth exist. The issue we had was that there would be so many deferred backups because they were hourly and would stack in cue and the cue would not flush no matter how old/stale a given object maybe; to make it worse, it would go in order of deferred object and it will time out if it has deferred too long. The simple solution if to uncheck the deferring option within EVault and this problem goes away but because we do not use deferring, a scheduled backup will run wither a server is synchronizing to the vault or not......

Speaking of the deferring option, it isn't obvious where it is, on the EVault WebUI, select a server and edit the backup and not the agent (the edit button is to the left of the backup name, next to Logs). A new window will appear, click the Schedule tab, click on a schedule than click the Edit button on the bottom left

A new window will appear on top of the current one called Schedule Details, click the Advance Schedule Options... button which will open yet another window (way to think it through EVault)

Finally in the last window you will see a check box called Deferring which is typically checked by default (unless it's a SQL agent) and un-check that box to remove deferring from your backup schedule.

While we are looking at the images above, EVault does not have a default option for hourly backups and you will need to create a custom schedule in the Schedule Details view. The Custom drop down lets you edit specific times for a backup to run, and you can see in the one I have set up a value of 15/6-18/*/*/* which means it is to run on the :15 of every hour (12:15, 1:15, 2:15 and so on) from 6 A.M. to 6 P.M. every single day. As you can see in the image, there is a explanation of the values to set up an even more specific schedule using days of the week, dates of the week, and months as additional editable values.

If you are like my setup and have deferring turned off and need to manually synchronize your backup, don't forget to disable your backup schedule and to terminate any running backup process before doing so. If you forget to do this, simply stop the services, than go to the agent installation path where the catalog files are and either rename or delete the folder for that backup (it will be names the same as you named for your backup).

Backup name and corresponding folder

About those .CAT files.....

Those .CAT files, or catalog files that makes it possible to have a fast WebUI is a localized indexes of backups, yes a catalog of backups; but these are just pointer files and not the entire backup (That's what an ERA is for and that is another conversation all together). What is not said is that the catalog files will be stored at wherever the agent installation is and the default installation path is 'C:\Program Files'. The problem with this is that the catalog files will grow as the more backups occur and depending on your retention plan, they may remain for a short time or can be there for a while as well the backup interval creating as many catalog files as there are backups. These catalog files vary in size depending on the size of the backup and if it was a deltized file or not where a deltized backup is typically smaller than a full backup. To give you an idea, on my Exchange server's mailbox level back up of 94.7GB, I have 4.06GB in .CAT files that range in size form 0KB out to 359,030KB at the largest, most being in the ~340,000KB range.

What the .CAT files look like

With the growing catalog files comes a shrinking disk space and if on C:\, that can be disastrous for your operating system and ultimately your server and at worst, your client facing applications. EVault backups do have a fail safe where if the drive containing the catalog files fall below 15% of the volume's overall capacity, the backups will cease but the error in the log is not obvious to this fact. Moving the installation to another path isn't obvious either, to move the path you will need the agent installer so if you've already deleted, go get the same agent version again. When you launch the installer, you are simply going to choose to change (I think that that is what the option is called) the installation versus a new installation and follow the prompts. At one point you will need to choose the installation path and that is how you move the agent installation. Before doing this, you will need to stop EVault services and stop all running processes and once you moved the path, copy the catalog files over than start services and synchronize the server on the WebUI.

I hope that this post has helped those of you that are setting up EVault and to those that are considering it, once you get past the initial growing pains of new technology in the enterprise realm, it's a great product. No matter which backup vendor you choose, you will have to work out some issues but EVault was by far the best to work with.