Getting Geppetto Version 4.3.0 working behind a proxy (a workaround)

Geppetto stands alone as the only Puppet IDE available as of this writing. Initially created by cloudsmith the IDE leverage Eclipse with custom components for developing modules and writing manifests. As of version 4.0 Geppetto has been maintained by Puppet Labs with one of the fixes being that it would work behind a proxy. The idea is that the Puppet components should obey proxy settings set within Eclipse and the reality is that his doesn't work. First of all there are sparse documentation on where and how to set the proxy within Eclipse. In case the next release fixes the bug that I filed regarding this issue, here is where you set the proxy setting within Eclipse;

-Click the Window tab and select Preferences which bring up a new window
-Select Network Connections in the left pane and change Action Provider drop down to Manual
-Edit the proxy info as necessary and hit apply then close window

What happens once the setting is crated is that certain functions does obey the proxy settings except for Java function calls within Eclipse. When an attempt is made to import a module from the forge the error log isn't clear on where it fails, but the log reveals a java process that is making a call to https://forgeapi.puppetlabs.com. The workaround I have in place makes use of Proxifier to direct any traffic going out to http://www.proxifier.com/download.htm to make two hops, the first through Proxifier which then redirects it to the corporate proxy address and port. Here are the steps and settings within Proxifier;

-Click the Profile tab and Select Proxy Server. Enter your corporate proxy's IP and port and select protocol and click OK.
-Click Profile tab again and this time select Proxification Rules. Create a new rule called Geppetto and under Target Hosts enter forgeapi.puppetlabs.com and change the Action drop down to the the Proxy Server you set up in the previous step.

Proxifier will not pipe all traffic that tries to reach forgeapi.puppetlabs.com to the corporate firewall. The con of this process is that it requires a second application to continuously run, but sure beats waiting for my bug to be resolved.

How to get proxy working for Ubuntu 14.04 without GUI (Works for all users and sudo)

By default proxy settings aren't enabled on Ubuntu servers and I've come to find that there are syntax changes in 14.04 that are drastically different compared to their last LTS (Long Term Support) release without it being very obvious. I've crawled the web and have come across various solutions, but those solutions only fix pieces like apt-get and what I wanted is something that works across all users and all calls even when using sudo. One of the things that I figured out was the when you elevate with sudo it doesn't necessarily mean that it will inherit user variables and that was a key discovery that ultimately pointed me on the right path. Without further ado here are the steps. Keep in mind that I prefer vim for a text editor and my examples will reflect that and you can use whichever editor you are comfortable with.

1.) Update your environmental variable

The first step is to create exports in your environmental variables and you can do that by editing /etc/environment

sudo vim /etc/environment

In the file you should add your exports below the PATH= line and the syntax will look like this for each type, http, https, and ftp. (In older versions of Ubuntu you could use all_proxy which I am not sure is still in use and haven't tested for). One you edit for the exports you need, save and exit.

export http_proxy=http://yourproxyserver:proxyserverport

2.) Create your proxy file

As I've mentioned that by default Ubuntu doesn't have proxy settings enabled and this means that there ins't a proxy setting file for apt-get either. You will need to create one and if you are new to Linux, the file be will automatically created when you open a text editor by the name you intend to use for the file.

sudo vim /etc/apt/apt.conf.d/30proxy

Keep in mind that the two digit number before the file name is arbitrary, it can be any number except for ones already in use. If you want to use a number and see what's there you can run this to see what other files are in that directory;

ls /etc/apt/apt.conf.d

Moving on to editing the file, the syntax is highly specific and I would suggest copying and pasting what I have written below. Short of that, pay close attention to double quotes (" ") and semicolons (;) and be sure to close every curly bracket used. Save the file once done and off to editing sudoers file next.

3.) Update sudoers file

The sudoers file needs to be edited to allow it to not only inherit, but to also keep the environmental variables being passed to it that you specified in /etc/environment. The best way to edit the sudoers file located /etc/sudoers is to use viduso. Keep in mind unlike vi or vim it doesn't require you to type "i" to input and to save you press ctrl+x, agree with "y" and take the default .tmp file that visudo wants to use to update the sudoers file. The section that you be editing is towards the top where it lists "Defaults". There you will add additional lines equal to each environmental variable and here is the syntax for adding the http_proxy example from before;

Defaults env_keep += http_proxy

Once you are done editing we will perform some tests to find out if sudo is inheriting the environmental variables. Run the following two commands and the output should be the same for both, if sudo is missing ant of the grep searches than try to reboot first and if still the same issue than double check your syntax in previous edits.

env | grep proxy

and sudo env | grep proxy

Finally, try to run sudo apt-get update or if you have ruby installed sudo gem update and see if the updates run successfully.

Mount NetApp NFS volume to Linux server (for Windows people)

I recently had to update the disk firmware on a NetApp filer running NFS only (CIFS would allow Windows/NTFS access but needs a license and setup, and certain versions of ONTAP does not recommend running both NFS and CIFS) and after reading the documentation, I realized that the doc fell short for those that are not as Linux/Unix savvy and it inspired me to write something about it here.

The gist of the upgrade for a NetApp filer is rather simple, mount the /vol/vol0/ and browse to /etc/disk_fw and copy over thew new firmware bits.....but if you haven't mounted a NFS volume to a Linux server before, you are likely searching Google right now chanting Whiskey Tango Foxtrot as you may have come across a link to this blog here. Time to fill in the gaps on that doc now;

1.) You are going to need a Linux box

I virtualize all of my servers and I happen to have a Fedora 13 VM running and that is what I used to mount the NFS volume to the Linux server. If you don't have a Linux server but have vitualization technology available to you, there are pre-made VMs called appliances and there are many Fedora builds available. If you are the hardware sort of guy or gal, I'll wait, go ahead and build your dinosaur of a physical server while I drink some coffee.

2.) Getting the NetApp NFS volume mounted to Linux (in my case Fedora 13)

You have likely already looked at several webpages and have an idea of what the syntax maybe but just not connecting? No worries, I will explain in full here. First you need to make sure that the Fedora server's /mnt directory has a valid sub directory for the NetApp volume to mount to. The Mount command is Mount [file type] [from where:what directory] [to where] and the [to where] parts needs an existing path. I used WinSCP for an easy GUI browse of the Fedora server and when you first log in, hit the drop down that reads 'root' and go to /[root] which will re-populate the right side split window with more folders and each of those folders is called a directory. Go into the one labeled 'mnt' than press F7 to create a new directory and call is netapp.

Now that you have created the /mnt/netapp directory, you are ready to mount the filer. I use putty to SSH to the Fedora server and here is the command once you have logged in;

mount -t nfs [IP of your netapp filer]:/vol/vol0 /mnt/netapp

The above command declares the file type as NFS by '-t nfs' and [IP of your filer]:/vol/vol0 is the from where:what directory on the NetApp filer and the /mnt/netapp defines what directory on the Fedora server that this volume will be presented to. As a note of clarification, there should be no space between your filer IP and the : above, (e.g. 192.168.1.100:/vol/vol0). As for why /vol/vol0? That's the default on the NetApp filer's directories of where the /etc folder is, which we will be getting to shortly.

To make sure that the NetApp volume mounted, browse the directory by entering ls /mnt/netapp/etc and if an output shows a bunch of files and directories, congratulations, you have mounted the volume successfully as you are viewing the /etc directory on the NetApp filer that has now been mounted to the /mnt/netapp directory on your Fedora server.

For sake of ease, since I already have WinSCP running, I browse to /mnt/netapp/etc/disk_fw and copy over the necessary disk firmware files using WinSCP. From here on out, use the NetApp disk firmware upgrade instructions. Once you have everything you need, run the following command to unmount the NFS volume;

umount -a -t nfs

There are methods to mount the NFS volume directly to Windows using third party software such as NekoDrive but had issues when copying over data as it seems to retain the NFS volume in memory and will require a high memory machine to be able to complete write operations without error. If you are running ONTAP version 8.1 or later, I have come across a blog post by Cosonok regarding how to enable access directly from WinSCP but since I do not have that ONTAP version available to test, I haven't confirmed it's functionality (I can tell you that the method listed on his blog does not work for ONTAP version 7.3.2). The method that I listed works on Fedora and Red Hat servers (tested Fedora 13, 15, and RHEL 6) and if you use other versions of Linux, keep in mind that some of the syntax may change (instead of mount, it may be mnt, mt, etc). Hope this helped solve your NFS mounting issues.

How to export a list of Exchange 2010 Distribution Groups and membership details

I was asked today to make a document which lists all if the email addresses, distribution groups (DG), and DG memberships. If it was a small organization with a small mail server this wouldn't be a huge issue but when the number of DGs are well past a hundred than the task may seem daunting. I had no time to sit here and look through each and every single DG to see who the members are and enter it all in a spreadsheet so I decided use the Exchange Power Shell to see if I can get all of this information.

A quick search on the web reveals that this is indeed possible but methods from MS show how to get the membership information from a single DG and is not dynamic enough to read across an entire organization as it requires you to know the name of the DG and enter it each time. After some trial and error, I finally found out which portion of AD, ADSI, and Exchange values that the Exchange Power Shell can read and here is the script that will output a text file that includes User, User Name, and email address associated with a given DG;

write-output “” > C:\outputDGmembers.txt get-distributiongroup | Sort -Property DisplayName | foreach { $name = $_.displayname $output = ‘Group Name: ‘ + $Name write-output $output >> C:\outputDGmembers.txt Get-DistributionGroupMember $name | Sort -Property DisplayName | Select DisplayName, Alias, primarysmtpaddress >> C:\outputDGmembers.txt write-output “” “” >> C:\outputDGmembers.txt }

If you need more information in the output, you can add the appropriate values after the word 'Select'. As for getting the email addresses, an excellent script is available at Flaming Keys' site which simply works like a charm. Below is a re-post from Flaming Keys, please check out the site and read the whole article which explains each step.

# ------------------------------------------------------------------------------- # Script: Get-AllEmailAddresses.ps1 # Author: Chris Brown http://www.flamingkeys.com/ # Date: 25/07/2011 00:03 # Keywords: Exchange, Email, SMTP # comments: # # Versioning # 25/07/2011 CJB Initial Script # # ------------------------------------------------------------------------------- # Import the E2010 modules if available, otherwise import 2007's. if (Get-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 -Registered -ErrorAction SilentlyContinue) { # Found 2010, add it Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 } else { # Add 2007 Add-PSSnapin -Name Microsoft.Exchange.Management.PowerShell.Admin } # Create an object to hold the results $addresses = @() # Get every mailbox in the Exchange Organisation $Mailboxes = Get-Mailbox -ResultSize Unlimited # Recurse through the mailboxes ForEach ($mbx in $Mailboxes) { # Recurse through every address assigned to the mailbox Foreach ($address in $mbx.EmailAddresses) { # If it starts with "SMTP:" then it's an email address. Record it if ($address.ToString().ToLower().StartsWith("smtp:")) { # This is an email address. Add it to the list $obj = "" | Select-Object Alias,EmailAddress $obj.Alias = $mbx.Alias $obj.EmailAddress = $address.ToString().SubString(5) $addresses += $obj } } } # Export the final object to a csv in the working directory $addresses | Export-Csv addresses.csv -NoTypeInformation # Open the csv with the default handler Invoke-Item addresses.csv

Now back to work on something else...

Unable to delete VMDKs in VMware (vSphere)

I have been trying to P2V a Win 7 desktop and had to make another attempt when it produced an error at 14% completion where the VMW KB workaround suggests to P2V a single volume at a time. The desktop in question has 3 virtual disks plus the Win 7 boot partition so on my next attempt, I went to P2V just the FAT32 boot partition and C:\ when I discovered that the LUN I created on the SAN shows insufficient space...WTF. I launch vSphere client and double check that I deleted the VM that was created during the previous P2V attempt and it was indeed gone. Suspicious, I check the volume to find that there remains 2 VMDKs from the previous P2V attempt that vSphere did not successfully delete nor did it spit out an error about it either. I right click one of the disks and select delete just to get an error that simply states: Cannot delete file....super. Typically an error like this is seen where a VM has locked the volume but there is no VM to speak of so the typical method to unlock a volume will not work here.

Meh.

The way to remove the locked volume where a VM does not exist is through shell and if you haven't poked around on the ESX shell or are unfamiliar with Linux, this may seem like a daunting task but I promise you that it is not. First if you have not already, enable remote shell access to your ESX or ESXi host and once that is established, use a program such as Putty to remotely connect to your target host. If you are running your hosts off of a SAN like myself, it honestly does not matter which host you access so long as that host has access to the SAN volume in question where the VMDKs are located. Once connected via putty, run the following command to figure out where the volume exists;

[root@yourhost ~]# lsof | grep name_of_vmdk.vmdk

The above command will reveal the path of the vmdk assuming that it does exist, the output should look something like this;

vix-async 8586 root 136u REG 0,18 319959334912 377047
/vmfs/volumes/915udud0-6b77a6b7-05a-992ba5sest69/name_of_vmdk.vmdk

Notice the 4 digit number in the above output after vix-async which is the ID number you will use to terminate it's running processes with the following command;

[root@yourhost ~]# kill -9 8586

With all processes pertaining to the ID terminated, we will change the working directory to where the VMDK is for simplicity's sake;

[root@yourhost ~]# cd /vmfs/volumes/915udud0-6b77a6b7-05a-992ba5sest69/

Once there you will delete the VMDK and you will be asked to confirm deletion as a redundancy check to ensure that the deletion is intended by typing Y or N;

[root@name_of_directory ~]# rm name_of_vmdk.vmdk
rm: remove regular file name_of_vmdk.vmdk? y

That should have removed the locked VMDK and have freed up space on your volume.

SQL 2008 R2 SP1 0x84B20001 error

I recently built out a SQL 2008 R2 server and went to apply Service pack 1 and came across error code 0x84B20001 where it specifically error on RS_Server_Adv. Initial digging on the web states that I should run a repair installation of the shared components path with the original media which did not fix the issue and still has the same error. The error occurs at the beginning of the installation of SP1 which suggests that it is checking the installation to verify that it meets requirements such as version and installed components.

The RS_Server_Adv is the Reporting Services component which I installed because it was a specific requirement for a project and one I rarely ever install. Since I suspected that the error was checking installed components first, I crawl the registry to find that there is indeed a registry key called RS_Server_Adv at HKLM\Software\Microsoft\Microsoft SQL Server\MSA=RS10_50.YOURINSTANCENAME with a value of 4. Since it is a brand new build on a virtual machine, I take a snapshot and start altering the value, starting with 0 which did not fix the issue. Rebooted and tried again, still no go. Back to the registry and changed the value to 1 than tried the installation and what do you know, the installation works!

the installation completes and I reboot and ran SQL to make sure everything was working after the value alteration and everything seems to be working just fine.

EVault .CAT issues and things that aren't mentioned in regards to Exchange 2010

The company that I work for have used LiveVault for it's backup needs and even when they were still part of Iron Mountain, it worked only OK most of the time. LiveVault was sold off twice with the final destination being Autonomy and the second part of that sale being that of Autonomy to HP........not that I have a problem with HP but I worry when HP feels like buying companies and entering a market segment such as SaaS (Software as a Service) back up just because they felt like it and had the resources to do so. Beyond the skepticism of HP entering the SaaS game, if you have not used LiveVault's web UI, let me sum it up in a few words: it sucks huge jungle-y d!#k; and it has not even changed at all since the buy-outs. The WebUI alone is enough to drive you away from the product (starting with your sanity) and the fact that it hasn't been improved upon is disturbing, and add to that the less than stellar customer service response and fixes and all the breakages it has.....it's time to change.

After checking out and speaking to a bunch of vendors, we settled on EVault as our new backup vendor since it really does offer very similar features to the competition but we liked the feature set currently available as well as what's in the pipeline from them. One of the requirements that we had that many vendors could not accommodate was that we perform hourly backups on some of our servers and EVault was one of the few who said that they could. After testing various vendors we settled on them.

Before I go on, I came across RestartIT, who as it turns out is a re-seller/vendor of other company's backup technology and do not offer their own and because they did not mention this fact; at one point I was testing RestartIT side by side with EVault and realized that it was really EVault re-branded as their own and I was now running trials of EVault and EVault.... weak. It's not that they are a re-seller that I have an issue with, it's the lack of transparency of the fact that bothered me the most; and in comparison to dealing with EVault directly for customer service and what RestartIT provides for customer service, hands down EVault was a better choice. The worst was when I called them out directly in a conference call with RestartIT that they were re-branding EVault, they kept trying to gloss over the point I made like I didn't even say it.

With any new technology being purchased, what we all hope as a consumer is that it works as advertised but when it comes to the enterprise arena of technology, that hope of it working as told seems to be far more optional than consumer technology. One of the first hurdles that we had to clear during the evaluation stage was the hourly backup requirement; just because it can be done doesn't mean it can be done well. All of the vendors we tried choked on this requirement, there simply is far too much data to process in the hourly backups and to be fair; LiveVault did this one thing well when it was working. Since we need that granularity of a hourly window we had to keep trying various fixes and workarounds and this here is why we chose EVault, they truly worked with us to solve this problem. There are many 'issues' that 'must use workarounds' which must be setup properly for the backup software to work properly, especially with an Exchange 2010 server that sadly is undocumented.

EVault can't use IPv6 on Exchange 2010 at this time.....

I had intermittent Exchange 2010 backup issues and after many hours with EVault support who were just awesome to deal with, finally came to a fix. The main thing was that the backups were inconsistent and would fail to a stoppage due to catalog files that did not sync properly and one of the unmentioned things is that IPv6 is not supported for Exchange 2010 backups. This means disabling IPv6 which I already do for internal corporate servers but it has to go beyond that, it has to go as far as creating a new DWORD value in the Exchange server's registry to stop IPv6 functions from even starting so that the IPv6 components aren't detected by the EVault agent.

There is a Microsoft KB regarding this topic that is an excellent source and here are the highlights of what you need to know;

Run command regedit

and browse down to HKLM\System\CurrentControlSet\Services\TcpIp6\Parameters

There create a 32-bit DWORD value called DisabledComponents and assign the value of ffffffff (yup, that's 8 f's) and restart the server for the new registry value to be read by the server.

Now that the IPv6 mess is sorted out, the vault has to re-sync and re-download the catalog files and the catalog files is the magic sauce that makes the EVault WebUI really snappy by having localized indexes to backups.

Collision during synchronization...

Since we run hourly backups, what can't happen during a synchronization is for a backup to run. One would think a fault tolerant system would know a synchronization is occurring and would not run the scheduled backup until the synchronization completes. Well, this is only partly true. When we first tested EVault, we had errors from deferring, which is exactly what it sounds like. The scheduled backup will defer based on bandwidth allotment as well as other resources and if another backup or synchronization event is running at that time and will defer a schedule until ideal resources/time/bandwidth exist. The issue we had was that there would be so many deferred backups because they were hourly and would stack in cue and the cue would not flush no matter how old/stale a given object maybe; to make it worse, it would go in order of deferred object and it will time out if it has deferred too long. The simple solution if to uncheck the deferring option within EVault and this problem goes away but because we do not use deferring, a scheduled backup will run wither a server is synchronizing to the vault or not......

Speaking of the deferring option, it isn't obvious where it is, on the EVault WebUI, select a server and edit the backup and not the agent (the edit button is to the left of the backup name, next to Logs). A new window will appear, click the Schedule tab, click on a schedule than click the Edit button on the bottom left

A new window will appear on top of the current one called Schedule Details, click the Advance Schedule Options... button which will open yet another window (way to think it through EVault)

Finally in the last window you will see a check box called Deferring which is typically checked by default (unless it's a SQL agent) and un-check that box to remove deferring from your backup schedule.

While we are looking at the images above, EVault does not have a default option for hourly backups and you will need to create a custom schedule in the Schedule Details view. The Custom drop down lets you edit specific times for a backup to run, and you can see in the one I have set up a value of 15/6-18/*/*/* which means it is to run on the :15 of every hour (12:15, 1:15, 2:15 and so on) from 6 A.M. to 6 P.M. every single day. As you can see in the image, there is a explanation of the values to set up an even more specific schedule using days of the week, dates of the week, and months as additional editable values.

If you are like my setup and have deferring turned off and need to manually synchronize your backup, don't forget to disable your backup schedule and to terminate any running backup process before doing so. If you forget to do this, simply stop the services, than go to the agent installation path where the catalog files are and either rename or delete the folder for that backup (it will be names the same as you named for your backup).

Backup name and corresponding folder

About those .CAT files.....

Those .CAT files, or catalog files that makes it possible to have a fast WebUI is a localized indexes of backups, yes a catalog of backups; but these are just pointer files and not the entire backup (That's what an ERA is for and that is another conversation all together). What is not said is that the catalog files will be stored at wherever the agent installation is and the default installation path is 'C:\Program Files'. The problem with this is that the catalog files will grow as the more backups occur and depending on your retention plan, they may remain for a short time or can be there for a while as well the backup interval creating as many catalog files as there are backups. These catalog files vary in size depending on the size of the backup and if it was a deltized file or not where a deltized backup is typically smaller than a full backup. To give you an idea, on my Exchange server's mailbox level back up of 94.7GB, I have 4.06GB in .CAT files that range in size form 0KB out to 359,030KB at the largest, most being in the ~340,000KB range.

What the .CAT files look like

With the growing catalog files comes a shrinking disk space and if on C:\, that can be disastrous for your operating system and ultimately your server and at worst, your client facing applications. EVault backups do have a fail safe where if the drive containing the catalog files fall below 15% of the volume's overall capacity, the backups will cease but the error in the log is not obvious to this fact. Moving the installation to another path isn't obvious either, to move the path you will need the agent installer so if you've already deleted, go get the same agent version again. When you launch the installer, you are simply going to choose to change (I think that that is what the option is called) the installation versus a new installation and follow the prompts. At one point you will need to choose the installation path and that is how you move the agent installation. Before doing this, you will need to stop EVault services and stop all running processes and once you moved the path, copy the catalog files over than start services and synchronize the server on the WebUI.

I hope that this post has helped those of you that are setting up EVault and to those that are considering it, once you get past the initial growing pains of new technology in the enterprise realm, it's a great product. No matter which backup vendor you choose, you will have to work out some issues but EVault was by far the best to work with.