Postfix queue management

Haven’t touched Postfix in a long time since I do very little administration work anymore, but recently found a server that had a ton of mail queued up.

The way I used to manage it was with qvmenu.pl. You could find it here – http://taz.net.au/postfix/scripts/qvmenu.pl – it shows a graphical (curses based) user interface that allows you to select messages, read them, delete them, etc.

What if I wanted to really delete a ton of messages though? I did a quick search and found http://www.howtoforge.com/delete-mails-to-or-from-a-specific-email-address-from-postfix-mail-queue and modified the command to work for me. I decided to run these commands:
mailq | tail +2 | awk ‘BEGIN { RS = “” } / MAILER-DAEMON*/ { print $1 }’ | tr -d ‘*!’ | postsuper -d -
mailq | tail +2 | awk ‘BEGIN { RS = “” } / root@wuhan\.shocknetwork\.com$/ { print $1 }’ | tr -d ‘*!’ | postsuper -d -

This way, I’m getting rid of all of the bounce and double bounce messages and also the ones from root that probably aren’t important.

if vCenter Server Heartbeat or Neverfail Heartbeat failover appears to be not working or taking a long time …

The main symptom of this problem would be that the vCenter Server Heartbeat console or Neverfail Management Client console would show that the services had failed over, but if you were to try to ping it, it wouldn’t respond.
Logically, there there are some hypotheses you could come up with:

1) Network packet filter isn’t revealed on the active server, so we can’t connect to it.
2) Something wrong w/ the service.
3) The console is wrong and on the backend, nothing failed over.

These would be all wrong.

What we found was that it was an issue with ARP caching on the switches. Because the VM or host abruptly fell off the network, the switches hadn’t expired the ARP entries and that they were stale. You would think that it would be fixed in a minute after the ARP entries expired, but I guess the chain could take a little longer.

Probably the best way to troubleshoot this would be to get on a host on the same network segment and try a ping. If that fails, you could run “arp -a” and check to see if you indeed have the right mac address of the host you want to connect to. If not, you could probably log into the switch to delete the entry or you can create a task to run the command during switchover:

“C:\Program Files\VMware\VMware vCenter Server Heartbeat\R2\bin>nfpktfltr.exe arp”

You should then see the switchover happen without the long delay.

What in the world is irlwinning.com? DDOS?

Recently, I set up my own DNS server. I hadn’t run a public DNS server in years. Since the tvpads recently had some DNS issues, I thought maybe I could help eliminate some support calls by running my own DNS server, pointing to the right servers. Boy was I wrong! For some reason, even though some others on comcast would point to my server as a DNS server, they would still get answers that were not the answers given from my server! It was so bizarre! I had never seen it before. If they ran nslookup and used “server <DNS Server IP>” and typed in the name they wanted to resolve, it’s almost as if the server statement prior was ignored and they were getting the IP that the ISP wanted to give them.

Anyways, that’s not the problem I’m writing about here. Surfing some websites became slow for some reason and I thought I would investigate. The first thing I went to see was what connections I had to the outside world. I went to the router and looked at the traffic. Here’s what I saw:
dns
Obviously, that’s DNS traffic. Well, go to the DNS server and what do I see? This:

10-Oct-2013 15:34:14.228 queries: client 54.252.236.155#58070: query: irlwinning.com IN ANY +E
10-Oct-2013 15:34:14.670 queries: client 206.220.43.92#26073: query: irlwinning.com IN ANY +E

Many different lines of the same exact query. What is it? I have no idea. I’ve decided just to shut down DNS queries for now, but if anyone knows anything about this, I’d be happy to hear from you.

Thanks!

WordPress stuck on “Briefly Unavailable for Scheduled Maintenance”

This was interesting …

Logged into WordPress and found that I had a few plugins that could be upgraded. I clicked to “upgrade all” and clicked away like I normally do – it usually just finishes upgrading with no big deal. Unfortunately, this time, it got stuck. It said “Briefly unavailable for scheduled maintenance. Check back in a minute.” Well, after about 10 minutes, I got a little impatient and hit google.

Turned out that when upgrading, WordPress creates a file called .maintenance at the root level of your install (in the directory containing wp-admin).

If you take a look at what’s in the file, it looks like this:

cat .maintenance 
<?php $upgrading = 1377922367; ?>

It’s probably safe to just delete the file, but I thought I would rename it just in case I might need it. I renamed it and the blog came back and it appeared that the updates completed too. After doing that however, I renamed it back and it didn’t do anything. Kind of strange, but looks like I don’t have a problem, so I’m happy with it.

OPS1 – VMware Management app for the iPhone – Fantastic!

I’ve been using this app for quite some time, but haven’t found the time to write about it.

If you use an iPhone or iPad and manage a vSphere environment, you’ll want this app. You can get it here: OPS1 – VMware and Amazon AWS Cloud Management for …

It’s made by a company called Spragos based out of Santa Clara, CA. You can find their website here: http://www.spragos.com.

It’s pretty awesome that I could manage my vSphere hosts and VMs without having to power on the laptop. Since I’m on a Mac, I don’t enjoy bringing up the vSphere thick client and even the web client takes quite some time to load. Most of the time, I just need to power on or off a VM or shutdown a host anyways. This app has allowed for me to do these things without having to power on my laptop or even if I’m on the laptop, I don’t need to start up Fusion for the client and I’m loving it.

Here are some screen shots. You can configure a single or multiple hosts – connect to vCenter or an ESX host directly. It will also cache credentials. Since I’m not necessarily in a super secure environment (my home lab), I don’t care much about security. I hate having to type my password in over and over just to log in or even my user name for that matter.

After logging in, here’s my home screen. From here, I usually head over to Virtual Machines or Hosts, depending on what I want to do.

IMG_2026

I’m I’m interested in what’s going on overall, I would navigate to Status. Here, I could see at a high level that everyone’s going just fine with my host.

IMG_2032

It’s not always this way though – see, it pulls events and alarms from vCenter Server.

IMG_2400-1

If you go into VMs, you can see a nice list of the VMs:

IMG_2027

Then, you can drill into the properties of the VM and see what’s going on, make changes, power on or off, etc

IMG_2029

IMG_2028

If a VM was suddenly unresponsive for some reason, maybe the CPU stats could give you a clue as to what was going on. In my case, I just had a couple of spikes.

IMG_2030

I think you get my point. It’s a great app! Download it free and try it yourself. I honestly feel that the value of the free version is well worth the measly $10 for upgrading to the Enterprise version. It’s probably saved me hours of time if you aggregate the couple of minutes it takes to start up the mac, start up fusion, start up the vSphere client and then logging into the ESX or vCenter server.

Here’s a few other screenshots just for eye candy’s sake.

Adventures in the quest to install Lion on my MacBook Pro …

This was pretty fun. I had to install a new HD in my Mac. Since Lion was out and I was still on Snow Leopard, I figured I might as well upgrade that too. I downloaded and burned it on a dvd. I figured, After swapping out the hard drive, I could install from DVD. Unfortunately, the DVD didn’t read. Damn! It was a pain to swap the disks and I didn’t want to do it again. So I tried booting from USB – attached the old disk to a usb enclosure and it booted! Linux couldn’t do that! Windows couldn’t do that! This was awesome. From there, I was able to just install Lion via the downloaded dmg file and pointed the install to the newly installed disk.

By the way, trying to burn a dmg file on Linux or Windows is a pain in the ass. I tried with both and gave up after about an hour of google searches and trying different converter products.

I’m starting to hate Sony laptops!

I guess it’s not just Sony. It’s just managing device drivers. Windows 7 has fixed most issues, but man … keyword = MOST!

A friend asked me to upgrade her laptop to Windows 7. It’s a Sony VGN-FJ67C/R. It’s a laptop from China, so you probably won’t run into the exact same one, but you might run into a similar issue with one from here.

So, I installed Windows and it found a few drivers – the ones that were missing were the Wireless network, video, sound, camera, and mass controller. I’ll thank God that Windows detected and installed the driver for the network. Had it not done that, I’m sure finding it on the net first would’ve been a pain. I started with a Windows update, then went through the device manager and started my search for drivers one at a time. The first was the video driver. Ven 8086 and dev 2592 sent me to Intel’s site and told me that it was a 915GM. Unfortunately, Intel didn’t make a driver for Windows 7 on it. Damn it! Searched around a little bit … couldn’t find it. Well, not a show stopper. Let’s move on. So, updates finished installing and I restart the machine. Somehow, magically, the video card driver starts to install. Awesome! Well, I find that the audio driver is installed too. The damn sound doesn’t work though! Oh man … this will be a pain … well, after an hour long exhaustive search (Sony doesn’t have drivers – HP does, but they don’t work), I just go to RealTek.com and install the RealTek HD drivers from there. Magically, it works! It’s Ven 10EC Dev 0260 if you’re interested. That was a pain in the ass! Last, but not least, I had the mass controller – ven 104c and dev AC8E – TI PCI-7×20/6×20. It wasn’t too bad. I just installed the Windows XP driver and it worked! Actually, some site said that they tried it, so I just followed suit. Superb! :)

This took a little longer than expected, but man… I guess it was satisfying. :)

Hope a person that reads this doesn’t spend as much time as I did! :)