networking | ShocKNetworK Blog

Troubleshooting a Mysterious Networking Issue in Windows 11 (NOT!)

Networking issues can be frustrating and time-consuming to troubleshoot. This was just one of my many experiences troubleshooting an interesting network issue that took me a while to solve.

The Problem: One day, I noticed that my computer’s network connection was acting up. The network interface card (NIC) was sending packets just fine, but it was receiving very few packets, and eventually, it would stop receiving packets altogether. At first, I suspected that the issue happened after I installed the Insider Preview of Windows 11, so I reset Windows. I updated the Realtek NIC driver to the latest version, hoping that it might help. The problem persisted.

The Troubleshooting: Next, I decided to reinstall Windows 11 from scratch, thinking that it might fix the issue. The problem still persisted even after the fresh install. Now I knew that the issue was likely to be hardware.

I boot into Linux from a USB drive. To my surprise, the issue persisted even in Linux. This ruled out any software or driver issues with Windows.

The Solution: I started to suspect that the issue might be with my Wi-Fi access point. I have a TP-Link Deco 6E mesh Wi-Fi system, and one of the access points acts as the main router. I decided to swap the problematic access point with another one, and to my relief, the issue disappeared instantly. My NIC was now sending and receiving packets normally, and I was back online.

Conclusion: Networking issues can be tricky to troubleshoot, and it’s easy to get lost in a sea of software and driver issues. Sometimes, the problem might not even be with your computer at all, but with your network equipment. If you’re experiencing a similar networking issue, try ruling out all software and driver issues first, and then focus on your network equipment. Hopefully, my experience will save you some time and frustration.

irqbalance or set_irq_affinity – interesting cause for a network performance issue.

When it comes to high-performance computing, squeezing every bit of performance out of the system is crucial. One of the critical factors in achieving high performance is reducing system latency. Interrupt requests (IRQs) are a type of signal generated by hardware devices that require attention from the CPU. By default, IRQs can be delivered to any CPU core in a multi-core system. This can lead to cache misses and contention, ultimately leading to increased latency. Fortunately, there are tools available to help manage IRQ affinity and reduce latency, such as irqbalance and set_irq_affinity. https://github.com/majek/ixgbe/blob/master/scripts/set_irq_affinity

irqbalance is a Linux daemon that helps to balance IRQs across multiple CPU cores to reduce latency. By default, irqbalance distributes IRQs across all CPU cores, which is a good starting point. However, depending on the system configuration, it may be necessary to adjust IRQ affinity further to optimize performance.

set_irq_affinity is a script that allows users to set IRQ affinity for specific hardware devices. The script can be used to specify which CPU cores should receive IRQs for a specific hardware device, reducing the chance of cache misses and contention. Set_irq_affinity requires root access to run and must be executed for each device on the system.

To use set_irq_affinity, first, identify the device’s IRQ number using the “cat /proc/interrupts” command. Once the IRQ number has been identified, run the set_irq_affinity script, specifying the IRQ number and the desired CPU cores. For example, to set the IRQ affinity for IRQ 16 to CPU cores 0 and 1, run the following command:

sudo set_irq_affinity.sh 16 0-1

This command tells the kernel to route IRQ 16 to CPU cores 0 and 1.

Keep in mind that setting IRQ affinity is a delicate balance. Setting IRQ affinity for too few CPU cores can result in increased latency due to increased contention for those cores. On the other hand, setting IRQ affinity for too many CPU cores can result in inefficient cache usage and increased latency due to cache misses.

In summary, managing IRQ affinity is an important aspect of optimizing system performance, particularly in high-performance computing environments. The irqbalance daemon can help to balance IRQs across multiple CPU cores, while set_irq_affinity allows users to specify the IRQ affinity for specific hardware devices. By carefully managing IRQ affinity, users can reduce latency and achieve better system performance.

What’s NonRootPortBinding? I just want to run my web server on port 443!

In the Unix world, privileged ports are 1-1024. As a non-root user, you’re not allowed to start a service and listen on them.

So, how do web servers work then? They usually use ports 80 and 443.

There are a few ways around this. The most common is that the process is started as root and then downgraded.

If you want to start a process without ever having root access though, the way to do it is with NonRootPortBinding. You can find information about it using Apache here.

Basically, for any process you want to start on a port under 1025, you can run:

setcap cap_net_bind_service=+ep <path to binary>

Following that, you can confirm that you’ve set the correct permission by running:

getcap <path to binary>

It should return with: cap_net_bind_service+ep

When you patch or update the binary, you will need to rerun the setcap command.

Hope this helps!

Rebroadcast your neighbor’s wifi for yourself (wifi extender) with Tomato firmware

My parents recently swapped Internet providers and since they didn’t know that it would take a week for the application to be completed, they were out of Internet service for about a week. The neighbors graciously allowed them to use theirs, but the signal didn’t reach the entire house. To make it reach, we configured the router to rebroadcast their wifi. If you’re going to be doing this, please make sure you get permission first!

The easiest way to do this is to just get one of those wifi extenders. We just didn’t happen to have any at the time. Since the router was Tomato compatible, I first flashed the router with tomato. The screenshots you’re seeing are Tomato by Shibby, just with a custom skin.

To do this, you first need to find out what IP address range you can use. I did this just by connecting a laptop to their wifi. Turned out that the IP address their DHCP server gave me was 192.168.7.x. I tried to ping 192.168.7.253 to make sure it wasn’t taken and sure enough, it wasn’t. I assigned 192.168.7.253 to my router.

Next, I needed to disable DHCP. You don’t want your DHCP competing with the neighbor’s. Lastly, use the default gateway that you get from their DHCP server. In my case, it was 192.168.7.1. You can use the DNS server from them also or you can use others. I like Quad9’s 9.9.9.9 or Cloudflare’s 1.1.1.1 or Google’s 8.8.8.8.

After that, you can match up your wifi settings with theirs’ so that it can connect. Use the exact same SSID, shared key, and use “Wireless Ethernet Bridge” for the Wireless Network Mode.

Lastly, optionally, you can put up any your own wifi settings as virtual wifi settings so that you don’t need to reconfigure any of your own devices.

The virtual setting is the wl0.1. Just add it and that’s it!

That’s all you need to do to make your own Tomato Wireless Extender. This has much better range than a regular wifi extender and was available at the time.

Protect your home network using TomatoUSB – how to only allow only HTTP/S out!

While we continue to see the WannaCry and other malware around, I thought I would secure my own network. Since I allow visitors onto their networks, I figured I would configure all new DHCP’d hosts to access the Internet only via HTTP and HTTPs and not allow them to use any DNS servers other than OpenDNS. Here’s how to do it:

The first thing I did was create an access restriction. I did this just to see what chain would be created and I would put subsequent rules into that chain.

The previous screenshot created this chain:

Chain rdev07 (1 references)
target prot opt source destination
DROP all -- 192.168.0.15 anywhere

With this chain, I can add additional rules. The first thing I want to do is allow only DNS access to OpenDNS servers and none other. For this, I would run the following commands:

iptables -A rdev07 -4 -p tcp -s 192.168.0.0/24 -d 208.67.222.222/32 --dport 53 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 192.168.0.0/24 -d 208.67.222.222/32 --dport 53 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 192.168.0.0/24 -d 208.67.220.220/32 --dport 53 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 192.168.0.0/24 -d 208.67.220.220/32 --dport 53 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 192.168.0.0/24 -d 0.0.0.0/0/0 --dport 53 -j REJECT
iptables -A rdev07 -4 -p udp -s 192.168.0.0/24 -d 0.0.0.0/0/0 --dport 53 -j REJECT

These rules basically allow DNS queries from my network to the 2 OpenDNS servers. The last 2 rules mean that no other DNS servers outside of those 2 servers can be queried. The reason I do this is because there is some malware out there that will change the DNS servers to query on Windows, effectively overriding the DHCP setting. An alternative to this would be to configure Tomato to intercept DNS requests, but I would rather do it this way.

I added the following rules because I had noticed for some reason that some connections coming back from OpenDNS were dropped. I think they’re optional, but I put them in.

iptables -A rdev07 -4 -p tcp -s 208.67.222.222/32 -d 192.168.0.0/24 --sport 53 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 208.67.222.222/32 -d 192.168.0.0/24 --sport 53 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 208.67.222.222/32 -d 192.168.0.0/24 --sport 53 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 208.67.222.222/32 -d 192.168.0.0/24 --sport 53 -j ACCEPT

Next, I go to create my whitelist – this would be my iPhone, iPad, android, etc – any hosts that I trust. I’m going to allow these host to go out to any host with TCP and UDP.

iptables -A rdev07 -4 -p tcp -s 192.168.0.3/32 -d 0.0.0.0/0 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 192.168.0.11/32 -d 0.0.0.0/0 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 192.168.0.31/32 -d 0.0.0.0/0 -j ACCEPT

iptables -A rdev07 -4 -p udp -s 192.168.0.3/32 -d 0.0.0.0/0 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 192.168.0.11/32 -d 0.0.0.0/0 -j ACCEPT
iptables -A rdev07 -4 -p udp -s 192.168.0.31/32 -d 0.0.0.0/0 -j ACCEPT

I know that they can still get viruses. I hope they don’t. They can only use OpenDNS for DNS services, but they can access basically anything outside on any port.

Lastly, I configure the rules to allow only HTTP and HTTPs out.

iptables -A rdev07 -4 -p tcp -s 192.168.0.0/24 -d 0.0.0.0/0 --dport 80 -j ACCEPT
iptables -A rdev07 -4 -p tcp -s 192.168.0.0/24 -d 0.0.0.0/0 --dport 443 -j ACCEPT
iptables -A rdev07 -4 -p all -s 192.168.0.0/24 -d 0.0.0.0/0 -j DROP

With this, anyone else on the network can connect to port 80 and 443 of any host on the Internet. Then, any traffic going out to any other port is dropped.

After testing all commands and seeing that they worked for me, I put them all into Administration/Scripts/Firewall.

Have fun and be safe! Please post any comments below.

Using 2 ISPs at home at the same time! Tomato MultiWAN – works great! (Video)

Why do you really need this? When Shibby first put out the firmware with MultiWAN support, I questioned why someone would pay for 2 service providers. Too much bandwidth utilization? If you need more bandwidth, just upgrade your line with your current ISP. It would be cheaper than getting a new line!

Do you need reliability? When you work from home and need to be connected to the Internet for your work and it’s not available, that’s when you might look into a solution like this one! I have Comcast Business. Does that help? No, not really when the problems is with the infrastructure and not a misconfiguration or something internal. It just means that when you call, you talk to someone a little more competent and you can get a person quicker than going through the phone system. You can have someone come on-site a little faster too. That said, the service is the same as that of all other consumers. After a few rains and a couple of Comcast outages (not exactly outages, but huge degradations in service), I started to think about getting a second provider. Luckily, in San Francisco, we have a few options for service providers here. I happen to be lucky enough to have access to 2 different cable providers, Comcast and Wave Broadband (formerly Astound). I used Astound before. It was not bad. My experience was not nearly as bad as what the Yelp reviews say. I’ve now had them for a couple of weeks and still have the same opinion. They seem to be just fine.

So, moving onto the implementation. As you can see from the screenshot, Shibby makes it easy! First configuration the VLAN. It points to a link for where to do it – in advanced settings. You can look at the next screenshot to see an example of the VLAN being set up. I’m using LAN port 1 for the 2nd WAN port.

In this screen however, You might notice my “Load Balance Weight”. The problem with my service providers is that Comcast gives me unlimited bandwidth. Wave Broadband does not. Because of this, I want more connections to go out of the first WAN link and Shibby gives us a couple of ways to do it. First is with “Load Balance Weight”. I’m just setting the 1st link to 2 and 2nd WAN link to 1. You can play with the numbers to try finding your desired balance.

Another place to do load balancing is by pinning a particular host to a particular WAN link. For example, I have some traffic I want out of 1 WAN link and some out of another. This way, I can tell my highest traffic hosts to go through WAN1 while some others through WAN2.

Here’s a status window to show that I have both WANs connected.

Lastly, you can see from different searches of what’s my IP, that both WAN links are being utilized.

Please post your comments and share!

Drobo connected to Asus router running TomatoUSB

Simple – Don’t do it! This was from a couple of years ago when running TomatoUSB on an ASUS RT-N16 router. DD-WRT had issues where the wifi would drop off periodically. I found that TomatoUSB (Shibby or Toastman) were both very good alternatives. So good that I’ve since moved onto Tomato for almost all of my routers.

Anyways, some of the issues I ran into were:

Value too large for defined data type …

Sep 24 00:01:08 unknown daemon.err smbd[1539]: disk_free: sys_fsusage() failed.
Error was : Value too large for defined data type
Sep 24 00:01:08 unknown daemon.err smbd[1539]: disk_free: sys_fsusage() failed.
Error was : Value too large for defined data type

root@unknown:/tmp/etc# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 5.6M 5.6M 0 100% /
tmpfs 62.0M 400.0K 61.6M 1% /tmp
devfs 62.0M 0 62.0M 0% /dev
df: /tmp/mnt/Monster: Value too large for defined data type

Along with this, the CPU on that router just couldn’t handle NTFS-3G well – CPU utilization was going through the roof on writes. I think reads were okay.

It would be better to just build a cheap NAS.

if vCenter Server Heartbeat or Neverfail Heartbeat failover appears to be not working or taking a long time …

The main symptom of this problem would be that the vCenter Server Heartbeat console or Neverfail Management Client console would show that the services had failed over, but if you were to try to ping it, it wouldn’t respond.
Logically, there there are some hypotheses you could come up with:

1) Network packet filter isn’t revealed on the active server, so we can’t connect to it.
2) Something wrong w/ the service.
3) The console is wrong and on the backend, nothing failed over.
…
These would be all wrong.

What we found was that it was an issue with ARP caching on the switches. Because the VM or host abruptly fell off the network, the switches hadn’t expired the ARP entries and that they were stale. You would think that it would be fixed in a minute after the ARP entries expired, but I guess the chain could take a little longer.

Probably the best way to troubleshoot this would be to get on a host on the same network segment and try a ping. If that fails, you could run “arp -a” and check to see if you indeed have the right mac address of the host you want to connect to. If not, you could probably log into the switch to delete the entry or you can create a task to run the command during switchover:

“C:\Program Files\VMware\VMware vCenter Server Heartbeat\R2\bin>nfpktfltr.exe arp”

You should then see the switchover happen without the long delay.

What in the world is irlwinning.com? DDOS?

Recently, I set up my own DNS server. I hadn’t run a public DNS server in years. Since the tvpads recently had some DNS issues, I thought maybe I could help eliminate some support calls by running my own DNS server, pointing to the right servers. Boy was I wrong! For some reason, even though some others on comcast would point to my server as a DNS server, they would still get answers that were not the answers given from my server! It was so bizarre! I had never seen it before. If they ran nslookup and used “server <DNS Server IP>” and typed in the name they wanted to resolve, it’s almost as if the server statement prior was ignored and they were getting the IP that the ISP wanted to give them.

Anyways, that’s not the problem I’m writing about here. Surfing some websites became slow for some reason and I thought I would investigate. The first thing I went to see was what connections I had to the outside world. I went to the router and looked at the traffic. Here’s what I saw:

Obviously, that’s DNS traffic. Well, go to the DNS server and what do I see? This:

10-Oct-2013 15:34:14.228 queries: client 54.252.236.155#58070: query: irlwinning.com IN ANY +E

10-Oct-2013 15:34:14.670 queries: client 206.220.43.92#26073: query: irlwinning.com IN ANY +E

Many different lines of the same exact query. What is it? I have no idea. I’ve decided just to shut down DNS queries for now, but if anyone knows anything about this, I’d be happy to hear from you.

Thanks!

apt-get cheatsheet

nixCraft » Debian Linux apt-get package management cheat sheet » Print

– nixCraft – http://www.cyberciti.biz/tips –

Debian Linux apt-get package management cheat sheet

Posted By LinuxTitli On May 9, 2005 @ 12:21 pm In Debian Linux, Howto, Linux, Sys admin, Tips, Ubuntu Linux | 6 Comments

^[1]

Both Debian and Ubuntu Linux provides a number of package management tools. This article summaries package management command along with it usage and examples for you.

(1) apt-get : APT is acronym for Advanced Package Tool. It supports installing packages over internet (ftp or http). You can also upgrade all packages in single operations, which makes it even more attractive.

(2) dpkg : Debian packaging tool which can be use to install, query, uninstall packages.

(3) Gui tools:

You can also try GUI based or high level interface to the Debian GNU/Linux package system. Following list summaries them:
(1) aptitude ^[2]: It is a text-based interface to the Debian GNU/Linux package system.
(2) synaptic ^[3]: GUI front end for APT

Red hat Linux package names generally end in .rpml similarly Debian package names end in .deb, for example:
apache_1.3.31-6_i386.deb

apache : Package name
1.3.31-6 : Version number
i386 : Hardware Platform on which this package will run (i386 == intel x86 based system)
.deb : Extension that suggest it is a Debian package

Remember whenever I refer .deb file it signifies complete file name, and whenever I refer package name it must be first part of .deb file. For example when I refer to package sudo it means sudo only and not the .deb file i.e. sudo_1.6.7p5-2_i386.deb. However do not worry you can find out complete debian package list with the following command:

apt-cache search {package-name}

apt-get add a new package

Add a new package called samba
Syntax: apt-get install {package-name}

# apt-get install samba

apt-get remove the package called samba but keep the configuration files

Syntax: apt-get remove {package-name}

# apt-get remove samba

apt-get remove (erase) package and configuration file

Syntax: apt-get –purge remove {package-name}

# apt-get --purge remove samba

apt-get Update (upgrade) package

Syntax: apt-get upgrade

To upgrade individual package called sudo, enter:
# apt-get install sudo

apt-get display available software updates

Following command will display the list of all available upgrades (updates) using -u option, if you decided to upgrade all of the shown packages just hit ‘y’

# apt-get upgrade samba

However if you just wish to upgrade individual package then use apt-get command and it will take care of rest of your worries:
Syntax: apt-get install {package-name}

dpkg command to get package information such as description of package, version etc.

Syntax: dpkg –info {.deb-package-name}

# dpkg --info sudo_1.6.7p5-2_i386.deb | less

List all installed packages

Syntax: dpkg -l

# dpkg -l

To list individual package try such as apache

# dpkg -l apache

You can also use this command to see (verify) if package sudo is install or not (note that if package is installed then it displays package name along with small description):

# dpkg -l | grep -i 'sudo'

To list packages related to the apache:

# dpkg -l '*apache*'

List files provided (or owned) by the installed package (for example what files are provided by the installed samba package)
Syntax: dpkg -L {package-name}

# dpkg -L samba

(H) List files provided (or owned) by the package (for example what files are provided by the uninstalled sudo package)

Syntax: dpkg –contents {.deb-package-name}

# dpkg --contents sudo_1.6.7p5-2_i386.deb

Find, what package owns the file /bin/netstat?

Syntax: dpkg -S {/path/to/file}

# dpkg -S /bin/netstat

Search for package or package description

Some times you don’t know package name but aware of some keywords to search the package. Once you got package name you can install it using apt-get -i {package-name} command:
Syntax: apt-cache search “Text-to-search”

Find out all the Debian package which can be used for Intrusion Detection

# apt-cache search "Intrusion Detection"

Find out all sniffer packages

# apt-cache search sniffer

Find out if Debian package is installed or not (status)

Syntax: dpkg -s {package-name} | grep Status

# dpkg -s samba| grep Status

List ach dependency a package has…

Display a listing of each dependency a package has and all the possible other packages that can fulfill that dependency. You hardly use this command as apt-get does decent job fulfill all package dependencies.

Syntax: apt-cache depends package

Display dependencies for lsof and mysql-server packages:

# apt-cache depends lsof
# apt-cache depends mysql-server

Tag: networking