Recently I was working on a network outage that only 2-3 ip addresses of a single network out of 15-20 different network segments was getting randomly blocked, then would be allowed. We spent a significant amount of time troubleshooting, pulling packets captures, rebuilding VPNs, replacing core devices, pretty much anything and everything you could think of. We had the manufacturer tier 2 support working on the issue for a few days as well. To make a long (very long) story short, the issue was with a device that sat in between two core pieces of network equipment. This in between device was updated a few weeks prior and that update was causing these IPs to be unavailable. All of our network diagrams was telling us that this device was not logically inline, though it was.
The importance of following the OSI model. I have always been taught that if you refer back to the OSI model from Layer 1 to Layer 7 you will almost always be able to find the issue. In this case, that was the exact case. We found the source mac address was differing from what we were expecting. We were expecting to see the firewalls Mac, but in turn saw another device. After working through the mac address tables, we found it was a web filter that was placed in line. Once this web filter was power cycled and the issue corrected the very odd IP issue went away as well.
Moral of the story, if you are hitting a wall troubleshooting a network issue, stop and start from Layer 1 and work your way back up.
Danny McLean, PEI
