A problem arose a few days ago at work with our Sybase database server dropping sessions from users that were trying to connect to a database in our home office.

The dropping was fixed when the name of the local service in the home office was changed on the server and the local clients. However, this didn't indicate why the trouble arose in the first place, and had to be tracked down ASAP to prevent future problems.

The Sybase service runs over the NETBIOS port (Port 139) to take requests from the Sybase client on the user's workstation. Our initial suspicion was that something had changed in the router configuration of one of the offices on our WAN to expose their own server with the same service name (because they were being used for the same purpose).

Since this theoretical rogue database server could be anywhere on our WAN, we had to have some proof before wasting someone's time in another office. The first thought was to watch the service initialization with a tcpdump from my workstation in my office, but the office switch prevented me from seeing any non-broadcast packets not intended for my systems.

Once I got my notebook on the same hub with the database server, it was a simple matter to monitor what workstations were exchanging UDP packets with the system, and even the content of certain packets.

At this point I realized that I'd have to watch a workstation in action to know if a rogue machine was reponding to a DB client request, because any reply packets to a workstation in another part of the network wouldn't have to travel the path to the local server.

The traffic monitoring was done with a combination of script, tcpdump 3.6 and grep. First, I called script with a filename to store the dump in, and then tcpdump so that I could watch and store the traffic at the same time.

$ script workstation-to-db.txt
Script started, file is workstation-to-db.txt
$ tcpdump –n
dump of network traffic
^C
$ exit
exit
Script done, file is workstation-to-db.txt

Once that was done, I used grep to filter out all but the traffic to and from the workstation.

The first test was to connect the workstation to new service name. We got a huge amount of communication the first time between the database server and the workstation, and nothing else.

The second test was to connect to the old database name, as the rogue server is ostensibly still going by the old name. The result was a chattering of one-line packets between the workstation and a specific remote office that only occurred when it was trying to connect to the service.

At that point we could contact the office in question and point out the IP and name of the server that was causing the problem, and close the issue.

Some problems with Sybase (for me) included that I wasn’t familiar with the nature of the Sybase services, or how to track them down. In any case, the service runs over the NETBIOS port to take requests from the Sybase client to the user's workstation. This makes them impossible to port-scan the network for using conventional Unix tools like nmap, and forces a Unix Geek like me to resort to more direct monitoring measures.

But I did prove that there aren't many network problems that can't be solved with a notebook and tcpdump.

No Comments | #1654

Unless noted, all content on epistolary.org is © Copyright 1999-2008 to Rob Carlson with all rights reserved. All information is verified when possible, cited as appropriate and applied in the real world at your own risk. Send all feedback to rob@vees.net.