Firewall expert required

[markTHEblake]

Any firewall experts here?
I need some assistance in solving an issue with one of my network appliances co-located in a large scale corporate network - google is not helping me. :(

If you know what ephemeral ports are apply within :wink:

[henno]

Talk to me.

[Webster]

Just use "in private browsing" Blakey and no-one will ever know....

[live4golf]

Not an expert, might be able to help somewhat....ephermal issues...running out of ports??

[markTHEblake]

Jack, thanks but its ephemeral not enema, your expertise not required.

Henno/L4G, read on. Pretend you are the security admin at a fictional University of Hard Knocks (UHK).

The UHK researchers want me (MB) to provide his specialised vending machine (VM) services to them, which means they will host this VM on the UHK network. The data application on the VM needs to connect to a Mysql server hosted at MB, so you implement this rule on the UHK firewall.
> Allow VM (local ip address) outgoing TCP 3306 to MB (remote ip address)

Now This works beautifully, because you know your shit, and the VM is secured because it cant connect to any other services or IP addresses, which makes you kind of happy because you really didnt want this foreign piece of junk on your network in the first place.

However under the hood unknown to myself there is some pretty funky magic happens down at the OSI layer (bit sized little elves, like mike) that goes something like this.
* VM connects to server on port 3306.
* server responds and says, 'wouldnt it be cool for us to continue this conversation on Port X (anywhere between 32768-61000)
* VM says "that does sound cool, lets do it".
* then the VM maintains its connection on port 32768+ forever or until he gets tired

My question is: If you needed to create a firewall rule in the first instance to allow the VM to connect 'outbound' from your network, why is the VM able to connect/continue a session on these ephemeral ports without a specific rule to allow it?

[mike]

I'll have a think about this and get back to you.

[markTHEblake]

i should have known to ask mike first about a short lived protocol.

[Peter]

Have you tried turning it off and back on again?

[henno]

Let me get a few things cleared up before we continue:

a) I assume by your questioning that you don't have access to change anything at either end (vending maching or server)?
b) What flavour firewall is in the middle? (Not that it really matters, for such a simple task, I suppose.)
c) MySQL has to run on port 3306?

This may be a stupid question but I always start at the easiest possibilities, but have you tried manually assigning it (even if only for 'testing' purposes) to a port above 32768 in the first instance and see if it "holds" it? University networks are strange places and there might be some other device/protocol using port 3306 and the server (and/or firewall) are being a bit too clever and working around it by suggesting something in the ridiculously high number range? Worth a shot, particularly if you can port-forward by incoming IP at the server end so you don't have to muck with the MySQL install.

[markTHEblake]

A)I have full remote access to the client. Server is in my office, behind my firewall
B) NFI
C) the client connects fine to us on port 3306. They are now blocking the outbound connections on 32768+



The problem is it then fails to maintain the connection, reconnects 15 seconds later, meaming that in no time Max user limit on mysql is hit. So I got to reduce time wait ridiculously low on mysql settings and that has other annoying side affects

Go back to my original question? I can't seem to be be able to communicate to them that they are denying my client how to communicate using normal TCP protocols. 39/40 Universoty networks don't have a problem with this

[henno]

Also (and this is where I get annoying by trying to solve an issue that hasn't even asked to be solved) why don't you open an SSH tunnel end-to-end and shoot your MySQL connection through there? Particularly on a university network. It'd probably be a good idea to use a pub/priv key combination and that the ssh user has very, very few permissions at the server end (i.e. MySQL and that's about it).

[henno]

Go back to my original question?


Any firewall experts here?

I think we answered your original question. ;)

[Yossarian]

Why are you creating a firewall for gold coast golfer?

[live4golf]

MB,

What exactly do you own? - MySQL DB, Server running MySQL, Firewall, Client?

This works in the exact same config elsewhere? - If this is the case then this might all be redundant as you will have checked it...but just in case:

So your MySQL is set to use port 3306, the client will contact the MySQL server on a port other than 3306 (50000 for example), the server will pick this up on port 3306 and do the data transfer across port 3306 with the FW translating this to port 50000 on the client...

CLIENT(50000)<>FW<>MYSQL(3306)

There are a couple of things to check:

1. Is the client communicating on a different port each time (50000, then 500010, etc.) or is it set to use the exact same port every time?

(From Wikipedia)


On servers, ephemeral ports may also be used as the port assignment on the server end of a communication. This is done to continue communications with a client that initially connected to one of the server's well-known service listening ports. File Transfer Protocol (FTP) and Remote Procedure Call (RPC) applications are two protocols that can behave in this manner. Note that the term "server" here includes workstations running services that receive connections initiated from other clients (such as Remote Desktop Protocol or RDP).
The allocations are temporary and only valid for the duration of the communication session. After completion of the communication session, the ports become available for reuse.[note 1] Since the ports are used on a per request basis they are also called dynamic ports.
The Internet Assigned Numbers Authority (IANA) suggests the range 49152 to 65535 (215+214 to 216−1) for dynamic or private ports.[1]
Many Linux kernels use the port range 32768 to 61000.[note 2] FreeBSD has used the IANA port range since release 4.6. Previous versions, including the Berkeley Software Distribution (BSD), use ports 1024 through 5000 as ephemeral ports.[2]
Microsoft Windows operating systems through XP use the range 1024 to 5000 as ephemeral ports by default.[3] Windows Vista, Windows 7, and Server 2008 use the IANA range by default.[4] Windows Server 2003 uses the range 1024 to 5000 by default, until Microsoft security update MS08-037 from 2008 is installed, after which it uses the IANA range by default.[5] Few Microsoft articles other than KB Article 956188 reference the new port range used by Windows Server 2003, leading to confusion. Of specific note, Windows Server 2008 with Exchange Server 2007 installed has a default port range of 1024 through 60000.[6]
In addition to the default range, all versions of Windows since Windows 2000 also allow the option to use a non-default port range with a maximum of 1024 to 65535.[7][8] Some Microsoft articles misleadingly list only this non-default range,[9] leading to a popular misconception that 1024 to 65535 is the default or required port range used.


2. Does the Firewall use SPI? - If it does and the client is set to use a random port then the connection will be dropped because the Firewall will only allow a single port to be used from the client

(From Wikipedia)
In computing, a stateful firewall (any firewall that performs stateful packet inspection (SPI) or stateful inspection) is a firewall that keeps track of the state of network connections (such as TCP streams, UDP communication) traveling across it. The firewall is programmed to distinguish legitimate packets for different types of connections. Only packets matching a known active connection will be allowed by the firewall; others will be rejected.

If the client uses a different random outgoing port after setting up the connection on 3306 AND the FW uses SPI, either disable SPI on the Firewall or configure the client to connect on the same port every time.

In a nutshell, if the client and mySQL do kick off the conversation but fail to transfer data or maintain the connection it is either:
1. client and/or server not set up to use the same port range
2. server using dynamic ports in IANA range
3. SPI on the FW and client using different outgoing ports each time
4. UHK have set up a silly rule on the FW to block the ephemeral port

[Courty]

That's what I was going up say.

[markTHEblake]

Also (and this is where I get annoying by trying to solve an issue that hasn't even asked to be solved)
I particularly hate that.


why don't you open an SSH tunnel end-to-end and shoot your MySQL connection through there?
That is on the white board,. so is about a dozen other things. I actually built it already, couldnt login, didnt have time to troubleshoot, went on 5 weeks holiday, and being flat out on other stuff ever since.

[markTHEblake]

I think we answered your original question. ;)
i really need to understand this - irrespective of the other good stuff being thrown into the discussion
"If you needed to create a firewall rule in the first instance to allow the VM to connect 'outbound' from your network, why is the VM able to connect/continue a session on these ephemeral ports without a specific rule to allow it?"


What exactly do you own? - MySQL DB, Server running MySQL, Firewall, Client?
My side is Mysql community, running on ubuntu 1204, on my lan, router is running Openwrt.


This works in the exact same config elsewhere? - If this is the case then this might all be redundant as you will have checked it...but just in case:
It only works seamlessly in about 39 other places, all universities, private and govt research institutes, & hospitals.


So your MySQL is set to use port 3306, the client will contact the MySQL server on a port other than 3306 (50000 for example)
no the client initiates the connection outwards on TCP 3306, as explained originally and their firewall alows that.
your wiki quotes explain what happens next.




In a nutshell, if the client and mySQL do kick off the conversation but fail to transfer data or maintain the connection it is either:
1. client and/or server not set up to use the same port range
2. server using dynamic ports in IANA range
3. SPI on the FW and client using different outgoing ports each time
4. UHK have set up a silly rule on the FW to block the ephemeral port

its 4. and the problem is I dont know how to tell them what they are doing is silly, apart from saying its silly.

[henno]

What they are doing is silly. That is normal MySQL operation.

[live4golf]

you can't just tell them they are silly? I mean, the FW does the port crap itself, that is what they are designed to do and every single app on earth does it this way...kick off comms on a port, then use another for actual transfer of data.

I took this to my firewall expert and we all agree the firewall guy at UHK needs to be fired.

[markTHEblake]

Thanks for telling me what i already know. :lol: This has been 6 months now. :(

I cant speak to the "firewall guy", like most uni IT dept that have several layers, I am speaking to the network support team for that school.

I just remembered, at the same time they broke this, I also could no longer VNC in, likely for the same reason. but now VNC is working again. I think I will point out "why are you letting me connect inbound on these crazy high ports, and you wont let the other machine connect out the same way to a trusted host?

I can see what happens then, clunk, there goes my VNC access

[mike]

What version suction clacker are you using?

[markTHEblake]

Problem solved: it's the old asymmetrical routing issue trick, that's the 2nd time I have fallen for it this week.

Not fixed yet though, they still haven't found the why, and it's affecting other stuff of theirs (obviously).

Thanks Henno, I just had to keep pushing them to dig deeper.