[FW-1] Path MTU problems through VPN with Nokia IP740's

[Mitchell Rowton <MRowton@FW1-NOSPAMROWTONCONSULTING.COM>]

All,

I apologize in advance for the length of this post...

We are in the process of upgrading our headquarters site from dual Cisco PIX
535's in failover mode to dual Nokia IP740's running VRRP with IPSO 3.5 fcs
4 and CheckPoint FP3.  During the attempted transition we ran into problems
with NetBIOS traffic, one specific problem to focus on is doing a
\\ipaddress\c$ share from headquarters to a remote site server.  Our network
topology consist of (summarized) HQ network segments connected Cisco 6509
core switch, this switch connected to the Cisco PIX firewall's, this PIX
pair connected to a Cisco 7206 border router.  The Cisco 7206 border router
has VPN's, set up to most of our remote locations.  The remote locations
consist of Cisco 2620 routers running firewall and IDS feature set, these
2600 routers terminate the VPN.

HQ segment ----> 6509 switch ---> dual PIX firewall ---> 7206 Router
(VPN) ------> (VPN) remote site 2620 router

This scenario (without the Nokia 740's) seems to work.  To illustrate this I
get on a HQ segment PC and do a \\ipaddress\c$ to a remote site server.
When I sniff this traffic, I see the HQ pc send an "syn" packet to the
remote server with the TCP Maximum Segment Size (MSS) set to 1460 bytes.
This is the default TCP MSS because 1460 + 20 bytes for IP, + 20 bytes for
TCP, + 14 bytes for ethernet = 1514 (ethernet MTU)  I then see the remote
server send a "syn ack" back to my HQ pc with the maximum segment size
lowered to 1380 bytes, my HQ pc then returns the "ack" agreeing to 1380
bytes and all is well.  By the remote server lowering the default maximum
segment size the traffic has enough room to pass through the VPN's.

Now bring in the Nokia IP740's...

Basically the same scenario takes place except the remote server never
downgrades the maximum segment size to 1380 bytes, so the "syn" "syn ack"
and "ack" all agree on 1460 bytes.  This means that I see most traffic
coming back, however the NetBIOS traffic always has the don't fragment (DF)
bit set, and is usually as large as it can possibly be.  Hence this NetBIOS
traffic does not make it back to the HQ PC.

I have recently read the RFC's for TCP maximum segment size (MSS) and Path
MTU discovery, as well as exhausted quite a bit of research material on the
web trying to figure this out.  It is my theory that the remote server would
downgrade the default maximum segment size because of PMTU discovery.  It
should do this because of the ICMP packet to big (type 3 code 4 I believe?)
However for some reason this does not work after I put the Nokia IP740's in.

A couple of things to keep in mind.

I did a "cpstop" and "ipsofwd on admin" on the Nokia boxes, this is why I
think it is probably not a checkpoint issue (like dropping the ICMP packets.
I don't see the ICMP packets getting dropped anywhere in between.  When I
lower the MTU on the inside interface of the Nokia boxes (this isn't a
supported feature) things seem to work.  However if I lower the MTU on the
routers running the VPN's, it doesn't seem to work. (im now questioning this
test)

I should be getting some higher level help from Nokia soon, but I'm just
looking for some heads up "i had a problem like this" from one of you smart
guys.  Are Nokia devices PMTU compliant or are they a PMTU black hole?  I
have developed quite a bit of information surrounding this issue so if there
is something more I should provide to the list for accurate analysis please
let me know.  Thanks in advance if you have read this far down  :)



Mitchell E Rowton
-------------------
Network Security Engineer
CISSP, CCNP, CCDP, CCSA, NSA IAM, Network+
----------------------------
Free security white papers
http://www.rowtonconsulting.com

=================================================
To set vacation, Out Of Office, or away messages,
send an email to LISTSERV@lists.us.checkpoint.com
in the BODY of the email add:
set fw-1-mailinglist nomail
=================================================
To unsubscribe from this mailing list,
please see the instructions at
http://www.checkpoint.com/services/mailing.html
=================================================
If you have any questions on how to change your
subscription options, email
fw-1-owner@ts.checkpoint.com
=================================================