[FW-1] IPSec traffic broken on HA pair after upgrade to SP6

["<Aaron Reynolds>" <Aaron.Reynolds@FW1-NOSPAMFRANKLINCOVEY.COM>]

Management - Solaris 2.6 / 4.1 SP6
Firewall HA -Pair - Upgraded from IPSO 3.3 / 4.1 SP3 to IPSO 3.5 FCS8 / 4.1
SP6

I upgraded a number of firewalls over this past weekend.  Some from SP5a,
SP3, and SP4.  All of them went well, until I got to my HA pair.  Upgraded
process per release notes went as follows:

-fwstop Secondary firewall
-copy sync.conf to sync.conf.save
-edit sync.conf -> one line which reads "SyncMode=no sync"
-fwstart Secondary firewall (so primary has something to fail over to while
it is fwstop'd in the next step).
-repeat above steps for Primary firewall
-On Secondary firewall do newpkg to upgrade to SP6 from SP3
-On Secondary firewall do newimage -Rkl ipso.tgz to install latest IPSO
image.
-reboot Secondary firewall
-follow the same steps for the Primary.
-On Secondary firewall - fwstop and cp sync.conf.save back to sync.conf
-fwstart secondary firewall
-repeat for primary firewall

At this point everything went well, with the exception that the primary
firewall hung on its bootup, after being upgraded.  It hung at the bootmgr
prompt.  I had to power off the IP650.  It came up fine after that and
firewall traffic appeared to be running smoothly.

I have a monitor script for all of our VPNs, which soon paged me saying they
were all down.  Sure enough all IPSec traffic, including SecuRemote traffic
was broken.  I read back over the release notes, looked at Phoneboy's site,
and nokia's support site.  It seemed that it had to be an issue with the
IPSec_cluster_nat (true) property changing from a global property to an
individual object property.  I edited objects.C on the management console,
re-pushed the policy, and nothing changed.  I could see all of my remote
gateways sending traffic to port 500 on my cluster address, but my firewall
would not even respond.  I failed over to my secondary firewall, same thing.
After hours and hours of pulling my hair out, I became weak and succumbed to
the thought of rebooting my HA pair.  I started with the secondary, and then
rebooted the primary.  Within seconds after logging in to the firewall, I
got a page stating that all of my VPNs had come back up.  SecuRemote traffic
was working too.  It was as if something didn't start after that initial
reboot, after upgrading.  I have found the following error in the
isakmpd.elg file and am wondering if isakmpd didn't ever start the first
time.  The error is as follows:

InvokeIsakmpServer: can't bind to UDP socket, port: 500: Operation not
permitted

I have seen some threads hinting to the fact that this could be a routing
issue between the mgmt and firewall.  This was not the case.  I could push
policies, and recieve logs the whole time my VPN traffic was failing.  Any
thoughts would be appreciated. Sorry for the novel.

-Aaron

=================================================
To set vacation, Out Of Office, or away messages,
send an email to LISTSERV@lists.us.checkpoint.com
in the BODY of the email add:
set fw-1-mailinglist nomail
=================================================
To unsubscribe from this mailing list,
please see the instructions at
http://www.checkpoint.com/services/mailing.html
=================================================
If you have any questions on how to change your
subscription options, email
fw-1-owner@ts.checkpoint.com
=================================================