How to avoid FortiGate entering conserve mode


threat_landscape“The system has entered conserve mode”

“Fortigate has reached connection limit for n seconds”

That is status field from the “Alert message control”  on System Dashboard. that status indicates the critical level from FortiGate device if it has entered conserve mode.

This problem happens when shared memory goes over 80%, to exit this conserve mode you have to wait (or kill some of the processes) until the memory goes under 70%. A FortiGate goes into the conserve mode state as a self protection measure when a memory shortage appears on the system. When entering conserve mode the FortiGate activates protection measures in order to recover memory space. When enough memory is recovered, the system is leaving/exiting the conserve mode state and releases the protection measures.

Antivirus fail-open is a safeguard feature that determines the behavior of the FortiGate AntiVirus system, when it becomes overloaded with high traffic.

to mitigate this you have more type of options:

# set av-failopen { off | on-shot | pass | idledrop}

Below we will describe what all of them do:

a. Off – if the FortiGate enters conserve mode, the FortiGate will stop accepting new AV sessions, but will continue to process currently active sessions

b. One-shot – if the FortiGate enters conserve mode, all new connections will bypass the AV system, but currently sessions will continue to be processed. This is the same as the “pass” options, but it will NOT turn off once the condition causing the av-failopen has stopped

c. Idle-drop – will drop connection based on the clients that has the most opened connection

d. Pass – this is the default option

Please keep in mind that with one-shot and pass option, NO content filtering of the traffic is done. The data stream could contain malicious content.

Below are some commands to troubleshoot when the system enters conserve mode:

a. Check if the system is in Conserve Mode:

# diagnose hardware sysinfo shm
SHM counter: 67
SHM allocated: 1556480
SHM total: 101220352
conservemode: 0     [conservemode 0 means not in conserve mode, 1 means on conserve mode, 2 means on kernel conserve mode]
shm last entered: n/a
system last entered: n/a
SHM FS total: 106827776
SHM FS free: 105205760
SHM FS avail: 105205760
SHM FS alloc: 1622016

b. Check if there any errors on the interfaces:

#diagnose hardware deviceinfo nic

So, If this problem occurs somehow we need to reduce shared memory usage on Fortigate, some optimization which I have try to improve performance on this box is:

1. Session timer optimizations
# config system global
set tcp-halfclose-timer 30         [ default 120 s ]
set tcp-halfopen-timer 30          [ default 60 s ]
set tcp-timewait-timer 0           [ default 120 s ]
set udp-idle-timer 60                [ default 120 s ]
end

# config system session-ttl
set default 300                  [ default 300 ]

config port
edit 0
set protocol 17
set timeout 10
set end-port 53
set start-port 53
end
end

2. Reduce the FortiGuard services for the cache
# config system fortiguard
set webfilter-cache-ttl 500 [ default 3600 ]
set antispam-cache-ttl 500 [ default 1800 ]
end

3. DNS cache optimization
# config system dns
set dns-cache-limit 300     [ default: 5000 ]
end

Optional:
3a. Disable options for DNS forwarding:
# config system dns
unset fwdintf
end

3b. Delete dns-udp session-helper:
# config system session-helper
delete 14     [ 14 is references number for dns-udp ]
end

4. Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering)
on FortiOS 5.0: System > Config > Features [ enable/disable ]

5. Turn off all non mandatory features such as Logging, archiving, data leak prevention, IPS
Display CPU/Memory usage:
# get system performance top <delay> <max_lines>
or
# diag sys top <delay> <max_lines>

And to kill process:
# diagnose sys kill 9

Restart any applications:
# diagnose test application <application> <option>

Restart IPS engine:
# diagnose test application ipsengine 99

6. Turn off DHCP-server services
# config sys dhcp server
delete (reference number for dhcp-server)
end

7. Reduce the maximum file size for antivirus scanning
On FortiOS 5.0: Go to Policy > Proxy Options > Common Options > Change: Amount (bytes)
On FortiOS 4.0: Go to Firewall > Policy > Protocol Options > reduce the file size threshold

If your firewall still entered conserve mode instead of still more RAM, replace with the higher model😀

4 thoughts on “How to avoid FortiGate entering conserve mode

  1. Ryszard Moron December 6, 2013 at 3:52 AM Reply

    shortest and very extensive manual.

  2. Dede Fadhilah August 15, 2014 at 11:02 AM Reply

    thanks mas artikel nya🙂

  3. landex February 21, 2015 at 3:13 PM Reply

    nice ! thx !

  4. Thyago Siebra September 21, 2016 at 9:41 PM Reply

    Hey man, tks… Work for me!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: