3. Starting HAProxy
HAProxy is started by invoking the “haproxy” program with a number of arguments passed on the command line. The actual syntax is:
where [<options>]* is any number of options. An option always starts with ‘-’ followed by one
of more letters, and possibly followed by one or multiple extra arguments. Without any option,
HAProxy displays the help page with a reminder about supported options. Available options may vary
slightly based on the operating system. A fair number of these options overlap with an equivalent
one in the “global” section. In this case, the command line always has precedence over the
configuration file, so that the command line can be used to quickly enforce some settings without
touching the configuration files. The current list of options is:
-- <cfgfile>*
all the arguments following “–” are paths to configuration file/directory to be loaded and processed in the declaration order. It is mostly useful when relying on the shell to load many files that are numerically ordered. See also “-f”. The difference between “–” and “-f” is that one “-f” must be placed before each file name, while a single “–” is needed before all file names. Both options can be used together, the command line ordering still applies. When more than one file is specified, each file must start on a section boundary, so the first keyword of each file must be one of “global”, “defaults”, “peers”, “listen”, “frontend”, “backend”, and so on. A file cannot contain just a server list for example.
-f <cfgfile|cfgdir>
adds <cfgfile> to the list of configuration files to be loaded. If <cfgdir> is a directory, all
the files (and only files) it contains are added in lexical order (using LC_COLLATE=C) to the list
of configuration files to be loaded; only files with “.cfg” extension are added, only non hidden
files (not prefixed with “.”) are added. Configuration files are loaded and processed in their
declaration order. This option may be specified multiple times to load multiple files. See also
“–”. The difference between “–” and “-f” is that one “-f” must be placed before each file name,
while a single “–” is needed before all file names. Both options can be used together, the command
line ordering still applies. When more than one file is specified, each file must start on a section
boundary, so the first keyword of each file must be one of “global”, “defaults”, “peers”, “listen”,
“frontend”, “backend”, and so on. A file cannot contain just a server list for example.
-C <dir>
changes to directory <dir> before loading configuration files. This is useful when using relative
paths. Warning when using wildcards after “–” which are in fact replaced by the shell before
starting haproxy.
-D
start as a daemon. The process detaches from the current terminal after forking, and errors are not reported anymore in the terminal. It is equivalent to the “daemon” keyword in the “global” section of the configuration. It is recommended to always force it in any init script so that a faulty configuration doesn’t prevent the system from booting.
-L <name>
change the local peer name to <name>, which defaults to the local hostname. This is used only with
peers replication. You can use the variable $HAPROXY_LOCALPEER in the configuration file to
reference the peer name.
-N <limit>
sets the default per-proxy maxconn to <limit> instead of the builtin default value (usually 2000).
Only useful for debugging.
-V
enable verbose mode (disables quiet mode). Reverts the effect of “-q” or “quiet”.
-W
master-worker mode. It is equivalent to the “master-worker” keyword in the “global” section of the configuration. This mode will launch a “master” which will monitor the “workers”. Using this mode, you can reload HAProxy directly by sending a SIGUSR2 signal to the master. The master-worker mode is compatible either with the foreground or daemon mode. It is recommended to use this mode with multiprocess and systemd.
-Ws
master-worker mode with support of notify type of systemd service.
-4
force DNS resolvers to query and accept IPv4 addresses only (“A” records). This can be used when facing difficulties in certain environments lacking end-to-end dual-stack connectivity. It overrides the global “dns-accept-family” directive and forces it to “ipv4”.
-c
only performs a check of the configuration files and exits before trying to bind. The exit status is zero if everything is OK, or non-zero if an error is encountered. Presence of warnings will be reported if any. By default this option does not report a success message. Combined with “-V” this will print the message “Configuration file is valid” upon success.
Scripts must use the exit status to determine the success of the command.
-cc
evaluates a condition as used within a conditional block of the configuration. The exit status is zero if the condition is true, 1 if the condition is false or 2 if an error is encountered.
-d
enable debug mode. This disables daemon mode, forces the process to stay in foreground and to show incoming and outgoing events. It must never be used in an init script.
-dA[file]
dump an archive of all dependencies detected at boot time in the designated file in tar format, immediately after the configuration is done loading. This is equivalent to “set-dumpable libs”, but instead of keeping the libs in memory, it dumps them into a file. This may be used after a core dump, in order to provide all necessary libraries to developers to permit them to exploit the core. This may not be available on all operating systems. It is highly recommended to use this with the regular configuration files, and optionally with “-c” when used manually, to make haproxy immediately exit after the dump, without starting. Example:
-dC[key]
dump the configuration file. It is performed after the lines are tokenized, so comments are stripped and indenting is forced. If a non-zero key is specified, lines are truncated before sensitive/confidential fields, and identifiers and addresses are emitted hashed with this key using the same algorithm as the one used by the anonymized mode on the CLI. This means that the output may safely be shared with a developer who needs it to figure what’s happening in a dump that was anonymized using the same key. Please also see the CLI’s “set anon” command.
-dD
enable diagnostic mode. This mode will output extra warnings about suspicious configuration statements. This will never prevent startup even in “zero-warning” mode nor change the exit status code.
-dF
disable data fast-forward. It is a mechanism to optimize the data forwarding by passing data directly from a side to the other one without waking the stream up. Thanks to this directive, it is possible to disable this optimization. Note it also disable any kernel tcp splicing. This command is not meant for regular use, it will generally only be suggested by developers along complex debugging sessions.
-dG
disable use of getaddrinfo() to resolve host names into addresses. It can be used when suspecting that getaddrinfo() doesn’t work as expected. This option was made available because many bogus implementations of getaddrinfo() exist on various systems and cause anomalies that are difficult to troubleshoot.
-dI
enable the insecure fork. This is the equivalent of the “insecure-fork-wanted” in the global section. It can be useful when running all the reg-tests with ASAN which need to fork addr2line to resolve the addresses.
-dK<class[,class]*>
dumps the list of registered keywords in each class. The list of classes is available with “-dKhelp”. All classes may be dumped using “-dKall”, otherwise a selection of those shown in the help can be specified as a comma-delimited list. The output format will vary depending on what class of keywords is being dumped (e.g. “cfg” will show the known configuration keywords in a format resembling the config file format while “smp” will show sample fetch functions prefixed with a compatibility matrix with each rule set). These may rarely be used as-is by humans but can be of great help for external tools that try to detect the appearance of new keywords at certain places to automatically update some documentation, syntax highlighting files, configuration parsers, API etc. The output format may evolve a bit over time so it is really recommended to use this output mostly to detect differences with previous archives. Note that not all keywords are listed because many keywords have existed long before the different keyword registration subsystems were created, and they do not appear there. However since new keywords are only added via the modern mechanisms, it’s reasonably safe to assume that this output may be used to detect language additions with a good accuracy. The keywords are only dumped after the configuration is fully parsed, so that even dynamically created keywords can be dumped. A good way to dump and exit is to run a silent config check on an existing configuration:
If no configuration file is available, using “-f /dev/null” will work as well to dump all default keywords, but then the return status will not be zero since there will be no listener, and will have to be ignored.
-dL
dumps the list of dynamic shared libraries that are loaded at the end of the config processing. This will generally also include deep dependencies such as anything loaded from Lua code for example, as well as the executable itself. The list is printed in a format that ought to be easy enough to sanitize to directly produce a tarball of all dependencies. Since it doesn’t stop the program’s startup, it is recommended to only use it in combination with “-c” and “-q” where only the list of loaded objects will be displayed (or nothing in case of error). In addition, keep in mind that when providing such a package to help with a core file analysis, most libraries are in fact symbolic links that need to be dereferenced when creating the archive:
When started in verbose mode (-V) the shared libraries’ address ranges are also enumerated, unless the quiet mode is in use (-q).
-dM[<byte>[,]][help|options,...]
forces memory poisoning, and/or changes memory other debugging options. Memory poisonning means that
each and every memory region allocated with malloc() or pool_alloc() will be filled with <byte>
before being passed to the caller. When <byte> is not specified, it defaults to 0x50 (‘P’). While
this slightly slows down operations, it is useful to reliably trigger issues resulting from missing
initializations in the code that cause random crashes. Note that -dM0 has the effect of turning any
malloc() into a calloc(). In any case if a bug appears or disappears when using this option it means
there is a bug in haproxy, so please report it. A number of other options are available either alone
or after a comma following the byte. The special option “help” will list the currently supported
options and their current value. Each debugging option may be forced on or off. The most optimal
options are usually chosen at build time based on the operating system and do not need to be
adjusted, unless suggested by a developer. Supported debugging options include (set/clear):
- fail / no-fail:
- no-merge / merge:
- cold-first / hot-first:
- integrity / no-integrity:
- backup / no-backup:
- no-global / global:
- no-cache / cache:
- caller / no-caller:
- tag / no-tag:
- poison / no-poison:
-dR
disable SO_REUSEPORT socket option on listening ports. It is equivalent to the “global” section’s “noreuseport” keyword. This may be applied in multi-threading scenarios, when load distribution issues observed among the haproxy threads (could be monitored with top).
-dS
disable use of the splice() system call. It is equivalent to the “global” section’s “nosplice” keyword. This may be used when splice() is suspected to behave improperly or to cause performance issues, or when using strace to see the forwarded data (which do not appear when using splice()).
-dT
disable the use of ktls. It is equivalent to the “global” section’s keyword “noktls”. It is mostly useful when suspecting a bug related to ktls.
-dV
disable SSL verify on the server side. It is equivalent to having “ssl-server-verify none” in the “global” section. This is useful when trying to reproduce production issues out of the production environment. Never use this in an init script as it degrades SSL security to the servers.
-dW
if set, haproxy will refuse to start if any warning was emitted while processing the configuration. This helps detect subtle mistakes and keep the configuration clean and portable across versions. It is recommended to set this option in service scripts when configurations are managed by humans, but it is recommended not to use it with generated configurations, which tend to emit more warnings. It may be combined with “-c” to cause warnings in checked configurations to fail. This is equivalent to global option “zero-warning”.
-dZ
disable forwarding of data in “zero-copy” mode. It is equivalent to the “global” section’s “tune.disable-zero-copy-forwarding” keyword. This may be helpful in case of issues with data loss or data integrity, or when using strace to see the forwarded data, as it also disables any kernel tcp splicing.
-db
disable background mode and multi-process mode. The process remains in foreground. It is mainly used during development or during small tests, as Ctrl-C is enough to stop the process. Never use it in an init script.
-dc
enable CPU affinity debugging. The list of selected and evicted CPUs as well as their topology will be reported before starting.
-de
disable the use of the “epoll” poller. It is equivalent to the “global” section’s keyword “noepoll”. It is mostly useful when suspecting a bug related to this poller. On systems supporting epoll, the fallback will generally be the “poll” poller.
-dk
disable the use of the “kqueue” poller. It is equivalent to the “global” section’s keyword “nokqueue”. It is mostly useful when suspecting a bug related to this poller. On systems supporting kqueue, the fallback will generally be the “poll” poller.
-dp
disable the use of the “poll” poller. It is equivalent to the “global” section’s keyword “nopoll”. It is mostly useful when suspecting a bug related to this poller. On systems supporting poll, the fallback will generally be the “select” poller, which cannot be disabled and is limited to 1024 file descriptors.
-dr
ignore server address resolution failures. It is very common when validating a configuration out of production not to have access to the same resolvers and to fail on server address resolution, making it difficult to test a configuration. This option simply appends the “none” method to the list of address resolution methods for all servers, ensuring that even if the libc fails to resolve an address, the startup sequence is not interrupted.
-dt [<trace_desc>,...]
activates traces on stderr. Without argument, this enables all trace sources on error level. This can notably be useful to detect protocol violations from clients or servers. An optional argument can be used to specify a list of various trace configurations using ‘,’ as separator. Each element activates one or all trace sources. Additionally, level and verbosity can be optionally specified on each element using ‘:’ as inner separator with trace name. When entering an invalid verbosity or level name, the list of available keywords is presented. For example it can be convenient to pass ‘help’ for each field to consult the list first.
-dv
disable the use of the “evports” poller. It is equivalent to the “global” section’s keyword “noevports”. It is mostly useful when suspecting a bug related to this poller. On systems supporting event ports (SunOS derived from Solaris 10 and later), the fallback will generally be the “poll” poller.
-m <limit>
limit allocatable memory, which is used to keep process’s data, to <limit> megabytes. This may
cause some connection refusals or some slowdowns depending on the amount of memory needed for normal
operations. This is mostly used to force haproxy process to work in a constrained resource
consumption scenario. It is important to note that the memory is not shared between haproxy
processes and a child process created via fork() system call inherits its parent’s resource limits.
So, in a master-worker mode this memory limit is separately applied to the master and its forked
worker process.
-n <limit>
limits the per-process connection limit to <limit>. This is equivalent to the global section’s
keyword “maxconn”. It has precedence over this keyword. This may be used to quickly force lower
limits to avoid a service outage on systems where resource limits are too low.
-p <file>
write all processes’ pids into <file> during startup. This is equivalent to the “global” section’s
keyword “pidfile”. The file is opened before entering the chroot jail, and after doing the chdir()
implied by “-C”. Each pid appears on its own line.
-q
set “quiet” mode. This disables the output messages. It can be used in combination with “-c” to just check if a configuration file is valid or not.
-S <bind>[,bind_options...]
in master-worker mode, bind a master CLI, which allows the access to every processes, running or leaving ones. For security reasons, it is recommended to bind the master CLI to a local UNIX socket. The bind options are the same as the keyword “bind” in the configuration file with words separated by commas instead of spaces.
Note that this socket can’t be used to retrieve the listening sockets from an old process during a seamless reload.
-sf <pid>*
send the “finish” signal (SIGUSR1) to older processes after boot completion to ask them to finish
what they are doing and to leave. <pid> is a list of pids to signal (one per argument). The list
ends on any option starting with a “-”. It is not a problem if the list of pids is empty, so that it
can be built on the fly based on the result of a command like “pidof” or “pgrep”.
-st <pid>*
send the “terminate” signal (SIGTERM) to older processes after boot completion to terminate them
immediately without finishing what they were doing. <pid> is a list of pids to signal (one per
argument). The list ends on any option starting with a “-”. It is not a problem if the list of pids
is empty, so that it can be built on the fly based on the result of a command like “pidof” or
“pgrep”.
-v
report the version and build date.
-vv
display the version, build options, libraries versions and usable pollers. This output is systematically requested when filing a bug report.
-x <unix_socket>
connect to the specified socket and try to retrieve any listening sockets from the old process, and use them instead of trying to bind new ones. This is useful to avoid missing any new connection when reloading the configuration on Linux.
Without master-worker mode, the capability must be enable on the stats socket using “expose-fd listeners” in your configuration.
In master-worker mode, it does not need “expose-fd listeners”, the master will use automatically this option upon a reload with the “sockpair@” syntax, which allows the master to connect directly to a worker without using any stats socket declared in the configuration. If you want to disable this, you can pass -x /dev/null.
A safe way to start HAProxy from an init file consists in forcing the daemon mode, storing existing pids to a pid file and using this pid file to notify older processes to finish before leaving:
When the configuration is split into a few specific files (eg: tcp vs http), it is recommended to use the “-f” option:
When an unknown number of files is expected, such as customer-specific files, it is recommended to assign them a name starting with a fixed-size sequence number and to use “–” to load them, possibly after loading some defaults:
Sometimes a failure to start may happen for whatever reason. Then it is important to verify if the version of HAProxy you are invoking is the expected version and if it supports the features you are expecting (eg: SSL, PCRE, compression, Lua, etc). This can be verified using “haproxy -vv”. Some important information such as certain build options, the target system and the versions of the libraries being used are reported there. It is also what you will systematically be asked for when posting a bug report:
HAProxy version 1.6-dev7-a088d3-4 2015/10/08 Copyright 2000-2015 Willy Tarreau willy@haproxy.org
Build options:
Default settings:
Encrypted password support via crypt(3): yes Built with zlib version: 1.2.6 Compression algorithms supported: identity(“identity”), deflate(“deflate”), \ raw-deflate(“deflate”), gzip(“gzip”) Built with OpenSSL version: OpenSSL 1.0.1o 12 Jun 2015 Running on OpenSSL version: OpenSSL 1.0.1o 12 Jun 2015 OpenSSL library supports TLS extensions: yes OpenSSL library supports SNI: yes OpenSSL library supports prefer-server-ciphers: yes Built with PCRE version: 8.12 2011-01-15 PCRE library supports JIT: no (USE_PCRE_JIT not set) Built with Lua version: Lua 5.3.1 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Available polling systems:
Total: 3 (3 usable), will use epoll.
The relevant information that many non-developer users can verify here are:
- the version
1.6-dev7-a088d3-4 above means the code is currently at commit ID “a088d3” which is the 4th one after after official version “1.6-dev7”. Version 1.6-dev7 would show as “1.6-dev7-8c1ad7”. What matters here is in fact “1.6-dev7”. This is the 7th development version of what will become version 1.6 in the future. A development version not suitable for use in production (unless you know exactly what you are doing). A stable version will show as a 3-numbers version, such as “1.5.14-16f863”, indicating the 14th level of fix on top of version 1.5. This is a production-ready version.
- the release date
2015/10/08. It is represented in the universal year/month/day format. Here this means August 8th, 2015. Given that stable releases are issued every few months (1-2 months at the beginning, sometimes 6 months once the product becomes very stable), if you’re seeing an old date here, it means you’re probably affected by a number of bugs or security issues that have since been fixed and that it might be worth checking on the official site.
- build options
they are relevant to people who build their packages themselves, they can explain why things are not behaving as expected. For example the development version above was built for Linux 2.6.28 or later, targeting a generic CPU (no CPU-specific optimizations), and lacks any code optimization (-O0) so it will perform poorly in terms of performance.
- libraries versions
zlib version is reported as found in the library itself. In general zlib is considered a very stable product and upgrades are almost never needed. OpenSSL reports two versions, the version used at build time and the one being used, as found on the system. These ones may differ by the last letter but never by the numbers. The build date is also reported because most OpenSSL bugs are security issues and need to be taken seriously, so this library absolutely needs to be kept up to date. Seeing a 4-months old version here is highly suspicious and indeed an update was missed. PCRE provides very fast regular expressions and is highly recommended. Certain of its extensions such as JIT are not present in all versions and still young so some people prefer not to build with them, which is why the build status is reported as well. Regarding the Lua scripting language, HAProxy expects version 5.3 which is very young since it was released a little time before HAProxy 1.6. It is important to check on the Lua web site if some fixes are proposed for this branch.
- Available polling systems will affect the process's scalability when
dealing with more than about one thousand of concurrent connections. These ones are only available when the correct system was indicated in the TARGET variable during the build. The “epoll” mechanism is highly recommended on Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them will result in poll() or even select() being used, causing a high CPU usage when dealing with a lot of connections.