HAProxy 3.4.4 Documentation on PIG.CENTER

Starter Guide

Mon, 01 Jan 0001 00:00:00 +0000

Load Balancing Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

This document is an introduction to HAProxy for all those who don’t know it, as well as for those who want to re-discover it when they know older versions. Its primary focus is to provide users with all the elements to decide if HAProxy is the product they’re looking for or not. Advanced users may find here some parts of solutions to some ideas they had just because they were not aware of a given new feature. Some sizing information is also provided, the product’s lifecycle is explained, and comparisons with partially overlapping products are provided.

What HAProxy Is and How It Works

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is written as “HAProxy” to designate the product, and as “haproxy” to designate the executable program, software package or a process. However, both are commonly used for both purposes, and are pronounced H-A-Proxy. Very early, “haproxy” used to stand for “high availability proxy” and the name was written in two separate words, though by now it means nothing else than “HAProxy”.

Basic Features

Mon, 01 Jan 0001 00:00:00 +0000

This section will enumerate a number of features that HAProxy implements, some of which are generally expected from any modern load balancer, and some of which are a direct benefit of HAProxy’s architecture. More advanced features will be detailed in the next section.

3.3.1. Basic features : Proxying

Proxying is the action of transferring data between a client and a server over two independent connections. The following basic features are supported by HAProxy regarding proxying and connection management:

Standard Features

Mon, 01 Jan 0001 00:00:00 +0000

In this section, some features that are very commonly used in HAProxy but are not necessarily present on other load balancers are enumerated.

3.4.1. Standard features : Sampling and converting information

HAProxy supports information sampling using a wide set of “sample fetch functions”. The principle is to extract pieces of information known as samples, for immediate use. This is used for stickiness, to build conditions, to produce information in logs or to enrich HTTP headers.

Advanced Features

Mon, 01 Jan 0001 00:00:00 +0000

3.5.1. Advanced features : Management

HAProxy is designed to remain extremely stable and safe to manage in a regular production environment. It is provided as a single executable file which doesn’t require any installation process. Multiple versions can easily coexist, meaning that it’s possible (and recommended) to upgrade instances progressively by order of importance instead of migrating all of them at once. Configuration files are easily versioned. Configuration checking is done off-line so it doesn’t require to restart a service that will possibly fail. During configuration checks, a number of advanced mistakes may be detected (e.g. a rule hiding another one, or stickiness that will not work) and detailed warnings and configuration hints are proposed to fix them. Backwards configuration file compatibility goes very far away in time, with version 1.5 still fully supporting configurations for versions 1.1 written 13 years before, and 1.6 only dropping support for almost unused, obsolete keywords that can be done differently. The configuration and software upgrade mechanism is smooth and non disruptive in that it allows old and new processes to coexist on the system, each handling its own connections. System status, build options, and library compatibility are reported on startup.

Sizing and Performance

Mon, 01 Jan 0001 00:00:00 +0000

Typical CPU usage figures show 15% of the processing time spent in HAProxy versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy versus 70% for the kernel in HTTP keep-alive mode. This means that the operating system and its tuning have a strong impact on the global performance.

Usages vary a lot between users, some focus on bandwidth, other ones on request rate, others on connection concurrency, others on SSL performance. This section aims at providing a few elements to help with this task.

Releases, Packages, and Upgrades

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is an open source project covered by the GPLv2 license, meaning that everyone is allowed to redistribute it provided that access to the sources is also provided upon request, especially if any modifications were made.

HAProxy evolves as a main development branch called “master” or “mainline”, from which new branches are derived once the code is considered stable. A lot of web sites run some development branches in production on a voluntarily basis, either to participate to the project or because they need a bleeding edge feature, and their feedback is highly valuable to fix bugs and judge the overall quality and stability of the version being developed.

Companion Products and Alternatives

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy integrates fairly well with certain products listed below, which is why they are mentioned here even if not directly related to HAProxy.

4.1. Apache HTTP server

Apache is the de-facto standard HTTP server. It’s a very complete and modular project supporting both file serving and dynamic contents. It can serve as a frontend for some application servers. It can even proxy requests and cache responses. In all of these use cases, a front load balancer is commonly needed. Apache can work in various modes, some being heavier than others. Certain modules still require the heavier pre-forked model and will prevent Apache from scaling well with a high number of connections. In this case HAProxy can provide a tremendous help by enforcing the per-server connection limits to a safe value and will significantly speed up the server and preserve its resources that will be better used by the application.

Documentation and Community

Mon, 01 Jan 0001 00:00:00 +0000

1. Available documentation

The complete HAProxy documentation is contained in the following documents. Please ensure to consult the relevant documentation to save time and to get the most accurate response to your needs. Also please refrain from sending questions to the mailing list whose responses are present in these documents.

intro.txt (this document): it presents the basics of load balancing, HAProxy as a product, what it does, what it doesn’t do, some known traps to avoid, some OS-specific limitations, how to get it, how it evolves, how to ensure you’re running with all known fixes, how to update it, complements and alternatives.

Configuration Manual

Mon, 01 Jan 0001 00:00:00 +0000

1. Quick Reminder About HTTP

Mon, 01 Jan 0001 00:00:00 +0000

This document covers the configuration language as implemented in the version specified above. It does not provide any hints, examples, or advice. For such documentation, please refer to the Reference Manual or the Architecture Manual. The numbered chapters are ordered in the flat HAProxy sidebar for direct navigation.

When HAProxy is running in HTTP mode, both the request and the response are fully analyzed and indexed, thus it becomes possible to build matching criteria on almost anything found in the contents.

2. Configuring HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

2.1. Configuration file format

HAProxy’s configuration process involves 3 major sources of parameters:

the arguments from the command-line, which always take precedence
the configuration file(s), whose format is described here
the running process’s environment, in case some environment variables are explicitly referenced

The configuration file follows a fairly simple hierarchical format which obey a few basic rules:

3. Global Section

Mon, 01 Jan 0001 00:00:00 +0000

Parameters in the “global” section are process-wide and often OS-specific. They are generally set once for all and do not need being changed once correct. Some of them have command-line equivalents.

The following keywords are supported in the “global” section:

Process management and security
- 51degrees-allow-unmatched
- 51degrees-cache-size
- 51degrees-data-file
- 51degrees-difference
- 51degrees-drift
- 51degrees-property-name-list
- 51degrees-property-separator
- 51degrees-use-performance-graph
- 51degrees-use-predictive-graph
- ca-base
- chroot
- cluster-secret
- cpu-affinity
- cpu-map
- cpu-policy
- cpu-set
- crt-base
- daemon
- default-path
- description
- deviceatlas-json-file
- deviceatlas-log-level
- deviceatlas-properties-cookie
- deviceatlas-separator
- dns-accept-family
- expose-deprecated-directives
- expose-experimental-directives
- external-check
- fd-hard-limit
- gid
- grace
- group
- h1-accept-payload-with-any-method
- h1-case-adjust
- h1-case-adjust-file
- h1-do-not-close-on-insecure-transfer-encoding
- h2-workaround-bogus-websocket-clients
- hard-stop-after
- harden.reject-privileged-ports.tcp
- harden.reject-privileged-ports.quic
- insecure-fork-wanted
- insecure-setuid-wanted
- issuers-chain-path
- jwt.decrypt_alg_list
- jwt.decrypt_enc_list
- key-base
- limited-quic
- localpeer
- log
- log-send-hostname
- log-tag
- lua-load
- lua-load-per-thread
- lua-prepend-path
- max-threads-per-group
- mworker-max-reloads
- nbthread
- node
- numa-cpu-mapping
- ocsp-update.disable
- ocsp-update.maxdelay
- ocsp-update.mindelay
- ocsp-update.httpproxy
- ocsp-update.mode
- pidfile
- pp2-never-send-local
- presetenv
- prealloc-fd
- resetenv
- set-dumpable
- set-var
- setenv
- ssl-default-bind-ciphers
- ssl-default-bind-ciphersuites
- ssl-default-bind-client-sigalgs
- ssl-default-bind-curves
- ssl-default-bind-options
- ssl-default-bind-sigalgs
- ssl-default-server-ciphers
- ssl-default-server-ciphersuites
- ssl-default-server-client-sigalgs
- ssl-default-server-curves
- ssl-default-server-options
- ssl-default-server-sigalgs
- ssl-dh-param-file
- ssl-propquery
- ssl-provider
- ssl-provider-path
- ssl-security-level
- ssl-server-verify
- ssl-skip-self-issued-ca
- stats
- stats-file
- strict-limits
- uid
- ulimit-n
- unix-bind
- unsetenv
- user
- wurfl-cache-size
- wurfl-data-file
- wurfl-information-list
- wurfl-information-list-separator
Performance tuning

4. Proxies

Mon, 01 Jan 0001 00:00:00 +0000

Proxy configuration can be located in a set of sections:

defaults [<name>] [ from <defaults_name> ]
frontend <name> [ from <defaults_name> ]
backend <name> [ from <defaults_name> ]
listen <name> [ from <defaults_name> ]

A “frontend” section describes a set of listening sockets accepting client connections.

A “backend” section describes a set of servers to which the proxy will connect to forward incoming connections.

5. Bind and Server Options

Mon, 01 Jan 0001 00:00:00 +0000

The “bind”, “server” and “default-server” keywords support a number of settings depending on some build options and on the system HAProxy was built on. These settings generally each consist in one word sometimes followed by a value, written on the same line as the “bind” or “server” line. All these options are described in this section.

5.1. Bind options

The “bind” keyword supports a certain number of settings which are all passed as arguments on the same line. The order in which those arguments appear makes no importance, provided that they appear after the bind address. All of these parameters are optional. Some of them consist in a single words (booleans), while other ones expect a value after them. In this case, the value must be provided immediately after the setting name.

6. Cache

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy provides a cache, which was designed to perform cache on small objects (favicon, css…). This is a minimalist low-maintenance cache which runs in RAM.

The cache is based on a memory area shared between all threads, and split in 1kB blocks.

If an object is not used anymore, it can be deleted to store a new object independently of its expiration date. The oldest objects are deleted first when we try to allocate a new one.

7. ACLs and Sample Fetching

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is capable of extracting data from request or response streams, from client or server information, from tables, environmental information etc… The action of extracting such data is called fetching a sample. Once retrieved, these samples may be used for various purposes such as a key to a stick-table, but most common usages consist in matching them against predefined constant data called patterns.

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

One of HAProxy’s strong points certainly lies is its precise logs. It probably provides the finest level of information available for such a product, which is very important for troubleshooting complex environments. Standard information provided in logs include client ports, TCP/HTTP state timers, precise stream state at termination and precise termination cause, information about decisions to direct traffic to a server, and of course the ability to capture arbitrary headers.

9. Supported Filters

Mon, 01 Jan 0001 00:00:00 +0000

Here are listed officially supported filters with the list of parameters they accept. Depending on compile options, some of these filters might be unavailable. The list of available filters is reported in haproxy -vv.

9.1. Trace

filter trace [name <name>] [random-forwarding] [max-fwd <max>] [hexdump]

Arguments:

<name> is an arbitrary name that will be reported in
 messages. If no name is provided, "TRACE" is used.

<quiet> inhibits trace messages.

<random-forwarding> enables the random forwarding of parsed data. By
 default, this filter forwards all previously parsed
 data. With this parameter, it only forwards a random
 amount of the parsed data.

<max> is the maximum amount of data that can be forwarded at
 a time. "max-fwd" option can be combined with the
 random forwarding. <max> must be an positive integer.
 0 means there is no limit.

<hexdump> dumps all forwarded data to the server and the client.

This filter can be used as a base to develop new filters. It defines all callbacks and print a message on the standard error stream (stderr) with useful information for all of them. It may be useful to debug the activity of other filters or, quite simply, HAProxy’s activity.

10. FastCGI Applications

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is able to send HTTP requests to Responder FastCGI applications. This feature was added in HAProxy 2.1. To do so, servers must be configured to use the FastCGI protocol (using the keyword “proto fcgi” on the server line) and a FastCGI application must be configured and used by the backend managing these servers (using the keyword “use-fcgi-app” into the proxy section). Several FastCGI applications may be defined, but only one can be used at a time by a backend.

11. Stick Tables and Peers

Mon, 01 Jan 0001 00:00:00 +0000

Stick-tables in HAProxy are a mechanism which permits to associate a certain number of information and metrics with a key of a certain type, and this for a certain duration after the last update. This can be seen as a multicolumn line in a table, where the line number is defined by the key value, and the columns all represent distinct criteria.

12. Other Sections

Mon, 01 Jan 0001 00:00:00 +0000

The sections described below are less commonly used and usually support only a few parameters. There is no implicit relation between any of them. They’re all started using a single keyword. None of them is permitted before a “global” section. The support for some of them might be conditioned by build options (e.g. anything SSL-related).

12.1. Traces

For debugging purpose, it is possible to activate traces on an HAProxy’s subsystem. This will dump debug messages about a specific subsystem. It is a very powerful tool to diagnose issues. Traces can be dynamically configured via the CLI. It is also possible to predefined some settings in the configuration file, in dedicated “traces” sections. More details about traces can be found in the management guide. It remains a developer tools used during complex debugging sessions. It is pretty verbose and have a cost, so use it with caution. And because it is a developer tool, there is no warranty about the backward compatibility of this section.

Management Guide

Mon, 01 Jan 0001 00:00:00 +0000

1. Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to start, stop, manage, and troubleshoot HAProxy, as well as some known limitations and traps to avoid. It does not describe how to configure it (for this please read configuration.txt ).

In this document it is assumed that the reader has sufficient administration skills on a UNIX-like operating system, uses the shell on a daily basis and is familiar with troubleshooting utilities such as strace and tcpdump.

2. HAProxy Architecture

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it uses event multiplexing to schedule all of its activities instead of relying on the system to schedule between multiple activities. Most of the time it runs as a single process, so the output of “ps aux” on a system will report only one “haproxy” process, unless a soft reload is in progress and an older process is finishing its job in parallel to the new one. It is thus always easy to trace its activity using the strace utility. In order to scale with the number of available processors, by default haproxy will start one worker thread per processor it is allowed to run on. Unless explicitly configured differently, the incoming traffic is spread over all these threads, all running the same event loop. A great care is taken to limit inter-thread dependencies to the strict minimum, so as to try to achieve near-linear scalability. This has some impacts such as the fact that a given connection is served by a single thread. Thus in order to use all available processing capacity, it is needed to have at least as many connections as there are threads, which is almost always granted.

3. Starting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is started by invoking the “haproxy” program with a number of arguments passed on the command line. The actual syntax is:

$ haproxy [<options>]*

where [<options>]* is any number of options. An option always starts with ‘-’ followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one in the “global” section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is:

4. Stopping and Restarting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed. The graceful stop is triggered when the SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding from listening ports, but continue to process existing connections until they close. Once the last connection is closed, the process leaves.

5. File-Descriptor Limitations

Mon, 01 Jan 0001 00:00:00 +0000

In order to ensure that all incoming connections will successfully be served, HAProxy computes at load time the total number of file descriptors that will be needed during the process’s life. A regular Unix process is generally granted 1024 file descriptors by default, and a privileged process can raise this limit itself. This is one reason for starting HAProxy as root and letting it adjust the limit. The default limit of 1024 file descriptors roughly allow about 500 concurrent connections to be processed. The computation is based on the global maxconn parameter which limits the total number of connections per process, the number of listeners, the number of servers which have a health check enabled, the agent checks, the peers, the loggers and possibly a few other technical requirements. A simple rough estimate of this number consists in simply doubling the maxconn value and adding a few tens to get the approximate number of file descriptors needed.

6. Memory Management

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy uses a simple and fast pool-based memory management. Since it relies on a small number of different object types, it’s much more efficient to pick new objects from a pool which already contains objects of the appropriate size than to call malloc() for each different size. The pools are organized as a stack or LIFO, so that newly allocated objects are taken from recently released objects still hot in the CPU caches. Pools of similar sizes are merged together, in order to limit memory fragmentation.

7. CPU Usage

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy normally spends most of its time in the system and a smaller part in userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end connection setups and closes per second at 100% CPU on a single core. When one core is saturated, typical figures are:

95% system, 5% user for long TCP connections or large HTTP objects
85% system and 15% user for short TCP connections or small HTTP objects in close mode
70% system and 30% user for small HTTP objects in keep-alive mode

The amount of rules processing and regular expressions will increase the user land part. The presence of firewall rules, connection tracking, complex routing tables in the system will instead increase the system part.

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

For logging, HAProxy always relies on a syslog server since it does not perform any file-system access. The standard way of using it is to send logs over UDP to the log server (by default on port 514). Very commonly this is configured to 127.0.0.1 where the local syslog daemon is running, but it’s also used over the network to log to a central server. The central server provides additional benefits especially in active-active scenarios where it is desirable to keep the logs merged in arrival order. HAProxy may also make use of a UNIX socket to send its logs to the local syslog daemon, but it is not recommended at all, because if the syslog server is restarted while haproxy runs, the socket will be replaced and new logs will be lost. Since HAProxy will be isolated inside a chroot jail, it will not have the ability to reconnect to the new socket. It has also been observed in field that the log buffers in use on UNIX sockets are very small and lead to lost messages even at very light loads. But this can be fine for testing however.

9. Statistics and Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket.

Statistics are regroup in categories labelled as domains, corresponding to the multiple components of HAProxy. There are two domains available: proxy and resolvers. If not specified, the proxy domain is selected. Note that only the proxy statistics are printed on the HTTP page.

10. Easier Configuration Management

Mon, 01 Jan 0001 00:00:00 +0000

It is very common that two HAProxy nodes constituting a cluster share exactly the same configuration modulo a few addresses. Instead of having to maintain a duplicate configuration for each node, which will inevitably diverge, it is possible to include environment variables in the configuration. Thus multiple configuration may share the exact same file with only a few different system wide environment variables. This started in version 1.5 where only addresses were allowed to include environment variables, and 1.6 goes further by supporting environment variables everywhere. The syntax is the same as in the UNIX shell, a variable starts with a dollar sign (’$’), followed by an opening curly brace (’{’), then the variable name followed by the closing brace (’}’). Except for addresses, environment variables are only interpreted in arguments surrounded with double quotes (this was necessary not to break existing setups using regular expressions involving the dollar symbol).

11. Well-Known Traps to Avoid

Mon, 01 Jan 0001 00:00:00 +0000

Once in a while, someone reports that after a system reboot, the haproxy service wasn’t started, and that once they start it by hand it works. Most often, these people are running a clustered IP address mechanism such as keepalived, to assign the service IP address to the master node only, and while it used to work when they used to bind haproxy to address 0.0.0.0, it stopped working after they bound it to the virtual IP address. What happens here is that when the service starts, the virtual IP address is not yet owned by the local node, so when HAProxy wants to bind to it, the system rejects this because it is not a local IP address. The fix doesn’t consist in delaying the haproxy service startup (since it wouldn’t stand a restart), but instead to properly configure the system to allow binding to non-local addresses. This is easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This is also needed in order to transparently intercept the IP traffic that passes through HAProxy for a specific target address.

12. Debugging and Performance Issues

Mon, 01 Jan 0001 00:00:00 +0000

When HAProxy is started with the “-d” option, it will stay in the foreground and will print one line per event, such as an incoming connection, the end of a connection, and for each request or response header line seen. This debug output is emitted before the contents are processed, so they don’t consider the local modifications. The main use is to show the request and response without having to run a network sniffer. The output is less readable when multiple connections are handled in parallel, though the “debug2ansi” and “debug2html” scripts found in the examples/ directory definitely help here by coloring the output.

13. Security Considerations

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is designed to run with very limited privileges. The standard way to use it is to isolate it into a chroot jail and to drop its privileges to a non-root user without any permissions inside this jail so that if any future vulnerability were to be discovered, its compromise would not affect the rest of the system.

In order to perform a chroot, it first needs to be started as a root user. It is pointless to build hand-made chroots to start the process there, these ones are painful to build, are never properly maintained and always contain way more bugs than the main file-system. And in case of compromise, the intruder can use the purposely built file-system. Unfortunately many administrators confuse “start as root” and “run as root”, resulting in the uid change to be done prior to starting haproxy, and reducing the effective security restrictions.