HAProxy 3.4.4 Documentation

• 1: Load Balancing Fundamentals
• 2: What HAProxy Is and How It Works
• 3: Basic Features
• 4: Standard Features
• 5: Advanced Features
• 6: Sizing and Performance
• 7: Releases, Packages, and Upgrades
• 8: Companion Products and Alternatives
• 9: Documentation and Community
• 10: 1. Quick Reminder About HTTP
• 11: 2. Configuring HAProxy
• 12: 3. Global Section
• 13: 4. Proxies
• 14: 5. Bind and Server Options
• 15: 6. Cache
• 16: 7. ACLs and Sample Fetching
• 17: 8. Logging
• 18: 9. Supported Filters
• 19: 10. FastCGI Applications
• 20: 11. Stick Tables and Peers
• 21: 12. Other Sections
• 22: 1. Prerequisites
• 23: 2. HAProxy Architecture
• 24: 3. Starting HAProxy
• 25: 4. Stopping and Restarting HAProxy
• 26: 5. File-Descriptor Limitations
• 27: 6. Memory Management
• 28: 7. CPU Usage
• 29: 8. Logging
• 30: 9. Statistics and Monitoring
• 31: 10. Easier Configuration Management
• 32: 11. Well-Known Traps to Avoid
• 33: 12. Debugging and Performance Issues
• 34: 13. Security Considerations

1 - Load Balancing Fundamentals

This document is an introduction to HAProxy for all those who don’t know it, as well as for those who want to re-discover it when they know older versions. Its primary focus is to provide users with all the elements to decide if HAProxy is the product they’re looking for or not. Advanced users may find here some parts of solutions to some ideas they had just because they were not aware of a given new feature. Some sizing information is also provided, the product’s lifecycle is explained, and comparisons with partially overlapping products are provided.

This document doesn’t provide any configuration help or hints, but it explains where to find the relevant documents. The guide is presented as a flat sequence of topic pages in the HAProxy sidebar.

Load balancing consists in aggregating multiple components in order to achieve a total processing capacity above each component’s individual capacity, without any intervention from the end user and in a scalable way. This results in more operations being performed simultaneously by the time it takes a component to perform only one. A single operation however will still be performed on a single component at a time and will not get faster than without load balancing. It always requires at least as many operations as available components and an efficient load balancing mechanism to make use of all components and to fully benefit from the load balancing. A good example of this is the number of lanes on a highway which allows as many cars to pass during the same time frame without increasing their individual speed.

Examples of load balancing:

• Process scheduling in multi-processor systems
• Link load balancing (e.g. EtherChannel, Bonding)
• IP address load balancing (e.g. ECMP, DNS round-robin)
• Server load balancing (via load balancers)

The mechanism or component which performs the load balancing operation is called a load balancer. In web environments these components are called a “network load balancer”, and more commonly a “load balancer” given that this activity is by far the best known case of load balancing.

A load balancer may act:

• at the link level: this is called link load balancing, and it consists in choosing what network link to send a packet to;

• at the network level: this is called network load balancing, and it consists in choosing what route a series of packets will follow;

• at the server level: this is called server load balancing and it consists in deciding what server will process a connection or request.

Two distinct technologies exist and address different needs, though with some overlapping. In each case it is important to keep in mind that load balancing consists in diverting the traffic from its natural flow and that doing so always requires a minimum of care to maintain the required level of consistency between all routing decisions.

The first one acts at the packet level and processes packets more or less individually. There is a 1-to-1 relation between input and output packets, so it is possible to follow the traffic on both sides of the load balancer using a regular network sniffer. This technology can be very cheap and extremely fast. It is usually implemented in hardware (ASICs) allowing to reach line rate, such as switches doing ECMP. Usually stateless, it can also be stateful (consider the session a packet belongs to and called layer4-LB or L4), may support DSR (direct server return, without passing through the LB again) if the packets were not modified, but provides almost no content awareness. This technology is very well suited to network-level load balancing, though it is sometimes used for very basic server load balancing at high speed.

The second one acts on session contents. It requires that the input streams is reassembled and processed as a whole. The contents may be modified, and the output stream is segmented into new packets. For this reason it is generally performed by proxies and they’re often called layer 7 load balancers or L7. This implies that there are two distinct connections on each side, and that there is no relation between input and output packets sizes nor counts. Clients and servers are not required to use the same protocol (for example IPv4 vs IPv6, clear vs SSL). The operations are always stateful, and the return traffic must pass through the load balancer. The extra processing comes with a cost so it’s not always possible to achieve line rate, especially with small packets. On the other hand, it offers wide possibilities and is generally achieved by pure software, even if embedded into hardware appliances. This technology is very well suited for server load balancing.

Packet-based load balancers are generally deployed in cut-through mode, so they are installed on the normal path of the traffic and divert it according to the configuration. The return traffic doesn’t necessarily pass through the load balancer. Some modifications may be applied to the network destination address in order to direct the traffic to the proper destination. In this case, it is mandatory that the return traffic passes through the load balancer. If the routes doesn’t make this possible, the load balancer may also replace the packets’ source address with its own in order to force the return traffic to pass through it.

Proxy-based load balancers are deployed as a server with their own IP addresses and ports, without architecture changes. Sometimes this requires to perform some adaptations to the applications so that clients are properly directed to the load balancer’s IP address and not directly to the server’s. Some load balancers may have to adjust some servers’ responses to make this possible (e.g. the HTTP Location header field used in HTTP redirects). Some proxy-based load balancers may intercept traffic for an address they don’t own, and spoof the client’s address when connecting to the server. This allows them to be deployed as if they were a regular router or firewall, in a cut-through mode very similar to the packet based load balancers. This is particularly appreciated for products which combine both packet mode and proxy mode. In this case DSR is obviously still not possible and the return traffic still has to be routed back to the load balancer.

A very scalable layered approach would consist in having a front router which receives traffic from multiple load balanced links, and uses ECMP to distribute this traffic to a first layer of multiple stateful packet-based load balancers (L4). These L4 load balancers in turn pass the traffic to an even larger number of proxy-based load balancers (L7), which have to parse the contents to decide what server will ultimately receive the traffic.

The number of components and possible paths for the traffic increases the risk of failure; in very large environments, it is even normal to permanently have a few faulty components being fixed or replaced. Load balancing done without awareness of the whole stack’s health significantly degrades availability. For this reason, any sane load balancer will verify that the components it intends to deliver the traffic to are still alive and reachable, and it will stop delivering traffic to faulty ones. This can be achieved using various methods.

The most common one consists in periodically sending probes to ensure the component is still operational. These probes are called “health checks”. They must be representative of the type of failure to address. For example a ping-based check will not detect that a web server has crashed and doesn’t listen to a port anymore, while a connection to the port will verify this, and a more advanced request may even validate that the server still works and that the database it relies on is still accessible. Health checks often involve a few retries to cover for occasional measuring errors. The period between checks must be small enough to ensure the faulty component is not used for too long after an error occurs.

Other methods consist in sampling the production traffic sent to a destination to observe if it is processed correctly or not, and to evict the components which return inappropriate responses. However this requires to sacrifice a part of the production traffic and this is not always acceptable. A combination of these two mechanisms provides the best of both worlds, with both of them being used to detect a fault, and only health checks to detect the end of the fault. A last method involves centralized reporting: a central monitoring agent periodically updates all load balancers about all components’ state. This gives a global view of the infrastructure to all components, though sometimes with less accuracy or responsiveness. It’s best suited for environments with many load balancers and many servers.

Layer 7 load balancers also face another challenge known as stickiness or persistence. The principle is that they generally have to direct multiple subsequent requests or connections from a same origin (such as an end user) to the same target. The best known example is the shopping cart on an online store. If each click leads to a new connection, the user must always be sent to the server which holds his shopping cart. Content-awareness makes it easier to spot some elements in the request to identify the server to deliver it to, but that’s not always enough. For example if the source address is used as a key to pick a server, it can be decided that a hash-based algorithm will be used and that a given IP address will always be sent to the same server based on a divide of the address by the number of available servers. But if one server fails, the result changes and all users are suddenly sent to a different server and lose their shopping cart. The solution against this issue consists in memorizing the chosen target so that each time the same visitor is seen, he’s directed to the same server regardless of the number of available servers. The information may be stored in the load balancer’s memory, in which case it may have to be replicated to other load balancers if it’s not alone, or it may be stored in the client’s memory using various methods provided that the client is able to present this information back with every request (cookie insertion, redirection to a sub-domain, etc). This mechanism provides the extra benefit of not having to rely on unstable or unevenly distributed information (such as the source IP address). This is in fact the strongest reason to adopt a layer 7 load balancer instead of a layer 4 one.

In order to extract information such as a cookie, a host header field, a URL or whatever, a load balancer may need to decrypt SSL/TLS traffic and even possibly to re-encrypt it when passing it to the server. This expensive task explains why in some high-traffic infrastructures, sometimes there may be a lot of load balancers.

Since a layer 7 load balancer may perform a number of complex operations on the traffic (decrypt, parse, modify, match cookies, decide what server to send to, etc), it can definitely cause some trouble and will very commonly be accused of being responsible for a lot of trouble that it only revealed. Often it will be discovered that servers are unstable and periodically go up and down, or for web servers, that they deliver pages with some hard-coded links forcing the clients to connect directly to one specific server without passing via the load balancer, or that they take ages to respond under high load causing timeouts. That’s why logging is an extremely important aspect of layer 7 load balancing. Once a trouble is reported, it is important to figure if the load balancer took a wrong decision and if so why so that it doesn’t happen anymore.

2 - What HAProxy Is and How It Works

HAProxy is written as “HAProxy” to designate the product, and as “haproxy” to designate the executable program, software package or a process. However, both are commonly used for both purposes, and are pronounced H-A-Proxy. Very early, “haproxy” used to stand for “high availability proxy” and the name was written in two separate words, though by now it means nothing else than “HAProxy”.

3.1. What HAProxy is and isn’t

HAProxy is:

• a TCP proxy: it can accept a TCP connection from a listening socket, connect to a server and attach these sockets together allowing traffic to flow in both directions; IPv4, IPv6 and even UNIX sockets are supported on either side, so this can provide an easy way to translate addresses between different families.

• an HTTP reverse-proxy (called a “gateway” in HTTP terminology): it presents itself as a server, receives HTTP requests over connections accepted on a listening TCP socket, and passes the requests from these connections to servers using different connections. It may use any combination of HTTP/1.x or HTTP/2 on any side and will even automatically detect the protocol spoken on each side when ALPN is used over TLS.

• an SSL terminator / initiator / offloader: SSL/TLS may be used on the connection coming from the client, on the connection going to the server, or even on both connections. A lot of settings can be applied per name (SNI), and may be updated at runtime without restarting. Such setups are extremely scalable and deployments involving tens to hundreds of thousands of certificates were reported.

• a TCP normalizer: since connections are locally terminated by the operating system, there is no relation between both sides, so abnormal traffic such as invalid packets, flag combinations, window advertisements, sequence numbers, incomplete connections (SYN floods), or so will not be passed to the other side. This protects fragile TCP stacks from protocol attacks, and also allows to optimize the connection parameters with the client without having to modify the servers’ TCP stack settings.

• an HTTP normalizer: when configured to process HTTP traffic, only valid complete requests are passed. This protects against a lot of protocol-based attacks. Additionally, protocol deviations for which there is a tolerance in the specification are fixed so that they don’t cause problem on the servers (e.g. multiple-line headers).

• an HTTP fixing tool: it can modify / fix / add / remove / rewrite the URL or any request or response header. This helps fixing interoperability issues in complex environments.

• a content-based switch: it can consider any element from the request to decide what server to pass the request or connection to. Thus it is possible to handle multiple protocols over a same port (e.g. HTTP, HTTPS, SSH).

• a server load balancer: it can load balance TCP connections and HTTP requests. In TCP mode, load balancing decisions are taken for the whole connection. In HTTP mode, decisions are taken per request.

• a traffic regulator: it can apply some rate limiting at various points, protect the servers against overloading, adjust traffic priorities based on the contents, and even pass such information to lower layers and outer network components by marking packets.

• a protection against DDoS and service abuse: it can maintain a wide number of statistics per IP address, URL, cookie, etc and detect when an abuse is happening, then take action (slow down the offenders, block them, send them to outdated contents, etc).

• an observation point for network troubleshooting: due to the precision of the information reported in logs, it is often used to narrow down some network-related issues.

• an HTTP compression offloader: it can compress responses which were not compressed by the server, thus reducing the page load time for clients with poor connectivity or using high-latency, mobile networks.

• a caching proxy: it may cache responses in RAM so that subsequent requests for the same object avoid the cost of another network transfer from the server as long as the object remains present and valid. It will however not store objects to any persistent storage. Please note that this caching feature is designed to be maintenance free and focuses solely on saving haproxy’s precious resources and not on save the server’s resources. Caches designed to optimize servers require much more tuning and flexibility. If you instead need such an advanced cache, please use Varnish Cache, which integrates perfectly with haproxy, especially when SSL/TLS is needed on any side.

• a FastCGI gateway: FastCGI can be seen as a different representation of HTTP, and as such, HAProxy can directly load-balance a farm comprising any combination of FastCGI application servers without requiring to insert another level of gateway between them. This results in resource savings and a reduction of maintenance costs.

HAProxy is not:

• an explicit HTTP proxy, i.e. the proxy that browsers use to reach the internet. There are excellent open-source software dedicated for this task, such as Squid. However HAProxy can be installed in front of such a proxy to provide load balancing and high availability.

• a data scrubber: it will not modify the body of requests nor responses.

• a static web server: during startup, it isolates itself inside a chroot jail and drops its privileges, so that it will not perform any single file-system access once started. As such it cannot be turned into a static web server (dynamic servers are supported through FastCGI however). There are excellent open-source software for this such as Apache or Nginx, and HAProxy can be easily installed in front of them to provide load balancing, high availability and acceleration.

• a packet-based load balancer: it will not see IP packets nor UDP datagrams, will not perform NAT or even less DSR. These are tasks for lower layers. Some kernel-based components such as IPVS (Linux Virtual Server) already do this pretty well and complement perfectly with HAProxy.

3.2. How HAProxy works

HAProxy is an event-driven, non-blocking engine combining a very fast I/O layer with a priority-based, multi-threaded scheduler. As it is designed with a data forwarding goal in mind, its architecture is optimized to move data as fast as possible with the least possible operations. It focuses on optimizing the CPU cache’s efficiency by sticking connections to the same CPU as long as possible. As such it implements a layered model offering bypass mechanisms at each level ensuring data doesn’t reach higher levels unless needed. Most of the processing is performed in the kernel, and HAProxy does its best to help the kernel do the work as fast as possible by giving some hints or by avoiding certain operation when it guesses they could be grouped later. As a result, typical figures show 15% of the processing time spent in HAProxy versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy versus 70% for the kernel in HTTP keep-alive mode.

A single process can run many proxy instances; configurations as large as 300000 distinct proxies in a single process were reported to run fine. A single core, single CPU setup is far more than enough for more than 99% users, and as such, users of containers and virtual machines are encouraged to use the absolute smallest images they can get to save on operational costs and simplify troubleshooting. However the machine HAProxy runs on must never ever swap, and its CPU must not be artificially throttled (sub-CPU allocation in hypervisors) nor be shared with compute-intensive processes which would induce a very high context-switch latency.

Threading allows to exploit all available processing capacity by using one thread per CPU core. This is mostly useful for SSL or when data forwarding rates above 40 Gbps are needed. In such cases it is critically important to avoid communications between multiple physical CPUs, which can cause strong bottlenecks in the network stack and in HAProxy itself. While counter-intuitive to some, the first thing to do when facing some performance issues is often to reduce the number of CPUs HAProxy runs on.

HAProxy only requires the haproxy executable and a configuration file to run. For logging it is highly recommended to have a properly configured syslog daemon and log rotations in place. Logs may also be sent to stdout/stderr, which can be useful inside containers. The configuration files are parsed before starting, then HAProxy tries to bind all listening sockets, and refuses to start if anything fails. Past this point it cannot fail anymore. This means that there are no runtime failures and that if it accepts to start, it will work until it is stopped.

Once HAProxy is started, it does exactly 3 things:

• process incoming connections;

• periodically check the servers’ status (known as health checks);

• exchange information with other haproxy nodes.

Processing incoming connections is by far the most complex task as it depends on a lot of configuration possibilities, but it can be summarized as the 9 steps below:

• accept incoming connections from listening sockets that belong to a configuration entity known as a “frontend”, which references one or multiple listening addresses;

• apply the frontend-specific processing rules to these connections that may result in blocking them, modifying some headers, or intercepting them to execute some internal applets such as the statistics page or the CLI;

• pass these incoming connections to another configuration entity representing a server farm known as a “backend”, which contains the list of servers and the load balancing strategy for this server farm;

• apply the backend-specific processing rules to these connections;

• decide which server to forward the connection to according to the load balancing strategy;

• apply the backend-specific processing rules to the response data;

• apply the frontend-specific processing rules to the response data;

• emit a log to report what happened in fine details;

• in HTTP, loop back to the second step to wait for a new request, otherwise close the connection.

Frontends and backends are sometimes considered as half-proxies, since they only look at one side of an end-to-end connection; the frontend only cares about the clients while the backend only cares about the servers. HAProxy also supports full proxies which are exactly the union of a frontend and a backend. When HTTP processing is desired, the configuration will generally be split into frontends and backends as they open a lot of possibilities since any frontend may pass a connection to any backend. With TCP-only proxies, using frontends and backends rarely provides a benefit and the configuration can be more readable with full proxies.

3 - Basic Features

This section will enumerate a number of features that HAProxy implements, some of which are generally expected from any modern load balancer, and some of which are a direct benefit of HAProxy’s architecture. More advanced features will be detailed in the next section.

3.3.1. Basic features : Proxying

Proxying is the action of transferring data between a client and a server over two independent connections. The following basic features are supported by HAProxy regarding proxying and connection management:

• Provide the server with a clean connection to protect them against any client-side defect or attack;

• Listen to multiple IP addresses and/or ports, even port ranges;

• Transparent accept: intercept traffic targeting any arbitrary IP address that doesn’t even belong to the local system;

• Server port doesn’t need to be related to listening port, and may even be translated by a fixed offset (useful with ranges);

• Transparent connect: spoof the client’s (or any) IP address if needed when connecting to the server;

• Provide a reliable return IP address to the servers in multi-site LBs;

• Offload the server thanks to buffers and possibly short-lived connections to reduce their concurrent connection count and their memory footprint;

• Optimize TCP stacks (e.g. SACK), congestion control, and reduce RTT impacts;

• Support different protocol families on both sides (e.g. IPv4/IPv6/Unix);

• Timeout enforcement: HAProxy supports multiple levels of timeouts depending on the stage the connection is, so that a dead client or server, or an attacker cannot be granted resources for too long;

• Protocol validation: HTTP, SSL, or payload are inspected and invalid protocol elements are rejected, unless instructed to accept them anyway;

• Policy enforcement: ensure that only what is allowed may be forwarded;

• Both incoming and outgoing connections may be limited to certain network namespaces (Linux only), making it easy to build a cross-container, multi-tenant load balancer;

• PROXY protocol presents the client’s IP address to the server even for non-HTTP traffic. This is an HAProxy extension that was adopted by a number of third-party products by now, at least these ones at the time of writing:

  • client: haproxy, stud, stunnel, exaproxy, ELB, squid
  • server: haproxy, stud, postfix, exim, nginx, squid, node.js, varnish

3.3.2. Basic features : SSL

HAProxy’s SSL stack is recognized as one of the most featureful according to Google’s engineers (http://istlsfastyet.com/ ). The most commonly used features making it quite complete are:

• SNI-based multi-hosting with no limit on sites count and focus on performance. At least one deployment is known for running 50000 domains with their respective certificates;

• support for wildcard certificates reduces the need for many certificates;

• certificate-based client authentication with configurable policies on failure to present a valid certificate. This allows to present a different server farm to regenerate the client certificate for example;

• authentication of the backend server ensures the backend server is the real one and not a man in the middle;

• authentication with the backend server lets the backend server know it’s really the expected haproxy node that is connecting to it;

• TLS NPN and ALPN extensions make it possible to reliably offload SPDY/HTTP2 connections and pass them in clear text to backend servers;

• OCSP stapling further reduces first page load time by delivering inline an OCSP response when the client requests a Certificate Status Request;

• Dynamic record sizing provides both high performance and low latency, and significantly reduces page load time by letting the browser start to fetch new objects while packets are still in flight;

• permanent access to all relevant SSL/TLS layer information for logging, access control, reporting etc. These elements can be embedded into HTTP header or even as a PROXY protocol extension so that the offloaded server gets all the information it would have had if it performed the SSL termination itself.

• Detect, log and block certain known attacks even on vulnerable SSL libs, such as the Heartbleed attack affecting certain versions of OpenSSL.

• support for stateless session resumption (RFC 5077 TLS Ticket extension). TLS tickets can be updated from CLI which provides them means to implement Perfect Forward Secrecy by frequently rotating the tickets.

3.3.3. Basic features : Monitoring

HAProxy focuses a lot on availability. As such it cares about servers state, and about reporting its own state to other network components:

• Servers’ state is continuously monitored using per-server parameters. This ensures the path to the server is operational for regular traffic;

• Health checks support two hysteresis for up and down transitions in order to protect against state flapping;

• Checks can be sent to a different address/port/protocol: this makes it easy to check a single service that is considered representative of multiple ones, for example the HTTPS port for an HTTP+HTTPS server.

• Servers can track other servers and go down simultaneously: this ensures that servers hosting multiple services can fail atomically and that no one will be sent to a partially failed server;

• Agents may be deployed on the server to monitor load and health: a server may be interested in reporting its load, operational status, administrative status independently from what health checks can see. By running a simple agent on the server, it’s possible to consider the server’s view of its own health in addition to the health checks validating the whole path;

• Various check methods are available: TCP connect, HTTP request, SMTP hello, SSL hello, LDAP, SQL, Redis, send/expect scripts, all with/without SSL;

• State change is notified in the logs and stats page with the failure reason (e.g. the HTTP response received at the moment the failure was detected). An e-mail can also be sent to a configurable address upon such a change;

• Server state is also reported on the stats interface and can be used to take routing decisions so that traffic may be sent to different farms depending on their sizes and/or health (e.g. loss of an inter-DC link);

• HAProxy can use health check requests to pass information to the servers, such as their names, weight, the number of other servers in the farm etc. so that servers can adjust their response and decisions based on this knowledge (e.g. postpone backups to keep more CPU available);

• Servers can use health checks to report more detailed state than just on/off (e.g. I would like to stop, please stop sending new visitors);

• HAProxy itself can report its state to external components such as routers or other load balancers, allowing to build very complete multi-path and multi-layer infrastructures.

3.3.4. Basic features : High availability

Just like any serious load balancer, HAProxy cares a lot about availability to ensure the best global service continuity:

• Only valid servers are used; the other ones are automatically evicted from load balancing farms; under certain conditions it is still possible to force to use them though;

• Support for a graceful shutdown so that it is possible to take servers out of a farm without affecting any connection;

• Backup servers are automatically used when active servers are down and replace them so that sessions are not lost when possible. This also allows to build multiple paths to reach the same server (e.g. multiple interfaces);

• Ability to return a global failed status for a farm when too many servers are down. This, combined with the monitoring capabilities makes it possible for an upstream component to choose a different LB node for a given service;

• Stateless design makes it easy to build clusters: by design, HAProxy does its best to ensure the highest service continuity without having to store information that could be lost in the event of a failure. This ensures that a takeover is the most seamless possible;

• Integrates well with standard VRRP daemon keepalived: HAProxy easily tells keepalived about its state and copes very well with floating virtual IP addresses. Note: only use IP redundancy protocols (VRRP/CARP) over cluster-based solutions (Heartbeat, …) as they’re the ones offering the fastest, most seamless, and most reliable switchover.

3.3.5. Basic features : Load balancing

HAProxy offers a fairly complete set of load balancing features, most of which are unfortunately not available in a number of other load balancing products:

• no less than 10 load balancing algorithms are supported, some of which apply to input data to offer an infinite list of possibilities. The most common ones are round-robin (for short connections, pick each server in turn), leastconn (for long connections, pick the least recently used of the servers with the lowest connection count), source (for SSL farms or terminal server farms, the server directly depends on the client’s source address), URI (for HTTP caches, the server directly depends on the HTTP URI), hdr (the server directly depends on the contents of a specific HTTP header field), first (for short-lived virtual machines, all connections are packed on the smallest possible subset of servers so that unused ones can be powered down);

• all algorithms above support per-server weights so that it is possible to accommodate from different server generations in a farm, or direct a small fraction of the traffic to specific servers (debug mode, running the next version of the software, etc);

• dynamic weights are supported for round-robin, leastconn and consistent hashing; this allows server weights to be modified on the fly from the CLI or even by an agent running on the server;

• slow-start is supported whenever a dynamic weight is supported; this allows a server to progressively take the traffic. This is an important feature for fragile application servers which require to compile classes at runtime as well as cold caches which need to fill up before being run at full throttle;

• hashing can apply to various elements such as client’s source address, URL components, query string element, header field values, POST parameter, RDP cookie;

• consistent hashing protects server farms against massive redistribution when adding or removing servers in a farm. That’s very important in large cache farms and it allows slow-start to be used to refill cold caches;

• a number of internal metrics such as the number of connections per server, per backend, the amount of available connection slots in a backend etc makes it possible to build very advanced load balancing strategies.

3.3.6. Basic features : Stickiness

Application load balancing would be useless without stickiness. HAProxy provides a fairly comprehensive set of possibilities to maintain a visitor on the same server even across various events such as server addition/removal, down/up cycles, and some methods are designed to be resistant to the distance between multiple load balancing nodes in that they don’t require any replication:

• stickiness information can be individually matched and learned from different places if desired. For example a JSESSIONID cookie may be matched both in a cookie and in the URL. Up to 8 parallel sources can be learned at the same time and each of them may point to a different stick-table;

• stickiness information can come from anything that can be seen within a request or response, including source address, TCP payload offset and length, HTTP query string elements, header field values, cookies, and so on.

• stick-tables are replicated between all nodes in a multi-master fashion;

• commonly used elements such as SSL-ID or RDP cookies (for TSE farms) are directly accessible to ease manipulation;

• all sticking rules may be dynamically conditioned by ACLs;

• it is possible to decide not to stick to certain servers, such as backup servers, so that when the nominal server comes back, it automatically takes the load back. This is often used in multi-path environments;

• in HTTP it is often preferred not to learn anything and instead manipulate a cookie dedicated to stickiness. For this, it’s possible to detect, rewrite, insert or prefix such a cookie to let the client remember what server was assigned;

• the server may decide to change or clean the stickiness cookie on logout, so that leaving visitors are automatically unbound from the server;

• using ACL-based rules it is also possible to selectively ignore or enforce stickiness regardless of the server’s state; combined with advanced health checks, that helps admins verify that the server they’re installing is up and running before presenting it to the whole world;

• an innovative mechanism to set a maximum idle time and duration on cookies ensures that stickiness can be smoothly stopped on devices which are never closed (smartphones, TVs, home appliances) without having to store them on persistent storage;

• multiple server entries may share the same stickiness keys so that stickiness is not lost in multi-path environments when one path goes down;

• soft-stop ensures that only users with stickiness information will continue to reach the server they’ve been assigned to but no new users will go there.

3.3.7. Basic features : Logging

Logging is an extremely important feature for a load balancer, first because a load balancer is often wrongly accused of causing the problems it reveals, and second because it is placed at a critical point in an infrastructure where all normal and abnormal activity needs to be analyzed and correlated with other components.

HAProxy provides very detailed logs, with millisecond accuracy and the exact connection accept time that can be searched in firewalls logs (e.g. for NAT correlation). By default, TCP and HTTP logs are quite detailed and contain everything needed for troubleshooting, such as source IP address and port, frontend, backend, server, timers (request receipt duration, queue duration, connection setup time, response headers time, data transfer time), global process state, connection counts, queue status, retries count, detailed stickiness actions and disconnect reasons, header captures with a safe output encoding. It is then possible to extend or replace this format to include any sampled data, variables, captures, resulting in very detailed information. For example it is possible to log the number of cumulative requests or number of different URLs visited by a client.

The log level may be adjusted per request using standard ACLs, so it is possible to automatically silent some logs considered as pollution and instead raise warnings when some abnormal behavior happen for a small part of the traffic (e.g. too many URLs or HTTP errors for a source address). Administrative logs are also emitted with their own levels to inform about the loss or recovery of a server for example.

Each frontend and backend may use multiple independent log outputs, which eases multi-tenancy. Logs are preferably sent over UDP, maybe JSON-encoded, and are truncated after a configurable line length in order to guarantee delivery. But it is also possible to send them to stdout/stderr or any file descriptor, as well as to a ring buffer that a client can subscribe to in order to retrieve them.

3.3.8. Basic features : Statistics

HAProxy provides a web-based statistics reporting interface with authentication, security levels and scopes. It is thus possible to provide each hosted customer with his own page showing only his own instances. This page can be located in a hidden URL part of the regular web site so that no new port needs to be opened. This page may also report the availability of other HAProxy nodes so that it is easy to spot if everything works as expected at a glance. The view is synthetic with a lot of details accessible (such as error causes, last access and last change duration, etc), which are also accessible as a CSV table that other tools may import to draw graphs. The page may self-refresh to be used as a monitoring page on a large display. In administration mode, the page also allows to change server state to ease maintenance operations.

A Prometheus exporter is also provided so that the statistics can be consumed in a different format depending on the deployment.

4 - Standard Features

In this section, some features that are very commonly used in HAProxy but are not necessarily present on other load balancers are enumerated.

3.4.1. Standard features : Sampling and converting information

HAProxy supports information sampling using a wide set of “sample fetch functions”. The principle is to extract pieces of information known as samples, for immediate use. This is used for stickiness, to build conditions, to produce information in logs or to enrich HTTP headers.

Samples can be fetched from various sources:

• constants: integers, strings, IP addresses, binary blocks;

• the process: date, environment variables, server/frontend/backend/process state, byte/connection counts/rates, queue length, random generator, …

• variables: per-session, per-request, per-response variables;

• the client connection: source and destination addresses and ports, and all related statistics counters;

• the SSL client session: protocol, version, algorithm, cipher, key size, session ID, all client and server certificate fields, certificate serial, SNI, ALPN, NPN, client support for certain extensions;

• request and response buffers contents: arbitrary payload at offset/length, data length, RDP cookie, decoding of SSL hello type, decoding of TLS SNI;

• HTTP (request and response): method, URI, path, query string arguments, status code, headers values, positional header value, cookies, captures, authentication, body elements;

A sample may then pass through a number of operators known as “converters” to experience some transformation. A converter consumes a sample and produces a new one, possibly of a completely different type. For example, a converter may be used to return only the integer length of the input string, or could turn a string to upper case. Any arbitrary number of converters may be applied in series to a sample before final use. Among all available sample converters, the following ones are the most commonly used:

• arithmetic and logic operators: they make it possible to perform advanced computation on input data, such as computing ratios, percentages or simply converting from one unit to another one;

• IP address masks are useful when some addresses need to be grouped by larger networks;

• data representation: URL-decode, base64, hex, JSON strings, hashing;

• string conversion: extract substrings at fixed positions, fixed length, extract specific fields around certain delimiters, extract certain words, change case, apply regex-based substitution;

• date conversion: convert to HTTP date format, convert local to UTC and conversely, add or remove offset;

• lookup an entry in a stick table to find statistics or assigned server;

• map-based key-to-value conversion from a file (mostly used for geolocation).

3.4.2. Standard features : Maps

Maps are a powerful type of converter consisting in loading a two-columns file into memory at boot time, then looking up each input sample from the first column and either returning the corresponding pattern on the second column if the entry was found, or returning a default value. The output information also being a sample, it can in turn experience other transformations including other map lookups. Maps are most commonly used to translate the client’s IP address to an AS number or country code since they support a longest match for network addresses but they can be used for various other purposes.

Part of their strength comes from being updatable on the fly either from the CLI or from certain actions using other samples, making them capable of storing and retrieving information between subsequent accesses. Another strength comes from the binary tree based indexation which makes them extremely fast even when they contain hundreds of thousands of entries, making geolocation very cheap and easy to set up.

3.4.3. Standard features : ACLs and conditions

Most operations in HAProxy can be made conditional. Conditions are built by combining multiple ACLs using logic operators (AND, OR, NOT). Each ACL is a series of tests based on the following elements:

• a sample fetch method to retrieve the element to test;

• an optional series of converters to transform the element;

• a list of patterns to match against;

• a matching method to indicate how to compare the patterns with the sample

For example, the sample may be taken from the HTTP “Host” header, it could then be converted to lower case, then matched against a number of regex patterns using the regex matching method.

Technically, ACLs are built on the same core as the maps, they share the exact same internal structure, pattern matching methods and performance. The only real difference is that instead of returning a sample, they only return “found” or or “not found”. In terms of usage, ACL patterns may be declared inline in the configuration file and do not require their own file. ACLs may be named for ease of use or to make configurations understandable. A named ACL may be declared multiple times and it will evaluate all definitions in turn until one matches.

About 13 different pattern matching methods are provided, among which IP address mask, integer ranges, substrings, regex. They work like functions, and just like with any programming language, only what is needed is evaluated, so when a condition involving an OR is already true, next ones are not evaluated, and similarly when a condition involving an AND is already false, the rest of the condition is not evaluated.

There is no practical limit to the number of declared ACLs, and a handful of commonly used ones are provided. However experience has shown that setups using a lot of named ACLs are quite hard to troubleshoot and that sometimes using anonymous ACLs inline is easier as it requires less references out of the scope being analyzed.

3.4.4. Standard features : Content switching

HAProxy implements a mechanism known as content-based switching. The principle is that a connection or request arrives on a frontend, then the information carried with this request or connection are processed, and at this point it is possible to write ACLs-based conditions making use of these information to decide what backend will process the request. Thus the traffic is directed to one backend or another based on the request’s contents. The most common example consists in using the Host header and/or elements from the path (sub-directories or file-name extensions) to decide whether an HTTP request targets a static object or the application, and to route static objects traffic to a backend made of fast and light servers, and all the remaining traffic to a more complex application server, thus constituting a fine-grained virtual hosting solution. This is quite convenient to make multiple technologies coexist as a more global solution.

Another use case of content-switching consists in using different load balancing algorithms depending on various criteria. A cache may use a URI hash while an application would use round-robin.

Last but not least, it allows multiple customers to use a small share of a common resource by enforcing per-backend (thus per-customer connection limits).

Content switching rules scale very well, though their performance may depend on the number and complexity of the ACLs in use. But it is also possible to write dynamic content switching rules where a sample value directly turns into a backend name and without making use of ACLs at all. Such configurations have been reported to work fine at least with 300000 backends in production.

3.4.5. Standard features : Stick-tables

Stick-tables are commonly used to store stickiness information, that is, to keep a reference to the server a certain visitor was directed to. The key is then the identifier associated with the visitor (its source address, the SSL ID of the connection, an HTTP or RDP cookie, the customer number extracted from the URL or from the payload, …) and the stored value is then the server’s identifier.

Stick tables may use 3 different types of samples for their keys: integers, strings and addresses. Only one stick-table may be referenced in a proxy, and it is designated everywhere with the proxy name. Up to 8 keys may be tracked in parallel. The server identifier is committed during request or response processing once both the key and the server are known.

Stick-table contents may be replicated in active-active mode with other HAProxy nodes known as “peers” as well as with the new process during a reload operation so that all load balancing nodes share the same information and take the same routing decision if client’s requests are spread over multiple nodes.

Since stick-tables are indexed on what allows to recognize a client, they are often also used to store extra information such as per-client statistics. The extra statistics take some extra space and need to be explicitly declared. The type of statistics that may be stored includes the input and output bandwidth, the number of concurrent connections, the connection rate and count over a period, the amount and frequency of errors, some specific tags and counters, etc. In order to support keeping such information without being forced to stick to a given server, a special “tracking” feature is implemented and allows to track up to 3 simultaneous keys from different tables at the same time regardless of stickiness rules. Each stored statistics may be searched, dumped and cleared from the CLI and adds to the live troubleshooting capabilities.

While this mechanism can be used to surclass a returning visitor or to adjust the delivered quality of service depending on good or bad behavior, it is mostly used to fight against service abuse and more generally DDoS as it allows to build complex models to detect certain bad behaviors at a high processing speed.

3.4.6. Standard features : Formatted strings

There are many places where HAProxy needs to manipulate character strings, such as logs, redirects, header additions, and so on. In order to provide the greatest flexibility, the notion of Formatted strings was introduced, initially for logging purposes, which explains why it’s still called “log-format”. These strings contain escape characters allowing to introduce various dynamic data including variables and sample fetch expressions into strings, and even to adjust the encoding while the result is being turned into a string (for example, adding quotes). This provides a powerful way to build header contents, to build response data or even response templates, or to customize log lines. Additionally, in order to remain simple to build most common strings, about 50 special tags are provided as shortcuts for information commonly used in logs.

3.4.7. Standard features : HTTP rewriting and redirection

Installing a load balancer in front of an application that was never designed for this can be a challenging task without the proper tools. One of the most commonly requested operation in this case is to adjust requests and response headers to make the load balancer appear as the origin server and to fix hard coded information. This comes with changing the path in requests (which is strongly advised against), modifying Host header field, modifying the Location response header field for redirects, modifying the path and domain attribute for cookies, and so on. It also happens that a number of servers are somewhat verbose and tend to leak too much information in the response, making them more vulnerable to targeted attacks. While it’s theoretically not the role of a load balancer to clean this up, in practice it’s located at the best place in the infrastructure to guarantee that everything is cleaned up.

Similarly, sometimes the load balancer will have to intercept some requests and respond with a redirect to a new target URL. While some people tend to confuse redirects and rewriting, these are two completely different concepts, since the rewriting makes the client and the server see different things (and disagree on the location of the page being visited) while redirects ask the client to visit the new URL so that it sees the same location as the server.

In order to do this, HAProxy supports various possibilities for rewriting and redirects, among which:

• regex-based URL and header rewriting in requests and responses. Regex are the most commonly used tool to modify header values since they’re easy to manipulate and well understood;

• headers may also be appended, deleted or replaced based on formatted strings so that it is possible to pass information there (e.g. client side TLS algorithm and cipher);

• HTTP redirects can use any 3xx code to a relative, absolute, or completely dynamic (formatted string) URI;

• HTTP redirects also support some extra options such as setting or clearing a specific cookie, dropping the query string, appending a slash if missing, and so on;

• a powerful “return” directive allows to customize every part of a response like status, headers, body using dynamic contents or even template files.

• all operations support ACL-based conditions;

3.4.8. Standard features : Server protection

HAProxy does a lot to maximize service availability, and for this it takes large efforts to protect servers against overloading and attacks. The first and most important point is that only complete and valid requests are forwarded to the servers. The initial reason is that HAProxy needs to find the protocol elements it needs to stay synchronized with the byte stream, and the second reason is that until the request is complete, there is no way to know if some elements will change its semantics. The direct benefit from this is that servers are not exposed to invalid or incomplete requests. This is a very effective protection against slowloris attacks, which have almost no impact on HAProxy.

Another important point is that HAProxy contains buffers to store requests and responses, and that by only sending a request to a server when it’s complete and by reading the whole response very quickly from the local network, the server side connection is used for a very short time and this preserves server resources as much as possible.

A direct extension to this is that HAProxy can artificially limit the number of concurrent connections or outstanding requests to a server, which guarantees that the server will never be overloaded even if it continuously runs at 100% of its capacity during traffic spikes. All excess requests will simply be queued to be processed when one slot is released. In the end, this huge resource savings most often ensures so much better server response times that it ends up actually being faster than by overloading the server. Queued requests may be redispatched to other servers, or even aborted in queue when the client aborts, which also protects the servers against the “reload effect”, where each click on “reload” by a visitor on a slow-loading page usually induces a new request and maintains the server in an overloaded state.

The slow-start mechanism also protects restarting servers against high traffic levels while they’re still finalizing their startup or compiling some classes.

Regarding the protocol-level protection, it is possible to relax the HTTP parser to accept non standard-compliant but harmless requests or responses and even to fix them. This allows bogus applications to be accessible while a fix is being developed. In parallel, offending messages are completely captured with a detailed report that help developers spot the issue in the application. The most dangerous protocol violations are properly detected and dealt with and fixed. For example malformed requests or responses with two Content-length headers are either fixed if the values are exactly the same, or rejected if they differ, since it becomes a security problem. Protocol inspection is not limited to HTTP, it is also available for other protocols like TLS or RDP.

When a protocol violation or attack is detected, there are various options to respond to the user, such as returning the common “HTTP 400 bad request”, closing the connection with a TCP reset, or faking an error after a long delay (“tarpit”) to confuse the attacker. All of these contribute to protecting the servers by discouraging the offending client from pursuing an attack that becomes very expensive to maintain.

HAProxy also proposes some more advanced options to protect against accidental data leaks and session crossing. Not only it can log suspicious server responses but it will also log and optionally block a response which might affect a given visitors’ confidentiality. One such example is a cacheable cookie appearing in a cacheable response and which may result in an intermediary cache to deliver it to another visitor, causing an accidental session sharing.

5 - Advanced Features

3.5.1. Advanced features : Management

HAProxy is designed to remain extremely stable and safe to manage in a regular production environment. It is provided as a single executable file which doesn’t require any installation process. Multiple versions can easily coexist, meaning that it’s possible (and recommended) to upgrade instances progressively by order of importance instead of migrating all of them at once. Configuration files are easily versioned. Configuration checking is done off-line so it doesn’t require to restart a service that will possibly fail. During configuration checks, a number of advanced mistakes may be detected (e.g. a rule hiding another one, or stickiness that will not work) and detailed warnings and configuration hints are proposed to fix them. Backwards configuration file compatibility goes very far away in time, with version 1.5 still fully supporting configurations for versions 1.1 written 13 years before, and 1.6 only dropping support for almost unused, obsolete keywords that can be done differently. The configuration and software upgrade mechanism is smooth and non disruptive in that it allows old and new processes to coexist on the system, each handling its own connections. System status, build options, and library compatibility are reported on startup.

Some advanced features allow an application administrator to smoothly stop a server, detect when there’s no activity on it anymore, then take it off-line, stop it, upgrade it and ensure it doesn’t take any traffic while being upgraded, then test it again through the normal path without opening it to the public, and all of this without touching HAProxy at all. This ensures that even complicated production operations may be done during opening hours with all technical resources available.

The process tries to save resources as much as possible, uses memory pools to save on allocation time and limit memory fragmentation, releases payload buffers as soon as their contents are sent, and supports enforcing strong memory limits above which connections have to wait for a buffer to become available instead of allocating more memory. This system helps guarantee memory usage in certain strict environments.

A command line interface (CLI) is available as a UNIX or TCP socket, to perform a number of operations and to retrieve troubleshooting information. Everything done on this socket doesn’t require a configuration change, so it is mostly used for temporary changes. Using this interface it is possible to change a server’s address, weight and status, to consult statistics and clear counters, dump and clear stickiness tables, possibly selectively by key criteria, dump and kill client-side and server-side connections, dump captured errors with a detailed analysis of the exact cause and location of the error, dump, add and remove entries from ACLs and maps, update TLS shared secrets, apply connection limits and rate limits on the fly to arbitrary frontends (useful in shared hosting environments), and disable a specific frontend to release a listening port (useful when daytime operations are forbidden and a fix is needed nonetheless). Updating certificates and their configuration on the fly is permitted, as well as enabling and consulting traces of every processing step of the traffic.

For environments where SNMP is mandatory, at least two agents exist, one is provided with the HAProxy sources and relies on the Net-SNMP Perl module. Another one is provided with the commercial packages and doesn’t require Perl. Both are roughly equivalent in terms of coverage.

It is often recommended to install 4 utilities on the machine where HAProxy is deployed:

• socat (in order to connect to the CLI, though certain forks of netcat can also do it to some extents);

• halog from the latest HAProxy version: this is the log analysis tool, it parses native TCP and HTTP logs extremely fast (1 to 2 GB per second) and extracts useful information and statistics such as requests per URL, per source address, URLs sorted by response time or error rate, termination codes etc. It was designed to be deployed on the production servers to help troubleshoot live issues so it has to be there ready to be used;

• tcpdump: this is highly recommended to take the network traces needed to troubleshoot an issue that was made visible in the logs. There is a moment where application and haproxy’s analysis will diverge and the network traces are the only way to say who’s right and who’s wrong. It’s also fairly common to detect bugs in network stacks and hypervisors thanks to tcpdump;

• strace: it is tcpdump’s companion. It will report what HAProxy really sees and will help sort out the issues the operating system is responsible for from the ones HAProxy is responsible for. Strace is often requested when a bug in HAProxy is suspected;

3.5.2. Advanced features : System-specific capabilities

Depending on the operating system HAProxy is deployed on, certain extra features may be available or needed. While it is supported on a number of platforms, HAProxy is primarily developed on Linux, which explains why some features are only available on this platform.

The transparent bind and connect features, the support for binding connections to a specific network interface, as well as the ability to bind multiple processes to the same IP address and ports are only available on Linux and BSD systems, though only Linux performs a kernel-side load balancing of the incoming requests between the available processes.

On Linux, there are also a number of extra features and optimizations including support for network namespaces (also known as “containers”) allowing HAProxy to be a gateway between all containers, the ability to set the MSS, Netfilter marks and IP TOS field on the client side connection, support for TCP FastOpen on the listening side, TCP user timeouts to let the kernel quickly kill connections when it detects the client has disappeared before the configured timeouts, TCP splicing to let the kernel forward data between the two sides of a connections thus avoiding multiple memory copies, the ability to enable the “defer-accept” bind option to only get notified of an incoming connection once data become available in the kernel buffers, and the ability to send the request with the ACK confirming a connect (sometimes called “piggy-back”) which is enabled with the “tcp-smart-connect” option. On Linux, HAProxy also takes great care of manipulating the TCP delayed ACKs to save as many packets as possible on the network.

Some systems have an unreliable clock which jumps back and forth in the past and in the future. This used to happen with some NUMA systems where multiple processors didn’t see the exact same time of day, and recently it became more common in virtualized environments where the virtual clock has no relation with the real clock, resulting in huge time jumps (sometimes up to 30 seconds have been observed). This causes a lot of trouble with respect to timeout enforcement in general. Due to this flaw of these systems, HAProxy maintains its own monotonic clock which is based on the system’s clock but where drift is measured and compensated for. This ensures that even with a very bad system clock, timers remain reasonably accurate and timeouts continue to work. Note that this problem affects all the software running on such systems and is not specific to HAProxy. The common effects are spurious timeouts or application freezes. Thus if this behavior is detected on a system, it must be fixed, regardless of the fact that HAProxy protects itself against it.

On Linux, a new starting process may communicate with the previous one to reuse its listening file descriptors so that the listening sockets are never interrupted during the process’s replacement.

3.5.3. Advanced features : Scripting

HAProxy can be built with support for the Lua embedded language, which opens a wide area of new possibilities related to complex manipulation of requests or responses, routing decisions, statistics processing and so on. Using Lua it is even possible to establish parallel connections to other servers to exchange information. This way it becomes possible (though complex) to develop an authentication system for example. Please refer to the documentation in the file “doc/lua-api/index.rst” for more information on how to use Lua.

3.5.4. Advanced features: Tracing

At any moment an administrator may connect over the CLI and enable tracing in various internal subsystems. Various levels of details are provided by default so that in practice anything between one line per request to 500 lines per request can be retrieved. Filters as well as an automatic capture on/off/pause mechanism are available so that it really is possible to wait for a certain event and watch it in detail. This is extremely convenient to diagnose protocol violations from faulty servers and clients, or denial of service attacks.

6 - Sizing and Performance

Typical CPU usage figures show 15% of the processing time spent in HAProxy versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy versus 70% for the kernel in HTTP keep-alive mode. This means that the operating system and its tuning have a strong impact on the global performance.

Usages vary a lot between users, some focus on bandwidth, other ones on request rate, others on connection concurrency, others on SSL performance. This section aims at providing a few elements to help with this task.

It is important to keep in mind that every operation comes with a cost, so each individual operation adds its overhead on top of the other ones, which may be negligible in certain circumstances, and which may dominate in other cases.

When processing the requests from a connection, we can say that:

• forwarding data costs less than parsing request or response headers;

• parsing request or response headers cost less than establishing then closing a connection to a server;

• establishing an closing a connection costs less than a TLS resume operation;

• a TLS resume operation costs less than a full TLS handshake with a key computation;

• an idle connection costs less CPU than a connection whose buffers hold data;

• a TLS context costs even more memory than a connection with data;

So in practice, it is cheaper to process payload bytes than header bytes, thus it is easier to achieve high network bandwidth with large objects (few requests per volume unit) than with small objects (many requests per volume unit). This explains why maximum bandwidth is always measured with large objects, while request rate or connection rates are measured with small objects.

Some operations scale well on multiple processes spread over multiple CPUs, and others don’t scale as well. Network bandwidth doesn’t scale very far because the CPU is rarely the bottleneck for large objects, it’s mostly the network bandwidth and data buses to reach the network interfaces. The connection rate doesn’t scale well over multiple processors due to a few locks in the system when dealing with the local ports table. The request rate over persistent connections scales very well as it doesn’t involve much memory nor network bandwidth and doesn’t require to access locked structures. TLS key computation scales very well as it’s totally CPU-bound. TLS resume scales moderately well, but reaches its limits around 4 processes where the overhead of accessing the shared table offsets the small gains expected from more power.

The performance numbers one can expect from a very well tuned system are in the following range. It is important to take them as orders of magnitude and to expect significant variations in any direction based on the processor, IRQ setting, memory type, network interface type, operating system tuning and so on.

The following numbers were found on a Core i7 running at 3.7 GHz equipped with a dual-port 10 Gbps NICs running Linux kernel 3.10, HAProxy 1.6 and OpenSSL 1.0.2. HAProxy was running as a single process on a single dedicated CPU core, and two extra cores were dedicated to network interrupts:

• 20 Gbps of maximum network bandwidth in clear text for objects 256 kB or higher, 10 Gbps for 41kB or higher;

• 4.6 Gbps of TLS traffic using AES256-GCM cipher with large objects;

• 83000 TCP connections per second from client to server;

• 82000 HTTP connections per second from client to server;

• 97000 HTTP requests per second in server-close mode (keep-alive with the client, close with the server);

• 243000 HTTP requests per second in end-to-end keep-alive mode;

• 300000 filtered TCP connections per second (anti-DDoS)

• 160000 HTTPS requests per second in keep-alive mode over persistent TLS connections;

• 13100 HTTPS requests per second using TLS resumed connections;

• 1300 HTTPS connections per second using TLS connections renegotiated with RSA2048;

• 20000 concurrent saturated connections per GB of RAM, including the memory required for system buffers; it is possible to do better with careful tuning but this result it easy to achieve.

• about 8000 concurrent TLS connections (client-side only) per GB of RAM, including the memory required for system buffers;

• about 5000 concurrent end-to-end TLS connections (both sides) per GB of RAM including the memory required for system buffers;

A more recent benchmark featuring the multi-thread enabled HAProxy 2.4 on a 64-core ARM Graviton2 processor in AWS reached 2 million HTTPS requests per second at sub-millisecond response time, and 100 Gbps of traffic:

https://www.haproxy.com/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/

Thus a good rule of thumb to keep in mind is that the request rate is divided by 10 between TLS keep-alive and TLS resume, and between TLS resume and TLS renegotiation, while it’s only divided by 3 between HTTP keep-alive and HTTP close. Another good rule of thumb is to remember that a high frequency core with AES instructions can do around 20 Gbps of AES-GCM per core.

Another good rule of thumb is to consider that on the same server, HAProxy will be able to saturate:

• about 5-10 static file servers or caching proxies;

• about 100 anti-virus proxies;

• and about 100-1000 application servers depending on the technology in use.

7 - Releases, Packages, and Upgrades

HAProxy is an open source project covered by the GPLv2 license, meaning that everyone is allowed to redistribute it provided that access to the sources is also provided upon request, especially if any modifications were made.

HAProxy evolves as a main development branch called “master” or “mainline”, from which new branches are derived once the code is considered stable. A lot of web sites run some development branches in production on a voluntarily basis, either to participate to the project or because they need a bleeding edge feature, and their feedback is highly valuable to fix bugs and judge the overall quality and stability of the version being developed.

The new branches that are created when the code is stable enough constitute a stable version and are generally maintained for several years, so that there is no emergency to migrate to a newer branch even when you’re not on the latest. Once a stable branch is issued, it may only receive bug fixes, and very rarely minor feature updates when that makes users’ life easier. All fixes that go into a stable branch necessarily come from the master branch. This guarantees that no fix will be lost after an upgrade. For this reason, if you fix a bug, please make the patch against the master branch, not the stable branch. You may even discover it was already fixed. This process also ensures that regressions in a stable branch are extremely rare, so there is never any excuse for not upgrading to the latest version in your current branch.

Branches are numbered with two digits delimited with a dot, such as “1.6”. Since 1.9, branches with an odd second digit are mostly focused on sensitive technical updates and more aimed at advanced users because they are likely to trigger more bugs than the other ones. They are maintained for about a year only and must not be deployed where they cannot be rolled back in emergency. A complete version includes one or two sub-version numbers indicating the level of fix. For example, version 1.5.14 is the 14th fix release in branch 1.5 after version 1.5.0 was issued. It contains 126 fixes for individual bugs, 24 updates on the documentation, and 75 other backported patches, most of which were needed to fix the aforementioned 126 bugs. An existing feature may never be modified nor removed in a stable branch, in order to guarantee that upgrades within the same branch will always be harmless.

HAProxy is available from multiple sources, at different release rhythms:

• The official community web site: http://www.haproxy.org/ : this site provides the sources of the latest development release, all stable releases, as well as nightly snapshots for each branch. The release cycle is not fast, several months between stable releases, or between development snapshots. Very old versions are still supported there. Everything is provided as sources only, so whatever comes from there needs to be rebuilt and/or repackaged;

• GitHub: https://github.com/haproxy/haproxy/ : this is the mirror for the development branch only, which provides integration with the issue tracker, continuous integration and code coverage tools. This is exclusively for contributors;

• A number of operating systems such as Linux distributions and BSD ports. These systems generally provide long-term maintained versions which do not always contain all the fixes from the official ones, but which at least contain the critical fixes. It often is a good option for most users who do not seek advanced configurations and just want to keep updates easy;

• Commercial versions from http://www.haproxy.com/ : these are supported professional packages built for various operating systems or provided as appliances, based on the latest stable versions and including a number of features backported from the next release for which there is a strong demand. It is the best option for users seeking the latest features with the reliability of a stable branch, the fastest response time to fix bugs, or simply support contracts on top of an open source product;

In order to ensure that the version you’re using is the latest one in your branch, you need to proceed this way:

• verify which HAProxy executable you’re running: some systems ship it by default and administrators install their versions somewhere else on the system, so it is important to verify in the startup scripts which one is used;

• determine which source your HAProxy version comes from. For this, it’s generally sufficient to type “haproxy -v”. A development version will appear like this, with the “dev” word after the branch number:

HAProxy version 2.4-dev18-a5357c-137 2021/05/09 - https://haproxy.org/

A stable version will appear like this, as well as unmodified stable

versions provided by operating system vendors:

HAProxy version 1.5.14 2015/07/02

And a nightly snapshot of a stable version will appear like this with an
hexadecimal sequence after the version, and with the date of the snapshot

instead of the date of the release:

HAProxy version 1.5.14-e4766ba 2015/07/29

Any other format may indicate a system-specific package with its own
patch set. For example HAProxy Enterprise versions will appear with the

following format (<branch>-<latest commit>-<revision>):

HAProxy version 1.5.0-994126-357 2015/07/02

Please note that historically versions prior to 2.4 used to report the
process name with a hyphen between "HA" and "Proxy", including those above
which were adjusted to show the correct format only, so better ignore this
word or use a relaxed match in scripts. Additionally, modern versions add
a URL linking to the project's home.

Finally, versions 2.1 and above will include a "Status" line indicating
whether the version is safe for production or not, and if so, till when, as
well as a link to the list of known bugs affecting this version.

• for system-specific packages, you have to check with your vendor’s package repository or update system to ensure that your system is still supported, and that fixes are still provided for your branch. For community versions coming from haproxy.org, just visit the site, verify the status of your branch and compare the latest version with yours to see if you’re on the latest one. If not you can upgrade. If your branch is not maintained anymore, you’re definitely very late and will have to consider an upgrade to a more recent branch (carefully read the README when doing so).

HAProxy will have to be updated according to the source it came from. Usually it follows the system vendor’s way of upgrading a package. If it was taken from sources, please read the README file in the sources directory after extracting the sources and follow the instructions for your operating system.

8 - Companion Products and Alternatives

HAProxy integrates fairly well with certain products listed below, which is why they are mentioned here even if not directly related to HAProxy.

4.1. Apache HTTP server

Apache is the de-facto standard HTTP server. It’s a very complete and modular project supporting both file serving and dynamic contents. It can serve as a frontend for some application servers. It can even proxy requests and cache responses. In all of these use cases, a front load balancer is commonly needed. Apache can work in various modes, some being heavier than others. Certain modules still require the heavier pre-forked model and will prevent Apache from scaling well with a high number of connections. In this case HAProxy can provide a tremendous help by enforcing the per-server connection limits to a safe value and will significantly speed up the server and preserve its resources that will be better used by the application.

Apache can extract the client’s address from the X-Forwarded-For header by using the “mod_rpaf” extension. HAProxy will automatically feed this header when “option forwardfor” is specified in its configuration. HAProxy may also offer a nice protection to Apache when exposed to the internet, where it will better resist a wide number of types of DoS attacks.

4.2. NGINX

NGINX is the second de-facto standard HTTP server. Just like Apache, it covers a wide range of features. NGINX is built on a similar model as HAProxy so it has no problem dealing with tens of thousands of concurrent connections. When used as a gateway to some applications (e.g. using the included PHP FPM) it can often be beneficial to set up some frontend connection limiting to reduce the load on the PHP application. HAProxy will clearly be useful there both as a regular load balancer and as the traffic regulator to speed up PHP by decongesting it. Also since both products use very little CPU thanks to their event-driven architecture, it’s often easy to install both of them on the same system. NGINX implements HAProxy’s PROXY protocol, thus it is easy for HAProxy to pass the client’s connection information to NGINX so that the application gets all the relevant information. Some benchmarks have also shown that for large static file serving, implementing consistent hash on HAProxy in front of NGINX can be beneficial by optimizing the OS’ cache hit ratio, which is basically multiplied by the number of server nodes.

4.3. Varnish

Varnish is a smart caching reverse-proxy, probably best described as a web application accelerator. Varnish doesn’t implement SSL/TLS and wants to dedicate all of its CPU cycles to what it does best. Varnish also implements HAProxy’s PROXY protocol so that HAProxy can very easily be deployed in front of Varnish as an SSL offloader as well as a load balancer and pass it all relevant client information. Also, Varnish naturally supports decompression from the cache when a server has provided a compressed object, but doesn’t compress however. HAProxy can then be used to compress outgoing data when backend servers do not implement compression, though it’s rarely a good idea to compress on the load balancer unless the traffic is low.

When building large caching farms across multiple nodes, HAProxy can make use of consistent URL hashing to intelligently distribute the load to the caching nodes and avoid cache duplication, resulting in a total cache size which is the sum of all caching nodes. In addition, caching of very small dumb objects for a short duration on HAProxy can sometimes save network round trips and reduce the CPU load on both the HAProxy and the Varnish nodes. This is only possible is no processing is done on these objects on Varnish (this is often referred to as the notion of “favicon cache”, by which a sizeable percentage of useless downstream requests can sometimes be avoided). However do not enable HAProxy caching for a long time (more than a few seconds) in front of any other cache, that would significantly complicate troubleshooting without providing really significant savings.

4.4. Alternatives

Linux Virtual Server (LVS or IPVS) is the layer 4 load balancer included within the Linux kernel. It works at the packet level and handles TCP and UDP. In most cases it’s more a complement than an alternative since it doesn’t have layer 7 knowledge at all.

Pound is another well-known load balancer. It’s much simpler and has much less features than HAProxy but for many very basic setups both can be used. Its author has always focused on code auditability first and wants to maintain the set of features low. Its thread-based architecture scales less well with high connection counts, but it’s a good product.

Pen is a quite light load balancer. It supports SSL, maintains persistence using a fixed-size table of its clients’ IP addresses. It supports a packet-oriented mode allowing it to support direct server return and UDP to some extents. It is meant for small loads (the persistence table only has 2048 entries).

NGINX can do some load balancing to some extents, though it’s clearly not its primary function. Production traffic is used to detect server failures, the load balancing algorithms are more limited, and the stickiness is very limited. But it can make sense in some simple deployment scenarios where it is already present. The good thing is that since it integrates very well with HAProxy, there’s nothing wrong with adding HAProxy later when its limits have been reached.

Varnish also does some load balancing of its backend servers and does support real health checks. It doesn’t implement stickiness however, so just like with NGINX, as long as stickiness is not needed that can be enough to start with. And similarly, since HAProxy and Varnish integrate so well together, it’s easy to add it later into the mix to complement the feature set.

9 - Documentation and Community

1. Available documentation

The complete HAProxy documentation is contained in the following documents. Please ensure to consult the relevant documentation to save time and to get the most accurate response to your needs. Also please refrain from sending questions to the mailing list whose responses are present in these documents.

• intro.txt (this document): it presents the basics of load balancing, HAProxy as a product, what it does, what it doesn’t do, some known traps to avoid, some OS-specific limitations, how to get it, how it evolves, how to ensure you’re running with all known fixes, how to update it, complements and alternatives.

• management.txt : it explains how to start haproxy, how to manage it at runtime, how to manage it on multiple nodes, and how to proceed with seamless upgrades.

• configuration.txt : the reference manual details all configuration keywords and their options. It is used when a configuration change is needed.

• coding-style.txt: this is for developers who want to propose some code to the project. It explains the style to adopt for the code. It is not very strict and not all the code base completely respects it, but contributions which diverge too much from it will be rejected.

• proxy-protocol.txt: this is the de-facto specification of the PROXY protocol which is implemented by HAProxy and a number of third party products.

• security.txt: how to report a security issue, and what does and does not qualify as a vulnerability.

• README: how to build HAProxy from sources

5. Contacts

If you want to contact the developers or any community member about anything, the best way to do it usually is via the mailing list by sending your message to haproxy@formilux.org . Please note that this list is public and its archives are public as well so you should avoid disclosing sensitive information. A thousand of users of various experience levels are present there and even the most complex questions usually find an optimal response relatively quickly. Suggestions are welcome too. For users having difficulties with e-mail, a Discourse platform is available at http://discourse.haproxy.org/ . However please keep in mind that there are less people reading questions there and that most are handled by a really tiny team. In any case, please be patient and respectful with those who devote their spare time helping others.

I you believe you’ve found a bug but are not sure, it’s best reported on the mailing list. If you’re quite convinced you’ve found a bug, that your version is up-to-date in its branch, and you already have a GitHub account, feel free to go directly to https://github.com/haproxy/haproxy/ and file an issue with all possibly available details. Again, this is public so be careful not to post information you might later regret. Since the issue tracker presents itself as a very long thread, please avoid pasting very long dumps (a few hundreds lines or more) and attach them instead.

If you believe you may have found a security issue, please refer to the file doc/security.txt. It explains what does and does not qualify as a vulnerability in HAProxy, and how to report a genuine one privately. Most suspected issues turn out to be ordinary bugs that are better reported as described above.

Complete local manual set

• Starter Guide
• Configuration Manual
• Management Guide

Edition provenance

• Edition: HAProxy 3.4.4, tag v3.4.4
• Source commit: 7f03ae65c286
• Rendered upstream set: docs.haproxy.org/3.4/
• Local license copies: HAProxy license notice and GPLv2

10 - 1. Quick Reminder About HTTP

This document covers the configuration language as implemented in the version specified above. It does not provide any hints, examples, or advice. For such documentation, please refer to the Reference Manual or the Architecture Manual. The numbered chapters are ordered in the flat HAProxy sidebar for direct navigation.

When HAProxy is running in HTTP mode, both the request and the response are fully analyzed and indexed, thus it becomes possible to build matching criteria on almost anything found in the contents.

However, it is important to understand how HTTP requests and responses are formed, and how HAProxy decomposes them. It will then become easier to write correct rules and to debug existing configurations.

First, HTTP is standardized by a series of RFC that HAProxy follows as closely as possible:

• RFC 9110: HTTP Semantics (explains the meaning of protocol elements)
• RFC 9111: HTTP Caching (explains the rules to follow for an HTTP cache)
• RFC 9112: HTTP/1.1 (representation, interoperability rules, security)
• RFC 9113: HTTP/2 (representation, interoperability rules, security)
• RFC 9114: HTTP/3 (representation, interoperability rules, security)

In addition to these, RFC 8999 to 9002 specify the QUIC transport layer used by the HTTP/3 protocol.

1.1. The HTTP transaction model

The HTTP protocol is transaction-driven. This means that each request will lead to one and only one response. Originally, with version 1.0 of the protocol, there was a single request per connection: a TCP connection is established from the client to the server, a request is sent by the client over the connection, the server responds, and the connection is closed. A new request then involves a new connection:

[CON1] [REQ1] ... [RESP1] [CLO1] [CON2] [REQ2] ... [RESP2] [CLO2] ...

In this mode, often called the “HTTP close” mode, there are as many connection establishments as there are HTTP transactions. Since the connection is closed by the server after the response, the client does not need to know the content length, it considers that the response is complete when the connection closes. This also means that if some responses are truncated due to network errors, the client could mistakenly think a response was complete, and this used to cause truncated images to be rendered on screen sometimes.

Due to the transactional nature of the protocol, it was possible to improve it to avoid closing a connection between two subsequent transactions. In this mode however, it is mandatory that the server indicates the content length for each response so that the client does not wait indefinitely. For this, a special header is used: “Content-length”. This mode is called the “keep-alive” mode, and arrived with HTTP/1.1 (some HTTP/1.0 agents support it), and connections that are reused between requests are called “persistent connections”:

[CON] [REQ1] ... [RESP1] [REQ2] ... [RESP2] [CLO] ...

Its advantages are a reduced latency between transactions, less processing power required on the server side, and the ability to detect a truncated response. It is generally faster than the close mode, but not always because some clients often limit their concurrent connections to a smaller value, and this compensates less for poor network connectivity. Also, some servers have to keep the connection alive for a long time waiting for a possible new request and may experience a high memory usage due to the high number of connections, and closing too fast may break some requests that arrived at the moment the connection was closed.

In this mode, the response size needs to be known upfront so that’s not always possible with dynamically generated or compressed contents. For this reason another mode was implemented, the “chunked mode”, where instead of announcing the size of the whole size at once, the sender only advertises the size of the next “chunk” of response it already has in a buffer, and can terminate at any moment with a zero-sized chunk. In this mode, the Content-Length header is not used.

Another improvement in the communications is the pipelining mode. It still uses keep-alive, but the client does not wait for the first response to send the second request. This is useful for fetching large number of images composing a page:

[CON] [REQ1] [REQ2] ... [RESP1] [RESP2] [CLO] ...

This can obviously have a tremendous benefit on performance because the network latency is eliminated between subsequent requests. Many HTTP agents do not correctly support pipelining since there is no way to associate a response with the corresponding request in HTTP. For this reason, it is mandatory for the server to reply in the exact same order as the requests were received. In practice, after several attempts by various clients to deploy it, it has been totally abandoned for its lack of reliability on certain servers. But it is mandatory for servers to support it.

The next improvement is the multiplexed mode, as implemented in HTTP/2 and HTTP/3. In this mode, multiple transactions (i.e. request-response pairs) are transmitted in parallel over a single connection, and they all progress at their own speed, independent from each other. With multiplexed protocols, a new notion of “stream” was introduced, to represent these parallel communications happening over the same connection. Each stream is generally assigned a unique identifier for a given connection, that is used by both endpoints to know where to deliver the data. It is fairly common for clients to start many (up to 100, sometimes more) streams in parallel over a same connection, and let the server sort them out and respond in any order depending on what response is available. The main benefit of the multiplexed mode is that it significantly reduces the number of round trips, and speeds up page loading time over high latency networks. It is sometimes visible on sites using many images, where all images appear to load in parallel.

These protocols have also improved their efficiency by adopting some mechanisms to compress header fields in order to reduce the number of bytes on the wire, so that without the appropriate tools, they are not realistically manipulable by hand nor readable to the naked eye like HTTP/1 was. For this reason, various examples of HTTP messages continue to be represented in literature (including this document) using the HTTP/1 syntax even for newer versions of the protocol.

HTTP/2 suffers from some design limitations, such as packet losses affecting all streams at once, and if a client takes too much time to retrieve an object (e.g. needs to store it on disk), it may slow down its retrieval and make it impossible during this time to access the data that is pending behind it. This is called “head of line blocking” or “HoL blocking” or sometimes just “HoL”.

HTTP/3 is implemented over QUIC, itself implemented over UDP. QUIC solves the head of line blocking at the transport level by means of independently handled streams. Indeed, when experiencing loss, an impacted stream does not affect the other streams, and all of them can be accessed in parallel. QUIC also provides connection migration support but currently haproxy does not support it.

By default HAProxy operates in keep-alive mode with regards to persistent connections: for each connection it processes each request and response, and leaves the connection idle on both sides between the end of a response and the start of a new request. When it receives HTTP/2 connections from a client, it processes all the requests in parallel and leaves the connection idling, waiting for new requests, just as if it was a keep-alive HTTP connection.

HAProxy essentially supports 3 connection modes:

• keep alive : all requests and responses are processed, and the client facing and server facing connections are kept alive for new requests. This is the default and suits the modern web and modern protocols (HTTP/2 and HTTP/3).

• server close : the server-facing connection is closed after the response.

• close : the connection is actively closed after end of response on both sides.

In addition to this, by default, the server-facing connection is reusable by any request from any client, as mandated by the HTTP protocol specification, so any information pertaining to a specific client has to be passed along with each request if needed (e.g. client’s source address etc). When HTTP/2 is used with a server, by default HAProxy will dedicate this connection to the same client to avoid the risk of head of line blocking between clients.

1.2. Terminology

Inside HAProxy, the terminology has evolved a bit over the ages to follow the evolutions of the HTTP protocol and its usages. While originally there was no significant difference between a connection, a session, a stream or a transaction, these ones clarified over time to match closely what exists in the modern versions of the HTTP protocol, though some terms remain visible in the configuration or the command line interface for the purpose of historical compatibility.

Here are some definitions that apply to the current version of HAProxy:

• connection: a connection is a single, bidiractional communication channel between a remote agent (client or server) and haproxy, at the lowest level possible. Usually it corresponds to a TCP socket established between a pair of IP and ports. On the client-facing side, connections are the very first entities that are instantiated when a client connects to haproxy, and rules applying at the connection level are the earliest ones that apply.

• session: a session adds some context information associated with a connection. This includes and information specific to the transport layer (e.g. TLS keys etc), or variables. This term has long been used inside HAProxy to denote end-to-end HTTP/1.0 communications between two ends, and as such it remains visible in the name of certain CLI commands or statistics, despite representing streams nowadays, but the help messages and descriptions try to make this unambiguous. It is still valid when it comes to network-level terminology (e.g. TCP sessions inside the operating systems, or TCP sessions across a firewall), or for non-HTTP user-level applications (e.g. a telnet session or an SSH session). It must not be confused with “application sessions” that are used to store a full user context in a cookie and require to be sent to the same server.

• stream: a stream exactly corresponds to an end-to-end bidirectional communication at the application level, where analysis and transformations may be applied. In HTTP, it contains a single request and its associated response, and is instantiated by the arrival of the request and is finished with the end of delivery of the response. In this context there is a 1:1 relation between such a stream and the stream of a multiplexed protocol. In TCP communications there is a single stream per connection.

• transaction: a transaction is only a pair of a request and the associated response. The term was used in conjunction with sessions before the streams but nowadays there is a 1:1 relation between a transaction and a stream. It is essentially visible in the variables’ scope “txn” which is valid during the whole transaction, hence the stream.

• request: it designates the traffic flowing from the client to the server. It is mainly used for HTTP to indicate where operations are performed. This term also exists for TCP operations to indicate where data are processed. Requests often appear in counters as a unit of traffic or activity. They do not always imply a response (e.g. due to errors), but since there is no spontaneous responses without requests, requests remain a relevant metric of the overall activity. In TCP there are as many requests as connections.

• response: this designates the traffic flowing from the server to the client, or sometimes from HAProxy to the client, when HAProxy produces the response itself (e.g. an HTTP redirect).

• service: this generally indicates some internal processing in HAProxy that does not require a server, such as the stats page, the cache, or some Lua code to implement a small application. A service usually reads a request, performs some operations and produces a response.

1.3. HTTP request

First, let’s consider this HTTP request:

Line     Contents
number
   1     GET /serv/login.php?lang=en&profile=2 HTTP/1.1
   2     Host: www.mydomain.com
   3     User-agent: my small browser
   4     Accept: image/jpeg, image/gif
   5     Accept: image/png

1.3.1. The Request line

Line 1 is the “request line”. It is always composed of 3 fields:

• a METHOD : GET
• a URI : /serv/login.php?lang=en&profile=2
• a version tag: HTTP/1.1

All of them are delimited by what the standard calls LWS (linear white spaces), which are commonly spaces, but can also be tabs or line feeds/carriage returns followed by spaces/tabs. The method itself cannot contain any colon (’:’) and is limited to alphabetic letters. All those various combinations make it desirable that HAProxy performs the splitting itself rather than leaving it to the user to write a complex or inaccurate regular expression.

The URI itself can have several forms:

• A “relative URI”:

  /serv/login.php?lang=en&profile=2

It is a complete URL without the host part. This is generally what is
received by servers, reverse proxies and transparent proxies.

• An “absolute URI”, also called a “URL”:

  http://192.168.0.12:8080/serv/login.php?lang=en&profile=2

It is composed of a "scheme" (the protocol name followed by '://'), a host
name or address, optionally a colon (':') followed by a port number, then
a relative URI beginning at the first slash ('/') after the address part.
This is generally what proxies receive, but a server supporting HTTP/1.1
must accept this form too.

• a star (’*’): this form is only accepted in association with the OPTIONS method and is not relayable. It is used to inquiry a next hop’s capabilities.

• an address:port combination: 192.168.0.12:80 This is used with the CONNECT method, which is used to establish TCP tunnels through HTTP proxies, generally for HTTPS, but sometimes for other protocols too.

In a relative URI, two sub-parts are identified. The part before the question mark is called the “path”. It is typically the relative path to static objects on the server. The part after the question mark is called the “query string”. It is mostly used with GET requests sent to dynamic scripts and is very specific to the language, framework or application in use.

HTTP/2 and HTTP/3 do not convey a version information with the request, so the version is assumed to be the same as the one of the underlying protocol (i.e. “HTTP/2”). In addition, these protocols do not send a request line as one part, but split it into individual fields called “pseudo-headers”, whose name start with a colon, and which are conveniently reassembled by HAProxy into an equivalent request line. For this reason, request lines found in logs may slightly differ between HTTP/1.x and HTTP/2 or HTTP/3.

1.3.2. The request headers

The headers start at the second line. They are composed of a name at the beginning of the line, immediately followed by a colon (’:’). Traditionally, an LWS is added after the colon but that’s not required. Then come the values. Multiple identical headers may be folded into one single line, delimiting the values with commas, provided that their order is respected. This is commonly encountered in the “Cookie:” field. A header may span over multiple lines if the subsequent lines begin with an LWS. In the example in 1.3, lines 4 and 5 define a total of 3 values for the “Accept:” header. Finally, all LWS at the beginning or at the end of a header are ignored and are not part of the value, as per the specification.

Contrary to a common misconception, header names are not case-sensitive, and their values are not either if they refer to other header names (such as the “Connection:” header). In HTTP/2 and HTTP/3, header names are always sent in lower case, as can be seen when running in debug mode. Internally, all header names are normalized to lower case so that HTTP/1.x and HTTP/2 or HTTP/3 use the exact same representation, and they are sent as-is on the other side. This explains why an HTTP/1.x request typed with camel case is delivered in lower case.

The end of the headers is indicated by the first empty line. People often say that it’s a double line feed, which is not exact, even if a double line feed is one valid form of empty line.

Fortunately, HAProxy takes care of all these complex combinations when indexing headers, checking values and counting them, so there is no reason to worry about the way they could be written, but it is important not to accuse an application of being buggy if it does unusual, valid things.

Important note:

As suggested by RFC7231, HAProxy normalizes headers by replacing line breaks
in the middle of headers by LWS in order to join multi-line headers. This
is necessary for proper analysis and helps less capable HTTP parsers to work
correctly and not to be fooled by such complex constructs.

1.4. HTTP response

An HTTP response looks very much like an HTTP request. Both are called HTTP messages. Let’s consider this HTTP response:

Line     Contents
number
   1     HTTP/1.1 200 OK
   2     Content-length: 350
   3     Content-Type: text/html

As a special case, HTTP supports so called “Informational responses” as status codes 1xx. These messages are special in that they don’t convey any part of the response, they’re just used as sort of a signaling message to ask a client to continue to post its request for instance. In the case of a status 100 response the requested information will be carried by the next non-100 response message following the informational one. This implies that multiple responses may be sent to a single request, and that this only works when keep-alive is enabled (1xx messages appeared in HTTP/1.1). HAProxy handles these messages and is able to correctly forward and skip them, and only process the next non-100 response. As such, these messages are neither logged nor transformed, unless explicitly state otherwise. Status 101 messages indicate that the protocol is changing over the same connection and that HAProxy must switch to tunnel mode, just as if a CONNECT had occurred. Then the Upgrade header would contain additional information about the type of protocol the connection is switching to.

1.4.1. The response line

Line 1 is the “response line”. It is always composed of 3 fields:

• a version tag: HTTP/1.1
• a status code: 200
• a reason : OK

The status code is always 3-digit. The first digit indicates a general status:

• 1xx = informational message to be skipped (e.g. 100, 101)
• 2xx = OK, content is following (e.g. 200, 206)
• 3xx = OK, no content following (e.g. 302, 304)
• 4xx = error caused by the client (e.g. 401, 403, 404)
• 5xx = error caused by the server (e.g. 500, 502, 503)

Status codes greater than 599 must not be emitted in communications, though certain agents may produce them in logs to report their internal statuses. Please refer to RFC9110 for the detailed meaning of all such codes. HTTP/2 and above do not have a version tag and use the “:status” pseudo-header to report the status code.

The “reason” field is just a hint, but is not parsed by clients. Anything can be found there, but it’s a common practice to respect the well-established messages. It can be composed of one or multiple words, such as “OK”, “Found”, or “Authentication Required”. It does not exist in HTTP/2 and above and is not emitted there. When a response from HTTP/2 or above is transmitted to an HTTP/1 client, HAProxy will produce such a common reason field that matches the status code.

HAProxy may emit the following status codes by itself:

Code  When / reason
 200  access to stats page, and when replying to monitoring requests
 301  when performing a redirection, depending on the configured code
 302  when performing a redirection, depending on the configured code
 303  when performing a redirection, depending on the configured code
 307  when performing a redirection, depending on the configured code
 308  when performing a redirection, depending on the configured code
 400  for an invalid or too large request
 401  when an authentication is required to perform the action (when
      accessing the stats page)
 403  when a request is forbidden by a "http-request deny" rule
 404  when the requested resource could not be found
 408  when the request timeout strikes before the request is complete
 410  when the requested resource is no longer available and will not
      be available again
 413  when a HTTP/1.0 GET/HEAD/DELETE requests has a payload, also see
      the "h1-accept-payload-with-any-method" option
 500  when HAProxy encounters an unrecoverable internal error, such as a
      memory allocation failure, which should never happen
 501 when HAProxy is unable to satisfy a client request because of an
     unsupported feature
 502  when the server returns an empty, invalid or incomplete response, or
      when an "http-response deny" rule blocks the response.
 503  when no server was available to handle the request, or in response to
      monitoring requests which match the "monitor fail" condition
 504  when the response timeout strikes before the server responds

The error 4xx and 5xx codes above may be customized (see “errorloc” in section 4.2 ). Other status codes can be emitted on purpose by specific actions (see the “deny”, “return” and “redirect” actions in section 4.3 for example).

1.4.2. The response headers

Response headers work exactly like request headers, and as such, HAProxy uses the same parsing function for both. Please refer to paragraph 1.3.2 for more details.

11 - 2. Configuring HAProxy

2.1. Configuration file format

HAProxy’s configuration process involves 3 major sources of parameters:

• the arguments from the command-line, which always take precedence
• the configuration file(s), whose format is described here
• the running process’s environment, in case some environment variables are explicitly referenced

The configuration file follows a fairly simple hierarchical format which obey a few basic rules:

1. a configuration file is an ordered sequence of statements

2. a statement is a single non-empty line before any unprotected "#" (hash)

3. a line is a series of tokens or "words" delimited by unprotected spaces or
   tab characters

4. the first word or sequence of words of a line is one of the keywords or
   keyword sequences listed in this document

5. all other words are all arguments of the first one, some being well-known
   keywords listed in this document, others being values, references to other
   parts of the configuration, or expressions

6. certain keywords delimit a section inside which only a subset of keywords
   are supported

7. a section ends at the end of a file or on a special keyword starting a new
   section

This is all that is needed to know to write a simple but reliable configuration generator, but this is not enough to reliably parse any configuration nor to figure how to deal with certain corner cases.

First, there are a few consequences of the rules above. Rule 6 and 7 imply that the keywords used to define a new section are valid everywhere and cannot have a different meaning in a specific section. These keywords are always a single word (as opposed to a sequence of words), and traditionally the section that follows them is designated using the same name. For example when speaking about the “global section”, it designates the section of configuration that follows the “global” keyword. This usage is used a lot in error messages to help locate the parts that need to be addressed.

A number of sections create an internal object or configuration space, which requires to be distinguished from other ones. In this case they will take an extra word which will set the name of this particular section. For some of them the section name is mandatory. For example “frontend foo” will create a new section of type “frontend” named “foo”. Usually a name is specific to its section and two sections of different types may use the same name, but this is not recommended as it tends to complexify configuration management.

A direct consequence of rule 7 is that when multiple files are read at once, each of them must start with a new section, and the end of each file will end a section. A file cannot contain sub-sections nor end an existing section and start a new one.

Rule 1 mentioned that ordering matters. Indeed, some keywords create directives that can be repeated multiple times to create ordered sequences of rules to be applied in a certain order. For example “tcp-request” can be used to alternate “accept” and “reject” rules on varying criteria. As such, a configuration file processor must always preserve a section’s ordering when editing a file. The ordering of sections usually does not matter except for the global section which must be placed before other sections, but it may be repeated if needed. In addition, some automatic identifiers may automatically be assigned to some of the created objects (e.g. proxies), and by reordering sections, their identifiers will change. These ones appear in the statistics for example. As such, the configuration below will assign “foo” an ID number smaller than its “bar” counterpart. This will be swapped if the two sections are reversed:

listen foo
    bind:80

listen bar
    bind:81

Another important point is that according to rules 2 and 3 above, empty lines, spaces, tabs, and comments following and unprotected “#” character are not part of the configuration as they are just used as delimiters. This implies that the following configurations are strictly equivalent:

    global#this is the global section
daemon#daemonize
    frontend         foo
mode             http   # or tcp

and:

global
    daemon

# this is the public web frontend
frontend foo
    mode http

The common practice is to align to the left only the keyword that initiates a new section, and indent (i.e. prepend a tab character or a few spaces) all other keywords so that it’s instantly visible that they belong to the same section (as done in the second example above). Placing comments before a new section helps the reader decide if it’s the desired one. Leaving a blank line at the end of a section also visually helps spotting the end when editing it.

Tabs are very convenient for indent but they do not copy-paste well. If spaces are used instead, it is recommended to avoid placing too many (2 to 4) so that editing in field doesn’t become a burden with limited editors that do not support automatic indent.

In the early days it used to be common to see arguments split at fixed tab positions because most keywords would not take more than two arguments. With modern versions featuring complex expressions this practice does not stand anymore, and is not recommended.

2.2. Quoting and escaping

In modern configurations, some arguments require the use of some characters that were previously considered as pure delimiters. In order to make this possible, HAProxy supports character escaping by prepending a backslash (’\’) in front of the character to be escaped, weak quoting with double quotes ("") around a piece of text, and strong quoting with single quotes (’’) around a piece of text.

This is pretty similar to what is done in a number of programming languages and very close to what is commonly encountered in Bourne shell. The principle is the following: while the configuration parser cuts the lines into words, it also takes care of quotes and backslashes to decide whether a character is a delimiter or is the raw representation of this character within the current word. The escape character is then removed, the quotes are removed, and the remaining word is used as-is as a keyword or argument for example.

If a backslash is needed in a word, it must either be escaped using itself (i.e. double backslash) or be strongly quoted.

Escaping outside quotes is achieved by preceding a special character by a backslash (’\’):

\    to mark a space and differentiate it from a delimiter
\#   to mark a hash and differentiate it from a comment
\\   to use a backslash
\'   to use a single quote and differentiate it from strong quoting
\"   to use a double quote and differentiate it from weak quoting

In addition, a few non-printable characters may be emitted using their usual C-language representation:

\n   to insert a line feed (LF, character \x0a or ASCII 10 decimal)
\r   to insert a carriage return (CR, character \x0d or ASCII 13 decimal)
\t   to insert a tab (character \x09 or ASCII 9 decimal)
\xNN to insert character having ASCII code hex NN (e.g \x0a for LF).

Weak quoting is achieved by surrounding double quotes ("") around the character or sequence of characters to protect. Weak quoting prevents the interpretation of:

     space or tab as a word separator
'    single quote as a strong quoting delimiter

#    hash as a comment start

Weak quoting permits the interpretation of environment variables (which are not evaluated outside of quotes) by preceding them with a dollar sign (’$’). If a dollar character is needed inside double quotes, it must be escaped using a backslash.

Strong quoting is achieved by surrounding single quotes (’’) around the character or sequence of characters to protect. Inside single quotes, nothing is interpreted, it’s the efficient way to quote regular expressions.

As a result, here is the matrix indicating how special characters can be entered in different contexts (unprintable characters are replaced with their name within angle brackets). Note that some characters that may only be represented escaped have no possible representation inside single quotes, hence its absence there:

  Character  |  Unquoted     |  Weakly quoted              |  Strongly quoted
  -----------+---------------+-----------------------------+-----------------
    <TAB>    |  \<TAB>, \x09 |  "<TAB>", "\<TAB>", "\x09"  |  '<TAB>'
  -----------+---------------+-----------------------------+-----------------
    <LF>     |  \n, \x0a     |  "\n", "\x0a"               |
  -----------+---------------+-----------------------------+-----------------
    <CR>     |  \r, \x0d     |  "\r", "\x0d"               |
  -----------+---------------+-----------------------------+-----------------
    <SPC>    |  \<SPC>, \x20 |  "<SPC>", "\<SPC>", "\x20"  |  '<SPC>'
  -----------+---------------+-----------------------------+-----------------
    "        |  \", \x22     |  "\"", "\x22"               |  '"'
  -----------+---------------+-----------------------------+-----------------
    #        |  \#, \x23     |  "#", "\#", "\x23"          |  '#'
  -----------+---------------+-----------------------------+-----------------
    $        |  $, \$, \x24  |  "\$", "\x24"               |  '$'
  -----------+---------------+-----------------------------+-----------------
    '        |  \', \x27     |  "'", "\'", "\x27"          |
  -----------+---------------+-----------------------------+-----------------
    \        |  \\, \x5c     |  "\\", "\x5c"               |  '\'
  -----------+---------------+-----------------------------+-----------------

Example:

# those are all strictly equivalent:
log-format %{+Q}o\ %t\ %s\ %{-Q}r
log-format "%{+Q}o %t %s %{-Q}r"
log-format '%{+Q}o %t %s %{-Q}r'
log-format "%{+Q}o %t"' %s %{-Q}r'
log-format "%{+Q}o %t"' %s'\ %{-Q}r

There is one particular case where a second level of quoting or escaping may be necessary. Some keywords take arguments within parenthesis, sometimes delimited by commas. These arguments are commonly integers or predefined words, but when they are arbitrary strings, it may be required to perform a separate level of escaping to disambiguate the characters that belong to the argument from the characters that are used to delimit the arguments themselves. A pretty common case is the “regsub” converter. It takes a regular expression in argument, and if a closing parenthesis is needed inside, this one will require to have its own quotes.

The keyword argument parser is exactly the same as the top-level one regarding quotes, except that the \#, \$, and \xNN escapes are not processed. But what is not always obvious is that the delimiters used inside must first be escaped or quoted so that they are not resolved at the top level.

Let’s take this example making use of the “regsub” converter which takes 3 arguments, one regular expression, one replacement string and one set of flags:

# replace all occurrences of "foo" with "blah" in the path:
http-request set-path %[path,regsub(foo,blah,g)]

Here no special quoting was necessary. But if now we want to replace either “foo” or “bar” with “blah”, we’ll need the regular expression “(foo|bar)”. We cannot write:

http-request set-path %[path,regsub((foo|bar),blah,g)]

because we would like the string to cut like this:

    http-request set-path %[path,regsub((foo|bar),blah,g)]
                                       |---------|----|-|
                                 arg1 _/         /    /
                                 arg2 __________/    /
                                 arg3 ______________/

but actually what is passed is a string between the opening and closing parenthesis then garbage:

    http-request set-path %[path,regsub((foo|bar),blah,g)]
                                       |--------|--------|
                        arg1=(foo|bar _/        /
                    trailing garbage  _________/

The obvious solution here seems to be that the closing parenthesis needs to be quoted, but alone this will not work, because as mentioned above, quotes are processed by the top-level parser which will resolve them before processing this word:

http-request set-path %[path,regsub("(foo|bar)",blah,g)]
------------ -------- ----------------------------------
   word1       word2    word3=%[path,regsub((foo|bar),blah,g)]

So we didn’t change anything for the argument parser at the second level which still sees a truncated regular expression as the only argument, and garbage at the end of the string. By escaping the quotes they will be passed unmodified to the second level:

    http-request set-path %[path,regsub(\"(foo|bar)\",blah,g)]
    ------------ -------- ------------------------------------
       word1       word2    word3=%[path,regsub("(foo|bar)",blah,g)]
                                                |---------||----|-|
                                arg1=(foo|bar) _/          /    /
                                    arg2=blah  ___________/    /
                                        arg3=g _______________/

Another approach consists in using single quotes outside the whole string and double quotes inside (so that the double quotes are not stripped again):

    http-request set-path '%[path,regsub("(foo|bar)",blah,g)]'
    ------------ --------  ----------------------------------
       word1       word2    word3=%[path,regsub("(foo|bar)",blah,g)]
                                                |---------||----|-|
                                arg1=(foo|bar) _/          /    /
                                          arg2 ___________/    /
                                          arg3 _______________/

But in this case it’s important to note that delimiters embedded into the higher level string remain pure characters and are not delimiters anymore. It particularly means that spaces and tabs around commas are part of the string. The example below is wrong on multiple points:

    http-request set-path '%[path, regsub("(foo|bar)", blah, g)]'
    ------------ --------  --------------------------------------
       word1       word2    word3=%[path, regsub("(foo|bar)", blah, g)]
                                        |--------|---------||-----|--|
                       converter=" regsub" _/        /         /   /
                                    arg1=(foo|bar) _/         /   /
                                     arg2=" blah" ___________/   /
                                        arg3=" g" ______________/

The single fact of surrounding commas with spaces resulted in the spaces being part of the field itself, hence the converter " regsub" (starting with a space), which won’t be found and will trigger an error, but more subtly, the replacement string " blah" will insert a space in the output. A good rule of thumb is to never insert unneeded spaces inside expressions.

When using regular expressions, it can happen that the dollar (’$’) character appears in the expression or that a backslash (’\’) is used in the replacement string. In this case these ones will also be processed inside the double quotes thus single quotes are preferred (or double escaping). Example:

    http-request set-path '%[path,regsub("^/(here)(/|$)","my/\1",g)]'
    ------------ --------  -----------------------------------------
       word1       word2    word3=%[path,regsub("^/(here)(/|$)","my/\1",g)]
                                                |-------------| |-----||-|
                              arg1=(here)(/|$) _/               /      /
                                    arg2=my/\1 ________________/      /
                                          arg3 ______________________/

Remember that backslashes are not escape characters within single quotes and that the whole word above is already protected against them using the single quotes. Conversely, if double quotes had been used around the whole expression, single the dollar character and the backslashes would have been resolved at top level, breaking the argument contents at the second level.

Unfortunately, since single quotes can’t be escaped inside of strong quoting, if you need to include single quotes in your argument, you will need to escape or quote them twice. There are a few ways to do this:

http-request set-var(txn.foo) str("\\'foo\\'")
http-request set-var(txn.foo) str(\"\'foo\'\")
http-request set-var(txn.foo) str(\\\'foo\\\')

When in doubt, simply do not use quotes anywhere, and start to place single or double quotes around arguments that require a comma or a closing parenthesis, and think about escaping these quotes using a backslash if the string contains a dollar or a backslash. Again, this is pretty similar to what is used under a Bourne shell when double-escaping a command passed to “eval”. For API writers the best is probably to place escaped quotes around each and every argument, regardless of their contents. Users will probably find that using single quotes around the whole expression and double quotes around each argument provides more readable configurations.

2.3. Environment variables

HAProxy’s configuration supports environment variables. Those variables are interpreted only within double quotes. Variables are expanded during the configuration parsing. Variable names must be preceded by a dollar ("$") and optionally enclosed with braces ("{}") similarly to what is done in Bourne shell. Variable names can contain alphanumerical characters or the character underscore ("_") but should not start with a digit. If the variable contains a list of several values separated by spaces, it can be expanded as individual arguments by enclosing the variable with braces and appending the suffix ‘[*]’ before the closing brace. It is also possible to specify a default value to use when the variable is not set, by appending that value after a dash ‘-’ next to the variable name. Note that the default value only replaces non existing variables, not empty ones.

Example:

bind "fd@${FD_APP1}"

log "${LOCAL_SYSLOG-127.0.0.1}:514" local0 notice  # send to local server

user "$HAPROXY_USER"

Some variables are defined by HAProxy, they can be used in the configuration file. These variables are listed in the matrix below, and they are classified among four categories:

• usable: the variable is accessible from the configuration, either to be resolved as-is, or used within conditional blocks or predicates to enable or disable this some configuration fragments, as described in section 2.4 “Conditional blocks”.

• modifiable: the variable can be redefined or unset in the configuration via “setenv”/“unsetenv” keywords.

• listed: the variable is listed in CLI’s “show env” command output, described in section 9.3 “Unix Sockets commands” of the management guide.

There also two subcategories “master” and “worker”, respectively marked ‘M’ and ‘W’ in the table below, showing the differences between the two processes when HAProxy is launched in master-worker mode.

• master: the variable is set and accessible from the master process. So, it will appear in the master CLI’s “show env” output and it can be used in conditional blocks or directives to enable some special settings for the master (see examples in section 2.4 “Conditional blocks”).

• worker: the variable is set and accessible from the worker process. It will appear in the worker CLI’s “show env” (or the master CLI’s “@1 show env”) and it may as well condition some worker process parameters (see examples from section 2.4 “Conditional blocks”).

In standalone mode (without “-W” option nor the “master-worker” keyword) the process behaves like a worker, except for variables “HAPROXY_MASTER_CLI” and “HAPROXY_MWORKER” which are not defined.

Some variables are marked as not usable and not modifiable:

• HAPROXY_CFGFILES
• HAPROXY_MWORKER
• HAPROXY_CLI
• HAPROXY_MASTER_CLI
• HAPROXY_LOCALPEER

Their values are undefined during configuration parsing, they are set later during the initialization. So, it’s recommended not to use these variables within conditional blocks and not to reference them in the global section’s “setenv”/“resetenv”/“unsetenv” keywords.

The table below summaries the status of each variable for the different working modes:

  +---------------------------+---------+------------+-----------+
  |          variable         | usable  | modifiable |  listed   |
  |                           +---------+------------+-----------+
  |                           |  M | W  |   M  |  W  |  M  |  W  |
  +---------------------------+----+----+------+-----+-----+-----+
  | HAPROXY_STARTUP_VERSION   |  X | X  |      |     |  X  |  X  |
  | HAPROXY_BRANCH            |  X | X  |      |     |  X  |  X  |
  | HAPROXY_CFGFILES          |    |    |      |     |  X  |  X  |
  | HAPROXY_MWORKER           |    |    |      |     |  X  |  X  |
  | HAPROXY_CLI               |    |    |      |     |     |  X  |
  | HAPROXY_MASTER_CLI        |    |    |      |     |  X  |     |
  | HAPROXY_LOCALPEER         |    | X  |      |     |     |  X  |
  | HAPROXY_HTTP_LOG_FMT      |    | X  |      |  X  |     |     |
  | HAPROXY_HTTP_CLF_LOG_FMT  |    | X  |      |  X  |     |     |
  | HAPROXY_HTTPS_LOG_FMT     |    | X  |      |  X  |     |     |
  | HAPROXY_TCP_LOG_FMT       |    | X  |      |  X  |     |     |
  | HAPROXY_TCP_CLF_LOG_FMT   |    | X  |      |  X  |     |     |
  | HAPROXY_KEYLOG_FC_LOG_FMT |    | X  |      |  X  |     |     |
  | HAPROXY_KEYLOG_BC_LOG_FMT |    | X  |      |  X  |     |     |
  +---------------------------+----+----+------+-----+-----+-----+

The variables in question are the following:

• HAPROXY_LOCALPEER: defined at the startup of the process which contains the name of the local peer. (See “-L” in the management guide.)

• HAPROXY_CFGFILES: list of the configuration files loaded by HAProxy, separated by semicolons. Can be useful in the case you specified a directory.

• HAPROXY_HTTP_LOG_FMT: contains the value of the default HTTP log format as defined in section 8.2.3 “HTTP log format”. It can be used to override the default log format without having to copy the whole original definition.

• HAPROXY_HTTP_CLF_LOG_FMT: contains the value of the default HTTP CLF log format as defined in section 8.2.3 “HTTP log format”. It can be used to override the default log format without having to copy the whole original definition.

Example:

# Add the rule that gave the final verdict to the log
log-format "${HAPROXY_TCP_LOG_FMT} lr=%[last_rule_file]:%[last_rule_line]"

• HAPROXY_HTTPS_LOG_FMT: similar to HAPROXY_HTTP_LOG_FMT but for HTTPS log format as defined in section 8.2.4 “HTTPS log format”.

• HAPROXY_TCP_LOG_FMT: similar to HAPROXY_HTTP_LOG_FMT but for TCP log format as defined in section 8.2.2 “TCP log format”.

• HAPROXY_TCP_CLF_LOG_FMT: similar to HAPROXY_HTTP_CLF_LOG_FMT but for TCP CLF log format as defined in section 8.2.2 “TCP log format”.

• HAPROXY_KEYLOG_FC_LOG_FMT: contains the keylog format for the frontend (client-facing) TLS connection, with key entries separated by newlines so it might not be compatible with your syslog server. “tune.ssl.keylog on” is required.

• HAPROXY_KEYLOG_BC_LOG_FMT: similar to HAPROXY_KEYLOG_FC_LOG_FMT but for the backend (server-facing) TLS connection. Key entries are separated by newlines so it might not be compatible with your syslog server. “tune.ssl.keylog on” is required.

• HAPROXY_MWORKER: In master-worker mode, this variable is set to 1.

• HAPROXY_CLI: configured listeners addresses of the stats socket of every processe, these addresses are separated by semicolons.

• HAPROXY_MASTER_CLI: In master-worker mode, listeners addresses of the master CLI, separated by semicolons.

• HAPROXY_STARTUP_VERSION: contains the version used to start, in master-worker mode this is the version which was used to start the master, even after updating the binary and reloading.

• HAPROXY_BRANCH: contains the HAProxy branch version (such as “2.8”). It does not contain the full version number. It can be useful in case of migration if resources (such as maps or certificates) are in a path containing the branch number.

In addition, some pseudo-variables are internally resolved and may be used as regular variables. Pseudo-variables always start with a dot (’.’), and are the only ones where the dot is permitted. The current list of pseudo-variables is:

• .FILE: the name of the configuration file currently being parsed.

• .LINE: the line number of the configuration file currently being parsed, starting at one.

• .SECTION: the name of the section currently being parsed, or its type if the section doesn’t have a name (e.g. “global”), or an empty string before the first section.

These variables are resolved at the location where they are parsed. For example if a “.LINE” variable is used in a “log-format” directive located in a defaults section, its line number will be resolved before parsing and compiling the “log-format” directive, so this same line number will be reused by subsequent proxies.

This way it is possible to emit information to help locate a rule in variables, logs, error statuses, health checks, header values, or even to use line numbers to name some config objects like servers for example.

2.4. Conditional blocks

It may sometimes be convenient to be able to conditionally enable or disable some arbitrary parts of the configuration, for example to enable/disable SSL or ciphers, enable or disable some pre-production listeners without modifying the configuration, or adjust the configuration’s syntax to support two distinct versions of HAProxy during a migration.. HAProxy brings a set of nestable preprocessor-like directives which allow to integrate or ignore some blocks of text. These directives must be placed on their own line and they act on the lines that follow them. Two of them support an expression, the other ones only switch to an alternate block or end a current level. The 4 following directives are defined to form conditional blocks:

• .if <condition>
• .elif <condition>
• .else
• .endif

The “.if” directive nests a new level, “.elif” stays at the same level, “.else” as well, and “.endif” closes a level. Each “.if” must be terminated by a matching “.endif”. The “.elif” may only be placed after “.if” or “.elif”, and there is no limit to the number of “.elif” that may be chained. There may be only one “.else” per “.if” and it must always be after the “.if” or the last “.elif” of a block.

Comments may be placed on the same line if needed after a ‘#’, they will be ignored. The directives are tokenized like other configuration directives, and as such it is possible to use environment variables in conditions.

Conditions can also be evaluated on startup with the -cc parameter. See “3. Starting HAProxy” in the management doc.

The conditions are either an empty string (which then returns false), or an expression made of any combination of:

• the integer zero (‘0’), always returns “false”
• a non-nul integer (e.g. ‘1’), always returns “true”.
• a predicate optionally followed by argument(s) in parenthesis.
• a condition placed between a pair of parenthesis ‘(’ and ‘)’
• an exclamation mark (’!’) preceding any of the non-empty elements above, and which will negate its status.
• expressions combined with a logical AND (’&&’), which will be evaluated from left to right until one returns false
• expressions combined with a logical OR (’||’), which will be evaluated from right to left until one returns true

The same line tokenizer and argument parser are used as for the rest of the configuration language. Words are split around consecutive series of one or more unquoted spaces or tabs, and are reassembled together using a single space to delimit them before evaluation, in order to save the user from having to quote the entire line. But this also means that spaces surrounding commas or parenthesis are definitely part of the value, which is not always expected. For example, the expression below:

.if defined( HAPROXY_MWORKER )

will test for the existence of variable " HAPROXY_MWORKER " (with spaces), and this one:

.if streq("$ENABLE_SSL",     1)

will compare the environment variable “ENABLE_SSL” to the value " 1" (with a single leading space). The reason is the line is first split into words like this:

   .if streq("$ENABLE_SSL",     1)
  |---|--------------------|   |--|
    1           2               3

then the weak quoting is applied and environment variable “$ENABLE_SSL” is resolved (let’s say for example that ENABLE_SSL=0), and finally the words are reassembled into a single string by placing a single space between the words:

   .if streq(0, 1)
  |---|-------|--|
    1     2     3

and only then it is parsed as a single expression. The space that was inserted between the comma and “1” is still part of the argument value, making this argument " 1":

   .if streq(0, 1)
  |---|-----|-|--|
    \    \    \  \_ argument2: " 1"
     \    \    \___ argument1: "0"
      \    \_______ function: "streq"
       \___________ directive: ".if"

It’s visible here that even if ENABLE_SSL had been equal to “1”, it wouldn’t have matched " 1" since the string would differ by one space.

Note: as explained in section “2.2. Quoting and escaping”, a good rule of thumb is to never insert unneeded spaces inside expressions.

Note that like in other languages, the AND operator has precedence over the OR operator, so that “A && B || C && D” evalues as “(A && B) || (C && D)”.

The list of currently supported predicates is the following:

• awslc_api_atleast(<ver>): returns true if the current awslc API number is at least as recent as <ver> otherwise false. Example: awslc_api_atleast(35)

• awslc_api_before(<ver>): returns true if the current awslc API number is strictly older than <ver> otherwise false. Example: awslc_api_before(26)

• defined(<name>) : returns true if an environment variable <name> exists, regardless of its contents

• feature(<name>) : returns true if feature <name> is listed as present in the features list reported by “haproxy -vv” (which means a <name> appears after a ‘+’)

• openssl_version_atleast(<ver>): returns true if the current openssl version is at least as recent as <ver> otherwise false. Libraries like LibreSSL, AWS-LC and WolfSSL also provide a pseudo OpenSSL version. Example:

ssllib_name_startswith(OpenSSL) && openssl_version_atleast(1.1.1)

• openssl_version_before(<ver>): returns true if the current openssl version is strictly older than <ver> otherwise false. Libraries like LibreSSL, AWS-LC and WolfSSL also provide a pseudo OpenSSL version. Example: openssl_version_before(3.5.0)

• ssllib_name_startswith(<name>) : return true if the SSL library name HAProxy was linked with, starts with <name>. Example: ssllib_name_startswith(wolfSSL)

• streq(<str1>,<str2>) : returns true only if the two strings are equal

• strneq(<str1>,<str2>): returns true only if the two strings differ

• strstr(<str1>,<str2>): returns true only if the second string is found in the first one.

• version_atleast(<ver>): returns true if the current haproxy version is at least as recent as <ver> otherwise false. The version syntax is the same as shown by “haproxy -v” and missing components are assumed as being zero.

• version_before(<ver>): returns true if the current haproxy version is strictly older than <ver> otherwise false. The version syntax is the same as shown by “haproxy -v” and missing components are assumed as being zero.

• enabled(<opt>) : returns true if the option <opt> is enabled at run-time. Only a subset of options are supported:

POLL, EPOLL, KQUEUE, EVPORTS, SPLICE,
GETADDRINFO, REUSEPORT, FAST-FORWARD,
SERVER-SSL-VERIFY-NONE

Example:

# 1. HAPROXY_MWORKER variable is set automatically by HAProxy in master and
# in worker process environments (see HAProxy variables matrix from
# 2.3. Environment variables). Its presence enables an additional listener.

global
  master-worker

.if defined(HAPROXY_MWORKER) listen mwcli_px bind:1111 … .endif

# 2. HAPROXY_BRANCH is set automatically by HAProxy in master and in worker
# process environments (see HAProxy variables matrix from 2.3. Environment
# variables). We check HAPROXY_BRANCH value and conditionally enable
# mworker-max-reloads parameter.

global
  master-worker

.if streq("$HAPROXY_BRANCH",3.1) mworker-max-reloads 5 .endif

# 3. Some arbitrary environment variables are set by user in the global
# section. If HAProxy is started in master-worker mode, they are presented in
# master and in worker process environments. We check values of these
# variables and conditionally enable ports 80 and 443. Environment variables
# checks can be mixed with features and version checks.

global
  setenv WITH_SSL yes
  unsetenv SSL_ONLY

.if strneq("$SSL_ONLY",yes) bind:80 .endif

.if streq("$WITH_SSL",yes) .if feature(OPENSSL) bind:443 ssl crt … .endif .endif

.if feature(OPENSSL) && (streq("$WITH_SSL",yes) || streq("$SSL_ONLY",yes)) bind:443 ssl crt … .endif

.if version_atleast(2.4-dev19) profiling.memory on .endif

.if !feature(OPENSSL) .alert “SSL support is mandatory” .endif

Four other directives are provided to report some status:

• .diag “message” : emit this message only when in diagnostic mode (-dD)
• .notice “message” : emit this message at level NOTICE
• .warning “message”: emit this message at level WARNING
• .alert “message” : emit this message at level ALERT

Messages emitted at level WARNING may cause the process to fail to start if “zero-warning” is enabled. Messages emitted at level ALERT will always cause a fatal error. These can be used to detect some inappropriate conditions and provide advice to the user.

Example:

.if "${A}"
  .if "${B}"
     .notice "A=1, B=1"
  .elif "${C}"
     .notice "A=1, B=0, C=1"
  .elif "${D}"
     .warning "A=1, B=0, C=0, D=1"
  .else
     .alert "A=1, B=0, C=0, D=0"
  .endif
.else
     .notice "A=0"
.endif

.diag "WTA/2021-05-07: replace 'redirect' with 'return' after switch to 2.4"
      http-request redirect location /goaway if ABUSE

2.5. Time format

Some parameters involve values representing time, such as timeouts. These values are generally expressed in milliseconds (unless explicitly stated otherwise) but may be expressed in any other unit by suffixing the unit to the numeric value. It is important to consider this because it will not be repeated for every keyword. Supported units are:

• us: microseconds. 1 microsecond = 1/1000000 second
• ms: milliseconds. 1 millisecond = 1/1000 second. This is the default.
• s : seconds. 1s = 1000ms
• m : minutes. 1m = 60s = 60000ms
• h : hours. 1h = 60m = 3600s = 3600000ms
• d : days. 1d = 24h = 1440m = 86400s = 86400000ms

2.6. Size format

Some parameters involve values representing size, such as bandwidth limits. These values are generally expressed in bytes (unless explicitly stated otherwise) but may be expressed in any other unit by suffixing the unit to the numeric value. It is important to consider this because it will not be repeated for every keyword. Supported units are case insensitive:

• k: kilobytes. 1 kilobyte = 1024 bytes
• m: megabytes. 1 megabyte = 1048576 bytes
• g: gigabytes. 1 gigabyte = 1073741824 bytes

Both time and size formats require integers, decimal notation is not allowed.

2.7. Name format for maps and ACLs

It is possible to use a list of pattern for maps or ACLs. A list of pattern is identified by its name and may be used at different places in the configuration. List of pattern are split on three categories depending on the name format:

• Lists of pattern based on regular files: It is the default case. The filename, absolute or relative, is used as name. The file must exist otherwise an error is triggered. But it may be empty. The “file@” prefix may also be specified but it is not part of the name identifying the list. A filename, with or without the prefix, references the same list of pattern.

• Lists of pattern based on optional files: The filename must be preceded by “opt@” prefix. The file existence is optional. If the file exists, its content is loaded but no error is reported if not. The prefix is not part of the name identifying the list. It means, for a given filename, Optional files and regular files reference the same list of pattern.

• Lists of pattern based on virtual files: The name is just an identifier. It is not a reference to any file. “virt@” prefix must be used. It is part of the name. Thus it cannot be mixed with other kind of lists.

Virtual files are useful when patterns are fully dynamically managed with no patterns on startup and on reload. Optional files may be used under the same conditions. But patterns can be dumped in the file, via an external script based on the “show map” CLI command for instance. This way, it is possible to keep patterns on reload.

Note: Even if it is unlikely, it means no regular file starting with “file@”, “opt@” or “virt@” can be loaded, except by adding “./” explicitly in front of the filename (for instance “file@./virt@map”).

2.8. Variables

In HAProxy configuration, variables can be used in sample fetch functions, converters, log-format strings or TCP/HTTP actions. Process-wide variables can be defined, globally accessible for the whole life of the process. Some others have a shorter lifespan. Variables are similar to those found in shell scripts. It is a symbolic name for a chunk of memory. The variables size is not limited and is dynamically allocated. So they must be used with caution, especially for an intensive usage. However, it is possible to limit the maximum amount of memory used by the variables by setting “tune.vars” global parameters.

Variables must be designated using the format “<scope>.<name>”. The <scope> is a single word indicating the life time of the variable. The <name> part, inside a scope, may only contain characters ‘a-z’, ‘A-Z’, ‘0-9’ and ‘_’. It is unique in this scope but the same name in different scopes can be used and refers to different variables. Supported scopes are:

• proc : for variables known during the whole process lifespan and globally accessible. “proc” variables can be manipulated from the CLI using “get var” and “set var” commands. They can also be set from “global” sections via “set-var” and “set-var-fmt” directives.

• sess : for variables known during the whole lifespan of a session. “sess” variables are private to a session, not visbile from outside it and not shared with other sessions.

• txn : for variables known during the whole lifespan of a transaction. “txn” variables are private to a stream, not visible from outside it and not shared with other streams.

• req : for variables known during the request processing for a specific stream. “req” variables are visible from the stream creation and until the first server connection attempt. They are private to a stream, not visible from outside it and not shared with other streams. There is no overlap at all between “req” and “res” variables.

• res : for variables known during the response processing for a specific stream. “res” variables are visible from the first server connection attempt and until the stream destruction. They are private to a stream, not visible from outside it and not shared with other streams. There is no overlap at all between “req” and “res” variables.

• check: for variables known during a health-check execution. “check” variables are private to a health-check, not visible from outside it and are not shared with other health-checks. They can be set using dedicated “tcp-check” or “http-check” directives.

Depending on the context, extra scopes referencing the parent of a current stream can be used:

• psess: same as “sess” but using the session of the parent stream, if any.

• ptxn : same as “txn” but using the transaction of the parent stream, if any.

• preq : same as “req” but using the parent stream, if any. “preq” variables are only accessible during request processing of the parent stream.

• pres : same as “res” but using the parent stream, if any. “pres” variables are only accessible during response processing of the parent stream.

Scopes referencing the parent stream are usable from the moment it is defined. Most of time, there is no parent stream. But, if applicable, this will be explicitly specified. For now, it is only possible to retrieve the value of variables defined in a scope of the parent stream. It is not possible to set nor unset such variables. Usually a child stream performs some processing for the parent at a precise moment and prevents it from making progress until the operation it does is completed. This means that the parent may be stopped in the middle of a request processing or a response processing for example. As such, certain scopes will not be available from the child stream. For example if a request is subject to some analysis performed by a child stream, this child stream will not find any variable in the “pres” scope since the parent is not processing a response, hence doesn’t have any variables in its “res” scope.

The content of a variable is the result of the evaluation of a sample fetch expression and it inherits of the output type of this expression. It is important when the variable is used because its type must be compatible with its usage. For instance a variable containing a string used in “add()” converter must be convertible to a valid integer to succeed. It is especially true when variables are compared to static value. The right matching method must be used.

2.9. Address formats

Several statements as “bind, “server”, “nameserver” and “log” requires an address.

This address can be a host name, an IPv4 address, an IPv6 address, or ‘’. The ‘’ is equal to the special address “0.0.0.0” and can be used, in the case of “bind” or “dgram-bind” to listen on all IPv4 of the system.The IPv6 equivalent is ‘::’.

Depending of the statement, a port or port range follows the IP address. This is mandatory on ‘bind’ statement, optional on ‘server’.

This address can also begin with a slash ‘/’. It is considered as the “unix” family, and ‘/’ and following characters must be present the path.

Default socket type or transport method “datagram” or “stream” depends on the configuration statement showing the address. Indeed, ‘bind’ and ‘server’ will use a “stream” socket type by default whereas ’log’, ’nameserver’ or ‘dgram-bind’ will use a “datagram”.

Optionally, a prefix could be used to force the address family and/or the socket type and the transport method.

2.9.1. Address family prefixes

‘abns@<name>’ following <name> is an abstract namespace (Linux only).

‘abnsz@<name>’ following <name> is a zero-terminated abstract namespace (Linux only).

‘fd@<n>’ following address is a file descriptor <n> inherited from the parent. The fd must be bound and may or may not already be listening.

‘ip@<address>[:port1[-port2]]’ following <address> is considered as an IPv4 or IPv6 address depending on the syntax. Depending on the statement using this address, a port or a port range may or must be specified.

‘ipv4@<address>[:port1[-port2]]’ following <address> is always considered as an IPv4 address. Depending on the statement using this address, a port or a port range may or must be specified.

‘ipv6@<address>[:port1[-port2]]’ following <address> is always considered as an IPv6 address. Depending on the statement using this address, a port or a port range may or must be specified.

‘sockpair@<n>’ following address is the file descriptor of a connected unix socket or of a socketpair. During a connection, the initiator creates a pair of connected sockets, and passes one of them over the FD to the other end. The listener waits to receive the FD from the unix socket and uses it as if it were the FD of an accept(). Should be used carefully.

           Bugs: This protocol is known to be unreliable on macOS because
           of an issue in the macOS sendmsg(2) implementation. The
           connection might not be accepted correctly.

‘unix@<path>’ following string is considered as a UNIX socket <path>. this prefix is useful to declare an UNIX socket path which don’t start by slash ‘/’.

2.9.2. Socket type prefixes

Previous “Address family prefixes” can also be prefixed to force the socket type and the transport method. The default depends of the statement using this address but in some cases the user may force it to a different one. This is the case for “log” statement where the default is syslog over UDP but we could force to use syslog over TCP.

Those prefixes were designed for internal purpose and users should instead use use aliases of the next section “2.9.3 Protocol prefixes”. However these can sometimes be convenient, for example in combination with inherited sockets known by their file descriptor number, in which case the address family is “fd” and the socket type must be declared.

If users need one those prefixes to perform what they expect because they can not configure the same using the protocol prefixes, they should report this to the maintainers.

‘stream+<family>@<address>’ forces socket type and transport method to “stream”

‘dgram+<family>@<address>’ forces socket type and transport method to “datagram”.

‘quic+<family>@<address>’ forces socket type to “datagram” and transport method to “stream”.

2.9.3. Protocol prefixes

‘quic4@<address>[:port1[-port2]]’ following <address> is always considered as an IPv4 address but socket type is forced to “datagram” and the transport method is forced to “stream”. Depending on the statement using this address, a UDP port or port range can or must be specified. It is equivalent to “quic+ipv4@”.

‘quic6@<address>[:port1[-port2]]’ following <address> is always considered as an IPv6 address but socket type is forced to “datagram” and the transport method is forced to “stream”. Depending on the statement using this address, a UDP port or port range can or must be specified. It is equivalent to “quic+ipv6@”.

’tcp@<address>[:port1[-port2]]’ following <address> is considered as an IPv4 or IPv6 address depending of the syntax but socket type and transport method is forced to “stream”. Depending on the statement using this address, a port or a port range can or must be specified. It is considered as an alias of ‘stream+ip@’.

’tcp4@<address>[:port1[-port2]]’ following <address> is always considered as an IPv4 address but socket type and transport method is forced to “stream”. Depending on the statement using this address, a port or port range can or must be specified. It is considered as an alias of ‘stream+ipv4@’.

’tcp6@<address>[:port1[-port2]]’ following <address> is always considered as an IPv6 address but socket type and transport method is forced to “stream”. Depending on the statement using this address, a port or port range can or must be specified. It is considered as an alias of ‘stream+ipv4@’.

‘mptcp@<address>[:port1[-port2]]’ following <address> is considered as an IPv4 or IPv6 address depending of the syntax but socket type and transport method is forced to “stream”, with the MPTCP protocol. Depending on the statement using this address, a port or a port range can or must be specified.

‘mptcp4@<address>[:port1[-port2]]’ following <address> is always considered as an IPv4 address but socket type and transport method is forced to “stream”, with the MPTCP protocol. Depending on the statement using this address, a port or port range can or must be specified.

‘mptcp6@<address>[:port1[-port2]]’ following <address> is always considered as an IPv6 address but socket type and transport method is forced to “stream”, with the MPTCP protocol. Depending on the statement using this address, a port or port range can or must be specified.

‘udp@<address>[:port1[-port2]]’ following <address> is considered as an IPv4 or IPv6 address depending of the syntax but socket type and transport method is forced to “datagram”. Depending on the statement using this address, a port or a port range can or must be specified. It is considered as an alias of ‘dgram+ip@’.

‘udp4@<address>[:port1[-port2]]’ following <address> is always considered as an IPv4 address but socket type and transport method is forced to “datagram”. Depending on the statement using this address, a port or port range can or must be specified. It is considered as an alias of ‘dgram+ipv4@’.

‘udp6@<address>[:port1[-port2]]’ following <address> is always considered as an IPv6 address but socket type and transport method is forced to “datagram”. Depending on the statement using this address, a port or port range can or must be specified. It is considered as an alias of ‘dgram+ipv4@’.

‘uxdg@<path>’ following string is considered as a unix socket <path> but transport method is forced to “datagram”. It is considered as an alias of ‘dgram+unix@’.

‘uxst@<path>’ following string is considered as a unix socket <path> but transport method is forced to “stream”. It is considered as an alias of ‘stream+unix@’.

In future versions, other prefixes could be used to specify protocols like QUIC which proposes stream transport based on socket of type “datagram”.

2.10. Examples

# Simple configuration for an HTTP proxy listening on port 80 on all
    # interfaces and forwarding requests to a single backend "servers" with a
    # single server "server1" listening on 127.0.0.1:8000
    global
        daemon
        maxconn 256

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    frontend http-in
        bind *:80
        default_backend servers

    backend servers
        server server1 127.0.0.1:8000 maxconn 32


    # The same configuration defined with a single listen block. Shorter but
    # less expressive, especially in HTTP mode.
    global
        daemon
        maxconn 256

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    listen http-in
        bind *:80
        server server1 127.0.0.1:8000 maxconn 32

Assuming haproxy is in $PATH, test these configurations in a shell with:

$ sudo haproxy -f configuration.conf -c

12 - 3. Global Section

Parameters in the “global” section are process-wide and often OS-specific. They are generally set once for all and do not need being changed once correct. Some of them have command-line equivalents.

The following keywords are supported in the “global” section:

• Process management and security

  • 51degrees-allow-unmatched
  • 51degrees-cache-size
  • 51degrees-data-file
  • 51degrees-difference
  • 51degrees-drift
  • 51degrees-property-name-list
  • 51degrees-property-separator
  • 51degrees-use-performance-graph
  • 51degrees-use-predictive-graph
  • ca-base
  • chroot
  • cluster-secret
  • cpu-affinity
  • cpu-map
  • cpu-policy
  • cpu-set
  • crt-base
  • daemon
  • default-path
  • description
  • deviceatlas-json-file
  • deviceatlas-log-level
  • deviceatlas-properties-cookie
  • deviceatlas-separator
  • dns-accept-family
  • expose-deprecated-directives
  • expose-experimental-directives
  • external-check
  • fd-hard-limit
  • gid
  • grace
  • group
  • h1-accept-payload-with-any-method
  • h1-case-adjust
  • h1-case-adjust-file
  • h1-do-not-close-on-insecure-transfer-encoding
  • h2-workaround-bogus-websocket-clients
  • hard-stop-after
  • harden.reject-privileged-ports.tcp
  • harden.reject-privileged-ports.quic
  • insecure-fork-wanted
  • insecure-setuid-wanted
  • issuers-chain-path
  • jwt.decrypt_alg_list
  • jwt.decrypt_enc_list
  • key-base
  • limited-quic
  • localpeer
  • log
  • log-send-hostname
  • log-tag
  • lua-load
  • lua-load-per-thread
  • lua-prepend-path
  • max-threads-per-group
  • mworker-max-reloads
  • nbthread
  • node
  • numa-cpu-mapping
  • ocsp-update.disable
  • ocsp-update.maxdelay
  • ocsp-update.mindelay
  • ocsp-update.httpproxy
  • ocsp-update.mode
  • pidfile
  • pp2-never-send-local
  • presetenv
  • prealloc-fd
  • resetenv
  • set-dumpable
  • set-var
  • setenv
  • ssl-default-bind-ciphers
  • ssl-default-bind-ciphersuites
  • ssl-default-bind-client-sigalgs
  • ssl-default-bind-curves
  • ssl-default-bind-options
  • ssl-default-bind-sigalgs
  • ssl-default-server-ciphers
  • ssl-default-server-ciphersuites
  • ssl-default-server-client-sigalgs
  • ssl-default-server-curves
  • ssl-default-server-options
  • ssl-default-server-sigalgs
  • ssl-dh-param-file
  • ssl-propquery
  • ssl-provider
  • ssl-provider-path
  • ssl-security-level
  • ssl-server-verify
  • ssl-skip-self-issued-ca
  • stats
  • stats-file
  • strict-limits
  • uid
  • ulimit-n
  • unix-bind
  • unsetenv
  • user
  • wurfl-cache-size
  • wurfl-data-file
  • wurfl-information-list
  • wurfl-information-list-separator

• Performance tuning

  • busy-polling
  • max-spread-checks
  • maxcompcpuusage
  • maxcomprate
  • maxconn
  • maxconnrate
  • maxpipes
  • maxsessrate
  • maxsslconn
  • maxsslrate
  • maxzlibmem
  • no-memory-trimming
  • noepoll
  • noevports
  • nogetaddrinfo
  • nokqueue
  • noktls
  • nopoll
  • noreuseport
  • nosplice
  • profiling.memory
  • profiling.tasks
  • server-state-base
  • server-state-file
  • spread-checks
  • ssl-engine
  • ssl-mode-async
  • tune.applet.zero-copy-forwarding
  • tune.buffers.limit
  • tune.buffers.reserve
  • tune.bufsize
  • tune.bufsize.large
  • tune.bufsize.small
  • tune.cli.max-payload-size
  • tune.comp.maxlevel
  • tune.defaults.purge
  • tune.disable-fast-forward
  • tune.disable-zero-copy-forwarding
  • tune.epoll.mask-events
  • tune.events.max-events-at-once
  • tune.fail-alloc
  • tune.fd.edge-triggered
  • tune.h1.be.glitches-threshold
  • tune.h1.fe.glitches-threshold
  • tune.h1.zero-copy-fwd-recv
  • tune.h1.zero-copy-fwd-send
  • tune.h2.be.glitches-threshold
  • tune.h2.be.initial-window-size
  • tune.h2.be.max-concurrent-streams
  • tune.h2.be.max-frames-at-once
  • tune.h2.be.rxbuf
  • tune.h2.fe.glitches-threshold
  • tune.h2.fe.initial-window-size
  • tune.h2.fe.max-concurrent-streams
  • tune.h2.fe.max-frames-at-once
  • tune.h2.fe.max-rst-at-once
  • tune.h2.fe.max-total-streams
  • tune.h2.fe.rxbuf
  • tune.h2.header-table-size
  • tune.h2.initial-window-size
  • tune.h2.max-concurrent-streams
  • tune.h2.max-frame-size
  • tune.h2.zero-copy-fwd-send
  • tune.http.cookielen
  • tune.http.logurilen
  • tune.http.maxhdr
  • tune.idle-pool.shared
  • tune.idletimer
  • tune.lua.bool-sample-conversion
  • tune.lua.burst-timeout
  • tune.lua.forced-yield
  • tune.lua.log.loggers
  • tune.lua.log.stderr
  • tune.lua.maxmem
  • tune.lua.openlibs
  • tune.lua.service-timeout
  • tune.lua.session-timeout
  • tune.lua.task-timeout
  • tune.max-checks-per-thread
  • tune.maxaccept
  • tune.maxpollevents
  • tune.maxrewrite
  • tune.max-rules-at-once
  • tune.memory.hot-size
  • tune.pattern.cache-size
  • tune.peers.max-updates-at-once
  • tune.pipesize
  • tune.pool-high-fd-ratio
  • tune.pool-low-fd-ratio
  • tune.pt.zero-copy-forwarding
  • tune.quic.be.cc.cubic-min-losses
  • tune.quic.be.cc.hystart
  • tune.quic.be.cc.max-frame-loss
  • tune.quic.be.cc.max-win-size
  • tune.quic.be.cc.reorder-ratio
  • tune.quic.be.max-idle-timeout
  • tune.quic.be.sec.glitches-threshold
  • tune.quic.be.stream.data-ratio
  • tune.quic.be.stream.max-concurrent
  • tune.quic.be.stream.rxbuf
  • tune.quic.be.tx.pacing
  • tune.quic.be.tx.udp-gso
  • tune.quic.cc.cubic.min-losses (deprecated)
  • tune.quic.cc-hystart (deprecated)
  • tune.quic.disable-tx-pacing (deprecated)
  • tune.quic.disable-udp-gso (deprecated)
  • tune.quic.fe.cc.cubic-min-losses
  • tune.quic.fe.cc.hystart
  • tune.quic.fe.cc.max-frame-loss
  • tune.quic.fe.cc.max-win-size
  • tune.quic.fe.cc.reorder-ratio
  • tune.quic.fe.max-idle-timeout
  • tune.quic.fe.sec.glitches-threshold
  • tune.quic.fe.sec.retry-threshold
  • tune.quic.fe.sock-per-conn
  • tune.quic.fe.stream.data-ratio
  • tune.quic.fe.stream.max-concurrent
  • tune.quic.fe.stream.max-total
  • tune.quic.fe.stream.rxbuf
  • tune.quic.fe.tx.pacing
  • tune.quic.fe.tx.udp-gso
  • tune.quic.frontend.max-data-size (deprecated)
  • tune.quic.frontend.max-idle-timeout (deprecated)
  • tune.quic.frontend.max-streams-bidi (deprecated)
  • tune.quic.frontend.max-tx-mem (deprecated)
  • tune.quic.frontend.stream-data-ratio (deprecated)
  • tune.quic.frontend.default-max-window-size (deprecated)
  • tune.quic.listen
  • tune.quic.max-frame-loss (deprecated)
  • tune.quic.mem.tx-max
  • tune.quic.reorder-ratio (deprecated)
  • tune.quic.retry-threshold (deprecated)
  • tune.quic.socket-owner (deprecated)
  • tune.quic.zero-copy-fwd-send
  • tune.renice.runtime
  • tune.renice.startup
  • tune.rcvbuf.backend
  • tune.rcvbuf.client
  • tune.rcvbuf.frontend
  • tune.rcvbuf.server
  • tune.recv_enough
  • tune.ring.queues
  • tune.runqueue-depth
  • tune.sched.low-latency
  • tune.sndbuf.backend
  • tune.sndbuf.client
  • tune.sndbuf.frontend
  • tune.sndbuf.server
  • tune.streams-elasticity
  • tune.stick-counters
  • tune.ssl.cachesize
  • tune.ssl.capture-buffer-size
  • tune.ssl.capture-cipherlist-size (deprecated)
  • tune.ssl.certificate-compression
  • tune.ssl.default-dh-param
  • tune.ssl.force-private-cache
  • tune.ssl.hard-maxrecord
  • tune.ssl.keylog
  • tune.ssl.keyupdate-rate-limit
  • tune.ssl.lifetime
  • tune.ssl.maxrecord
  • tune.ssl.ssl-ctx-cache-size
  • tune.ssl.ocsp-update.maxdelay (deprecated)
  • tune.ssl.ocsp-update.mindelay (deprecated)
  • tune.takeover-other-tg-connections
  • tune.vars.global-max-size
  • tune.vars.proc-max-size
  • tune.vars.reqres-max-size
  • tune.vars.sess-max-size
  • tune.vars.txn-max-size
  • tune.zlib.memlevel
  • tune.zlib.windowsize

• Debugging

  • anonkey
  • debug.counters
  • force-cfg-parser-pause
  • quiet
  • warn-blocked-traffic-after
  • zero-warning

• HTTPClient

  • httpclient.resolvers.disabled
  • httpclient.resolvers.id
  • httpclient.resolvers.prefer
  • httpclient.retries
  • httpclient.ssl.ca-file
  • httpclient.ssl.verify
  • httpclient.timeout.connect

3.1. Process management and security

51degrees-data-file <file path>

51degrees-data-file <file path>

The path of the 51Degrees data file to provide device detection services. The file should be unzipped and accessible by HAProxy with relevant permissions.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES.

51degrees-property-name-list [<string> ...]

51degrees-property-name-list [<string> ...]

A list of 51Degrees property names to be load from the dataset. A full list of names is available on the 51Degrees website: https://51degrees.com/resources/property-dictionary

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES.

51degrees-property-separator <char>

51degrees-property-separator <char>

A char that will be appended to every property value in a response header containing 51Degrees results. If not set that will be set as ‘,’.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES.

51degrees-cache-size <number>

51degrees-cache-size <number>

Sets the size of the 51Degrees converter cache to <number> entries. This is an LRU cache which reminds previous device detections and their results. By default, this cache is disabled.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES.

51degrees-use-performance-graph { on | off }

51degrees-use-performance-graph { on | off }

Enables (‘on’) or disables (‘off’) the use of the performance graph in the detection process. The default value depends on 51Degrees library.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES and 51DEGREES_VER=4.

51degrees-use-predictive-graph { on | off }

51degrees-use-predictive-graph { on | off }

Enables (‘on’) or disables (‘off’) the use of the predictive graph in the detection process. The default value depends on 51Degrees library.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES and 51DEGREES_VER=4.

51degrees-drift <number>

51degrees-drift <number>

Sets the drift value that a detection can allow.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES and 51DEGREES_VER=4.

51degrees-difference <number>

51degrees-difference <number>

Sets the difference value that a detection can allow.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES and 51DEGREES_VER=4.

51degrees-allow-unmatched { on | off }

51degrees-allow-unmatched { on | off }

Enables (‘on’) or disables (‘off’) the use of unmatched nodes in the detection process. The default value depends on 51Degrees library.

Please note that this option is only available when HAProxy has been compiled with USE_51DEGREES and 51DEGREES_VER=4.

acme.scheduler { auto | off }

acme.scheduler { auto | off }

Enable or disable the ACME scheduler.

The ACME scheduler starts at HAProxy startup, it will loop over the certificates and start an ACME renewal task when the notAfter value is past curtime + (notAfter - notBefore) / 12, or 7 days if notBefore is not defined. The scheduler will then sleep and wakeup after 12 hours.

The default value is “auto”.

See also: acme

ca-base <dir>

ca-base <dir>

Assigns a default directory to fetch SSL CA certificates and CRLs from when a relative path is used with “ca-file”, “ca-verify-file” or “crl-file” directives. Absolute locations specified in “ca-file”, “ca-verify-file” and “crl-file” prevail and ignore “ca-base”.

chroot { <jail dir> | auto }

chroot { <jail dir> | auto }

Changes current directory to <jail dir> and performs a chroot() there before dropping privileges. This increases the security level in case an unknown vulnerability would be exploited, since it would make it very hard for the attacker to exploit the system. It is important to ensure that <jail dir> is both empty and non-writable to anyone. When the process is started with superuser privileges, the chroot() is performed directly. On Linux, when started unprivileged, haproxy attempts to perform it from inside a new user namespace created with unshare(CLONE_NEWUSER); if that mechanism is unavailable the chroot() will fail with the usual error.

As a special case, <jail dir> may be set to “auto”, in which case haproxy creates an anonymous temporary directory, unlinks it, and chroots into it. The resulting jail has no name in the filesystem and is empty and read-only, removing the need to prepare a dedicated jail directory.

When starting with superuser privileges, a warning will be displayed if no chroot is used, in order to encourage users to always use the mechanism. If for any reason there is a compelling reason not to use chroot (e.g. access to a server via a UNIX socket with an unconvenient path), it remains possible to silence the warning by adding an explicit “chroot /”, which has the benefit of being visible in a configuration.

close-spread-time <time>

close-spread-time <time>

Define a time window during which idle connections and active connections closing is spread in case of soft-stop. After a SIGUSR1 is received and the grace period is over (if any), the idle connections will all be closed at once if this option is not set, and active HTTP or HTTP2 connections will be ended after the next request is received, either by appending a “Connection: close” line to the HTTP response, or by sending a GOAWAY frame in case of HTTP2. When this option is set, connection closing will be spread over this set <time>. If the close-spread-time is set to “infinite”, active connection closing during a soft-stop will be disabled. The “Connection: close” header will not be added to HTTP responses (or GOAWAY for HTTP2) anymore and idle connections will only be closed once their timeout is reached (based on the various timeouts set in the configuration).

Arguments:

<time>  is a time window (by default in milliseconds) during which
        connection closing will be spread during a soft-stop operation, or
        "infinite" if active connection closing should be disabled.

It is recommended to set this setting to a value lower than the one used in the “hard-stop-after” option if this one is used, so that all connections have a chance to gracefully close before the process stops.

See also: grace, hard-stop-after, idle-close-on-response

cluster-secret <secret>

cluster-secret <secret>

Define an ASCII string secret shared between several nodes belonging to the same cluster. It could be used for different usages. It is at least used to derive stateless reset tokens for all the QUIC connections instantiated by this process. This is also the case to derive secrets used to encrypt Retry tokens.

If this parameter is not set, a random value will be selected on process startup. This allows to use features which rely on it, albeit with some limitations.

cpu-map [auto:]<thread-group>[/<thread-set>] <cpu-set>[,...] [...]

cpu-map [auto:]<thread-group>[/<thread-set>] <cpu-set>[,...] [...]

On some operating systems, it is possible to bind a thread group or a thread to a specific CPU set. This means that the designated threads will never run on other CPUs. The “cpu-map” directive specifies CPU sets for individual threads or thread groups. The first argument is a thread group range, optionally followed by a thread set. These ranges have the following format:

all | odd | even | number[-[number]]

<number> must be a number between 1 and 32 or 64, depending on the machine’s word size. Any group IDs above ’thread-groups’ and any thread IDs above the machine’s word size are ignored. All thread numbers are relative to the group they belong to. It is possible to specify a range with two such number delimited by a dash (’-’). It also is possible to specify all threads at once using “all”, only odd numbers using “odd” or even numbers using “even”, just like with the “thread” bind directive. The second and forthcoming arguments are CPU sets. Each CPU set is either a unique number starting at 0 for the first CPU or a range with two such numbers delimited by a dash (’-’). These CPU numbers and ranges may be repeated by delimiting them with commas or by passing more ranges as new arguments on the same line. Outside of Linux and BSD operating systems, there may be a limitation on the maximum CPU index to either 31 or 63. Multiple “cpu-map” directives may be specified, but each “cpu-map” directive will replace the previous ones when they overlap.

Ranges can be partially defined. The higher bound can be omitted. In such case, it is replaced by the corresponding maximum value, 32 or 64 depending on the machine’s word size.

The prefix “auto:” can be added before the thread set to let HAProxy automatically bind a set of threads to a CPU by incrementing threads and CPU sets. To be valid, both sets must have the same size. No matter the declaration order of the CPU sets, it will be bound from the lowest to the highest bound. Having both a group and a thread range with the “auto:” prefix is not supported. Only one range is supported, the other one must be a fixed number.

Note that group ranges are supported for historical reasons. Nowadays, a lone number designates a thread group and must be 1 if thread-groups are not used, and specifying a thread range or number requires to prepend “1/” in front of it if thread groups are not used. Finally, “1” is strictly equivalent to “1/all” and designates all threads in the group.

Examples:

cpu-map 1/all 0-3 # bind all threads of the first group on the
                  # first 4 CPUs

cpu-map 1/1- 0-   # will be replaced by "cpu-map 1/1-64 0-63"
                  # or "cpu-map 1/1-32 0-31" depending on the machine's
                  # word size.

# all these lines bind thread 1 to the cpu 0, the thread 2 to cpu 1
# and so on.
cpu-map auto:1/1-4   0-3
cpu-map auto:1/1-4   0-1 2-3
cpu-map auto:1/1-4   3 2 1 0
cpu-map auto:1/1-4   3,2,1,0

# bind each thread to exactly one CPU using all/odd/even keyword
cpu-map auto:1/all   0-63
cpu-map auto:1/even  0-31
cpu-map auto:1/odd   32-63

# invalid cpu-map because thread and CPU sets have different sizes.
cpu-map auto:1/1-4   0    # invalid
cpu-map auto:1/1     0-3  # invalid

# map 40 threads of those 4 groups to individual CPUs
cpu-map auto:1/1-10   0-9
cpu-map auto:2/1-10   10-19
cpu-map auto:3/1-10   20-29
cpu-map auto:4/1-10   30-39

# Map 80 threads to one physical socket and 80 others to another socket
# without forcing assignment. These are split into 4 groups since no
# group may have more than 64 threads.
cpu-map 1/1-40   0-39,80-119    # node0, siblings 0 & 1
cpu-map 2/1-40   0-39,80-119
cpu-map 3/1-40   40-79,120-159  # node1, siblings 0 & 1
cpu-map 4/1-40   40-79,120-159

cpu-affinity <affinity>

cpu-affinity <affinity>

Defines how you want threads to be bound to cpus. It currently accepts the following values:

• per-core: each thread will be bound to all the hardware threads of one core.
• per-group: each thread will be bound to all the hardware threads of the group. This is the default unless “threads-per-core 1” is used in “cpu-policy”. “per-group” accepts an optional argument, to specify how CPUs should be allocated. When a list of CPUs is larger than the maximum allowed number of CPUs per group and has to be split between multiple groups, an extra option allows to choose how the groups will be bound to those CPUs:
  • auto: each thread group will only be assigned a fair share of contiguous CPU cores that are dedicated to it and not shared with other groups. This is the default as it generally is more optimal.
  • loose: each group will still be allowed to use any CPU in the list. This generally causes more contention, but may sometimes help deal better with parasitic loads running on the same CPUs.
• auto: “per-group” will be used, unless “threads-per-core 1” is used in “cpu-policy”, in which case “per-core” will be used. This is the default.
• per-thread: that will bind one thread to one hardware thread only. If “threads-per-core 1” is used in “cpu-policy”, then each thread will be bound to one hardware thread of a different core.
• per-ccx: each thread will be bound to all the hardware threads of a CCX.

cpu-policy <policy> [threads-per-core 1 | auto]

cpu-policy <policy> [threads-per-core 1 | auto]

Selects the CPU allocation policy to be used.

On multi-CPU systems, there can be plenty of reasons for not using all available CPU cores, and/or for grouping them into different thread groups, for performance, latency, cost, or system-wide resource management. The “cpu-set” directive already allows to evict a number of them, but once done, it is necessary to decide how to assign the remaining ones to threads and thread groups.

This mapping is normally performed using the “cpu-map” directive, though it can be particularly difficult to maintain on heterogeneous systems.

The “cpu-policy” directive chooses between a small number of allocation policies which one to use instead, when “cpu-map” is not used. The following policies are currently supported, with “performance” being the default one:

• none no particular post-selection is performed. All enabled CPUs will be usable, and if the number of threads is not set, it will be set to the number of available CPUs but no more than 32 for 32-bit systems or 64 for 64-bit systems, per thread-group. The number of thread-groups, if not set, will be set to 1.

• efficiency exactly like “group-by-ccx” below, except that CPU clusters composed of cores whose performance is more than 25% above that of the next less performant one are evicted. These are typically “big” or “performance” cores. This means that if more than one type of CPU cores are detected, only the efficient one will be used. This can make sense for use with moderate loads when the most powerful cores need to be available to the application or a security component. Some modern CPUs have a large number of such efficient CPU cores which can collectively deliver a decent level of performance while using less power.

• first-usable-node if the CPUs were not previously restricted at boot (for example using the “taskset” utility), and if the “nbthread” directive was not set, then the first NUMA node with enabled CPUs will be used, and this number of CPUs will be used as the number of threads. A single thread group will be enabled with all of them, within the limit of 32 or 64 depending on the system.

• group-by-2-ccx same as “group-by-ccx” below but create a group every two CCX. This can make sense on CPUs having many CCX of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between CCX is slow. This is generally recommended against.

• group-by-2-clusters same as “group-by-cluster” but create a group every two clusters. This can make sense on CPUs having many clusters of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between clusters is slow. This is generally recommended against.

• group-by-3-ccx same as “group-by-ccx” below but create a group every three CCX. This can make sense on CPUs having many CCX of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between CCX is slow. This is generally recommended against.

• group-by-3-clusters same as “group-by-cluster” but create a group every three clusters. This can make sense on CPUs having many clusters of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between clusters is slow. This is generally recommended against.

• group-by-4-ccx same as “group-by-ccx” below but create a group every four CCX. This can make sense on CPUs having many CCX of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between CCX is slow. This is generally recommended against.

• group-by-4-clusters same as “group-by-cluster” but create a group every four clusters. This can make sense on CPUs having many clusters of few cores each, to avoid creating many groups, or to smooth the distribution a little bit when not all cores are in use. Please note that it can have very bad performance effects when the communication between clusters is slow. This is generally recommended against.

• group-by-ccx if neither “nbthread” not “nbtgroups” were set, then one thread group is created for each CPU core complex (“CCX”) with available CPUs, each with as many threads as CPUs. A CCX groups CPUs having a similarly fast access to the last level cache (“LLC”), typically the L3 cache. On most modern machines, it is critical for performance not to mix CPUs from distant CCX in the same thread group. All threads of a group are then bound to all CPUs of the CCX so that intra-group communications remain local to the CCX without enforcing too strong a binding. The per-group thread limits and thread-group limits are respected. This is recommended on multi-socket and NUMA systems, as well as CPUs with bad inter-CCX latencies.

• group-by-cluster if neither “nbthread” not “nbtgroups” were set, then one thread group is created for each CPU cluster with available CPUs, each with as many threads as CPUs. All threads of a group are bound to all CPUs of the cluster so that intra-group communications remain local to the cluster without enforcing too strong a binding. The per-group thread limits and thread-group limits are respected. This is recommended on multi-socket and NUMA systems, as well as CPUs with bad inter-CCX latencies. On most server machines, clusters and CCX are the same, but on heterogeneous machines (“performance” vs “efficiency” or “big” vs “little”), a cluster will generally be made of only a part of a CCX composed only of very similar CPUs (same type, +/-5% frequency difference max). The difference is visible on modern laptops and desktop machines used by developers and admins to validate setups.

• performance exactly like “group-by-ccx” above, except that CPU clusters composed of cores whose performance is less than 80% of those of the next more performant one are evicted. These are typically “little” or “efficient” cores, whose addition generally doesn’t bring significant gains and can easily be counter-productive (e.g. TLS handshakes). Often, keeping such cores for other tasks such as network handling is much more effective. On development systems, these can also be used to run auxiliary tools such as load generators and monitoring tools. This is the default policy.

• resource this is like “group-by-cluster” above, except that only the smallest and most efficient CPU cluster will be used, while all other ones will be ignored. This can be used to limit the resource usage to the strict minimum that still delivers decent performance, for example to try to further reduce power consumption or minimize the number of cores needed on some rented systems for a sidecar setup, in order to scale the system down more easily. Note that if a single cluster is present, it will still be fully used.

An optional keyword can be added, “threads-per-core”. It can accept two values, “1” and “auto”. If set to 1, then only one thread per core will be created, unrespective of how many hardware threads the core has. If set to auto, then one thread per hardware thread will be created. If no affinity is specified, and threads-per-core 1 is used, then by default the affinity will be per-core.

See also: “cpu-map”, “cpu-set”, “nbthread”

cpu-set <directive>...

cpu-set <directive>...

Allows to symbolically describe what sets of CPUs to run on. The directive supports the following keyword: - reset this undoes any previous limitation that could have been inherited by a service manager or a “taskset” command for example. - drop-cpu <set> do not bind to CPUs in this set - only-cpu <set> do not bind to CPUs not in this set - drop-node <set> do not bind to CPUs belonging to this NUMA node - only-node <set> do not bind to CPUs not belonging to this NUMA node - drop-cluster <set> do not bind to CPUs on this hardware cluster number - only-cluster <set> do not bind to CPUs on other hardware cluster number - drop-core <set> do not bind to CPUs on this hardware core number - only-core <set> do not bind to CPUs on other hardware core number - drop-thread <set> do not bind to CPUs on this hardware thread number - only-thread <set> do not bind to CPUs on other hardware thread number See also: “cpu-policy”

crt-base <dir>

crt-base <dir>

Assigns a default directory to fetch SSL certificates from when a relative path is used with “crtfile” or “crt” directives. Absolute locations specified prevail and ignore “crt-base”.

daemon

daemon

Makes the process fork into background. This is the recommended mode of operation. It is equivalent to the command line “-D” argument. It can be disabled by the command line “-db” argument. This option is ignored in systemd mode.

default-path { current | config | parent | origin <path> }

default-path { current | config | parent | origin <path> }

By default HAProxy loads all files designated by a relative path from the location the process is started in. In some circumstances it might be desirable to force all relative paths to start from a different location just as if the process was started from such locations. This is what this directive is made for. Technically it will perform a temporary chdir() to the designated location while processing each configuration file, and will return to the original directory after processing each file. It takes an argument indicating the policy to use when loading files whose path does not start with a slash (’/’): - “current” indicates that all relative files are to be loaded from the directory the process is started in; this is the default.

- "config" indicates that all relative files should be loaded from the
  directory containing the configuration file. More specifically, if the
  configuration file contains a slash ('/'), the longest part up to the
  last slash is used as the directory to change to, otherwise the current
  directory is used. This mode is convenient to bundle maps, errorfiles,
  certificates and Lua scripts together as relocatable packages. When
  multiple configuration files are loaded, the directory is updated for
  each of them.

- "parent" indicates that all relative files should be loaded from the
  parent of the directory containing the configuration file. More
  specifically, if the configuration file contains a slash ('/'), ".."
  is appended to the longest part up to the last slash is used as the
  directory to change to, otherwise the directory is "..". This mode is
  convenient to bundle maps, errorfiles,  certificates and Lua scripts
  together as relocatable packages, but where each part is located in a
  different subdirectory (e.g. "config/", "certs/", "maps/", ...).

- "origin" indicates that all relative files should be loaded from the
  designated (mandatory) path. This may be used to ease management of
  different HAProxy instances running in parallel on a system, where each
  instance uses a different prefix but where the rest of the sections are
  made easily relocatable.

Each “default-path” directive instantly replaces any previous one and will possibly result in switching to a different directory. While this should always result in the desired behavior, it is really not a good practice to use multiple default-path directives, and if used, the policy ought to remain consistent across all configuration files.

Warning: some configuration elements such as maps or certificates are uniquely identified by their configured path. By using a relocatable layout, it becomes possible for several of them to end up with the same unique name, making it difficult to update them at run time, especially when multiple configuration files are loaded from different directories. It is essential to observe a strict collision-free file naming scheme before adopting relative paths. A robust approach could consist in prefixing all files names with their respective site name, or in doing so at the directory level.

description <text>

description <text>

Add a text that describes the instance.

Please note that it is required to escape certain characters (# for example) and this text is inserted into a html page so you should avoid using “<” and “>” characters.

deviceatlas-json-file <path>

deviceatlas-json-file <path>

Sets the path of the DeviceAtlas JSON data file to be loaded by the API. The path must be a valid JSON data file and accessible by HAProxy process.

deviceatlas-log-level <value>

deviceatlas-log-level <value>

Sets the level of information returned by the API. This directive is optional and set to 0 by default if not set.

deviceatlas-properties-cookie <name>

deviceatlas-properties-cookie <name>

Sets the client cookie’s name used for the detection if the DeviceAtlas Client-side component was used during the request. This directive is optional and set to DAPROPS by default if not set.

deviceatlas-separator <char>

deviceatlas-separator <char>

Sets the character separator for the API properties results. This directive is optional and set to | by default if not set.

dns-accept-family <family>[,...]

dns-accept-family <family>[,...]

By default, DNS resolvers accept both IPv4 and IPv6 addresses. This can be influenced by the “resolve-prefer” keywords on server lines as well as the family argument to the “do-resolve” action, but that is only a preference, which does not block the other family from being used when it’s alone. In some environments where dual-stack is not usable, stumbling on an unreachable IPv6-only DNS record can cause significant trouble as it will replace a previous IPv4 one which would possibly have continued to work till next request. The “dns-accept-family” global option permits to enforce usage of only one (or both) address families. The argument is a comma-delimited list of the following words: - “ipv4”: query and accept IPv4 addresses (“A” records) - “ipv6”: query and accept IPv6 addresses (“AAAA” records) - “auto”: use IPv4, and IPv6 if the system has a default gateway for it. The result of the last check is cached for 30 seconds.

When a single family is used, no request will be sent to resolvers for the other family, and any response for the other family will be ignored. The default value since 3.3 is “auto”, which effectively enables both families only once IPv6 has been proven to be routable, otherwise sticks to IPv4. See also: “resolve-prefer”, “do-resolve”

expose-deprecated-directives

expose-deprecated-directives

This statement must appear before using some directives tagged as deprecated to silent warnings and make sure the config file will not be rejected. Not all deprecated directives are concerned, only those without any alternative solution.

expose-experimental-directives

expose-experimental-directives

This statement must appear before using directives tagged as experimental or the config file will be rejected. Please note that features covered by this option are not guaranteed to work well and may break during the maintenance cycle. Developers will maintain them in best effort mode while the next version is being worked on, and will deploy any reasonable effort to avoid breaking them but with no guarantee. For these reasons, these features are not expected to be supported beyond the release of the next LTS release. Users who want to try experimental features are expected to upgrade quickly to benefit from the improvements made to that feature. In order to know if this directive is still needed, it’s easy: if it is enabled without being used by any such feature, a warning will be emitted suggesting to turn it off. So without any warning, it means it’s still needed.

external-check [preserve-env]

external-check [preserve-env]

Allows the use of an external agent to perform health checks. This is disabled by default as a security precaution, and even when enabled, checks may still fail unless “insecure-fork-wanted” is enabled as well. If the program launched makes use of a setuid executable (it should really not), you may also need to set “insecure-setuid-wanted” in the global section. By default, the checks start with a clean environment which only contains variables defined in the “external-check” command in the backend section. It may sometimes be desirable to preserve the environment though, for example when complex scripts retrieve their extra paths or information there. This can be done by appending the “preserve-env” keyword. In this case however it is strongly advised not to run a setuid nor as a privileged user, as this exposes the check program to potential attacks. See “option external-check”, and “insecure-fork-wanted”, and “insecure-setuid-wanted” for extra details.

fd-hard-limit <number>

fd-hard-limit <number>

Sets an upper bound to the maximum number of file descriptors that the process will use, regardless of system limits. While “ulimit-n” and “maxconn” may be used to enforce a value, when they are not set, the process will be limited to the hard limit of the RLIMIT_NOFILE setting as reported by “ulimit -n -H”. But some modern operating systems are now allowing extremely large values here (in the order of 1 billion), which will consume way too much RAM for regular usage. The fd-hard-limit setting is provided to enforce a possibly lower bound to this limit. This means that it will always respect the system-imposed limits when they are below <number> but the specified value will be used if system-imposed limits are higher. By default fd-hard-limit is set to 1048576. This default could be changed via DEFAULT_MAXFD compile-time variable, that could serve as the maximum (kernel) system limit, if RLIMIT_NOFILE hard limit is extremely large. fd-hard-limit set in global section allows to temporarily override the value provided via DEFAULT_MAXFD at the build-time. In the example below, no other setting is specified and the maxconn value will automatically adapt to the lower of “fd-hard-limit” and the RLIMIT_NOFILE limit:

global
    # use as many FDs as possible but no more than 50000
    fd-hard-limit 50000

See also: ulimit-n, maxconn

gid <number>

gid <number>

Changes the process’s group ID to <number>. It is recommended that the group ID is dedicated to HAProxy or to a small set of similar daemons. HAProxy must be started with a user belonging to this group, or with superuser privileges. Note that if HAProxy is started from a user having supplementary groups, it will only be able to drop these groups if started with superuser privileges. See also “group” and “uid”.

grace <time>

grace <time>

Defines a delay between SIGUSR1 and real soft-stop.

Arguments:

<time>  is an extra delay (by default in milliseconds) after receipt of the
        SIGUSR1 signal that will be waited for before proceeding with the
        soft-stop operation.

This is used for compatibility with legacy environments where the haproxy process needs to be stopped but some external components need to detect the status before listeners are unbound. The principle is that the internal “stopping” variable (which is reported by the “stopping” sample fetch function) will be turned to true, but listeners will continue to accept connections undisturbed, until the delay expires, after what the regular soft-stop will proceed. This must not be used with processes that are reloaded, or this will prevent the old process from unbinding, and may prevent the new one from starting, or simply cause trouble.

Example:

global
  grace 10s

# Returns 200 OK until stopping is set via SIGUSR1
frontend ext-check
  bind:9999
  monitor-uri /ext-check
  monitor fail if { stopping }

Please note that a more flexible and durable approach would instead consist for an orchestration system in setting a global variable from the CLI, use that variable to respond to external checks, then after a delay send the SIGUSR1 signal.

Example:

# Returns 200 OK until proc.stopping is set to non-zero. May be done
# from HTTP using set-var(proc.stopping) or from the CLI using:
# > set var proc.stopping int(1)
frontend ext-check
  bind:9999
  monitor-uri /ext-check
  monitor fail if { var(proc.stopping) -m int gt 0 }

See also: hard-stop-after, monitor

group <group name>

group <group name>

Similar to “gid” but uses the GID of group name <group name> from /etc/group. See also “gid” and “user”.

h1-accept-payload-with-any-method

h1-accept-payload-with-any-method

Does not reject HTTP/1.0 GET/HEAD/DELETE requests with a payload with a 413 Payload Too Large HTTP response.

While It is explicitly allowed in HTTP/1.1, HTTP/1.0 is not clear on this point and some old servers don’t expect any payload and never look for body length (via Content-Length or Transfer-Encoding headers). It means that some intermediaries may properly handle the payload for HTTP/1.0 GET/HEAD/DELETE requests, while some others may totally ignore it. That may lead to security issues because a request smuggling attack is possible. Thus, by default, HAProxy rejects HTTP/1.0 GET/HEAD/DELETE requests with a payload.

However, it may be an issue with some old clients. In this case, this global option may be set.

h1-do-not-close-on-insecure-transfer-encoding

h1-do-not-close-on-insecure-transfer-encoding

As mandated by the HTTP/1.1 specification (RFC9112#6.1), the presence of both a Transfer-Encoding header field and a Content-Length header field in the same message represents a serious risk of conveying a content smuggling attack if there are any HTTP/1.0 agent anywhere in the upstream of downstream chain, and when facing this, an agent must absolutely close the connection after the response so as to prevent any exploitation. But this may have a performance impact on some very old clients, especially if they need to renegotiate a TLS connection for every request. This option is present to ask HAProxy not to enforce this rule, and to just sanitize the message but leave the connection alive after the response. This may only be done when absolutely certain that no HTTP/1.0 agents are present in the chain and that all implementations before HAProxy are fully HTTP/1.1 compliant regarding the rules that apply to these header fields. In any case, HAProxy will continue to ignore and drop the extraneous Content-Length header so as not to confuse the next hop.

When enabling this option to work around an old broken client or server, it is important to understand that regardless of the need or not for this option, such an agent violating this rule faces a risk to see its messages truncated by old agents that would consider Content-Length and ignore Transfer-Encoding, since the cumulated size of the encoded chunk sizes are not being accounted for. As such, the rule above is not just a matter of security but also of taking care of getting rid of agents that may face communication trouble due to incompatibilities with older ones.

h1-case-adjust <from> <to>

h1-case-adjust <from> <to>

Defines the case adjustment to apply, when enabled, to the header name <from>, to change it to <to> before sending it to HTTP/1 clients or servers. <from> must be in lower case, and <from> and <to> must not differ except for their case. It may be repeated if several header names need to be adjusted. Duplicate entries are not allowed. If a lot of header names have to be adjusted, it might be more convenient to use “h1-case-adjust-file”. Please note that no transformation will be applied unless “option h1-case-adjust-bogus-client” or “option h1-case-adjust-bogus-server” is specified in a proxy.

There is no standard case for header names because, as stated in RFC7230, they are case-insensitive. So applications must handle them in a case-insensitive manner. But some bogus applications violate the standards and erroneously rely on the cases most commonly used by browsers. This problem becomes critical with HTTP/2 because all header names must be exchanged in lower case, and HAProxy follows the same convention. All header names are sent in lower case to clients and servers, regardless of the HTTP version.

Applications which fail to properly process requests or responses may require to temporarily use such workarounds to adjust header names sent to them for the time it takes the application to be fixed. Please note that an application which requires such workarounds might be vulnerable to content smuggling attacks and must absolutely be fixed.

Example:

global
  h1-case-adjust content-length Content-Length

See “h1-case-adjust-file”, “option h1-case-adjust-bogus-client” and “option h1-case-adjust-bogus-server”.

h1-case-adjust-file <hdrs-file>

h1-case-adjust-file <hdrs-file>

Defines a file containing a list of key/value pairs used to adjust the case of some header names before sending them to HTTP/1 clients or servers. The file <hdrs-file> must contain 2 header names per line. The first one must be in lower case and both must not differ except for their case. Lines which start with ‘#’ are ignored, just like empty lines. Leading and trailing tabs and spaces are stripped. Duplicate entries are not allowed. Please note that no transformation will be applied unless “option h1-case-adjust-bogus-client” or “option h1-case-adjust-bogus-server” is specified in a proxy.

If this directive is repeated, only the last one will be processed. It is an alternative to the directive “h1-case-adjust” if a lot of header names need to be adjusted. Please read the risks associated with using this.

See “h1-case-adjust”, “option h1-case-adjust-bogus-client” and “option h1-case-adjust-bogus-server”.

h2-workaround-bogus-websocket-clients

h2-workaround-bogus-websocket-clients

This disables the announcement of the support for h2 websockets to clients. This can be use to overcome clients which have issues when implementing the relatively fresh RFC8441, such as Firefox 88. To allow clients to automatically downgrade to http/1.1 for the websocket tunnel, specify h2 support on the bind line using “alpn” without an explicit “proto” keyword. If this statement was previously activated, this can be disabled by prefixing the keyword with “no”.

hard-stop-after <time>

hard-stop-after <time>

Defines the maximum time allowed to perform a clean soft-stop.

Arguments:

<time>  is the maximum time (by default in milliseconds) for which the
        instance will remain alive when a soft-stop is received via the
        SIGUSR1 signal.

This may be used to ensure that the instance will quit even if connections remain opened during a soft-stop (for example with long timeouts for a proxy in tcp mode). It applies both in TCP and HTTP mode.

Example:

global
  hard-stop-after 30s

See also: grace

harden.reject-privileged-ports.tcp { on | off }

harden.reject-privileged-ports.tcp { on | off }
harden.reject-privileged-ports.quic { on | off }

Toggle per protocol protection which forbid communication with clients which use privileged ports as their source port. This range of ports is defined according to RFC 6335. By default, protection is active for QUIC protocol as this behavior is suspicious and may be used as a spoofing or DNS/NTP amplification attack.

http-err-codes [+-]<range>[,...] [...]

http-err-codes [+-]<range>[,...] [...]

Replace, reduce or extend the list of status codes that define an error as considered by the termination codes and the “http_err_cnt” counter in stick tables. The default range for errors is 400 to 499, but in certain contexts some users prefer to exclude specific codes, especially when tracking client errors (e.g. 404 on systems with dynamically generated contents). See also “http-fail-codes” and “http_err_cnt”.

A range specified without ‘+’ nor ‘-’ redefines the existing range to the new one. A range starting with ‘+’ extends the existing range to also include the specified one, which may or may not overlap with the existing one. A range starting with ‘-’ removes the specified range from the existing one. A range consists in a number from 100 to 599, optionally followed by “-” followed by another number greater than or equal to the first one to indicate the high boundary of the range. Multiple ranges may be delimited by commas for a same add/del/ replace operation.

Example:

http-err-codes 400,402-444,446-480,490   # sets exactly these codes
http-err-codes 400-499 -450 +500         # sets 400 to 500 except 450
http-err-codes -450-459                  # removes 450 to 459 from range
http-err-codes +501,505                  # adds 501 and 505 to range

http-fail-codes [+-]<range>[,...] [...]

http-fail-codes [+-]<range>[,...] [...]

Replace, reduce or extend the list of status codes that define a failure as considered by the termination codes and the “http_fail_cnt” counter in stick tables. The default range for failures is 500 to 599 except 501 and 505 which can be triggered by clients, and normally indicate a failure from the server to process the request. Some users prefer to exclude certain codes in certain contexts where it is known they’re not relevant, such as 500 in certain SOAP environments as it doesn’t translate a server fault there. The syntax is exactly the same as for http-err-codes above. See also “http-err-codes” and “http_fail_cnt”.

insecure-fork-wanted

insecure-fork-wanted

By default HAProxy tries hard to prevent any thread and process creation after it starts. Doing so is particularly important when using Lua files of uncertain origin, and when experimenting with development versions which may still contain bugs whose exploitability is uncertain. And generally speaking it’s good hygiene to make sure that no unexpected background activity can be triggered by traffic. But this prevents external checks from working, and may break some very specific Lua scripts which actively rely on the ability to fork. This option is there to disable this protection. Note that it is a bad idea to disable it, as a vulnerability in a library or within HAProxy itself will be easier to exploit once disabled. In addition, forking from Lua or anywhere else is not reliable as the forked process may randomly embed a lock set by another thread and never manage to finish an operation. As such it is highly recommended that this option is never used and that any workload requiring such a fork be reconsidered and moved to a safer solution (such as agents instead of external checks). This option supports the “no” prefix to disable it. This can also be activated with “-dI” on the haproxy command line.

insecure-setuid-wanted

insecure-setuid-wanted

HAProxy doesn’t need to call executables at run time (except when using external checks which are strongly recommended against), and is even expected to isolate itself into an empty chroot. As such, there basically is no valid reason to allow a setuid executable to be called without the user being fully aware of the risks. In a situation where HAProxy would need to call external checks and/or disable chroot, exploiting a vulnerability in a library or in HAProxy itself could lead to the execution of an external program. On Linux it is possible to lock the process so that any setuid bit present on such an executable is ignored. This significantly reduces the risk of privilege escalation in such a situation. This is what HAProxy does by default. In case this causes a problem to an external check (for example one which would need the “ping” command), then it is possible to disable this protection by explicitly adding this directive in the global section. If enabled, it is possible to turn it back off by prefixing it with the “no” keyword.

issuers-chain-path <dir>

issuers-chain-path <dir>

Assigns a directory to load certificate chain for issuer completion. All files must be in PEM format. For certificates loaded with “crt” or “crt-list”, if certificate chain is not included in PEM (also commonly known as intermediate certificate), HAProxy will complete chain if the issuer of the certificate corresponds to the first certificate of the chain loaded with “issuers-chain-path”. A “crt” file with PrivateKey+Certificate+IntermediateCA2+IntermediateCA1 could be replaced with PrivateKey+Certificate. HAProxy will complete the chain if a file with IntermediateCA2+IntermediateCA1 is present in “issuers-chain-path” directory. All other certificates with the same issuer will share the chain in memory.

The OCSP features are able to use the completed chain when no .issuer was used, or no chain was provided in the PEM.

jwt.decrypt_alg_list <list>

jwt.decrypt_alg_list <list>

Set the list of algorithms allowed in the jwt_decrypt_XXX converters. JWT tokens using an unsupported or disabled algorithms will never be decrypted. The specified algorithms must have the same format as in section 4.1 of RFC7518 and must be colon-separated. The special “ALL” name can be used to enable all the supported algorithms (see “jwt_decrypt_jwk” converter for a complete list) and a ‘!’ can be appended to an algorithm name to explicitly disable it. Please note that unless “ALL” is specified, using this option will disable any algorithm that is not explicitly mentioned in the provided list.

Examples:

# Enable all algorithms but the "ECDH-ES" one
jwt.decrypt_alg_list ALL:!ECDH-ES

# Only enable ECDH-ES algorithms
jwt.decrypt_alg_list ECDH-ES:ECDH-ES+A128KW:ECDH-ES+A192KW:ECDH-ES+A256KW

jwt.decrypt_enc_list <list>

jwt.decrypt_enc_list <list>

Set the list of encryption algorithms allowed in the jwt_decrypt_XXX converters. JWT tokens using an unsupported or disabled encryption algorithms will never be decrypted. The specified algorithms must have the same format as in section 5.1 of RFC7518 and must be colon-separated. The special “ALL” name can be used to enable all the supported algorithms (see “jwt_decrypt_jwk” converter for a complete list) and a ‘!’ can be appended to an algorithm name to explicitly disable it. Please note that unless “ALL” is specified, using this option will disable any algorithm that is not explicitly mentioned in the provided list.

Examples:

# Enable only AES GCM encrypting algorithms
jwt.decrypt_enc_list A128GCM:A192GCM:A256GCM

key-base <dir>

key-base <dir>

Assigns a default directory to fetch SSL private keys from when a relative path is used with “key” directives. Absolute locations specified prevail and ignore “key-base”. This option only works with a crt-store load line.

limited-quic

limited-quic

This setting must be used to explicitly enable the QUIC listener bindings when haproxy is compiled with a version of OpenSSL without QUIC support. It activates an haproxy internal compatibility layer which must have been selected at build time with USE_QUIC_OPENSSL_COMPAT=1. This compatibility layer supports most of the necessary TLS operations, albeit without QUIC 0-RTT capability.

This feature is primarily targeted for OpenSSL prior to version 3.5.2, where QUIC API was not implemented or only partially. The compatibility layer can still be activated for version 3.5.2 and above, but this is probably unnecessary.

If limited-quic is set but the compatibility layer was not selected at build time, the option is silently ignored and QUIC TLS operations rely on the TLS library.

localpeer <name>

localpeer <name>

Sets the local instance’s peer name. It will be ignored if the “-L” command line argument is specified or if used after “peers” section definitions. In such cases, a warning message will be emitted during the configuration parsing.

This option will also set the HAPROXY_LOCALPEER environment variable. See also “-L” in the management guide and “peers” section below.

log <target> [len <length>] [format <format>] [sample <ranges>:<sample_size>]

log <target> [len <length>] [format <format>] [sample <ranges>:<sample_size>]
    [profile <prof>] <facility> [max level [min level]]

Adds a global syslog server. Several global servers can be defined. They will receive logs for starts and exits, as well as all logs from proxies configured with “log global”. See “log” option for proxies for more details.

log-send-hostname [<string>]

log-send-hostname [<string>]

Sets the hostname field in the syslog header. If optional “string” parameter is set the header is set to the string contents, otherwise uses the hostname of the system. Generally used if one is not relaying logs through an intermediate syslog server or for simply customizing the hostname printed in the logs.

log-tag <string>

log-tag <string>

Sets the tag field in the syslog header to this string. It defaults to the program name as launched from the command line, which usually is “haproxy”. Sometimes it can be useful to differentiate between multiple processes running on the same host. See also the per-proxy “log-tag” directive.

lua-load <file> [ <arg1> [ <arg2> [ ... ] ] ]

lua-load <file> [ <arg1> [ <arg2> [ ... ] ] ]

This global directive loads and executes a Lua file in the shared context that is visible to all threads. Any variable set in such a context is visible from any thread. This is the easiest and recommended way to load Lua programs but it will not scale well if a lot of Lua calls are performed, as only one thread may be running on the global state at a time. A program loaded this way will always see 0 in the “core.thread” variable. This directive can be used multiple times.

args are available in the lua file using the code below in the body of the file. Do not forget that Lua arrays start at index 1. A “local” variable declared in a file is available in the entire file and not available on other files.

 local args = table.pack(...)

lua-load-per-thread <file> [ <arg1> [ <arg2> [ ... ] ] ]

lua-load-per-thread <file> [ <arg1> [ <arg2> [ ... ] ] ]

This global directive loads and executes a Lua file into each started thread. Any global variable has a thread-local visibility so that each thread could see a different value. As such it is strongly recommended not to use global variables in programs loaded this way. An independent copy is loaded and initialized for each thread, everything is done sequentially and in the thread’s numeric order from 1 to nbthread. If some operations need to be performed only once, the program should check the “core.thread” variable to figure what thread is being initialized. Programs loaded this way will run concurrently on all threads and will be highly scalable. This is the recommended way to load simple functions that register sample-fetches, converters, actions or services once it is certain the program doesn’t depend on global variables. For the sake of simplicity, the directive is available even if only one thread is used and even if threads are disabled (in which case it will be equivalent to lua-load). This directive can be used multiple times.

See lua-load for usage of args.

lua-prepend-path <string> [<type>]

lua-prepend-path <string> [<type>]

Prepends the given string followed by a semicolon to Lua’s package.<type> variable. <type> must either be “path” or “cpath”. If <type> is not given it defaults to “path”.

Lua’s paths are semicolon delimited lists of patterns that specify how the require function attempts to find the source file of a library. Question marks (?) within a pattern will be replaced by module name. The path is evaluated left to right. This implies that paths that are prepended later will be checked earlier.

As an example by specifying the following path:

lua-prepend-path /usr/share/haproxy-lua/?/init.lua
lua-prepend-path /usr/share/haproxy-lua/?.lua

When require "example" is being called Lua will first attempt to load the /usr/share/haproxy-lua/example.lua script, if that does not exist the /usr/share/haproxy-lua/example/init.lua will be attempted and the default paths if that does not exist either.

See https://www.lua.org/pil/8.1.html for the details within the Lua documentation.

master-worker (deprecated)

master-worker (deprecated)

Master-worker mode. It is equivalent to the command line “-W” argument.

This keyword is deprecated, please start in master-worker mode using “-W” or “-Ws”.

This mode will launch a “master” which will fork a “worker” after reading the configuration to process the traffic. The master is used as a process manager which will monitor the “workers”.

Using this mode, you can reload HAProxy directly by sending a SIGUSR2 signal to the master. Reloading will ask the master to read the configuration again and fork a new worker. The previous worker will be kept until the end of its jobs.

The master-worker mode is compatible either with the foreground or daemon mode.

By default, if a worker exits with a bad return code, in the case of a segfault for example, all workers will be killed, and the master will leave. It is convenient to combine this behavior with Restart=on-failure in a systemd unit file in order to relaunch the whole process. If you don’t want this behavior, you must use the keyword “no-exit-on-failure”.

See also “-W” in the management guide.

master-worker no-exit-on-failure

master-worker no-exit-on-failure

In master-worker mode, by default, if a worker exits with a bad return code, in the case of a segfault for example, all workers will be killed, and the master will leave. It is convenient to combine this behavior with Restart=on-failure in a systemd unit file in order to relaunch the whole process.

This keyword allows to keep the remaining processes alive when a worker crashed instead of killing everything. This need to be used with caution as it is only meant for debugging and could put the master process in an abnormal state.

max-threads-per-group <number>

max-threads-per-group <number>

Defines the maximum number of threads in a thread group. Unless the number of thread groups is fixed with the “thread-groups” directive, haproxy will create as many thread groups as needed to satisfy the requested number of threads. The minimum value is 1, and the maximum value is 64 (on 64-bit systems, or 32 on 32-bit systems). Lower values reduce contention caused by atomic operations on shared states, but can increase the number of sockets needed to create all listeners and to hold idle backend connections. Higher values will reduce these costs, at the expense of higher CPU usage under contented situations, and lower connection rates. The default value is 16, which provides the best tradeoff that was experimentally found on various tested systems, including x86_64 processors from multiple vendors, and large Arm64 systems, both on bare metal and hypervisors.

mworker-max-reloads <number>

mworker-max-reloads <number>

In master-worker mode, this option limits the number of time a worker can survive to a reload. If the worker did not leave after a reload, once its number of reloads is greater than this number, the worker will receive a SIGTERM. This option helps to keep under control the number of workers. See also “show proc” in the Management Guide.

By default this value is set to 50.

nbthread <number>

nbthread <number>

This setting is only available when support for threads was built in. It makes HAProxy run on <number> threads. “nbthread” also works when HAProxy is started in foreground. On some platforms supporting CPU affinity, the default “nbthread” value is automatically set to the number of CPUs the process is bound to upon startup. This means that the thread count can easily be adjusted from the calling process using commands like “taskset” or “cpuset”. Otherwise, this value defaults to 1. The default value is reported in the output of “haproxy -vv”. Note that values set here or automatically detected are subject to the limit set by “thread-hard-limit” (if set).

numa-cpu-mapping

numa-cpu-mapping

When running on a NUMA-aware platform, this enables the “cpu-policy” directive to inspect the topology and figure the best set of CPUs to use and the corresponding number of threads. However, if the applied binding is non optimal on a particular architecture, it can be disabled with the statement ’no numa-cpu-mapping’. This automatic binding is also not applied if a ’nbthread’ statement is present in the configuration, if the affinity of the process is already specified, for example via the ‘cpu-map’ directive or the taskset utility, or if the cpu-policy is set to any other value. See also “cpu-map”, “cpu-policy”, “cpu-set”.

ocsp-update.disable [ on | off ]

ocsp-update.disable [ on | off ]

Disable completely the ocsp-update in HAProxy. Any ocsp-update configuration will be ignored. Default is “off”. See option “ocsp-update” for more information about the auto update mechanism.

ocsp-update.httpproxy <address>[:port]

ocsp-update.httpproxy <address>[:port]

Allow to use an HTTP proxy for the OCSP updates. This only works with HTTP, HTTPS is not supported. This option will allow the OCSP updater to send absolute URI in the request to the proxy.

ocsp-update.maxdelay <number>

ocsp-update.maxdelay <number>
tune.ssl.ocsp-update.maxdelay <number> (deprecated)

Sets the maximum interval between two automatic updates of the same OCSP response. This time is expressed in seconds and defaults to 3600 (1 hour). It must be set to a higher value than “ocsp-update.mindelay”. See option “ocsp-update” for more information about the auto update mechanism.

ocsp-update.mindelay <number>

ocsp-update.mindelay <number>
tune.ssl.ocsp-update.mindelay <number> (deprecated)

Sets the minimum interval between two automatic updates of the same OCSP response. This time is expressed in seconds and defaults to 300 (5 minutes). It is particularly useful for OCSP response that do not have explicit expiration times. It must be set to a lower value than “ocsp-update.maxdelay”. See option “ocsp-update” for more information about the auto update mechanism.

ocsp-update.mode [ on | off ]

ocsp-update.mode [ on | off ]

Sets the default ocsp-update mode for all certificates used in the configuration. This global option can be superseded by the crt-list “ocsp-update” option. This option is set to “off” by default. See option “ocsp-update” for more information about the auto update mechanism.

pidfile <pidfile>

pidfile <pidfile>

Writes PIDs of all daemons into file <pidfile> when daemon mode or writes PID of master process into file <pidfile> when master-worker mode. This option is equivalent to the “-p” command line argument. The file must be accessible to the user starting the process. See also “daemon” and “master-worker”.

pp2-never-send-local

pp2-never-send-local

A bug in the PROXY protocol v2 implementation was present in HAProxy up to version 2.1, causing it to emit a PROXY command instead of a LOCAL command for health checks. This is particularly minor but confuses some servers’ logs. Sadly, the bug was discovered very late and revealed that some servers which possibly only tested their PROXY protocol implementation against HAProxy fail to properly handle the LOCAL command, and permanently remain in the “down” state when HAProxy checks them. When this happens, it is possible to enable this global option to revert to the older (bogus) behavior for the time it takes to contact the affected components’ vendors and get them fixed. This option is disabled by default and acts on all servers having the “send-proxy-v2” statement.

presetenv <name> <value>

presetenv <name> <value>

Sets environment variable <name> to value <value>. If the variable exists, it is NOT overwritten. The changes immediately take effect so that the next line in the configuration file sees the new value. See also “setenv”, “resetenv”, and “unsetenv”.

prealloc-fd

prealloc-fd

Performs a one-time open of the maximum file descriptor which results in a pre-allocation of the kernel’s data structures. This prevents short pauses when nbthread>1 and HAProxy opens a file descriptor which requires the kernel to expand its data structures.

resetenv [<name> ...]

resetenv [<name> ...]

Removes all environment variables except the ones specified in argument. It allows to use a clean controlled environment before setting new values with setenv or unsetenv. Please note that some internal functions may make use of some environment variables, such as time manipulation functions, but also OpenSSL or even external checks. This must be used with extreme care and only after complete validation. The changes immediately take effect so that the next line in the configuration file sees the new environment. See also “setenv”, “presetenv”, and “unsetenv”.

server-state-base <directory>

server-state-base <directory>

Specifies the directory prefix to be prepended in front of all servers state file names which do not start with a ‘/’. See also “server-state-file”, “load-server-state-from-file” and “server-state-file-name”.

server-state-file <file>

server-state-file <file>

Specifies the path to the file containing state of servers. If the path starts with a slash (’/’), it is considered absolute, otherwise it is considered relative to the directory specified using “server-state-base” (if set) or to the current directory. Before reloading HAProxy, it is possible to save the servers’ current state using the stats command “show servers state”. The output of this command must be written in the file pointed by <file>. When starting up, before handling traffic, HAProxy will read, load and apply state for each server found in the file and available in its current running configuration. See also “server-state-base” and “show servers state”, “load-server-state-from-file” and “server-state-file-name”

set-dumpable [ on | off | libs ]

set-dumpable [ on | off | libs ]

This option helps choose the core dump behavior in case of process crash. Available options are:

• on this enables core dumping at the process level if it was previously disabled.

• off this disables a previously enabled core dumping.

• libs this enables core dumping with an embedded copy of the binaries and libraries that are required for debugging. This may be requested by developers. In this case haproxy will try to load the libraries it depends on into memory and keep them preciously. If the process crashes, they will be dumped into the core so there is no need for retrieving them from the file system anymore and no risk that they do not match the core. This takes a few megabytes to a few tens of megabytes of additional RAM, so it is better not to use it on small systems.

This option is better left disabled by default and enabled only upon a developer’s request. By default it is disabled. Without argument, it defaults to “on”. If it has been enabled, it may still be forcibly disabled by prefixing it with the “no” keyword or by setting it to “off”. It has no impact on performance nor stability but will try hard to re-enable core dumps that were possibly disabled by file size limitations (ulimit -f), core size limitations (ulimit -c), or “dumpability” of a process after changing its UID/GID (such as /proc/sys/fs/suid_dumpable on Linux). Core dumps might still be limited by the current directory’s permissions (check what directory the file is started from), the chroot directory’s permission (it may be needed to temporarily disable the chroot directive or to move it to a dedicated writable location), or any other system-specific constraint. For example, some Linux flavours are notorious for replacing the default core file with a path to an executable not even installed on the system (check /proc/sys/kernel/core_pattern). Often, simply writing “core”, “core.%p” or “/var/log/core/core.%p” addresses the issue. When trying to enable this option waiting for a rare issue to re-appear, it’s often a good idea to first try to obtain such a dump by issuing, for example, “kill -11” to the “haproxy” process and verify that it leaves a core where expected when dying.

set-var <var-name> <expr>

set-var <var-name> <expr>

Sets the process-wide variable ‘<var-name>’ to the result of the evaluation of the sample expression <expr>. The variable ‘<var-name>’ may only be a process-wide variable (using the ‘proc.’ prefix). It works exactly like the ‘set-var’ action in TCP or HTTP rules except that the expression is evaluated at configuration parsing time and that the variable is instantly set. The sample fetch functions and converters permitted in the expression are only those using internal data, typically ‘int(value)’ or ‘str(value)’. It is possible to reference previously allocated variables as well. These variables will then be readable (and modifiable) from the regular rule sets.

Example:

global
    set-var proc.current_state str(primary)
    set-var proc.prio int(100)
    set-var proc.threshold int(200),sub(proc.prio)

set-var-fmt <var-name> <fmt>

set-var-fmt <var-name> <fmt>

Sets the process-wide variable ‘<var-name>’ to the string resulting from the evaluation of the log-format <fmt>. The variable ‘<var-name>’ may only be a process-wide variable (using the ‘proc.’ prefix). It works exactly like the ‘set-var-fmt’ action in TCP or HTTP rules except that the expression is evaluated at configuration parsing time and that the variable is instantly set. The sample fetch functions and converters permitted in the expression are only those using internal data, typically ‘int(value)’ or ‘str(value)’. It is possible to reference previously allocated variables as well. These variables will then be readable (and modifiable) from the regular rule sets. Please see section 8.2.6 for details on the Custom log format syntax.

Example:

global
    set-var-fmt proc.current_state "primary"
    set-var-fmt proc.bootid        "%pid|%t"

setcap <name>[,<name>...]

setcap <name>[,<name>...]

Sets a list of capabilities that must be preserved when starting and running either as a non-root user (uid > 0), or when starting with uid 0 (root) and switching then to a non-root. By default all permissions are lost by the uid switch, but some are often needed when trying to connect to a server from a foreign address during transparent proxying, or when binding to a port below 1024, e.g. when using “tune.quic.fe.sock-per-conn default-on”, resulting in setups running entirely under uid 0. Setting capabilities generally is a safer alternative, as only the required capabilities will be preserved. The feature is OS-specific and only enabled on Linux when USE_LINUX_CAP=1 is set at build time. The list of supported capabilities also depends on the OS and is enumerated by the error message displayed when an invalid capability name or an empty one is passed. Multiple capabilities may be passed, delimited by commas. Among those commonly used, “cap_net_raw” allows to transparently bind to a foreign address, and “cap_net_bind_service” allows to bind to a privileged port and may be used by QUIC. If the process is started and run under the same non-root user, needed capabilities should be set on haproxy binary file with setcap along with this keyword. For more details about setting capabilities on haproxy binary, please see chapter 13.1 Linux capabilities support in the Management guide.

Example:

global
    setcap cap_net_bind_service,cap_net_admin

setenv <name> <value>

setenv <name> <value>

Sets environment variable <name> to value <value>. If the variable exists, it is overwritten. The changes immediately take effect so that the next line in the configuration file sees the new value. See also “presetenv”, “resetenv”, and “unsetenv”.

shm-stats-file <name>

shm-stats-file <name>

When this directive is set, it enables the use of shared memory for storing stats counters. <name> is used as argument to shm_open() to open the shared memory at a unique location. It also means that the directive is only available on systems which support shm_open(). When SHM is used for stats, all shareable counters for frontends, backends, listeners and servers will be stored in the SHM, provided that they have a GUID set. When reloading haproxy, new process will try to scan the SHM for objects that could be associated to objects defined in the configuration based on GUID and type, the goal is to be able to preserve some counters’ values upon reload. On the other hand, when haproxy is properly stopped, the SHM objects are released, which means counters are effectively reset. It is also possible to manually remove the file before starting a fresh process to force a reset.

See also “guid”, “guid-prefix” and “shm-stats-file-max-objects”

shm-stats-file-max-objects <number>

shm-stats-file-max-objects <number>

This setting defines the maximum number of objects the shared memory used for shared counters will be able to store per thread group. It is directly related to the maximum memory size of the shm and is used to “premap” the shm to a given size in order to avoid runtime re-mapping. It defaults to 2k, which should suit for most setups without risking unsuitable memory usage, but can be easily changed if needed. haproxy will complain during startup if this value is to low to register objects that are expected to be stored in the shared memory. It is only relevant when “shm-stats-file” was defined.

See also “thread-groups”

ssl-default-bind-ciphers <ciphers>

ssl-default-bind-ciphers <ciphers>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of cipher algorithms (“cipher suite”) that are negotiated during the SSL/TLS handshake up to TLSv1.2 for all “bind” lines which do not explicitly define theirs. The format of the string is defined in “man 1 ciphers” from OpenSSL man pages. For background information and recommendations see e.g. (https://wiki.mozilla.org/Security/Server_Side_TLS ) and (https://mozilla.github.io/server-side-tls/ssl-config-generator/ ). For TLSv1.3 cipher configuration, please check the “ssl-default-bind-ciphersuites” keyword. Please check the “bind” keyword for more information.

ssl-default-bind-ciphersuites <ciphersuites>

ssl-default-bind-ciphersuites <ciphersuites>

This setting is only available when support for OpenSSL was built in and OpenSSL 1.1.1 or later was used to build HAProxy. It sets the default string describing the list of cipher algorithms (“cipher suite”) that are negotiated during the TLSv1.3 handshake for all “bind” lines which do not explicitly define theirs. The format of the string is defined in “man 1 ciphers” from OpenSSL man pages under the section “ciphersuites”. For cipher configuration for TLSv1.2 and earlier, please check the “ssl-default-bind-ciphers” keyword. This setting might accept TLSv1.2 ciphersuites however this is an undocumented behavior and not recommended as it could be inconsistent or buggy. The default TLSv1.3 ciphersuites of OpenSSL are: “TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256”

TLSv1.3 only supports 5 ciphersuites:

• TLS_AES_128_GCM_SHA256
• TLS_AES_256_GCM_SHA384
• TLS_CHACHA20_POLY1305_SHA256
• TLS_AES_128_CCM_SHA256
• TLS_AES_128_CCM_8_SHA256

Please check the “bind” keyword for more information.

Example:

global
    ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

ssl-default-bind-client-sigalgs <sigalgs>

ssl-default-bind-client-sigalgs <sigalgs>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of signature algorithms related to client authentication for all “bind” lines which do not explicitly define theirs. The format of the string is a colon-delimited list of signature algorithms. Each signature algorithm can use one of two forms: TLS1.3 signature scheme names (“rsa_pss_rsae_sha256”) or the public key algorithm + digest form (“ECDSA+SHA256”). A list can contain both forms. For more information on the format, see SSL_CTX_set1_client_sigalgs(3). A list of signature algorithms is also available in RFC8446 section 4.2.3 and in OpenSSL in the ssl/t1_lib.c file. This setting is not applicable to TLSv1.1 and earlier versions of the protocol as the signature algorithms aren’t separately negotiated in these versions. It is not recommended to change this setting unless compatibility with a middlebox is required.

ssl-default-bind-curves <curves>

ssl-default-bind-curves <curves>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of elliptic curves algorithms (“curve suite”) that are negotiated during the SSL/TLS handshake with ECDHE. The format of the string is a colon-delimited list of curve name. Please check the “bind” keyword for more information.

ssl-default-bind-options [<option>]...

ssl-default-bind-options [<option>]...

This setting is only available when support for OpenSSL was built in. It sets default ssl-options to force on all “bind” lines. Please check the “bind” keyword to see available options.

Example:

global
   ssl-default-bind-options ssl-min-ver TLSv1.0 no-tls-tickets

ssl-default-bind-sigalgs <sigalgs>

ssl-default-bind-sigalgs <sigalgs>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of signature algorithms that are negotiated during the TLSv1.2 and TLSv1.3 handshake for all “bind” lines which do not explicitly define theirs. The format of the string is a colon-delimited list of signature algorithms. Each signature algorithm can use one of two forms: TLS1.3 signature scheme names (“rsa_pss_rsae_sha256”) or the public key algorithm + digest form (“ECDSA+SHA256”). A list can contain both forms. For more information on the format, see SSL_CTX_set1_sigalgs(3). A list of signature algorithms is also available in RFC8446 section 4.2.3 and in OpenSSL in the ssl/t1_lib.c file. This setting is not applicable to TLSv1.1 and earlier versions of the protocol as the signature algorithms aren’t separately negotiated in these versions. It is not recommended to change this setting unless compatibility with a middlebox is required.

ssl-default-server-ciphers <ciphers>

ssl-default-server-ciphers <ciphers>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of cipher algorithms that are negotiated during the SSL/TLS handshake up to TLSv1.2 with the server, for all “server” lines which do not explicitly define theirs. The format of the string is defined in “man 1 ciphers” from OpenSSL man pages. For background information and recommendations see e.g. (https://wiki.mozilla.org/Security/Server_Side_TLS ) and (https://mozilla.github.io/server-side-tls/ssl-config-generator/ ). For TLSv1.3 cipher configuration, please check the “ssl-default-server-ciphersuites” keyword. Please check the “server” keyword for more information.

ssl-default-server-ciphersuites <ciphersuites>

ssl-default-server-ciphersuites <ciphersuites>

This setting is only available when support for OpenSSL was built in and OpenSSL 1.1.1 or later was used to build HAProxy. It sets the default string describing the list of cipher algorithms that are negotiated during the TLSv1.3 handshake with the server, for all “server” lines which do not explicitly define theirs. The format of the string is defined in “man 1 ciphers” from OpenSSL man pages under the section “ciphersuites”. For cipher configuration for TLSv1.2 and earlier, please check the “ssl-default-server-ciphers” keyword. Please check the “server” keyword for more information.

ssl-default-server-client-sigalgs <sigalgs>

ssl-default-server-client-sigalgs <sigalgs>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of signature algorithms related to client authentication for all “server” lines which do not explicitly define theirs. The format of the string is a colon-delimited list of signature algorithms. Each signature algorithm can use one of two forms: TLS1.3 signature scheme names (“rsa_pss_rsae_sha256”) or the public key algorithm + digest form (“ECDSA+SHA256”). A list can contain both forms. For more information on the format, see SSL_CTX_set1_client_sigalgs(3). A list of signature algorithms is also available in RFC8446 section 4.2.3 and in OpenSSL in the ssl/t1_lib.c file. This setting is not applicable to TLSv1.1 and earlier versions of the protocol as the signature algorithms aren’t separately negotiated in these versions. It is not recommended to change this setting unless compatibility with a middlebox is required.

ssl-default-server-curves <curves>

ssl-default-server-curves <curves>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of elliptic curves algorithms (“curve suite”) that are negotiated during the SSL/TLS handshake with ECDHE. The format of the string is a colon-delimited list of curve name. Please check the “server” keyword for more information.

ssl-default-server-options [<option>]...

ssl-default-server-options [<option>]...

This setting is only available when support for OpenSSL was built in. It sets default ssl-options to force on all “server” lines. Please check the “server” keyword to see available options.

ssl-default-server-sigalgs <sigalgs>

ssl-default-server-sigalgs <sigalgs>

This setting is only available when support for OpenSSL was built in. It sets the default string describing the list of signature algorithms that are negotiated during the TLSv1.2 and TLSv1.3 handshake for all “server” lines which do not explicitly define theirs. The format of the string is a colon-delimited list of signature algorithms. Each signature algorithm can use one of two forms: TLS1.3 signature scheme names (“rsa_pss_rsae_sha256”) or the public key algorithm + digest form (“ECDSA+SHA256”). A list can contain both forms. For more information on the format, see SSL_CTX_set1_sigalgs(3). A list of signature algorithms is also available in RFC8446 section 4.2.3 and in OpenSSL in the ssl/t1_lib.c file. This setting is not applicable to TLSv1.1 and earlier versions of the protocol as the signature algorithms aren’t separately negotiated in these versions. It is not recommended to change this setting unless compatibility with a middlebox is required.

ssl-dh-param-file <file>

ssl-dh-param-file <file>

This setting is only available when support for OpenSSL was built in. It sets the default DH parameters that are used during the SSL/TLS handshake when ephemeral Diffie-Hellman (DHE) key exchange is used, for all “bind” lines which do not explicitly define theirs. It will be overridden by custom DH parameters found in a bind certificate file if any. If custom DH parameters are not specified either by using ssl-dh-param-file or by setting them directly in the certificate file, DHE ciphers will not be used, unless tune.ssl.default-dh-param is set. In this latter case, pre-defined DH parameters of the specified size will be used. Custom parameters are known to be more secure and therefore their use is recommended. Custom DH parameters may be generated by using the OpenSSL command “openssl dhparam <size>”, where size should be at least 2048, as 1024-bit DH parameters should not be considered secure anymore.

ssl-passphrase-cmd <cmd> <args> ...

ssl-passphrase-cmd <cmd> <args> ...

This settings is only available when support for OpenSSL was built in. It allows to define a full command line that will be called when an encrypted certificate is loaded during init. The command could be a script or any other program. It will be provided with the encrypted private key path as first parameter and the user-defined “args” parameters then and should dump the passphrase that allows to decode the encrypted private key on the standard output. For every new encrypted private key loaded during init, HAProxy will first try every other already known passphrase to decode the private key and will ultimately call the passphrase command again if none works.

ssl-propquery <query>

ssl-propquery <query>

This setting is only available when support for OpenSSL was built in and when OpenSSL’s version is at least 3.0. It allows to define a default property string used when fetching algorithms in providers. It behave the same way as the openssl propquery option and it follows the same syntax (described in https://www.openssl.org/docs/man3.0/man7/property.html ). For instance, if you have two providers loaded, the foo one and the default one, the propquery “?provider=foo” allows to pick the algorithm implementations provided by the foo provider by default, and to fallback on the default provider’s one if it was not found.

ssl-provider <name>

ssl-provider <name>

This setting is only available when support for OpenSSL was built in and when OpenSSL’s version is at least 3.0. It allows to load a provider during init. If loading is successful, any capabilities provided by the loaded provider might be used by HAProxy. Multiple ‘ssl-provider’ options can be specified in a configuration file. The providers will be loaded in their order of appearance.

Please note that loading a provider explicitly prevents OpenSSL from loading the ‘default’ provider automatically. OpenSSL also allows to define the providers that should be loaded directly in its configuration file (openssl.cnf for instance) so it is not necessary to use this ‘ssl-provider’ option to load providers. The “show ssl providers” CLI command can be used to show all the providers that were successfully loaded.

The default search path of OpenSSL provider can be found in the output of the “openssl version -a” command. If the provider is in another directory, you can set the OPENSSL_MODULES environment variable, which takes the directory where your provider can be found.

See also “ssl-propquery” and “ssl-provider-path”.

ssl-provider-path <path>

ssl-provider-path <path>

This setting is only available when support for OpenSSL was built in and when OpenSSL’s version is at least 3.0. It allows to specify the search path that is to be used by OpenSSL for looking for providers. It behaves the same way as the OPENSSL_MODULES environment variable. It will be used for any following ‘ssl-provider’ option or until a new ‘ssl-provider-path’ is defined. See also “ssl-provider”.

ssl-load-extra-del-ext

ssl-load-extra-del-ext

This setting allows to configure the way HAProxy does the lookup for the extra SSL files. By default HAProxy adds a new extension to the filename. (ex: with “foobar.crt” load “foobar.crt.key”). With this option enabled, HAProxy removes the extension before adding the new one (ex: with “foobar.crt” load “foobar.key”).

Your crt file must have a “.crt” extension for this option to work.

This option is not compatible with bundle extensions (.ecdsa, .rsa. .dsa) and won’t try to remove them.

This option is disabled by default. See also “ssl-load-extra-files”.

ssl-load-extra-files <none|all|bundle|sctl|ocsp|issuer|key>*

ssl-load-extra-files <none|all|bundle|sctl|ocsp|issuer|key>*

This setting alters the way HAProxy will look for unspecified files during the loading of the SSL certificates. This option applies to certificates associated to “bind” lines as well as “server” lines but some of the extra files will not have any functional impact for “server” line certificates.

By default, HAProxy discovers automatically a lot of files not specified in the configuration, and you may want to disable this behavior if you want to optimize the startup time.

“none”: Only load the files specified in the configuration. Don’t try to load a certificate bundle if the file does not exist. In the case of a directory, it won’t try to bundle the certificates if they have the same basename.

“all”: This is the default behavior, it will try to load everything, bundles, sctl, ocsp, issuer, key.

“bundle”: When a file specified in the configuration does not exist, HAProxy will try to load a “cert bundle”. Certificate bundles are only managed on the frontend side and will not work for backend certificates.

Starting from HAProxy 2.3, the bundles are not loaded in the same OpenSSL certificate store, instead it will loads each certificate in a separate store which is equivalent to declaring multiple “crt”. OpenSSL 1.1.1 is required to achieve this. Which means that bundles are now used only for backward compatibility and are not mandatory anymore to do an hybrid RSA/ECC bind configuration.

To associate these PEM files into a “cert bundle” that is recognized by HAProxy, they must be named in the following way: All PEM files that are to be bundled must have the same base name, with a suffix indicating the key type. Currently, three suffixes are supported: rsa, dsa and ecdsa. For example, if www.example.com has two PEM files, an RSA file and an ECDSA file, they must be named: “example.pem.rsa” and “example.pem.ecdsa”. The first part of the filename is arbitrary; only the suffix matters. To load this bundle into HAProxy, specify the base name only:

Example: bind:8443 ssl crt example.pem

Note that the suffix is not given to HAProxy; this tells HAProxy to look for a cert bundle.

HAProxy will load all PEM files in the bundle as if they were configured separately in several “crt”.

The bundle loading does not have an impact anymore on the directory loading since files are loading separately.

On the CLI, bundles are seen as separate files, and the bundle extension is required to commit them.

OCSP files (.ocsp), issuer files (.issuer), Certificate Transparency (.sctl) as well as private keys (.key) are supported with multi-cert bundling.

“sctl”: Try to load “<basename>.sctl” for each crt keyword. If provided for a backend certificate, it will be loaded but will not have any functional impact.

“ocsp”: Try to load “<basename>.ocsp” for each crt keyword. If provided for a backend certificate, it will be loaded but will not have any functional impact.

“issuer”: Try to load “<basename>.issuer” if the issuer of the OCSP file is not provided in the PEM file. If provided for a backend certificate, it will be loaded but will not have any functional impact.

“key”: If the private key was not provided by the PEM file, try to load a file “<basename>.key” containing a private key.

The default behavior is “all”.

Example:

ssl-load-extra-files bundle sctl
ssl-load-extra-files sctl ocsp issuer
ssl-load-extra-files none

See also: “crt”, section 5.1 about bind options and section 5.2 about server options.

ssl-security-level <number>

ssl-security-level <number>

This directive allows to chose the OpenSSL security level as described in https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_security_level.html The security level will be applied to every SSL contextes in HAProxy. Only a value between 0 and 5 is supported.

The default value depends on your OpenSSL version, distribution and how was compiled the library.

This directive requires at least OpenSSL 1.1.1.

ssl-server-verify [none|required]

ssl-server-verify [none|required]

The default behavior for SSL verify on servers side. If specified to ’none’, servers certificates are not verified. The default is ‘required’ except if forced using cmdline option ‘-dV’.

ssl-skip-self-issued-ca

ssl-skip-self-issued-ca

Self issued CA, aka x509 root CA, is the anchor for chain validation: as a server is useless to send it, client must have it. Standard configuration need to not include such CA in PEM file. This option allows you to keep such CA in PEM file without sending it to the client. Use case is to provide issuer for ocsp without the need for ‘.issuer’ file and be able to share it with ‘issuers-chain-path’. This concerns all certificates without intermediate certificates. It’s useless for BoringSSL, .issuer is ignored because ocsp bits does not need it. Requires at least OpenSSL 1.0.2.

stats calculate-max-counters [on|off]

stats calculate-max-counters [on|off]

Activates or deactivates the calculation of stats max counters. If you don’t need them, deactivating them may increase performances a bit. The default is on.

stats maxconn <connections>

stats maxconn <connections>

By default, the stats socket is limited to 10 concurrent connections. It is possible to change this value with “stats maxconn”.

stats socket [<address:port>|<path>] [param*]

stats socket [<address:port>|<path>] [param*]

Binds a UNIX socket to <path> or a TCPv4/v6 address to <address:port>. Connections to this socket will return various statistics outputs and even allow some commands to be issued to change some runtime settings. Please consult section 9.3 “Unix Socket commands” of Management Guide for more details.

All parameters supported by “bind” lines are supported, for instance to restrict access to some users or their access rights. Please consult section 5.1 for more information.

stats timeout <timeout, in milliseconds>

stats timeout <timeout, in milliseconds>

The default timeout on the stats socket is set to 10 seconds. It is possible to change this value with “stats timeout”. The value must be passed in milliseconds, or be suffixed by a time unit among { us, ms, s, m, h, d }.

stats-file <path>

stats-file <path>

Path to a generated haproxy stats-file. On startup haproxy will preload the values to its internal counters. Use the CLI command “dump stats-file” to produce such stats-file. See the management manual for more details.

stress-level <level>

stress-level <level>

Activate alternative code to stress haproxy binary. Level is an integer from 0 to 9. The default value 0 disable any stressing execution. Levels from 1 to 9 will increase the stress pressure on the haproxy binary. Note that using any positive level can significantly hurt performance. As such it should never be activated unless for debugging purpose and on a developer request.

strict-limits

strict-limits

Makes process fail at startup when a setrlimit fails. HAProxy tries to set the best setrlimit according to what has been calculated. If it fails, it will emit a warning. This option is here to guarantee an explicit failure of HAProxy when those limits fail. It is enabled by default. It may still be forcibly disabled by prefixing it with the “no” keyword.

thread-group <group> [<thread-range>...]

thread-group <group> [<thread-range>...]

This setting is only available when support for threads was built in. It enumerates the list of threads that will compose thread group <group>. Thread numbers and group numbers start at 1. Thread ranges are defined either using a single thread number at once, or by specifying the lower and upper bounds delimited by a dash ‘-’ (e.g. “1-16”). Unassigned threads will be automatically assigned to unassigned thread groups, and thread groups defined with this directive will never receive more threads than those defined. Defining the same group multiple times overrides previous definitions with the new one. See also “nbthread” and “thread-groups”.

thread-groups <number>

thread-groups <number>

This setting is only available when support for threads was built in. It makes HAProxy split its threads into <number> independent groups. At the moment, the default value is 1. Thread groups make it possible to reduce sharing between threads to limit contention, at the expense of some extra configuration efforts. It is also the only way to use more than 64 threads since up to 64 threads per group may be configured. The maximum number of groups is configured at compile time and defaults to 16. See also “nbthread”.

thread-hard-limit <number>

thread-hard-limit <number>

This setting is used to enforce a limit to the number of threads, either detected, or configured. This is particularly useful on operating systems where the number of threads is automatically detected, where a number of threads lower than the number of CPUs is desired in generic and portable configurations. Indeed, while “nbthread” enforces a number of threads that will result in a warning and bad performance if higher than CPUs available, thread-hard-limit will only cap the maximum value and automatically limit the number of threads to no higher than this value, but will not raise lower values. If “nbthread” is forced to a higher value, thread-hard-limit wins, and a warning is emitted in so that the configuration anomaly can be fixed. By default there is no limit. See also “nbthread”.

uid <number>

uid <number>

Changes the process’s user ID to <number>. It is recommended that the user ID is dedicated to HAProxy or to a small set of similar daemons. HAProxy must be started with superuser privileges in order to be able to switch to another one. See also “gid” and “user”.

ulimit-n <number>

ulimit-n <number>

Sets the maximum number of per-process file-descriptors to <number>. By default, it is automatically computed, so it is recommended not to use this option. If the intent is only to limit the number of file descriptors, better use “fd-hard-limit” instead.

Note that the dynamic servers are not taken into account in this automatic resource calculation. If using a large number of them, it may be needed to manually specify this value.

See also: fd-hard-limit, maxconn

unix-bind [ prefix <prefix> ] [ mode <mode> ] [ user <user> ] [ uid <uid> ] [ group <group> ] [ gid <gid> ]

Fixes common settings to UNIX listening sockets declared in “bind” statements. This is mainly used to simplify declaration of those UNIX sockets and reduce the risk of errors, since those settings are most commonly required but are also process-specific. The <prefix> setting can be used to force all socket path to be relative to that directory. This might be needed to access another component’s chroot. Note that those paths are resolved before HAProxy chroots itself, so they are absolute. The <mode>, <user>, <uid>, <group> and <gid> all have the same meaning as their homonyms used by the “bind” statement. If both are specified, the “bind” statement has priority, meaning that the “unix-bind” settings may be seen as process-wide default settings.

unsetenv [<name> ...]

unsetenv [<name> ...]

Removes environment variables specified in arguments. This can be useful to hide some sensitive information that are occasionally inherited from the user’s environment during some operations. Variables which did not exist are silently ignored so that after the operation, it is certain that none of these variables remain. The changes immediately take effect so that the next line in the configuration file will not see these variables. See also “setenv”, “presetenv”, and “resetenv”.

user <user name>

user <user name>

Similar to “uid” but uses the UID of user name <user name> from /etc/passwd. See also “uid” and “group”.

node <name>

node <name>

Only letters, digits, hyphen and underscore are allowed, like in DNS names.

This statement is useful in HA configurations where two or more processes or servers share the same IP address. By setting a different node-name on all nodes, it becomes easy to immediately spot what server is handling the traffic.

wurfl-cache-size <size>

wurfl-cache-size <size>

Sets the WURFL Useragent cache size. For faster lookups, already processed user agents are kept in a LRU cache:

• “0” : no cache is used.
• <size> : size of lru cache in elements.

Please note that this option is only available when HAProxy has been compiled with USE_WURFL=1.

wurfl-data-file <file path>

wurfl-data-file <file path>

The path of the WURFL data file to provide device detection services. The file should be accessible by HAProxy with relevant permissions.

Please note that this option is only available when HAProxy has been compiled with USE_WURFL=1.

wurfl-information-list [<capability>]*

wurfl-information-list [<capability>]*

A space-delimited list of WURFL capabilities, virtual capabilities, property names we plan to use in injected headers. A full list of capability and virtual capability names is available on the Scientiamobile website:

https://www.scientiamobile.com/wurflCapability

Valid WURFL properties are:

• wurfl_id Contains the device ID of the matched device.

• wurfl_root_id Contains the device root ID of the matched device.

• wurfl_isdevroot Tells if the matched device is a root device. Possible values are “TRUE” or “FALSE”.

• wurfl_useragent The original useragent coming with this particular web request.

• wurfl_api_version Contains a string representing the currently used Libwurfl API version.

• wurfl_info A string containing information on the parsed wurfl.xml and its full path.

• wurfl_last_load_time Contains the UNIX timestamp of the last time WURFL has been loaded successfully.

• wurfl_normalized_useragent The normalized useragent.

Please note that this option is only available when HAProxy has been compiled with USE_WURFL=1.

wurfl-information-list-separator <char>

wurfl-information-list-separator <char>

A char that will be used to separate values in a response header containing WURFL results. If not set that a comma (’,’) will be used by default.

Please note that this option is only available when HAProxy has been compiled with USE_WURFL=1.

wurfl-patch-file [<file path>]

wurfl-patch-file [<file path>]

A list of WURFL patch file paths. Note that patches are loaded during startup thus before the chroot.

Please note that this option is only available when HAProxy has been compiled with USE_WURFL=1.

3.2. Performance tuning

busy-polling

busy-polling

In some situations, especially when dealing with low latency on processors supporting a variable frequency or when running inside virtual machines, each time the process waits for an I/O using the poller, the processor goes back to sleep or is offered to another VM for a long time, and it causes excessively high latencies. This option provides a solution preventing the processor from sleeping by always using a null timeout on the pollers. This results in a significant latency reduction (30 to 100 microseconds observed) at the expense of a risk to overheat the processor. It may even be used with threads, in which case improperly bound threads may heavily conflict, resulting in a worse performance and high values for the CPU stolen fields in “show info” output, indicating which threads are misconfigured. It is important not to let the process run on the same processor as the network interrupts when this option is used. It is also better to avoid using it on multiple CPU threads sharing the same core. This option is disabled by default. If it has been enabled, it may still be forcibly disabled by prefixing it with the “no” keyword. It is ignored by the “select” and “poll” pollers.

This option is automatically disabled on old processes in the context of seamless reload; it avoids too much cpu conflicts when multiple processes stay around for some time waiting for the end of their current connections.

max-spread-checks <delay in milliseconds>

max-spread-checks <delay in milliseconds>

By default, HAProxy tries to spread the start of health checks across the smallest health check interval of all the servers in a farm. The principle is to avoid hammering services running on the same server. But when using large check intervals (10 seconds or more), the last servers in the farm take some time before starting to be tested, which can be a problem. This parameter is used to enforce an upper bound on delay between the first and the last check, even if the servers’ check intervals are larger. When servers run with shorter intervals, their intervals will be respected though.

maxcompcpuusage <number>

maxcompcpuusage <number>

Sets the maximum CPU usage HAProxy can reach before stopping the compression for new requests or decreasing the compression level of current requests. It works like ‘maxcomprate’ but measures CPU usage instead of incoming data bandwidth. The value is expressed in percent of the CPU used by HAProxy. A value of 100 disable the limit. The default value is 100. Setting a lower value will prevent the compression work from slowing the whole process down and from introducing high latencies.

maxcomprate <number>

maxcomprate <number>

Sets the maximum per-process input compression rate to <number> kilobytes per second. For each stream, if the maximum is reached, the compression level will be decreased during the stream. If the maximum is reached at the beginning of a stream, the stream will not compress at all. If the maximum is not reached, the compression level will be increased up to tune.comp.maxlevel. A value of zero means there is no limit, this is the default value.

maxconn <number>

maxconn <number>

Sets the maximum per-process number of concurrent connections to <number>. It is equivalent to the command-line argument “-n”. The value provided in command-line argument via “-n” takes the precedence over the maxconn value set in the global section. Haproxy process could be also compiled with SYSTEM_MAXCONN compile-time variable, which is served in this case as the system maxconn maximum. Again, the command-line “-n” argument allows at runtime to bypass SYSTEM_MAXCONN limit, if set. Proxies will stop accepting connections when maxconn is reached. The process soft file descriptor limit (could be obtained with “ulimit -n” command) is automatically adjusted according to provided maxconn. See also “ulimit-n”. Note: the “select” poller cannot reliably use more than 1024 file descriptors on some platforms. If your platform only supports select and reports “select FAILED” on startup, you need to reduce the maxconn until it works (slightly below 500 in general). If maxconn value is not set, it will be automatically calculated based on the current file descriptors limits, reported by the “ulimit -nH” command (we take the maximum between the hard and soft values), then automatic value will be possibly reduced by “fd-hard-limit” and by memory limit, if the latter was enforced via “-m” command line option. Automatic value is also dependent from the buffer size, memory allocated to compression, SSL cache size, and the use or not of SSL and the associated maxsslconn (which can also be automatic).

See also: fd-hard-limit, ulimit-n

maxconnrate <number>

maxconnrate <number>

Sets the maximum per-process number of connections per second to <number>. Proxies will stop accepting connections when this limit is reached. It can be used to limit the global capacity regardless of each frontend capacity. It is important to note that this can only be used as a service protection measure, as there will not necessarily be a fair share between frontends when the limit is reached, so it’s a good idea to also limit each frontend to some value close to its expected share. Also, lowering tune.maxaccept can improve fairness.

maxpipes <number>

maxpipes <number>

Sets the maximum per-process number of pipes to <number>. Currently, pipes are only used by kernel-based tcp splicing. Since a pipe contains two file descriptors, the “ulimit-n” value will be increased accordingly. The default value is maxconn/4, which seems to be more than enough for most heavy usages. The splice code dynamically allocates and releases pipes, and can fall back to standard copy, so setting this value too low may only impact performance.

maxsessrate <number>

maxsessrate <number>

Sets the maximum per-process number of sessions per second to <number>. Proxies will stop accepting connections when this limit is reached. It can be used to limit the global capacity regardless of each frontend capacity. It is important to note that this can only be used as a service protection measure, as there will not necessarily be a fair share between frontends when the limit is reached, so it’s a good idea to also limit each frontend to some value close to its expected share. Also, lowering tune.maxaccept can improve fairness.

maxsslconn <number>

maxsslconn <number>

Sets the maximum per-process number of concurrent SSL connections to <number>. By default there is no SSL-specific limit, which means that the global maxconn setting will apply to all connections. Setting this limit avoids having openssl use too much memory and crash when malloc returns NULL (since it unfortunately does not reliably check for such conditions). Note that the limit applies both to incoming and outgoing connections, so one connection which is deciphered then ciphered accounts for 2 SSL connections. If this value is not set, but a memory limit is enforced, this value will be automatically computed based on the memory limit, maxconn, the buffer size, memory allocated to compression, SSL cache size, and use of SSL in either frontends, backends or both. If neither maxconn nor maxsslconn are specified when there is a memory limit, HAProxy will automatically adjust these values so that 100% of the connections can be made over SSL with no risk, and will consider the sides where it is enabled (frontend, backend, both).

maxsslrate <number>

maxsslrate <number>

Sets the maximum per-process number of SSL sessions per second to <number>. SSL listeners will stop accepting connections when this limit is reached. It can be used to limit the global SSL CPU usage regardless of each frontend capacity. It is important to note that this can only be used as a service protection measure, as there will not necessarily be a fair share between frontends when the limit is reached, so it’s a good idea to also limit each frontend to some value close to its expected share. It is also important to note that the sessions are accounted before they enter the SSL stack and not after, which also protects the stack against bad handshakes. Also, lowering tune.maxaccept can improve fairness.

maxzlibmem <number>

maxzlibmem <number>

Sets the maximum amount of RAM in megabytes per process usable by the zlib. When the maximum amount is reached, future streams will not compress as long as RAM is unavailable. When sets to 0, there is no limit. The default value is 0. The value is available in bytes on the UNIX socket with “show info” on the line “MaxZlibMemUsage”, the memory used by zlib is “ZlibMemUsage” in bytes.

no-memory-trimming

no-memory-trimming

Disables memory trimming (“malloc_trim”) at a few moments where attempts are made to reclaim lots of memory (on memory shortage or on reload). Trimming memory forces the system’s allocator to scan all unused areas and to release them. This is generally seen as nice action to leave more available memory to a new process while the old one is unlikely to make significant use of it. But some systems dealing with tens to hundreds of thousands of concurrent connections may experience a lot of memory fragmentation, that may render this release operation extremely long. During this time, no more traffic passes through the process, new connections are not accepted anymore, some health checks may even fail, and the watchdog may even trigger and kill the unresponsive process, leaving a huge core dump. If this ever happens, then it is suggested to use this option to disable trimming and stop trying to be nice with the new process. Note that advanced memory allocators usually do not suffer from such a problem.

noepoll

noepoll

Disables the use of the “epoll” event polling system on Linux. It is equivalent to the command-line argument “-de”. The next polling system used will generally be “poll”. See also “nopoll”.

noevports

noevports

Disables the use of the event ports event polling system on SunOS systems derived from Solaris 10 and later. It is equivalent to the command-line argument “-dv”. The next polling system used will generally be “poll”. See also “nopoll”.

nogetaddrinfo

nogetaddrinfo

Disables the use of getaddrinfo(3) for name resolving. It is equivalent to the command line argument “-dG”. Deprecated gethostbyname(3) will be used.

nokqueue

nokqueue

Disables the use of the “kqueue” event polling system on BSD. It is equivalent to the command-line argument “-dk”. The next polling system used will generally be “poll”. See also “nopoll”.

noktls

noktls

Disables the use of ktls. It is equivalent to the command line argument “-dT”.

nopoll

nopoll

Disables the use of the “poll” event polling system. It is equivalent to the command-line argument “-dp”. The next polling system used will be “select”. It should never be needed to disable “poll” since it’s available on all platforms supported by HAProxy. See also “nokqueue”, “noepoll” and “noevports”.

noreuseport

noreuseport

Disables the use of SO_REUSEPORT - see socket(7). It is equivalent to the command line argument “-dR”.

nosplice

nosplice

Disables the use of kernel tcp splicing between sockets on Linux. It is equivalent to the command line argument “-dS”. Data will then be copied using conventional and more portable recv/send calls. Kernel tcp splicing is limited to some very recent instances of kernel 2.6. Most versions between 2.6.25 and 2.6.28 are buggy and will forward corrupted data, so they must not be used. This option makes it easier to globally disable kernel splicing in case of doubt. See also “option splice-auto”, “option splice-request” and “option splice-response”.

profiling.memory { on | off }

profiling.memory { on | off }

Enables (‘on’) or disables (‘off’) per-function memory profiling. This will keep usage statistics of malloc/calloc/realloc/free calls anywhere in the process (including libraries) which will be reported on the CLI using the “show profiling” command. This is essentially meant to be used when an abnormal memory usage is observed that cannot be explained by the pools and other info are required. The performance hit will typically be around 1%, maybe a bit more on highly threaded machines, so it is normally suitable for use in production. The same may be achieved at run time on the CLI using the “set profiling memory” command, please consult the management manual.

profiling.tasks { auto | on | off | lock | no-lock | memory | no-memory }*

profiling.tasks { auto | on | off | lock | no-lock | memory | no-memory }*

Enables (‘on’) or disables (‘off’) per-task CPU profiling. When set to ‘auto’ the profiling automatically turns on a thread when it starts to suffer from an average latency of 1000 microseconds or higher as reported in the “avg_loop_us” activity field, and automatically turns off when the latency returns below 990 microseconds (this value is an average over the last 1024 loops so it does not vary quickly and tends to significantly smooth short spikes). It may also spontaneously trigger from time to time on overloaded systems, containers, or virtual machines, or when the system swaps (which must absolutely never happen on a load balancer).

When task profiling is enabled, HAProxy can also collect the time each task spends with a lock held or waiting for a lock, as well as the time spent waiting for a memory allocation to succeed in case of a pool cache miss. This can sometimes help understand certain causes of latency. For this, the extra keywords “lock” (to enable lock time collection), “no-lock” (to disable it), “memory” (to enable memory allocation time collection) or “no-memory” (to disable it) may additionally be passed. By default they are not enabled since they can have a non-negligible CPU impact on highly loaded systems (3-10%). Note that the overhead is only taken when profiling is effectively running, so that when running in “auto” mode, it will only appear when HAProxy decides to turn it on.

CPU profiling per task can be very convenient to report where the time is spent and which requests have what effect on which other request. Enabling it will typically affect the overall’s performance by less than 1%, thus it is recommended to leave it to the default ‘auto’ value so that it only operates when a problem is identified. This feature requires a system supporting the clock_gettime(2) syscall with clock identifiers CLOCK_MONOTONIC and CLOCK_THREAD_CPUTIME_ID, otherwise the reported time will be zero. This option may be changed at run time using “set profiling” on the CLI.

spread-checks <0..50, in percent>

spread-checks <0..50, in percent>

Sometimes it is desirable to avoid sending agent and health checks to servers at exact intervals, for instance when many logical servers are located on the same physical server. With the help of this parameter, it becomes possible to add some randomness in the check interval between 0 and +/- 50%. A value between 2 and 5 seems to show good results. The default value remains at 0.

ssl-engine <name> [algo <comma-separated list of algorithms>]

ssl-engine <name> [algo <comma-separated list of algorithms>]

Sets the OpenSSL engine to <name>. List of valid values for <name> may be obtained using the command “openssl engine”. This statement may be used multiple times, it will simply enable multiple crypto engines. Referencing an unsupported engine will prevent HAProxy from starting. Note that many engines will lead to lower HTTPS performance than pure software with recent processors. The optional command “algo” sets the default algorithms an ENGINE will supply using the OPENSSL function ENGINE_set_default_string(). A value of “ALL” uses the engine for all cryptographic operations. If no list of algo is specified then the value of “ALL” is used. A comma-separated list of different algorithms may be specified, including: RSA, DSA, DH, EC, RAND, CIPHERS, DIGESTS, PKEY, PKEY_CRYPTO, PKEY_ASN1. This is the same format that openssl configuration file uses: https://www.openssl.org/docs/man1.0.2/apps/config.html

HAProxy Version 2.6 disabled the support for engines in the default build. This option is only available when HAProxy has been built with support for it. In case the ssl-engine is required HAProxy can be rebuild with the USE_ENGINE=1 flag.

ssl-mode-async

ssl-mode-async

Adds SSL_MODE_ASYNC mode to the SSL context. This enables asynchronous TLS I/O operations if asynchronous capable SSL engines are used. The current implementation supports a maximum of 32 engines. The Openssl ASYNC API doesn’t support moving read/write buffers and is not compliant with HAProxy’s buffer management. So the asynchronous mode is disabled on read/write operations (it is only enabled during initial and renegotiation handshakes).

tune.applet.zero-copy-forwarding { on | off }

tune.applet.zero-copy-forwarding { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy forwarding of data for the applets. It is enabled by default.

See also: tune.disable-zero-copy-forwarding.

tune.buffers.limit <number>

tune.buffers.limit <number>

Sets a hard limit on the number of buffers which may be allocated per process. The default value is zero which means unlimited. The limit will automatically be re-adjusted to satisfy the reserved buffers for emergency situations so that the user doesn’t have to perform complicated calculations. Forcing this value can be particularly useful to limit the amount of memory a process may take, while retaining a sane behavior. When this limit is reached, a task that requests a buffer waits for another one to be released first. Most of the time the waiting time is very short and not perceptible provided that limits remain reasonable. However, some historical limitations have weakened this mechanism over versions and it is known that in certain situations of sustained shortage, some tasks may freeze until their timeout expires, so it is safer to avoid using this when not strictly necessary.

tune.buffers.reserve <number>

tune.buffers.reserve <number>

Sets the number of per-thread buffers which are pre-allocated and reserved for use only during memory shortage conditions resulting in failed memory allocations. The minimum value is 0 and the default is 4. There is no reason a user would want to change this value, unless a core developer suggests to change it for a very specific reason.

tune.bufsize <size>

tune.bufsize <size>

Sets the buffer size to this size (in bytes). Lower values allow more streams to coexist in the same amount of RAM, and higher values allow some applications with very large cookies to work. The default value is 16384 and can be changed at build time. It is strongly recommended not to change this from the default value, as very low values will break some services such as statistics, and values larger than default size will increase memory usage, possibly causing the system to run out of memory. At least the global maxconn parameter should be decreased by the same factor as this one is increased. In addition, use of HTTP/2 mandates that this value must be 16384 or more. If an HTTP request is larger than (tune.bufsize - tune.maxrewrite), HAProxy will return HTTP 400 (Bad Request) error. Similarly if an HTTP response is larger than this size, HAProxy will return HTTP 502 (Bad Gateway). Note that the value set using this parameter will automatically be rounded up to the next multiple of 8 on 32-bit machines and 16 on 64-bit machines.

tune.bufsize.large <size>

tune.bufsize.large <size>

Sets the size in bytes for large buffers. By defaults, support for large buffers is not enabled, it must explicitly be enable by setting this value.

These buffers are designed to be used in some specific contexts where more data must be bufferized without changing the size of regular buffers. The large buffers are not implicitly used.

Note that when large buffers are configured, three special large buffers will be allocated for each threads during startup for internal usage.

tune.bufsize.small <size>

tune.bufsize.small <size>

Sets the size in bytes for small buffers. The defaults value is 1024.

These buffers are designed to be used in some specific contexts where memory consumption is restrained but it seems unnecessary to allocate a full buffer. If however a small buffer is not sufficient, a reallocation is automatically done to switch to a standard size buffer.

For the moment, it is automatically used only by HTTP/3 protocol to emit the response headers. Otherwise, small buffers support can be enabled for specific proxies via the “use-small-buffers” option.

See also: option use-small-buffers

tune.cli.max-payload-size <size>

tune.cli.max-payload-size <size>

Sets the maximum size allowed for the payload passed to a command on the CLI.

On the CLI, a command line is limited by the buffer size. It means all commands and their arguments must fit in a buffer to be processed, excluding the payload that can be passed to the last command of the command line. This payload can be allocated into a dedicated area if necessary. Its size is limited by this parameter. The default value is 128KB.

While it should be high enough for most usage, if this value is changed, it must be carefully chosen. A huge value can have impact on the HAProxy performance. Depending on the command, a huge payload can be quite long to process and can possibly trigger the watchdog.

Please consult the management manual for details about the CLI.

tune.comp.maxlevel <number>

tune.comp.maxlevel <number>

Sets the maximum compression level. The compression level affects CPU usage during compression. This value affects CPU usage during compression. Each stream using compression initializes the compression algorithm with this value. The default value is 1.

tune.defaults.purge

tune.defaults.purge

For dynamic backends support, all named defaults sections are now kept in memory after parsing. This is necessary as backend added at runtime must be based on a named defaults for its configuration.

This may consume significant memory if the number of defaults instances is important. In this case and if dynamic backend feature is unnecessary, it’s possible to use this option to force deletion of defaults section after parsing. It is still mandatory though to keep referenced defaults section which contain settings whose cannot be copied by their referencing proxies. For example, this is the case if the defaults section defines TCP/HTTP rules or a tcpcheck ruleset.

tune.disable-fast-forward

tune.disable-fast-forward

Disables the data fast-forwarding. It is a mechanism to optimize the data forwarding by passing data directly from a side to the other one without waking the stream up. Thanks to this directive, it is possible to disable this optimization. Note it also disable any kernel tcp splicing but also the zero-copy forwarding. This command is not meant for regular use, it will generally only be suggested by developers along complex debugging sessions.

tune.disable-zero-copy-forwarding

tune.disable-zero-copy-forwarding

Globally disables the zero-copy forwarding of data. It is a mechanism to optimize the data fast-forwarding by avoiding to use the channel’s buffer. Thanks to this directive, it is possible to disable this optimization. Note it also disable any kernel tcp splicing.

See also: tune.pt.zero-copy-forwarding, tune.applet.zero-copy-forwarding, tune.h1.zero-copy-fwd-recv, tune.h1.zero-copy-fwd-send, tune.h2.zero-copy-fwd-send, tune.quic.zero-copy-fwd-send

tune.epoll.mask-events <event[,...]>

tune.epoll.mask-events <event[,...]>

Along HAProxy’s history, a few complex issues were met that were caused by bugs in the epoll mechanism in the Linux kernel. These ones usually are very rare and unreproducible outside the reporter’s environment, and may only be worked around by disabling epoll and switching to poll instead, which is not very satisfying for high performance environments. Each time, issues affect only very specific (and rare) event types, and offering the ability to mask them can constitute a more acceptable work-around. This options offers this possibility by permitting to silently ignore events a few uncommon events and replace them with an input (which reports an unspecified incoming event). The effect is to avoid the fast error processing paths in certain places and only use the common paths. This should never be used unless being invited to do so by an expert in order to diagnose or work around a kernel bug.

The option takes a single argument which is a comma-delimited list of words each designating an event to be masked. The currently supported list of events is: - “err”: mask the EPOLLERR event - “hup”: mask the EPOLLHUP events - “rdhup”: mask the EPOLLRDHUP events

Example:

# mask all non-traffic epoll events:
tune.epoll.mask-events err,hup,rdhup

tune.events.max-events-at-once <number>

tune.events.max-events-at-once <number>

Sets the number of events that may be processed at once by an asynchronous task handler (from event_hdl API). <number> should be included between 1 and 10000. Large number could cause thread contention as a result of the task doing heavy work without interruption, and on the other hand, small number could result in the task being constantly rescheduled because it cannot consume enough events per run and is not able to catch up with the event producer. The default value may be forced at build time, otherwise defaults to 100.

tune.fail-alloc

tune.fail-alloc

If compiled with DEBUG_FAIL_ALLOC or started with “-dMfail”, gives the percentage of chances an allocation attempt fails. Must be between 0 (no failure) and 100 (no success). This is useful to debug and make sure memory failures are handled gracefully. When not set, the ratio is 0. However the command-line “-dMfail” option automatically sets it to 1% failure rate so that it is not necessary to change the configuration for testing.

tune.fd.edge-triggered { on | off } [ EXPERIMENTAL ]

tune.fd.edge-triggered { on | off }  [ EXPERIMENTAL ]

Enables (‘on’) or disables (‘off’) the edge-triggered polling mode for FDs that support it. This is currently only support with epoll. It may noticeably reduce the number of epoll_ctl() calls and slightly improve performance in certain scenarios. This is still experimental, it may result in frozen connections if bugs are still present, and is disabled by default.

tune.glitches.kill.cpu-usage <number>

tune.glitches.kill.cpu-usage <number>

Sets the minimum CPU usage between 0 and 100, at which connections showing too many glitches will be killed. This applies to connections that have reached their glitches-threshold limit. In environments where very long connections often behave badly without causing any performance impact, it might be desirable to keep them regardless of their misbehavior as long as they do not hurt, and to only start to kill such connections when the CPU is getting busy. This parameters allows to specify that a connection reaching its glitches threshold will be actively killed when the CPU usage is at this level or above, but never when it’s below. Note that the CPU usage is measured per thread, so a single misbehaving connection might be killed. The default is zero, meaning that a connection reaching its glitches-threshold will automatically get killed. A rule of thumb would be to set this value to twice the usually observed CPU usage, or the commonly observed CPU usage plus half the idle one (i.e. if CPU commonly reaches 60%, setting 80 here can make sense). This parameter has no effect without tune.h2.fe.glitches-threshold, tune.quic.fe.sec.glitches-threshold or tune.h1.fe.glitches-threshold. See also the global parameters “tune.h2.fe.glitches-threshold”, “tune.h1.fe.glitches-threshold” and “tune.quic.fe.sec.glitches-threshold”.

tune.h1.be.glitches-threshold <number>

tune.h1.be.glitches-threshold <number>

Sets the threshold for the number of glitches on a HTTP/1 backend connection, after which that connection will automatically be killed. This allows to automatically kill misbehaving connections without having to write explicit rules for them. The default value is zero, indicating that no threshold is set so that no event will cause a connection to be closed. Typical events include improperly formatted headers that had been nevertheless accepted by “accept-unsafe-violations-in-http-response”. Any non-zero value here should probably be in the hundreds or thousands to be effective without affecting slightly bogus servers. It is also possible to only kill connections when the CPU usage crosses a certain level, by using “tune.glitches.kill.cpu-usage”. Note that a graceful close is attempted at 75% of the configured threshold by advertising a GOAWAY for a future stream. This ensures that a slightly faulty connection will stop being used after some time without risking to interrupt ongoing transfers.

See also: tune.h1.fe.glitches-threshold, bc_glitches, and tune.glitches.kill.cpu-usage

tune.h1.fe.glitches-threshold <number>

tune.h1.fe.glitches-threshold <number>

Sets the threshold for the number of glitches on a HTTP/1 frontend connection after which that connection will automatically be killed. This allows to automatically kill misbehaving connections without having to write explicit rules for them. The default value is zero, indicating that no threshold is set so that no event will cause a connection to be closed. Typical events include improperly formatted headers that had been nevertheless accepted by “accept-unsafe-violations-in-http-request”. Any non-zero value here should probably be in the hundreds or thousands to be effective without affecting slightly bogus clients. It is also possible to only kill connections when the CPU usage crosses a certain level, by using “tune.glitches.kill.cpu-usage”. Note that a graceful close is attempted at 75% of the configured threshold by advertising a GOAWAY for a future stream. This ensures that a slightly non-compliant client will have the opportunity to create a new connection and continue to work unaffected without ever triggering the hard close thus risking to interrupt ongoing transfers.

See also: tune.h1.be.glitches-threshold, fc_glitches, and tune.glitches.kill.cpu-usage

tune.h1.zero-copy-fwd-recv { on | off }

tune.h1.zero-copy-fwd-recv { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy receives of data for the H1 multiplexer. It is enabled by default.

See also: tune.disable-zero-copy-forwarding, tune.h1.zero-copy-fwd-send

tune.h1.zero-copy-fwd-send { on | off }

tune.h1.zero-copy-fwd-send { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy sends of data for the H1 multiplexer. It is enabled by default.

See also: tune.disable-zero-copy-forwarding, tune.h1.zero-copy-fwd-recv

tune.h2.be.glitches-threshold <number>

tune.h2.be.glitches-threshold <number>

Sets the threshold for the number of glitches on a backend connection, where that connection will automatically be killed. This allows to automatically kill misbehaving connections without having to write explicit rules for them. The default value is zero, indicating that no threshold is set so that no event will cause a connection to be closed. Beware that some H2 servers may occasionally cause a few glitches over long lasting connection, so any non-zero value here should probably be in the hundreds or thousands to be effective without affecting slightly bogus servers. It is also possible to only kill connections when the CPU usage crosses a certain level, by using “tune.glitches.kill.cpu-usage”. Note that a graceful close is attempted at 75% of the configured threshold by advertising a GOAWAY for a future stream. This ensures that a slightly faulty connection will stop being used after some time without risking to interrupt ongoing transfers.

See also: tune.h2.fe.glitches-threshold, bc_glitches, and tune.glitches.kill.cpu-usage

tune.h2.be.initial-window-size <number>

tune.h2.be.initial-window-size <number>

Sets the HTTP/2 initial window size for outgoing connections, which is the number of bytes the server can respond before waiting for an acknowledgment from HAProxy. This setting only affects payload contents, not headers. When not set, the common default value set by tune.h2.initial-window-size applies. It can make sense to slightly increase this value to allow faster downloads or to reduce CPU usage on the servers, at the expense of creating unfairness between clients. It is better to use tune.h2.be.rxbuf instead, which does not cause any unfairness. It doesn’t affect resource usage.

See also: tune.h2.initial-window-size.

tune.h2.be.max-concurrent-streams <number>

tune.h2.be.max-concurrent-streams <number>

Sets the HTTP/2 maximum number of concurrent streams per outgoing connection (i.e. the number of outstanding requests on a single connection to a server). When not set, the default set by tune.h2.max-concurrent-streams applies. A smaller value than the default 100 may improve a site’s responsiveness at the expense of maintaining more established connections to the servers. When the “http-reuse” setting is set to “always”, it is recommended to reduce this value so as not to mix too many different clients over the same connection, because if a client is slower than others, a mechanism known as “head of line blocking” tends to cause cascade effect on download speed for all clients sharing a connection (keep tune.h2.be.initial-window-size low in this case). It is highly recommended not to increase this value; some might find it optimal to run at low values (1..5 typically).

tune.h2.be.max-frames-at-once <number>

tune.h2.be.max-frames-at-once <number>

Sets the maximum number of HTTP/2 incoming frames that will be processed at once on a backend connection. It can be useful to set this to a low value (a few tens to a few hundreds) when dealing with very large buffers in order to maintain a low latency and a better fairness between multiple connections. The default value is zero, which means that no limitation is enforced.

tune.h2.be.rxbuf <size>

tune.h2.be.rxbuf <size>

Sets the HTTP/2 receive buffer size for outgoing connections, in bytes. This size will be rounded up to the next multiple of tune.bufsize and will be shared between all streams uploading data (both HEADERS and DATA frames). In any case, one buffer will always be granted to each stream, and 7/8 of the unused buffers will be shared between streams downloading payload, allowing to significantly improve upload performance and avoid head-of-line blocking (HoL) on backend connections shared between multiple clients when http-reuse is set to “always”. The advertised per-stream window is automatically adjusted to reflect the available space so that in practice it should not be required to touch tune.h2.be.initial-window-size. If less than the size required to deal with all streams is set, this minimum will be used. The default value is about 1600k (100 streams with 16kB buffers each).

See also: tune.h2.be.initial-window-size, tune.h2.fe.rxbuf, http-reuse.

tune.h2.fe.glitches-threshold <number>

tune.h2.fe.glitches-threshold <number>

Sets the threshold for the number of glitches on a frontend connection, where that connection will automatically be killed. This allows to automatically kill misbehaving connections without having to write explicit rules for them. The default value is zero, indicating that no threshold is set so that no event will cause a connection to be closed. Beware that some H2 clientss may occasionally cause a few glitches over long lasting connection, so any non-zero value here should probably be in the hundreds or thousands to be effective without affecting slightly bogus clients. It is also possible to only kill connections when the CPU usage crosses a certain level, by using “tune.glitches.kill.cpu-usage”. Note that a graceful close is attempted at 75% of the configured threshold by advertising a GOAWAY for a future stream. This ensures that a slightly non-compliant client will have the opportunity to create a new connection and continue to work unaffected without ever triggering the hard close thus risking to interrupt ongoing transfers.

See also: tune.h2.be.glitches-threshold, fc_glitches, and tune.glitches.kill.cpu-usage

tune.h2.fe.initial-window-size <number>

tune.h2.fe.initial-window-size <number>

Sets the HTTP/2 initial window size for incoming connections, which is the number of bytes the client can upload before waiting for an acknowledgment from HAProxy. This setting only affects payload contents (i.e. the body of POST requests), not headers. When not set, the common default value set by tune.h2.initial-window-size applies. It can make sense to increase this value to allow faster uploads. The default value equals tune.bufsize (16384) and allows at least 1.25 Mbps of bandwidth per stream over a 100 ms ping time, and 125 Mbps for 1 ms ping time. It doesn’t affect resource usage. Using too large values may cause clients to experience a lack of responsiveness if pages are accessed in parallel to large uploads. It is better to use tune.h2.fe.rxbuf instead, which does not cause any unfairness.

See also: tune.h2.initial-window-size.

tune.h2.fe.max-concurrent-streams <number> [args...]

tune.h2.fe.max-concurrent-streams <number> [args...]

Sets the HTTP/2 maximum number of concurrent streams per incoming connection (i.e. the number of outstanding requests on a single connection from a client). When not set, the default set by tune.h2.max-concurrent-streams applies. A larger value than the default 100 may sometimes slightly improve the page load time for complex sites with lots of small objects over high latency networks but can also result in using more memory by allowing a client to allocate more resources at once. The default value of 100 is generally good and it is recommended not to change this value. A larger concurrency also has an impact on the processing load and latency when dealing with large numbers of connections which are themselves using many streams, and it may lower the barrier to denial of service attacks. The command supports the following optional arguments after the number:

• rq-load { <number> | auto | ignore }:

The optional argument "rq-load" permits to dynamically adjust the
advertised concurrency based on the executing thread's run-queue load:
as long as the thread's load remains below the indicated threshold, the
configured streams limit will be advertised. When the thread's load
increases beyond the configured limit, the advertised streams limit will be
decreased proportionally to the square of the excess ratio. Target load
levels between 50 and 100 generally show very good moderation under heavy
loads. Alternately, instead of specifying an explicit number, the keyword
accepts "ignore", which is the default and means that the thread's
run-queue load will not be considered to moderate the advertised streams
limit, and "auto", which sets the limit to the "tune.runqueue-depth"
value, which generally provides good results without having to tweak
the configuration any further.

• min <number>:

This sets the minimum advertised concurrency level when rq-load is used,
even if this results in a higher load than the configured target. This
allows to maintain a good level of interactivity on a site under very
heavy load. The minimum and default value is 1, but values between 5
and 15 can improve user experience.

Example:

tune.h2.fe.max-concurrent-streams 100 rq-load auto min 15

tune.h2.fe.max-frames-at-once <number>

tune.h2.fe.max-frames-at-once <number>

Sets the maximum number of HTTP/2 incoming frames that will be processed at once on a frontend connection. It can be useful to set this to a low value (a few tens to a few hundreds) when dealing with very large buffers in order to maintain a low latency and a better fairness between multiple connections. The default value is zero, which means that no limitation is enforced.

tune.h2.fe.max-rst-at-once <number>

tune.h2.fe.max-rst-at-once <number>

Sets the maximum number of HTTP/2 incoming RST_STREAM that will be processed at once on a frontend connection. Once the specified number of RST_STREAM frames are received, the connection handler will be placed in a low priority queue and be processed after all other tasks. It can be useful to set this to a very low value (1 or a few units) to significantly reduce the impacts of RST_STREAM floods. RST_STREAM do happen when a user clicks on the Stop button in their browser, but the few extra milliseconds caused by this requeuing are generally unnoticeable, however they are generally effective at significantly lowering the load caused from such floods. The default value is zero, which means that no limitation is enforced.

tune.h2.fe.max-total-streams <number>

tune.h2.fe.max-total-streams <number>

Sets the HTTP/2 maximum number of total streams processed per incoming connection. Once this limit is reached, HAProxy will send a graceful GOAWAY frame informing the client that it will close the connection after all pending streams have been closed. In practice, clients tend to close as fast as possible when receiving this, and to establish a new connection for next requests. Doing this is sometimes useful and desired in situations where clients stay connected for a very long time and cause some imbalance inside a farm. For example, in some highly dynamic environments, it is possible that new load balancers are instantiated on the fly to adapt to a load increase, and that once the load goes down they should be stopped without breaking established connections. By setting a limit here, the connections will have a limited lifetime and will be frequently renewed, with some possibly being established to other nodes, so that existing resources are quickly released.

It’s important to understand that there is an implicit relation between this limit and “tune.h2.fe.max-concurrent-streams” above. Indeed, HAProxy will always accept to process any possibly pending streams that might be in flight between the client and the frontend, so the advertised limit will always automatically be raised by the value configured in max-concurrent-streams, and this value will serve as a hard limit above which a violation by a non-compliant client will result in the connection being closed. Thus when counting the number of requests per connection from the logs, any number between max-total-streams and (max-total-streams + max-concurrent-streams) may be observed depending on how fast streams are created by the client.

The default value is zero, which enforces no limit beyond those implied by the protocol (2^30 ~= 1.07 billion). Values around 1000 may already cause frequent connection renewal without causing any perceptible latency to most clients. Setting it too low may result in an increase of CPU usage due to frequent TLS reconnections, in addition to increased page load time. Please note that some load testing tools do not support reconnections and may report errors with this setting; as such it may be needed to disable it when running performance benchmarks. See also “tune.h2.fe.max-concurrent-streams”.

tune.h2.fe.rxbuf <size>

tune.h2.fe.rxbuf <size>

Sets the HTTP/2 receive buffer size for incoming connections, in bytes. This size will be rounded up to the next multiple of tune.bufsize and will be shared between all streams uploading data (both HEADERS and DATA frames). In any case, one buffer will always be granted to each stream, and 7/8 of the unused buffers will be shared between streams uploading payload, allowing to significantly improve upload performance. The advertised per-stream window is automatically adjusted to reflect the available space so that in practice it should not be required to touch tune.h2.fe.initial-window-size. If less than the size required to deal with all streams is set, this minimum will be used. The default value of 1600k (100 streams with 16kB buffers each) permits roughly 130 Mbps of upload speed for a client with a 100ms RTT.

See also: tune.h2.fe.initial-window-size and tune.h2.be.rxbuf.

tune.h2.header-table-size <number>

tune.h2.header-table-size <number>

Sets the HTTP/2 dynamic header table size. It defaults to 4096 bytes and cannot be larger than 65536 bytes. A larger value may help certain clients send more compact requests, depending on their capabilities. This amount of memory is consumed for each HTTP/2 connection. It is recommended not to change it.

tune.h2.initial-window-size <number>

tune.h2.initial-window-size <number>

Sets the default value for the HTTP/2 initial window size, on both incoming and outgoing connections. This value is used for incoming connections when tune.h2.fe.initial-window-size is not set, and by outgoing connections when tune.h2.be.initial-window-size is not set. This setting is used both as the initial value and as a minimum per stream. The default value equals 16384 (tune.bufsize), which for uploads roughly allows at least 1.25 Mbps of bandwidth per stream over a network showing a 100 ms ping time, or 125 Mbps over a 1-ms local network. When less receive buffers than the maximum are in use, within the limits defined by tune.h2.be.rxbuf and tune.h2.fe.rxbuf, unused buffers will be shared between receiving streams. As such there is normally no point in changing this default setting. Given that changing this default value will both increase upload speeds and cause more unfairness between clients on downloads, it is recommended to instead use the side-specific settings tune.h2.fe.initial-window-size and tune.h2.be.initial-window-size.

tune.h2.log-errors { none | connection | stream }

tune.h2.log-errors { none | connection | stream }

Sets the level of errors in the H2 demultiplexer that will generate a log. The default is “stream”, which means that any decoding error encountered in the demultiplexer will lead to the emission of a log. The “connection” value indicates that only logs that result in invalidating the connection will produce a log. Finally, “none” indicates that no decoding error will produce any log. It is recommended to set at least “connection” in order to detect protocol anomalies, even if this means temporarily switching to “none” during difficult periods.

tune.h2.max-concurrent-streams <number>

tune.h2.max-concurrent-streams <number>

Sets the default HTTP/2 maximum number of concurrent streams per connection (i.e. the number of outstanding requests on a single connection). This value is used for incoming connections when tune.h2.fe.max-concurrent-streams is not set, and for outgoing connections when tune.h2.be.max-concurrent-streams is not set. The default value is 100. The impact varies depending on the side so please see the two settings above for more details. It is recommended not to use this setting and to switch to the per-side ones instead. A value of zero disables the limit so a single client may create as many streams as allocatable by HAProxy. It is highly recommended not to change this value.

tune.h2.max-frame-size <number>

tune.h2.max-frame-size <number>

Sets the HTTP/2 maximum frame size that HAProxy announces it is willing to receive to its peers. The default value is the largest between 16384 and the buffer size (tune.bufsize). In any case, HAProxy will not announce support for frame sizes larger than buffers. The main purpose of this setting is to allow to limit the maximum frame size setting when using large buffers. Too large frame sizes might have performance impact or cause some peers to misbehave. It is highly recommended not to change this value.

tune.h2.zero-copy-fwd-send { on | off }

tune.h2.zero-copy-fwd-send { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy sends of data for the H2 multiplexer. It is enabled by default.

See also: tune.disable-zero-copy-forwarding

tune.http.cookielen <number>

tune.http.cookielen <number>

Sets the maximum length of captured cookies. This is the maximum value that the “capture cookie xxx len yyy” will be allowed to take, and any upper value will automatically be truncated to this one. It is important not to set too high a value because all cookie captures still allocate this size whatever their configured value (they share a same pool). This value is per request per response, so the memory allocated is twice this value per connection. When not specified, the limit is set to 63 characters. It is recommended not to change this value.

tune.http.logurilen <number>

tune.http.logurilen <number>

Sets the maximum length of request URI in logs. This prevents truncating long request URIs with valuable query strings in log lines. This is not related to syslog limits. If you increase this limit, you may also increase the ’log … len yyy’ parameter. Your syslog daemon may also need specific configuration directives too. The default value is 1024.

tune.http.maxhdr <number>

tune.http.maxhdr <number>

Sets the maximum number of headers allowed in received HTTP messages. When a message comes with a number of headers greater than this value (including the first line), it is rejected with a “400 Bad Request” status code for a request, or “502 Bad Gateway” for a response. The default value is 101, which is enough for all usages, considering that the widely deployed Apache server uses the same limit. It can be useful to push this limit further to temporarily allow a buggy application to work by the time it gets fixed. The accepted range is 1..32767. Keep in mind that each new header consumes 32bits of memory for each stream, so don’t push this limit too high.

Note that HTTP/1.1 is a text protocol, so there is no special limit when the message is sent. The limit during the message parsing is sufficient. HTTP/2 and HTTP/3 are binary protocols and require an encoding step. A limit is set too when headers are encoded to comply to limitation imposed by the protocols. This limit is large enough but not documented on purpose. The same limit is applied on the first steps of the decoding for the same reason.

tune.idle-pool.shared { full | on | off }

tune.idle-pool.shared { full | on | off }

Controls sharing idle connection pools between threads for a same server. It can be enabled for all threads in a same thread group (‘on’), enabled for all threads (‘full’) or disabled (‘off’). The default is to share them between threads in the same thread group (‘on’), in order to minimize the number of persistent connections to a server, and to optimize the connection reuse rate. Sharing with threads from other thread groups can have a performance impact, and is not enabled by default, but can be useful if maximizing connection reuse is a priority. To help with debugging or when suspecting a bug in HAProxy around connection reuse, it can be convenient to forcefully disable this idle pool sharing between multiple threads, and force this option to “off”. It is strongly recommended against disabling this option without setting a conservative value on “pool-low-conn” for all servers relying on connection reuse to achieve a high performance level, otherwise connections might be closed very often as the thread count increases.

tune.idletimer <timeout>

tune.idletimer <timeout>

Sets the duration after which HAProxy will consider that an empty buffer is probably associated with an idle stream. This is used to optimally adjust some packet sizes while forwarding large and small data alternatively. The decision to use splice() or to send large buffers in SSL is modulated by this parameter. The value is in milliseconds between 0 and 65535. A value of zero means that HAProxy will not try to detect idle streams. The default is 1000, which seems to correctly detect end user pauses (e.g. read a page before clicking). There should be no reason for changing this value. Please check tune.ssl.maxrecord below.

tune.listener.default-shards { by-process | by-thread | by-group }

tune.listener.default-shards { by-process | by-thread | by-group }

Normally, all “bind” lines will create a single shard, that is, a single socket that all threads of the process will listen to. With many threads, this is not very efficient, and may even induce some important overhead in the kernel for updating the polling state or even distributing events to the various threads. Modern operating systems support balancing of incoming connections, a mechanism that will consist in permitting multiple sockets to be bound to the same address and port, and to evenly distribute all incoming connections to these sockets so that each thread only sees the connections that are waiting in the socket it is bound to. This significantly reduces kernel-side overhead and increases performance in the incoming connection path. This is usually enabled in HAProxy using the “shards” setting on “bind” lines, which defaults to 1, meaning that each listener will be unique in the process. On systems with many processors, it may be more convenient to change the default setting to “by-thread” in order to always create one listening socket per thread, or “by-group” in order to always create one listening socket per thread group. Be careful about the file descriptor usage with “by-thread” as each listener will need as many sockets as there are threads. Also some operating systems (e.g. FreeBSD) are limited to no more than 256 sockets on a same address. Note that “by-group” will remain equivalent to “by-process” for default configurations involving a single thread group, and will fall back to sharing the same socket on systems that do not support this mechanism. The default is “by-group” with a fallback to “by-process” for systems or socket families that do not support multiple bindings.

tune.listener.multi-queue { on | fair | off }

tune.listener.multi-queue { on | fair | off }

Enables (‘on’ / ‘fair’) or disables (‘off’) the listener’s multi-queue accept which spreads the incoming traffic to all threads a “bind” line is allowed to run on instead of taking them for itself. This provides a smoother traffic distribution and scales much better, especially in environments where threads may be unevenly loaded due to external activity (network interrupts colliding with one thread for example). The default mode, “on”, optimizes the choice of a thread by picking in a sample the one with the less connections. It is often the best choice when connections are long-lived as it manages to keep all threads busy. A second mode, “fair”, instead cycles through all threads regardless of their instant load level. It can be better suited for short-lived connections, or on machines with very large numbers of threads where the probability to find the least loaded thread with the first mode is low. Finally it is possible to forcefully disable the redistribution mechanism using “off” for troubleshooting, or for situations where connections are short-lived and it is estimated that the operating system already provides a good enough distribution. The default is “on”.

tune.lua.bool-sample-conversion { normal | pre-3.1-bug }

tune.lua.bool-sample-conversion { normal | pre-3.1-bug }

Explicitly tell haproxy how haproxy sample objects should be handled when pushed to Lua. Indeed, when leveraging native converters, sample fetches or variables from Lua script (to name a few), haproxy converts the internal smp type to equivalent Lua type. Because of historical implementation, there is an ambiguity around boolean handling: when doing Lua -> haproxy smp conversion, booleans are properly preserved, but when doing haproxy smp -> Lua conversion, booleans were converted to integers by mistake. This means that a sample fetch or converter returning a boolean would return an integer 0 or 1 when leveraged from Lua. Unfortunately, in Lua, booleans and integers are not interchangeable. Thus, to avoid ambiguities, “tune.lua.bool-sample-conversion” must explicitly be set to either “normal” (which means dropping the historical behavior for better consistency) or “pre-3.1-bug” (enforce historical behavior to prevent existing script logic from misbehaving). If the option is not set explicitly and a Lua script is loaded from the configuration, haproxy will emit a warning, and the option will implicitly default to “pre-3.1-bug” to match with the historical behavior. It is recommended to set this option to “normal” after ensuring that in-use Lua scripts are properly handling bool haproxy samples as booleans.

This setting must be set before any “lua-load” or “lua-load-per-thread” directive for it to be considered, else it is ignored.

tune.lua.burst-timeout <timeout>

tune.lua.burst-timeout <timeout>

The “burst” execution timeout applies to any Lua handler. If the handler fails to finish or yield before timeout is reached, it will be aborted to prevent thread contention, to prevent traffic from not being served for too long, and ultimately to prevent the process from crashing because of the watchdog kicking in. Unlike other lua timeouts which are yield-cumulative, burst-timeout will ensure that the time spent in a single lua execution window does not exceed the configured timeout.

Yielding here means that the lua execution is effectively interrupted either through an explicit call to lua-yielding function such as core.(m)sleep() or core.yield(), or following an automatic forced-yield (see tune.lua.forced-yield) and that it will be resumed later when the related task is set for rescheduling. Not all lua handlers may yield: we have to make a distinction between yieldable handlers and unyieldable handlers.

For yieldable handlers (tasks, actions..), reaching the timeout means “tune.lua.forced-yield” might be too high for the system, reducing it could improve the situation, but it could also be a good idea to check if adding manual yields at some key points within the lua function helps or not. It may also indicate that the handler is spending too much time in a specific lua library function that cannot be interrupted.

For unyieldable handlers (lua converters, sample fetches), it could simply indicate that the handler is doing too much computation, which could result from an improper design given that such handlers, which often block the request execution flow, are expected to terminate quickly to allow the request processing to go through. A common resolution approach here would be to try to better optimize the lua function for speed since decreasing “tune.lua.forced-yield” won’t help.

This timeout only counts the pure Lua runtime. If the Lua does a core.sleep, the sleeping time is not taken in account. The default timeout is 1000ms.

Note: if a lua GC cycle is initiated from the handler (either explicitly requested or automatically triggered by lua after some time), the GC cycle time will also be accounted for.

Indeed, there is no way to deduce the GC cycle time, so this could lead to some false positives on saturated systems (where GC is having hard time to catch up and consumes most of the available execution runtime). If it were to be the case, here are some resolution leads:

- checking if the script could be optimized to reduce lua memory footprint
- fine-tuning lua GC parameters and / or requesting manual GC cycles
  (see: https://www.lua.org/manual/5.4/manual.html#pdf-collectgarbage)
- increasing tune.lua.burst-timeout

Setting value to 0 completely disables this protection.

tune.lua.forced-yield <number>

tune.lua.forced-yield <number>

This directive forces the Lua engine to execute a yield each <number> of instructions executed. This permits interrupting a long script and allows the HAProxy scheduler to process other tasks like accepting connections or forwarding traffic. The default value is 10000 instructions for scripts loaded using “lua-load-per-thread” and MAX(500, 10000 / nbthread) instructions for scripts loaded using “lua-load” (it was found to be an optimal value for performance while taking care of not creating thread contention with multiple threads competing for the global lua lock).

If HAProxy often executes some Lua code but more responsiveness is required, this value can be lowered. If the Lua code is quite long and its result is absolutely required to process the data, the <number> can be increased, but the value should be set wisely as in multithreading context it could increase contention.

tune.lua.log.loggers { on | off }

tune.lua.log.loggers { on | off }

Enables (‘on’) or disables (‘off’) logging the output of LUA scripts via the loggers applicable to the current proxy, if any.

Defaults to ‘on’.

tune.lua.log.stderr { on | auto | off }

tune.lua.log.stderr { on | auto | off }

Enables (‘on’) or disables (‘off’) logging the output of LUA scripts via stderr. When set to ‘auto’, logging via stderr is conditionally ‘on’ if any of:

- tune.lua.log.loggers is set to 'off'
- the script is executed in a non-proxy context with no global logger
- the script is executed in a proxy context with no logger attached

Please note that, when enabled, this logging is in addition to the logging configured via tune.lua.log.loggers.

Defaults to ‘auto’.

tune.lua.maxmem <number>

tune.lua.maxmem <number>

Sets the maximum amount of RAM in megabytes per process usable by Lua. By default it is zero which means unlimited. It is important to set a limit to ensure that a bug in a script will not result in the system running out of memory.

tune.lua.openlibs [all | none | <lib>[,<lib>...]]

tune.lua.openlibs [all | none | <lib>[,<lib>...]]

Selects which Lua standard libraries are loaded when initialising the Lua state. The argument is a comma-separated list of library names taken from the following set: table, io, os, string, math, utf8, package, debug. The special values “all” and “none” may be used instead of a list. “none” cannot be combined with library names. The default value is “all”.

The base and coroutine libraries are always loaded regardless of this setting: base provides core Lua functions that HAProxy relies on, and coroutine is required because HAProxy overrides coroutine.create() with its own safe implementation.

Note that fork() and new thread creation are already blocked by default in HAProxy regardless of this setting, and can only be re-enabled via the “insecure-fork-wanted” global directive. Restricting the set of loaded libraries further reduces the attack surface exposed to Lua scripts. In particular: - omitting “os” prevents os.execute() and os.exit() - omitting “io” prevents io.open() and io.popen() - omitting “package” prevents loading native C modules via require() - omitting “debug” prevents introspection of HAProxy internals via debug.getupvalue(), debug.getmetatable(), or debug.sethook()

Examples:

tune.lua.openlibs none                    # only base + coroutine
tune.lua.openlibs string,math,table,utf8  # safe subset, no I/O or OS
tune.lua.openlibs all                     # default, load everything

This setting must be set before any “lua-load”, “lua-load-per-thread” or “lua-prepend-path” directive, otherwise a parse error is returned.

tune.lua.service-timeout <timeout>

tune.lua.service-timeout <timeout>

This is the execution timeout for the Lua services. This is useful for preventing infinite loops or spending too much time in Lua. This timeout counts only the pure Lua runtime. If the Lua does a sleep, the sleep is not taken in account. The default timeout is 4s.

tune.lua.session-timeout <timeout>

tune.lua.session-timeout <timeout>

This is the execution timeout for the Lua sessions. This is useful for preventing infinite loops or spending too much time in Lua. This timeout counts only the pure Lua runtime. If the Lua does a sleep, the sleep is not taken in account. The default timeout is 4s.

tune.lua.task-timeout <timeout>

tune.lua.task-timeout <timeout>

Purpose is the same as “tune.lua.session-timeout”, but this timeout is dedicated to the tasks. By default, this timeout isn’t set because a task may remain alive during of the lifetime of HAProxy. For example, a task used to check servers.

tune.max-checks-per-thread <number>

tune.max-checks-per-thread <number>

Sets the number of active checks per thread above which a thread will actively try to search a less loaded thread to run the health check, or queue it until the number of active checks running on it diminishes. The default value is zero, meaning no such limit is set. It may be needed in certain environments running an extremely large number of expensive checks with many threads when the load appears unequal and may make health checks to randomly time out on startup, typically when using OpenSSL 3.0 which is about 20 times more CPU-intensive on health checks than older ones. This will have for result to try to level the health check work across all threads. The vast majority of configurations do not need to touch this parameter. Please note that too low values may significantly slow down the health checking if checks are slow to execute.

tune.maxaccept <number>

tune.maxaccept <number>

Sets the maximum number of consecutive connections a process may accept in a row before switching to other work. In single process mode, higher numbers used to give better performance at high connection rates, though this is not the case anymore with the multi-queue. This value applies individually to each listener, so that the number of processes a listener is bound to is taken into account. This value defaults to 4 which showed best results. If a significantly higher value was inherited from an ancient config, it might be worth removing it as it will both increase performance and lower response time. In multi-process mode, it is divided by twice the number of processes the listener is bound to. Setting this value to -1 completely disables the limitation. It should normally not be needed to tweak this value.

tune.maxpollevents <number>

tune.maxpollevents <number>

Sets the maximum amount of events that can be processed at once in a call to the polling system. The default value is adapted to the operating system. It has been noticed that reducing it below 200 tends to slightly decrease latency at the expense of network bandwidth, and increasing it above 200 tends to trade latency for slightly increased bandwidth. The configured value must be lower than or equal to 1000000.

tune.maxrewrite <number>

tune.maxrewrite <number>

Sets the reserved buffer space to this size in bytes. The reserved space is used for header rewriting or appending. The first reads on sockets will never fill more than bufsize-maxrewrite. Historically it has defaulted to half of bufsize, though that does not make much sense since there are rarely large numbers of headers to add. Setting it too high prevents processing of large requests or responses. Setting it too low prevents addition of new headers to already large requests or to POST requests. It is generally wise to set it to about 1024. It is automatically readjusted to half of bufsize if it is larger than that. This means you don’t have to worry about it when changing bufsize.

tune.max-rules-at-once <number>

tune.max-rules-at-once <number>

Sets the maximum number of rules that can be evaluated at once in ruleset evaluating functions, provided that they support yielding. Indeed, it is not rare to see configurations with a large number of “tcp-request content” or “http-request” rules for instance. A large number of rules combined with cpu-demanding actions (e.g.: actions that work on content) may create thread contention as all the rules from a given ruleset are evaluated under the same polling loop if the evaluation is not interrupted. This option ensures that no more than <number> number of rules may be executed under the same polling loop for content-oriented rulesets (those that already support yielding due to content inspection). What it does is that it forces the evaluating function to yield, so that it comes back on the next polling loop to continues the evaluation.

Affected rulesets are:

• “tcp-request content”
• “tcp-response content”
• “http-request”
• “http-response”

The default value is 50.

tune.memory.hot-size <number>

tune.memory.hot-size <number>

Sets the per-thread amount of memory that will be kept hot in the local cache and will never be recoverable by other threads. Access to this memory is very fast (lockless), and having enough is critical to maintain a good performance level under extreme thread contention. The value is expressed in bytes, and the default value is configured at build time via CONFIG_HAP_POOL_CACHE_SIZE which defaults to 524288 (512 kB). A larger value may increase performance in some usage scenarios, especially when performance profiles show that memory allocation is stressed a lot. Experience shows that a good value sits between once to twice the per CPU core L2 cache size. Too large values will have a negative impact on performance by making inefficient use of the L3 caches in the CPUs, and will consume larger amounts of memory. It is recommended not to change this value, or to proceed in small increments. In order to completely disable the per-thread CPU caches, using a very small value could work, but it is better to use “-dMno-cache” on the command-line.

tune.notsent-lowat.client <size>

tune.notsent-lowat.client <size>
tune.notsent-lowat.server <size>

Adjusts the kernel’s per-socket buffering so as to report that the sending side of a socket is full once the amount of buffered data equals this value plus the measured window size. The principle is to let the strict minimum needed amount of bytes in socket buffers, plus a small margin corresponding to what would be sent by the time haproxy tries to send again. Setting this to a low value (typically around tune.bufsize) allows to significantly reduce the memory consumption in system buffers, and reduce the application level latency incurred by flushing buffered data. This generally represents a more effective and more accurate setting than tune.sndbuf.client and tune.sndbuf.client for systems supporting it. This applies per connection (connection from a client or connection to a server depending on the setting) and is only used by TCP connections. The default is zero, which means unlimited. This is only available on Linux.

tune.pattern.cache-size <number>

tune.pattern.cache-size <number>

Sets the size of the pattern lookup cache to <number> entries. This is an LRU cache which reminds previous lookups and their results. It is used by ACLs and maps on slow pattern lookups, namely the ones using the “sub”, “reg”, “dir”, “dom”, “end”, “bin” match methods as well as the case-insensitive strings. It applies to pattern expressions which means that it will be able to memorize the result of a lookup among all the patterns specified on a configuration line (including all those loaded from files). It automatically invalidates entries which are updated using HTTP actions or on the CLI. The default cache size is set to 10000 entries, which limits its footprint to about 5 MB per process/thread on 32-bit systems and 8 MB per process/thread on 64-bit systems, as caches are thread/process local. There is a very low risk of collision in this cache, which is in the order of the size of the cache divided by 2^64. Typically, at 10000 requests per second with the default cache size of 10000 entries, there’s 1% chance that a brute force attack could cause a single collision after 60 years, or 0.1% after 6 years. This is considered much lower than the risk of a memory corruption caused by aging components. If this is not acceptable, the cache can be disabled by setting this parameter to 0.

tune.peers.max-updates-at-once <number>

tune.peers.max-updates-at-once <number>

Sets the maximum number of stick-table updates that haproxy will try to process at once when sending messages. Retrieving the data for these updates requires some locking operations which can be CPU intensive on highly threaded machines if unbound, and may also increase the traffic latency during the initial batched transfer between an older and a newer process. Conversely low values may also incur higher CPU overhead, and take longer to complete. The default value is 200 and it is suggested not to change it.

tune.pipesize <size>

tune.pipesize <size>

Sets the kernel pipe buffer size to this size (in bytes). By default, pipes are the default size for the system. But sometimes when using TCP splicing, it can improve performance to increase pipe sizes, especially if it is suspected that pipes are not filled and that many calls to splice() are performed. This has an impact on the kernel’s memory footprint, so this must not be changed if impacts are not understood.

tune.pool-high-fd-ratio <number>

tune.pool-high-fd-ratio <number>

This setting sets the max number of file descriptors (in percentage) used by HAProxy globally against the maximum number of file descriptors HAProxy can use before we start killing idle connections when we can’t reuse a connection and we have to create a new one. The default is 25 (one quarter of the file descriptor will mean that roughly half of the maximum front connections can keep an idle connection behind, anything beyond this probably doesn’t make much sense in the general case when targeting connection reuse).

tune.pool-low-fd-ratio <number>

tune.pool-low-fd-ratio <number>

This setting sets the max number of file descriptors (in percentage) used by HAProxy globally against the maximum number of file descriptors HAProxy can use before we stop putting connection into the idle pool for reuse. The default is 20.

tune.pt.zero-copy-forwarding { on | off }

tune.pt.zero-copy-forwarding { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy forwarding of data for the pass-through multiplexer. To be used, the kernel splicing must also be configured. It is enabled by default.

See also: tune.disable-zero-copy-forwarding, option splice-auto, option splice-request and option splice-response

tune.quic.be.cc.cubic-min-losses <number>

tune.quic.be.cc.cubic-min-losses <number>
tune.quic.fe.cc.cubic-min-losses <number>

Defines how many lost packets are needed for the Cubic congestion control algorithm to really consider a loss event. Normally, any loss event is considered as the result of a congestion and is sufficient for Cubic to restart from a smaller window. But experiments show that there can be a variety of causes for losses that are not at all caused by congestion and that can simply be qualified of spurious losses, and for which adjusting the window will have no effect, except slowing communication down. Poor radio signal, out-of-order delivery, high CPU usage on a client causing random delays, as well as system timer imprecision can be among the common causes for this. This setting allows to make Cubic a bit more tolerant to spurious losses, by changing the minimum number of cumulated losses between two ACKs to be considered as a loss event, which defaults to 1. Some significant gains have been observed experimentally, but always accompanied with an aggravation of the bandwidth wasted on retransmits, and an increased risk of saturation of congested links. The value 2 may be used for short periods of time to compare some metrics. Never go beyond 2 without an expert’s prior analysis of the situation. The default and minimum value is 1. Always use 1.

tune.quic.cc.cubic.min-losses <number> (deprecated)

tune.quic.cc.cubic.min-losses <number> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.cc.hystart { on | off }

tune.quic.be.cc.hystart { on | off }
tune.quic.fe.cc.hystart { on | off }

Enables (‘on’) or disabled (‘off’) the HyStart++ (RFC 9406) algorithm for QUIC connections used as a replacement for the slow start phase of congestion control algorithms which may cause high packet loss. It is disabled by default.

tune.quic.cc-hystart { on | off } (deprecated)

tune.quic.cc-hystart { on | off } (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.cc.max-frame-loss <number>

tune.quic.be.cc.max-frame-loss <number>
tune.quic.fe.cc.max-frame-loss <number>

Sets the limit for which a single QUIC frame can be marked as lost. If exceeded, the connection is considered as failing and is closed immediately.

The default value is 10.

tune.quic.max-frame-loss <number> (deprecated)

tune.quic.max-frame-loss <number> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.cc.max-win-size <size>

tune.quic.be.cc.max-win-size <size>
tune.quic.fe.cc.max-win-size <size>

Sets the default maximum window size for the congestion controller of a single QUIC connection either on frontend or backend side. The value must be written as an integer with an optional suffix ‘k’, ’m’ or ‘g’. It must be between 10k and 4g.

QUIC multiplexer also uses the current congestion window size to determine if it can allocate new stream buffers on data emission. As such, the maximum congestion window size also serves as a limit on this allocator.

The default value is 480k.

See also the “quic-cc-algo” bind and server options.

tune.quic.frontend.default-max-window-size <size> (deprecated)

tune.quic.frontend.default-max-window-size <size> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.cc.reorder-ratio <0..100, in percent>

tune.quic.be.cc.reorder-ratio <0..100, in percent>
tune.quic.fe.cc.reorder-ratio <0..100, in percent>

The ratio applied to the packet reordering threshold calculated. It may trigger a high packet loss detection when too small.

The default value is 50.

tune.quic.reorder-ratio <0..100, in percent> (deprecated)

tune.quic.reorder-ratio <0..100, in percent> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.max-idle-timeout <timeout>

tune.quic.be.max-idle-timeout <timeout>
tune.quic.fe.max-idle-timeout <timeout>

Sets the QUIC max_idle_timeout transport parameters on either frontend or backend side. It follows the HAProxy time format and is expressed in milliseconds. This determines the period of time after which a connection is silently closed if it has remained inactive during an effective period of time. Both endpoints relies on the same negotiated value: - the minimum of the two parameters if both are not null, - the maximum if only one of them is not null, - if both parameters are null, this feature is disabled.

The default value is 30s.

tune.quic.frontend.max-idle-timeout <timeout> (deprecated)

tune.quic.frontend.max-idle-timeout <timeout> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.sec.glitches-threshold <number>

tune.quic.be.sec.glitches-threshold <number>
tune.quic.fe.sec.glitches-threshold <number>

Sets the threshold for the number of glitches per connection either on frontend or backend side, where that connection will automatically be killed. This allows to automatically kill misbehaving connections without having to write explicit rules for them. The default value is zero, indicating that no threshold is set so that no event will cause a connection to be closed. Beware that some QUIC clients may occasionally cause a few glitches over long lasting connection, so any non- zero value here should probably be in the hundreds or thousands to be effective without affecting slightly bogus clients. It is also possible to only kill connections when the CPU usage crosses a certain level, by using “tune.glitches.kill.cpu-usage”.

See also: fc_glitches, tune.glitches.kill.cpu-usage

tune.quic.frontend.glitches-threshold <number> (deprecated)

tune.quic.frontend.glitches-threshold <number> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.fe.sec.retry-threshold <number>

tune.quic.fe.sec.retry-threshold <number>

Dynamically enables the Retry feature for all the configured QUIC listeners as soon as this number of half open connections is reached. A half open connection is a connection whose handshake has not already successfully completed or failed. To be functional this setting needs a cluster secret to be set, if not it will be silently ignored (see “cluster-secret” setting). This setting will be also silently ignored if the use of QUIC Retry was forced (see “quic-force-retry”).

The default value is 100.

See https://www.rfc-editor.org/rfc/rfc9000.html#section-8.1.2 for more information about QUIC retry.

tune.quic.retry-threshold <number> (deprecated)

tune.quic.retry-threshold <number> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.fe.sock-per-conn { default-on | force-off }

tune.quic.fe.sock-per-conn { default-on | force-off }

Specifies globally how QUIC frontend connections will use socket for receive/send operations. Connections can share listener socket or each connection can allocate its own socket.

The default value is “default-on”. This is used to allocate a dedicated socket for every QUIC connections. This option is the preferred one to achieve the best performance with a large QUIC traffic. This is also the only way to ensure soft-stop is conducted properly without data loss for QUIC connections and cases of transient errors during sendto() operation are handled efficiently. However, this relies on some advanced features from the UDP network stack. If your platform is deemed not compatible, haproxy will automatically switch to “force-off” mode on startup. Please note that QUIC listeners running on privileged ports may require to run as uid 0, or some OS-specific tuning to permit the target uid to bind such ports, such as system capabilities. See also the “setcap” global directive.

The “force-off” value indicates that QUIC transfers will occur on the shared listener socket. This option can be a good compromise for small traffic as it allows to reduce FD consumption. However, performance won’t be optimal due to a higher CPU usage if listeners are shared across a lot of threads or a large number of QUIC connections can be used simultaneously.

This setting is applied in conjunction with each “quic-socket” bind options. If “default-on” mode is used on global tuning, it will be activated for each listener, except for the ones with “quic-socket listener”. However, if “force-off” is used globally, it will be applied on every listener instance, regardless of their individual configuration.

tune.quic.socket-owner { connection | listener } (deprecated)

tune.quic.socket-owner { connection | listener } (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. The newer option is named “tune.quic.fe.sock-per-conn”, with legacy value “connection” corresponding to “default-on” and “listener” to “force-off”.

tune.quic.be.stream.data-ratio <0..100, in percent>

tune.quic.be.stream.data-ratio <0..100, in percent>
tune.quic.fe.stream.data-ratio <0..100, in percent>

This setting allows to configure the hard limit of the number of data bytes in flight over each stream. It is expressed as a percentage relative to the QUIC stream rxbuf connection setting, with the result rounded up to bufsize.

The default value is 90. This is suitable with the most frequent web scenario, where uploads is performed only for one or a few streams, whereas the rest are used for download only. If the stream rxbuf connection limit remains at a reasonable level, it ensures that only a portion of opened streams can allocate to their maximum capacity.

In the case of an application using many uploading streams in parallel and suffering from unfairness between these streams, it can make sense to reduce this ratio, to increase fairness and reduce the per-stream bandwidth.

See also: “tune.quic.be.stream.rxbuf”, “tune.quic.fe.stream.rxbuf”, “tune.quic.be.stream.max-concurrent”, “tune.quic.fe.stream.max-concurrent”

tune.quic.frontend.stream-data-ratio <0..100, in percent> (deprecated)

tune.quic.frontend.stream-data-ratio  <0..100, in percent> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.stream.max-concurrent <number>

tune.quic.be.stream.max-concurrent <number>
tune.quic.fe.stream.max-concurrent <number>

On frontend side, this is used as the value for the advertised initial_max_streams_bidi transport parameter. This is enforced as the maximum number of bidirectional streams that the remote peer will be authorized to open concurrently during the connection lifetime. This effectively limits the number of concurrent HTTP/3 client requests.

The default value is 100. Note that if you reduces it, it can restrict the buffering capabilities of streams on receive, which would result in poor upload throughput. It can be corrected by increasing the QUIC stream rxbuf connection setting.

On backend side, this is enforced locally by haproxy to limit the number of concurrent requests multiplexed over a single connection. This may be further restricted by the peer flow control. It may be necessary to reduce the default value of 100 to improve a site’s responsiveness at the expense of a higher number of opened backend connections. Similarly to the frontend side, this setting also directly impacts the Rx buffering capability, this time though limiting the HTTP download capacity. QUIC stream rxbuf setting can be increased when dealing mostly with HTTP responses larger than “tune.bufsize”.

See also: “tune.quic.be.stream.rxbuf”, “tune.quic.fe.stream.rxbuf”, “tune.quic.be.stream.data-ratio”, “tune.quic.fe.stream.data-ratio”

tune.quic.fe.stream.max-total <number>

tune.quic.fe.stream.max-total <number>

Sets the maximum number of requests that can be handled by a single QUIC connection. Once this total is reached, the connection will be gracefully shutdown. In HTTP/3, this translates to a GOAWAY frame. The connection is finally closed when all remaining transfers are completed.

This setting is applied as a hard limit on the connection via the QUIC flow control mechanism. If a peer violates it, the connection will be immediately closed.

This setting can be used to force clients to open new connections once in a while to continue the emission of requests and avoid maintaining connections for too many times. However, low values will increase latency on the client side, as well as CPU consumption on both sides due to TLS handshakes.

The default value is 0 which implies no specific limit outside of the QUIC protocol encoding limitation (2^60, more than a billion billion).

tune.quic.frontend.max-streams-bidi <number> (deprecated)

tune.quic.frontend.max-streams-bidi <number> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.stream.rxbuf <size>

tune.quic.be.stream.rxbuf <size>
tune.quic.fe.stream.rxbuf <size>

This setting is the hard limit for the number of data bytes in flight over a QUIC frontend connection. It is reused as the value for the initial_max_data transport parameter. It directly impacts the upload bandwidth for the peer depending on the latency and the per-connection memory consumption in haproxy.

By default, the value is set to 0, which indicates that it must be automatically generated as the product between max-concurrent and bufsize. This can be increased for example if a backend application relies on massive uploads over high latency networks.

See also: “tune.quic.be.stream.max-concurrent”, “tune.quic.fe.stream.max-concurrent”, “tune.quic.be.stream.data-ratio”, “tune.quic.fe.stream.data-ratio”

tune.quic.frontend.max-data-size <size> (deprecated)

tune.quic.frontend.max-data-size <size> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.tx.pacing { on | off }

tune.quic.be.tx.pacing { on | off }
tune.quic.fe.tx.pacing { on | off }

Enables (‘on’) or disables (‘off’) pacing support for QUIC emission. By default, it is active. The purpose of pacing is to smooth emission of data to reduce network losses. In most scenario, it will significantly improve network throughput by avoiding retransmissions. However, it can be useful to deactivate it for networks with very high bandwidth/low latency characteristics to prevent unwanted delay and reduce CPU consumption.

See also the “quic-cc-algo” bind and server options.

tune.quic.disable-tx-pacing (deprecated)

tune.quic.disable-tx-pacing (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.be.tx.udp-gso { on | off }

tune.quic.be.tx.udp-gso { on | off }
tune.quic.fe.tx.udp-gso { on | off }

Enables (‘on’) or disables (‘off’) UDP GSO support for QUIC emission. By default, it is active. This kernel feature allows to emit multiple datagrams via a single system call which is more efficient for large transfer. It may be useful to disable it on developers suggestion when suspecting an issue on emission.

tune.quic.disable-udp-gso (deprecated)

tune.quic.disable-udp-gso (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.listen { on | off }

tune.quic.listen { on | off }

Disable QUIC transport protocol on the frontend side. All the QUIC listeners will still be created, but they won’t listen for incoming datagrams. Hence, no QUIC traffic will be processed by haproxy on the frontend side.

The default value is “on”. If an issue is suspected with QUIC traffic, this option can be used to easily toggle QUIC listeners without messing with each individual config lines.

See also “quic_enabled” sample fetch.

tune.quic.mem.tx-max <size>

tune.quic.mem.tx-max <size>

Sets the maximum amount of memory usable by QUIC stack at the transport layer for emission. This serves both as a limit of in flight bytes and multiplexer output buffers. Note that to prevent threads contention this limit is not strictly enforced so it can be exceeded on some occasions. Also, each connection will always be able to use a window of at least 2 datagrams, so a proper maxconn should be used in conjunction.

tune.quic.frontend.max-tx-mem <size> (deprecated)

tune.quic.frontend.max-tx-mem <size> (deprecated)

This keyword has been deprecated in 3.3 and will be removed in 3.5. It is part of the streamlining process apply on QUIC configuration. If used, this setting will only be applied on frontend connections.

tune.quic.zero-copy-fwd-send { on | off }

tune.quic.zero-copy-fwd-send { on | off }

Enables (‘on’) of disabled (‘off’) the zero-copy sends of data for the QUIC multiplexer. It is enabled by default.

See also: tune.disable-zero-copy-forwarding

tune.renice.runtime <number>

tune.renice.runtime <number>

This configuration option takes a value between -20 and 19. It applies a scheduling priority as documented in man 2 setpriority. This priority is applied after the configuration parsing, which means only the worker or the standalone process will apply it. It is usually configured to set a higher priority than a process doing configuration parsing (tune.renice.startup).

See also: tune.renice.startup

tune.renice.startup <number>

tune.renice.startup <number>

This configuration option takes a value between -20 and 19. It applies a scheduling priority as documented in man 2 setpriority. This priority is applied before applying the rest of the configuration which can be useful if you want to lower the priority for configuration parsing. This is applied on the standalone process or the worker before configuration parsing. Once the configuration is parsed, the previous priority is restored unless tune.renice.runtime is used.

See also: tune.renice.runtime

tune.rcvbuf.backend <size>

tune.rcvbuf.backend  <size>
tune.rcvbuf.frontend <size>

For the kernel socket receive buffer size on non-connected sockets to this size. This can be used QUIC in listener mode and log-forward on the frontend. The default system buffers might sometimes be too small for sockets receiving lots of aggregated traffic, causing some losses and possibly retransmits (in case of QUIC), possibly slowing down connection establishment under heavy traffic. The value is expressed in bytes, applied to each socket. In listener mode, sockets are shared between all connections, and the total number of sockets depends on the “shards” value of the “bind” line. There’s no good value, a good one corresponds to an expected size per connection multiplied by the expected number of connections. The kernel may trim large values. See also “tune.rcvbuf.client” and “tune.rcvbuf.server” for their connected socket counter parts, as well as “tune.sndbuf.backend” and “tune.sndbuf.frontend” for the send setting.

tune.rcvbuf.client <size>

tune.rcvbuf.client <size>
tune.rcvbuf.server <size>

Forces the kernel socket receive buffer size on the client or the server side to the specified value in bytes. This value applies to all TCP/HTTP frontends and backends. It should normally never be set, and the default size (0) lets the kernel auto-tune this value depending on the amount of available memory. However it can sometimes help to set it to very low values (e.g. 4096) in order to save kernel memory by preventing it from buffering too large amounts of received data. Lower values will significantly increase CPU usage though.

tune.recv_enough <size>

tune.recv_enough <size>

HAProxy uses some hints to detect that a short read indicates the end of the socket buffers. One of them is that a read returns more than <recv_enough> bytes, which defaults to 10136 (7 segments of 1448 each). This default value may be changed by this setting to better deal with workloads involving lots of short messages such as telnet or SSH sessions.

tune.ring.queues <number>

tune.ring.queues <number>

Sets the number of write queues in front of ring buffers. This can have an effect on the CPU usage of traces during debugging sessions, and both too low or too large a value can have an important effect. The good value was determined experimentally by developers and there should be no reason to try to change it unless instructed to do so in order to try to address specific issues. Such a setting should not be left in the configuration across version upgrades because its optimal value may evolve over time.

tune.runqueue-depth <number>

tune.runqueue-depth <number>

Sets the maximum amount of task that can be processed at once when running tasks. The default value depends on the number of threads but sits between 35 and 280, which tend to show the highest request rates and lowest latencies. Increasing it may incur latency when dealing with I/Os, making it too small can incur extra overhead. Higher thread counts benefit from lower values. When experimenting with much larger values, it may be useful to also enable tune.sched.low-latency and possibly tune.fd.edge-triggered to limit the maximum latency to the lowest possible.

tune.sched.low-latency { on | off }

tune.sched.low-latency { on | off }

Enables (‘on’) or disables (‘off’) the low-latency task scheduler. By default HAProxy processes tasks from several classes one class at a time as this is the most efficient. But when running with large values of tune.runqueue-depth this can have a measurable effect on request or connection latency. When this low-latency setting is enabled, tasks of lower priority classes will always be executed before other ones if they exist. This will permit to lower the maximum latency experienced by new requests or connections in the middle of massive traffic, at the expense of a higher impact on this large traffic. For regular usage it is better to leave this off. The default value is off.

tune.sndbuf.backend <size>

tune.sndbuf.backend  <size>
tune.sndbuf.frontend <size>

For the kernel socket send buffer size on non-connected sockets to this size. This can be used for UNIX socket and UDP logging on the backend side, and for QUIC in listener mode on the frontend. The default system buffers might sometimes be too small for sockets shared between many connections (or log senders), causing some losses and possibly retransmits, slowing down new connection establishment under high traffic. The value is expressed in bytes, applied to each socket. In listener mode, sockets are shared between all connections, and the total number of sockets depends on the “shards” value of the “bind” line. There’s no good value, a good one corresponds to an expected size per connection multiplied by the expected number of connections. The kernel may trim large values. See also “tune.sndbuf.client” and “tune.sndbuf.server” for their connected socket counter parts, as well as “tune.rcvbuf.backend” and “tune.rcvbuf.frontend” for the receive setting.

tune.sndbuf.client <size>

tune.sndbuf.client <size>
tune.sndbuf.server <size>

Forces the kernel socket send buffer size on the client or the server side to the specified value in bytes. This value applies to all TCP/HTTP frontends and backends. It should normally never be set, and the default size (0) lets the kernel auto-tune this value depending on the amount of available memory. However it can sometimes help to set it to very low values (e.g. 4096) in order to save kernel memory by preventing it from buffering too large amounts of received data. Lower values will significantly increase CPU usage though. Another use case is to prevent write timeouts with extremely slow clients due to the kernel waiting for a large part of the buffer to be read before notifying HAProxy again. See also tune.notsent-lowat.client and tune.notsent-lowat.server for more effective settings to more finely control memory usage and responsiveness on Linux without hurting performance.

tune.ssl.cachesize <number>

tune.ssl.cachesize <number>

Sets the size of the global SSL session cache, in a number of blocks. A block is large enough to contain an encoded session without peer certificate. An encoded session with peer certificate is stored in multiple blocks depending on the size of the peer certificate. A block uses approximately 200 bytes of memory (based on sizeof(struct sh_ssl_sess_hdr) + SHSESS_BLOCK_MIN_SIZE calculation used for shctx_init function). The default value may be forced at build time, otherwise defaults to 20000. When the cache is full, the most idle entries are purged and reassigned. Higher values reduce the occurrence of such a purge, hence the number of CPU-intensive SSL handshakes by ensuring that all users keep their session as long as possible. All entries are pre-allocated upon startup. Setting this value to 0 disables the SSL session cache.

tune.ssl.capture-buffer-size <number>

tune.ssl.capture-buffer-size <number>
tune.ssl.capture-cipherlist-size <number> (deprecated)

Sets the maximum size of the buffer used for capturing client hello cipher list, extensions list, elliptic curves list and elliptic curve point formats. If the value is 0 (default value) the capture is disabled, otherwise a buffer is allocated for each SSL/TLS connection.

tune.ssl.certificate-compression { auto | off }

tune.ssl.certificate-compression { auto | off }

This setting allows to configure the certificate compression support which is an extension (RFC 8879) to TLS 1.3.

When set to “auto” it uses the default value of the TLS library.

With “off” it tries to explicitly disable the support of the feature. HAProxy won’t try to send compressed certificates anymore nor accept compressed certificates.

Configures both backend and frontend sides.

This keyword is supported by OpenSSL >= 3.2.0.

The default value is auto.

tune.ssl.default-dh-param <number>

tune.ssl.default-dh-param <number>

Sets the maximum size of the Diffie-Hellman parameters used for generating the ephemeral/temporary Diffie-Hellman key in case of DHE key exchange. The final size will try to match the size of the server’s RSA (or DSA) key (e.g, a 2048 bits temporary DH key for a 2048 bits RSA key), but will not exceed this maximum value. Only 1024 or higher values are allowed. Higher values will increase the CPU load, and values greater than 1024 bits are not supported by Java 7 and earlier clients. This value is not used if static Diffie-Hellman parameters are supplied either directly in the certificate file or by using the ssl-dh-param-file parameter. If there is neither a default-dh-param nor a ssl-dh-param-file defined, and if the server’s PEM file of a given frontend does not specify its own DH parameters, then DHE ciphers will be unavailable for this frontend.

tune.ssl.force-private-cache

tune.ssl.force-private-cache

This option disables SSL session cache sharing between all processes. It should normally not be used since it will force many renegotiations due to clients hitting a random process. But it may be required on some operating systems where none of the SSL cache synchronization method may be used. In this case, adding a first layer of hash-based load balancing before the SSL layer might limit the impact of the lack of session sharing.

tune.ssl.hard-maxrecord <number>

tune.ssl.hard-maxrecord <number>

Sets the maximum amount of bytes passed to SSL_write() at any time. Default value 0 means there is no limit. In contrast to tune.ssl.maxrecord this settings will not be adjusted dynamically. Smaller records may decrease throughput, but may be required when dealing with low-footprint clients.

tune.ssl.keylog { on | off }

tune.ssl.keylog { on | off }

This option activates the logging of the TLS keys. It should be used with care as it will consume more memory per SSL session and could decrease performances. This is disabled by default.

These sample fetches should be used to generate the SSLKEYLOGFILE that is required to decipher traffic with wireshark.

https://tlswg.org/sslkeylogfile/draft-ietf-tls-keylogfile.html

The SSLKEYLOG is a series of lines which are formatted this way:

<Label> <space> <ClientRandom> <space> <Secret>

The ClientRandom is provided by the %[ssl_fc_client_random,hex] sample fetch, the secret and the Label could be find in the array below. You need to generate a SSLKEYLOGFILE with all the labels in this array.

The following sample fetches are hexadecimal strings and does not need to be converted.

  SSLKEYLOGFILE Label             |  Sample fetches for the Secrets
  --------------------------------|-----------------------------------------
  CLIENT_EARLY_TRAFFIC_SECRET     |  %[ssl_xx_client_early_traffic_secret]
  CLIENT_HANDSHAKE_TRAFFIC_SECRET |  %[ssl_xx_client_handshake_traffic_secret]
  SERVER_HANDSHAKE_TRAFFIC_SECRET |  %[ssl_xx_server_handshake_traffic_secret]
  CLIENT_TRAFFIC_SECRET_0         |  %[ssl_xx_client_traffic_secret_0]
  SERVER_TRAFFIC_SECRET_0         |  %[ssl_xx_server_traffic_secret_0]
  EXPORTER_SECRET                 |  %[ssl_xx_exporter_secret]
  EARLY_EXPORTER_SECRET           |  %[ssl_xx_early_exporter_secret]

These fetches exists for frontend (fc) or backend (bc) sides, replace “xx” by “fc” or “bc” to use the right side.

This is only available with OpenSSL 1.1.1, and useful with TLS1.3 session.

If you want to generate the content of a SSLKEYLOGFILE with TLS < 1.3, you only need this line:

“CLIENT_RANDOM %[ssl_fc_client_random,hex] %[ssl_fc_session_key,hex]”

A complete keylog could be generate with a log-format these way, even though this is not ideal for syslog:

log-format "CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret]\n
            CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret]\n
            SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret]\n
            CLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0]\n
            SERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0]\n
            EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret]\n
            EARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"

HAProxy also provides the above formats as predefined environment variables that can be used directly in a “log-format” directive:

$HAPROXY_KEYLOG_FC_LOG_FMT   frontend (client-facing) connection keys
$HAPROXY_KEYLOG_BC_LOG_FMT   backend (server-facing) connection keys

tune.ssl.keyupdate-rate-limit <limit>

tune.ssl.keyupdate-rate-limit <limit>

Limit the amount of KeyUpdate per second we’re willing to accept to <limit> before considering it flood, and killing the connection. Dealing with KeyUpdate is cpu-expensive, and there is little reason to receive a lot of them. Using a value of “0” disables the rate limiting. The default value is 100.

tune.ssl.lifetime <timeout>

tune.ssl.lifetime <timeout>

Sets how long a cached SSL session may remain valid. This time is expressed in seconds and defaults to 300 (5 min). It is important to understand that it does not guarantee that sessions will last that long, because if the cache is full, the longest idle sessions will be purged despite their configured lifetime. The real usefulness of this setting is to prevent sessions from being used for too long.

tune.ssl.maxrecord <number>

tune.ssl.maxrecord <number>

Sets the maximum amount of bytes passed to SSL_write() at the beginning of the data transfer. Default value 0 means there is no limit. Over SSL/TLS, the client can decipher the data only once it has received a full record. With large records, it means that clients might have to download up to 16kB of data before starting to process them. Limiting the value can improve page load times on browsers located over high latency or low bandwidth networks. It is suggested to find optimal values which fit into 1 or 2 TCP segments (generally 1448 bytes over Ethernet with TCP timestamps enabled, or 1460 when timestamps are disabled), keeping in mind that SSL/TLS add some overhead. Typical values of 1419 and 2859 gave good results during tests. Use “strace -e trace=write” to find the best value. HAProxy will automatically switch to this setting after an idle stream has been detected (see tune.idletimer above). See also tune.ssl.hard-maxrecord.

tune.ssl.ssl-ctx-cache-size <number>

tune.ssl.ssl-ctx-cache-size <number>

Sets the size of the cache used to store generated certificates to <number> entries. This is a LRU cache. Because generating a SSL certificate dynamically is expensive, they are cached. The default cache size is set to 1000 entries.

tune.streams-elasticity <number>

tune.streams-elasticity <number>

Defines a target percentage of streams per frontend connection relative to the maximum number of concurrent connections (maxconn) when all connections are established. This metric applies to multiplexed protocols like HTTP/2 or QUIC, where each connection may receive multiple streams. At least one is always guaranteed, so the percentage must be at least 100%. During connection setup, HAProxy dynamically advertises additional streams up to the configured limit, maintaining the target ratio. At connection establishment, every frontend connection receives at least one stream; extra streams are assigned based on the target percentage and configured stream limits. This ensures efficient stream allocation under varying load conditions (more streams at low loads, fewer at high loads).

Highly dynamic sites with many objects per page benefit from high ratios, enabling many streams per connection. Sites using fewer streams on average (WebSocket, application code) may prefer small ratios closer to 120 or 150 (20 to 50% more streams than connections) preventing excessive stream counts under sustained loads.

The default value is 0, meaning no enforcement at this level, so only H2 and QUIC configurations apply (with the default setting of 100 streams per connection, this corresponds to 10000%). This remains the recommended setting for small deployments (maxconn around a thousand). Moderately sized setups (few thousands to tens of thousands connections) typically set the ratio between 1000 and 5000, allowing 10 to 50 streams per connection at full load. Large-scale deployments (hundreds of thousands to millions connections) might use lower values (120 to 200) to support 1.2 to 2 streams per connection on average at full load.

Contrary to HTTP/2, QUIC is capable to dynamically adjust the number of concurrent streams during the connection lifetime. However, QUIC flow control is stricter than HTTP/2, thus it is preferable when using it to specify values big enough to prevent extra latency on the connection. There is also a limitation for QUIC listeners with enabled 0-RTT. In this case, the initial value advertised to the peer will ignore stream elasticity and instead rely solely on the “tune.quic.fe.stream.max-concurrent” setting. However, the stream elasticity principle will still be effective past this initial annoucement during the connection lifetime.

Monitoring the total number of active streams on backends, including queues, provides a practical indicator of a sustainable target load and helps avoid over-provisioning.

tune.stick-counters <number>

tune.stick-counters <number>

Sets the number of stick-counters that may be tracked at the same time by a connection or a request via “track-sc*” actions in “tcp-request” or “http-request” rules. The default value is set at build time by the macro MAX_SESS_STK_CTR, and defaults to 3. With this setting it is possible to change the value and ignore the one passed at build time, but it cannot be set to a value greater than 100. Increasing this value may be needed when porting complex configurations to haproxy, but users are warned against the costs: each entry takes 16 bytes per connection and 16 bytes per request, all of which need to be allocated and zeroed for all requests even when not used. As such a value of 10 will inflate the memory consumption per request by 320 bytes and will cause this memory to be erased for each request, which does have measurable CPU impacts. Conversely, when no “track-sc” rules are used, the value may be lowered (0 being valid to entirely disable stick-counters).

tune.takeover-other-tg-connections <value>

tune.takeover-other-tg-connections <value>

By default, we won’t attempt to use idle connections from other thread groups. This can however be changed. Valid values for <value> are: “none”, the default, if used, no attempt will be made to use idle connections from other thread groups, “restricted” where we will only attempt to get an idle connection from another thread if we’re using protocols that can’t create new connections, such as reverse http, as well as when using strict-maxconn, and “full” where we will always look in other thread groups for idle connections. Note that using connections from other thread groups can occur performance penalties, so it should not be used unless really needed. Note that this behavior is now controlled by tune.idle-pool.shared, and this keyword is just there for compatibility with older configurations, and will be deprecated.

tune.vars.global-max-size <size>

tune.vars.global-max-size <size>
tune.vars.proc-max-size <size>
tune.vars.reqres-max-size <size>
tune.vars.sess-max-size <size>
tune.vars.txn-max-size <size>

These five tunes help to manage the maximum amount of memory used by the variables system. “global” limits the overall amount of memory available for all scopes. “proc” limits the memory for the process scope, “sess” limits the memory for the session scope, “txn” for the transaction scope, and “reqres” limits the memory for each request or response processing. Memory accounting is hierarchical, meaning more coarse grained limits include the finer grained ones: “proc” includes “sess”, “sess” includes “txn”, and “txn” includes “reqres”.

For example, when “tune.vars.sess-max-size” is limited to 100, “tune.vars.txn-max-size” and “tune.vars.reqres-max-size” cannot exceed 100 either. If we create a variable “txn.var” that contains 100 bytes, all available space is consumed. Notice that exceeding the limits at runtime will not result in an error message, but values might be cut off or corrupted. So make sure to accurately plan for the amount of space needed to store all your variables.

tune.zlib.memlevel <number>

tune.zlib.memlevel <number>

Sets the memLevel parameter in zlib initialization for each stream. It defines how much memory should be allocated for the internal compression state. A value of 1 uses minimum memory but is slow and reduces compression ratio, a value of 9 uses maximum memory for optimal speed. Can be a value between 1 and 9. The default value is 8.

tune.zlib.windowsize <number>

tune.zlib.windowsize <number>

Sets the window size (the size of the history buffer) as a parameter of the zlib initialization for each stream. Larger values of this parameter result in better compression at the expense of memory usage. Can be a value between 8 and 15. The default value is 15.

3.3. Debugging

anonkey <key>

anonkey <key>

This sets the global anonymizing key to <key>, which must be a 32-bit number between 0 and 4294967295. This is the key that will be used by default by CLI commands when anonymized mode is enabled. This key may also be set at runtime from the CLI command “set anon global-key”. See also command line argument “-dC” in the management manual.

debug.counters { on | off }

debug.counters { on | off }

Enables (‘on’) or disables (‘off’) the updating of event counters in the code. These are the counters reported under the type “CNT” in the CLI command “debug counters”. These counters are only available when the code was build with DEBUG_COUNTERS set to a value 1 or above. With the value 1, the counters are not updated by default (“debug.counters off”), and with value 2, they are updated by default (“debug.counters on”). There is normally no reason to change this setting unless a developer requests it, or unless it is suspected to consume abnormal amounts of CPU (in which case a report to developers is necessary with a dump of the counters). It is also possible to change this status at run time using the “debug counters” CLI command. Please consult the management manual.

force-cfg-parser-pause <timeout>

force-cfg-parser-pause <timeout>

This command is pausing the configuration parser for <timeout> milliseconds. This is useful for development or for testing timeouts of init scripts, particularly to simulate a very long reload. It requires the expose-experimental-directives to be set.

<timeout> is the timeout value specified in milliseconds by default, but can be in any other unit if the number is suffixed by the unit, as explained at the top of this document.

Example:

global
    expose-experimental-directives
    force-cfg-parser-pause 10s

quick-exit

quick-exit

This speeds up the old process exit upon reload by skipping the releasing of memory objects and listeners, since all of these are reclaimed by the operating system at the process’ death. The gains are only marginal (in the order of a few hundred milliseconds for huge configurations at most). The main target usage in fact is when a bug is spotted in the deinit() code, as this allows to bypass it. It is better not to use this unless instructed to do so by developers.

quiet

quiet

Do not display any message during startup. It is equivalent to the command-line argument “-q”.

warn-blocked-traffic-after <time>

warn-blocked-traffic-after <time>

This allows to adjust the delay after which a stuck task blocking the traffic will trigger the emission of a warning on the standard error output. The delay is expressed in milliseconds and defaults to 100 ms. Permitted values must be comprised between 1 ms and 1000 ms included. Lower values will trigger warnings frequently and higher ones will rarely. The watchdog will kill a runaway task that fails to respond twice for one second anyway, so a 1000 ms warning delay will normally not trigger any warning. It is recommended to stay with values between 10 and 100ms to detect configuration anomalies that may degrade the user’s experience, causing long response times or jerkiness on interactive sessions. For example, a poorly designed Lua sample-fetch function doing heavy computations, or a very large map_reg or map_regm map file with a very high evaluation cost may cause such trouble. For comparison a TLS handshake can eat between one and two milliseconds, and compressing a 16kB HTTP response buffer is around one millisecond. The output contains a thread dump of the offending task with a backtrace and some context that helps figure where the time is being spent.

zero-warning

zero-warning

When this option is set, HAProxy will refuse to start if any warning was emitted while processing the configuration and applying it. It means that warnings about bad combinations of parameters, warnings about very high limits that couldn’t be set, and so on, make the process exit with an error during startup. A few late startup warnings cannot be caught by this option, such as the failure to drop supplementary groups when changing the group ID in “daemon” or “master-worker” modes, or the failure to mark the process dumpable after the fork(). This option does not catch warnings emitted at runtime. It is highly recommended to set this option on configurations that are not changed often, as it helps to detect subtle mistakes and keep the configuration clean and forward-compatible. Note that “haproxy -c” will also report errors in such a case. This option is equivalent to command line argument “-dW”.

3.4. HTTPClient tuning

HTTPClient is an internal HTTP library, it can be used by various subsystems, for example in LUA scripts. HTTPClient is not used in the data path, in other words it has nothing with HTTP traffic passing through HAProxy.

httpclient.resolvers.disabled <on|off>

httpclient.resolvers.disabled <on|off>

Disable the DNS resolution of the httpclient. Prevent the creation of the “default” resolvers section.

Default value is off.

httpclient.resolvers.id <resolvers id>

httpclient.resolvers.id <resolvers id>

This option defines the resolvers section with which the httpclient will try to resolve.

Default option is the “default” resolvers ID. By default, if this option is not used, it will simply disable the resolving if the section is not found.

However, when this option is explicitly enabled it will trigger a configuration error if it fails to load.

httpclient.resolvers.prefer <ipv4|ipv6>

httpclient.resolvers.prefer <ipv4|ipv6>

This option allows to chose which family of IP you want when resolving, which is convenient when IPv6 is not available on your network. Default option is “ipv6”.

httpclient.retries <number>

httpclient.retries <number>

This option allows to configure the number of retries attempt of the httpclient when a request failed. This does the same as the “retries” keyword in a backend.

Default value is 3.

httpclient.ssl.ca-file <cafile>

httpclient.ssl.ca-file <cafile>

This option defines the ca-file which should be used to verify the server certificate. It takes the same parameters as the “ca-file” option on the server line.

By default and when this option is not used, the value is “@system-ca” which tries to load the CA of the system. If it fails the SSL will be disabled for the httpclient.

However, when this option is explicitly enabled it will trigger a configuration error if it fails.

httpclient.ssl.verify [none|required]

httpclient.ssl.verify [none|required]

Works the same way as the verify option on server lines. If specified to ’none’, servers certificates are not verified. Default option is “required”.

By default and when this option is not used, the value is “required”. If it fails the SSL will be disabled for the httpclient.

However, when this option is explicitly enabled it will trigger a configuration error if it fails.

httpclient.timeout.connect <timeout>

httpclient.timeout.connect <timeout>

Set the maximum time to wait for a connection attempt by default for the httpclient.

Arguments:

<timeout> is the timeout value specified in milliseconds by default, but
          can be in any other unit if the number is suffixed by the unit,
          as explained at the top of this document.

The default value is 5000ms.

13 - 4. Proxies

Proxy configuration can be located in a set of sections:

• defaults [<name>] [ from <defaults_name> ]
• frontend <name> [ from <defaults_name> ]
• backend <name> [ from <defaults_name> ]
• listen <name> [ from <defaults_name> ]

A “frontend” section describes a set of listening sockets accepting client connections.

A “backend” section describes a set of servers to which the proxy will connect to forward incoming connections.

A “listen” section defines a complete proxy with its frontend and backend parts combined in one section. It is generally useful for TCP-only traffic.

A “defaults” section resets all settings to the documented ones and presets new ones for use by subsequent sections. All of “frontend”, “backend” and “listen” sections always take their initial settings from a defaults section, by default the latest one that appears before the newly created section. It is possible to explicitly designate a specific “defaults” section to load the initial settings from by indicating its name on the section line after the optional keyword “from”. While “defaults” section do not impose a name, this use is encouraged for better readability. It is also the only way to designate a specific section to use instead of the default previous one. Since “defaults” section names are optional, by default a very permissive check is applied on their name and these are even permitted to overlap. However if a “defaults” section is referenced by any other section, its name must comply with the syntax imposed on all proxy names, and this name must be unique among the defaults sections. Please note that regardless of what is currently permitted, it is recommended to avoid duplicate section names in general and to respect the same syntax as for proxy names. This rule might be enforced in a future version. In addition, a warning is emitted if a defaults section is explicitly used by a proxy while it is also implicitly used by another one because it is the last one defined. It is highly encouraged to not mix both usages by always using explicit references or by adding a last common defaults section reserved for all implicit uses.

Note that it is even possible for a defaults section to take its initial settings from another one, and as such, inherit settings across multiple levels of defaults sections. This can be convenient to establish certain configuration profiles to carry groups of default settings (e.g. TCP vs HTTP or short vs long timeouts) but can quickly become confusing to follow.

By default, named defaults sections are preserved after configuration parsing. This allows to reuse them for dynamic backends creation. This behavior can be changed globally via “tune.defaults.purge” keyword.

All proxy names must be formed from upper and lower case letters, digits, ‘-’ (dash), ‘_’ (underscore) , ‘.’ (dot) and ‘:’ (colon). ACL names are case-sensitive, which means that “www” and “WWW” are two different proxies.

Historically, all proxy names could overlap when certain conditions were met (e.g. when not having the same frontend/backend capabilities), but it used to cause too many problems in the logs as well as confusion on CLI operations, stick-tables naming and stats retrieval. It is now mandatory that two proxies have different names, regardless of their respective capabilities.

Right now, two major proxy modes are supported: “tcp”, also known as layer 4, and “http”, also known as layer 7. In layer 4 mode, HAProxy simply forwards bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the protocol, and can interact with it by allowing, blocking, switching, adding, modifying, or removing arbitrary contents in requests or responses, based on arbitrary criteria.

In HTTP mode, the processing applied to requests and responses flowing over a connection depends in the combination of the frontend’s HTTP options and the backend’s. HAProxy supports 3 connection modes:

• KAL: keep alive (“option http-keep-alive”) which is the default mode: all requests and responses are processed, and connections remain open but idle between responses and new requests.

• SCL: server close (“option http-server-close”): the server-facing connection is closed after the end of the response is received, but the client-facing connection remains open.

• CLO: close (“option httpclose”): the connection is closed after the end of the response and “Connection: close” appended in both directions.

The effective mode that will be applied to a connection passing through a frontend and a backend can be determined by both proxy modes according to the following matrix, but in short, the modes are symmetric, keep-alive is the weakest option and close is the strongest.

               Backend mode

                | KAL | SCL | CLO
            ----+-----+-----+----
            KAL | KAL | SCL | CLO
            ----+-----+-----+----
   mode     SCL | SCL | SCL | CLO
            ----+-----+-----+----
            CLO | CLO | CLO | CLO

It is possible to chain a TCP frontend to an HTTP backend. It is pointless if only HTTP traffic is handled. But it may be used to handle several protocols within the same frontend. In this case, the client’s connection is first handled as a raw tcp connection before being upgraded to HTTP. Before the upgrade, the content processings are performed on raw data. Once upgraded, data is parsed and stored using an internal representation called HTX and it is no longer possible to rely on raw representation. There is no way to go back.

There are two kind of upgrades, in-place upgrades and destructive upgrades. The first ones involves a TCP to HTTP/1 upgrade. In HTTP/1, the request processings are serialized, thus the applicative stream can be preserved. The second one involves a TCP to HTTP/2 upgrade. Because it is a multiplexed protocol, the applicative stream cannot be associated to any HTTP/2 stream and is destroyed. New applicative streams are then created when HAProxy receives new HTTP/2 streams at the lower level, in the H2 multiplexer. It is important to understand this difference because that drastically changes the way to process data. When an HTTP/1 upgrade is performed, the content processings already performed on raw data are neither lost nor reexecuted while for an HTTP/2 upgrade, applicative streams are distinct and all frontend rules are evaluated systematically on each one. And as said, the first stream, the TCP one, is destroyed, but only after the frontend rules were evaluated.

There is another important point to understand when HTTP processings are performed from a TCP proxy. While HAProxy is able to parse HTTP/1 in-fly from tcp-request content rules, it is not possible for HTTP/2. Only the HTTP/2 preface can be parsed. This is a huge limitation regarding the HTTP content analysis in TCP. Concretely it is only possible to know if received data are HTTP. For instance, it is not possible to choose a backend based on the Host header value while it is trivial in HTTP/1. Hopefully, there is a solution to mitigate this drawback.

There are two ways to perform an HTTP upgrade. The first one, the historical method, is to select an HTTP backend. The upgrade happens when the backend is set. Thus, for in-place upgrades, only the backend configuration is considered in the HTTP data processing. For destructive upgrades, the applicative stream is destroyed, thus its processing is stopped. With this method, possibilities to choose a backend with an HTTP/2 connection are really limited, as mentioned above, and a bit useless because the stream is destroyed. The second method is to upgrade during the tcp-request content rules evaluation, thanks to the “switch-mode http” action. In this case, the upgrade is performed in the frontend context and it is possible to define HTTP directives in this frontend. For in-place upgrades, it offers all the power of the HTTP analysis as soon as possible. It is not that far from an HTTP frontend. For destructive upgrades, it does not change anything except it is useless to choose a backend on limited information. It is of course the recommended method. Thus, testing the request protocol from the tcp-request content rules to perform an HTTP upgrade is enough. All the remaining HTTP manipulation may be moved to the frontend http-request ruleset. But keep in mind that tcp-request content rules remains evaluated on each streams, that can’t be changed.

4.1. Proxy keywords matrix

The following list of keywords is supported. Most of them may only be used in a limited set of section types. Some of them are marked as “deprecated” because they are inherited from an old syntax which may be confusing or functionally limited, and there are new recommended keywords to replace them. Keywords marked with “(*)” can be optionally inverted using the “no” prefix, e.g. “no option contstats”. This makes sense when the option has been enabled by default and must be disabled for a specific instance. Such options may also be prefixed with “default” in order to restore default settings regardless of what has been specified in a previous “defaults” section. Keywords supported in defaults sections marked with “(!)” are only supported in named defaults sections, not anonymous ones.

Note: Some dangerous and not recommended directives are intentionnaly not listed in the following matrix. It is on purpose. These directives are documentated. But by not listing them below is one more way to discourage anyone to use it.

 keyword                              defaults   frontend   listen    backend
------------------------------------+----------+----------+---------+---------
acl                                       X (!)      X         X         X
backlog                                   X          X         X         -
balance                                   X          -         X         X
be-unpublished                            -          -         X         X
bind                                      -          X         X         -
capture cookie                            -          X         X         -
capture request header                    -          X         X         -
capture response header                   -          X         X         -
clitcpka-cnt                              X          X         X         -
clitcpka-idle                             X          X         X         -
clitcpka-intvl                            X          X         X         -
compression                               X          X         X         X
cookie                                    X          -         X         X
crt                                       -          X         X         -
declare capture                           -          X         X         -
default-server                            X          -         X         X
default_backend                           X          X         X         -
description                               -          X         X         X
disabled                                  X          X         X         X
dispatch                    (deprecated)  -          -         X         X
email-alert from                          X          X         X         X
email-alert level                         X          X         X         X
email-alert mailers                       X          X         X         X
email-alert myhostname                    X          X         X         X
email-alert to                            X          X         X         X
enabled                                   X          X         X         X
errorfile                                 X          X         X         X
errorfiles                                X          X         X         X
errorloc                                  X          X         X         X
errorloc302                               X          X         X         X
-- keyword -------------------------- defaults - frontend - listen -- backend -
errorloc303                               X          X         X         X
error-log-format                          X          X         X         -
external-check command                    X          -         X         X
external-check path                       X          -         X         X
force-persist                             -          -         X         X
force-be-switch                           -          X         X         -
filter                                    -          X         X         X
filter-sequence                           -          X         X         X
fullconn                                  X          -         X         X
guid                                      -          X         X         X
hash-balance-factor                       X          -         X         X
hash-preserve-affinity                    X          -         X         X
hash-type                                 X          -         X         X
http-after-response                       X (!)      X         X         X
http-check comment                        X          -         X         X
http-check connect                        X          -         X         X
http-check disable-on-404                 X          -         X         X
http-check expect                         X          -         X         X
http-check send                           X          -         X         X
http-check send-state                     X          -         X         X
http-check set-var                        X          -         X         X
http-check unset-var                      X          -         X         X
http-error                                X          X         X         X
http-request                              X (!)      X         X         X
http-response                             X (!)      X         X         X
http-reuse                                X          -         X         X
http-send-name-header                     X          -         X         X
id                                        -          X         X         X
ignore-persist                            -          -         X         X
load-server-state-from-file               X          -         X         X
log                                  (*)  X          X         X         X
log-format                                X          X         X         -
log-format-sd                             X          X         X         -
log-tag                                   X          X         X         X
log-steps                                 X          X         X         -
max-keep-alive-queue                      X          -         X         X
max-session-srv-conns                     X          X         X         -
maxconn                                   X          X         X         -
mode                                      X          X         X         X
monitor fail                              -          X         X         -
monitor-uri                               X          X         X         -
option abortonclose                  (*)  X          X         X         X
option allbackups                    (*)  X          -         X         X
option checkcache                    (*)  X          -         X         X
option clitcpka                      (*)  X          X         X         -
option contstats                     (*)  X          X         X         -
option disable-h2-upgrade            (*)  X          X         X         -
option dontlog-normal                (*)  X          X         X         -
option dontlognull                   (*)  X          X         X         -
-- keyword -------------------------- defaults - frontend - listen -- backend -
option external-check                     X          -         X         X
option forwardfor                         X          X         X         X
option forwarded                     (*)  X          -         X         X
option h1-case-adjust-bogus-client   (*)  X          X         X         -
option h1-case-adjust-bogus-server   (*)  X          -         X         X
option http-buffer-request           (*)  X          X         X         X
option http-drop-request-trailers    (*)  X          -         -         X
option http-drop-response-trailers   (*)  X          -         X         -
option http-ignore-probes            (*)  X          X         X         -
option http-keep-alive               (*)  X          X         X         X
option http-no-delay                 (*)  X          X         X         X
option http-pretend-keepalive        (*)  X          -         X         X
option http-restrict-req-hdr-names        X          X         X         X
option http-server-close             (*)  X          X         X         X
option http-use-proxy-header         (*)  X          X         X         -
option httpchk                            X          -         X         X
option httpclose                     (*)  X          X         X         X
option httplog                            X          X         X         -
option httpslog                           X          X         X         -
option idle-close-on-response        (*)  X          X         X         -
option independent-streams           (*)  X          X         X         X
option ldap-check                         X          -         X         X
option log-health-checks             (*)  X          -         X         X
option log-separate-errors           (*)  X          X         X         -
option logasap                       (*)  X          X         X         -
option mysql-check                        X          -         X         X
option nolinger                      (*)  X          X         X         X
option originalto                         X          X         X         X
option persist                       (*)  X          -         X         X
option pgsql-check                        X          -         X         X
option prefer-last-server            (*)  X          -         X         X
option redispatch                    (*)  X          -         X         X
option redis-check                        X          -         X         X
option smtpchk                            X          -         X         X
option socket-stats                  (*)  X          X         X         -
option splice-auto                   (*)  X          X         X         X
option splice-request                (*)  X          X         X         X
option splice-response               (*)  X          X         X         X
option spop-check                         X          -         X         X
option srvtcpka                      (*)  X          -         X         X
option ssl-hello-chk                      X          -         X         X
-- keyword -------------------------- defaults - frontend - listen -- backend -
option tcp-check                          X          -         X         X
option tcp-smart-accept              (*)  X          X         X         -
option tcp-smart-connect             (*)  X          -         X         X
option tcpka                              X          X         X         X
option tcplog                             X          X         X         -
option transparent      (deprecated) (*)  X          -         X         X
option use-small-buffers             (*)  X          -         X         X
persist rdp-cookie                        X          -         X         X
quic-initial                              X (!)      X         X         -
rate-limit sessions                       X          X         X         -
redirect                                  -          X         X         X
-- keyword -------------------------- defaults - frontend - listen -- backend -
retries                                   X          -         X         X
retry-on                                  X          -         X         X
server                                    -          -         X         X
server-state-file-name                    X          -         X         X
server-template                           -          -         X         X
source                                    X          -         X         X
srvtcpka-cnt                              X          -         X         X
srvtcpka-idle                             X          -         X         X
srvtcpka-intvl                            X          -         X         X
stats admin                               -          X         X         X
stats auth                                X          X         X         X
stats enable                              X          X         X         X
stats hide-version                        X          X         X         X
stats http-request                        -          X         X         X
stats realm                               X          X         X         X
stats refresh                             X          X         X         X
stats scope                               X          X         X         X
stats show-desc                           X          X         X         X
stats show-legends                        X          X         X         X
stats show-node                           X          X         X         X
stats show-version                        X          X         X         X
stats uri                                 X          X         X         X
-- keyword -------------------------- defaults - frontend - listen -- backend -
stick match                               -          -         X         X
stick on                                  -          -         X         X
stick store-request                       -          -         X         X
stick store-response                      -          -         X         X
stick-table                               -          X         X         X
tcp-check comment                         X          -         X         X
tcp-check connect                         X          -         X         X
tcp-check expect                          X          -         X         X
tcp-check send                            X          -         X         X
tcp-check send-lf                         X          -         X         X
tcp-check send-binary                     X          -         X         X
tcp-check send-binary-lf                  X          -         X         X
tcp-check set-var                         X          -         X         X
tcp-check unset-var                       X          -         X         X
tcp-request connection                    X (!)      X         X         -
tcp-request content                       X (!)      X         X         X
tcp-request inspect-delay                 X (!)      X         X         X
tcp-request session                       X (!)      X         X         -
tcp-response content                      X (!)      -         X         X
tcp-response inspect-delay                X (!)      -         X         X
timeout check                             X          -         X         X
timeout client                            X          X         X         -
timeout client-fin                        X          X         X         -
timeout client-hs                         X          X         X         -
timeout connect                           X          -         X         X
timeout http-keep-alive                   X          X         X         X
timeout http-request                      X          X         X         X
timeout queue                             X          -         X         X
timeout server                            X          -         X         X
timeout server-fin                        X          -         X         X
timeout tarpit                            X          X         X         X
timeout tunnel                            X          -         X         X
transparent                 (deprecated)  X          -         X         X
unique-id-format                          X          X         X         X
unique-id-header                          X          X         X         -
use_backend                               -          X         X         -
use-fcgi-app                              -          -         X         X
use-server                                -          -         X         X
------------------------------------+----------+----------+---------+---------
 keyword                              defaults   frontend   listen    backend

4.2. Alphabetically sorted keywords reference

This section provides a description of each keyword and its usage.

acl <aclname> <criterion> [flags] [operator] <value> ...

acl <aclname> <criterion> [flags] [operator] <value> ...

Declare or complete an access list.

May be used in the following contexts: tcp, http

May be used in sections: defaults | frontend | listen | backend yes(!) | yes | yes | yes

This directive is only available from named defaults sections, not anonymous ones. ACLs defined in a defaults section are not visible from other sections using it.

Example:

acl invalid_src  src          0.0.0.0/7 224.0.0.0/3
acl invalid_src  src_port     0:1023
acl local_dst    hdr(host) -i localhost

See section 7 about ACL usage.

backlog <conns>

backlog <conns>

Give hints to the system about the approximate listen backlog desired size

May be used in the following contexts: tcp, http

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | no

Arguments:

<conns>   is the number of pending connections. Depending on the operating
          system, it may represent the number of already acknowledged
          connections, of non-acknowledged ones, or both.

This option is only meaningful for stream listeners, including QUIC ones. Its behavior however is not identical with QUIC instances.

For all listeners but QUIC, in order to protect against SYN flood attacks, one solution is to increase the system’s SYN backlog size. Depending on the system, sometimes it is just tunable via a system parameter, sometimes it is not adjustable at all, and sometimes the system relies on hints given by the application at the time of the listen() syscall. By default, HAProxy passes the frontend’s maxconn value to the listen() syscall. On systems which can make use of this value, it can sometimes be useful to be able to specify a different value, hence this backlog parameter.

On Linux 2.4, the parameter is ignored by the system. On Linux 2.6, it is used as a hint and the system accepts up to the smallest greater power of two, and never more than some limits (usually 32768).

For QUIC listeners, backlog sets a shared limits for both the maximum count of active handshakes and connections waiting to be accepted. The handshake phase relies primarily of the network latency with the remote peer, whereas the second phase depends solely on haproxy load. When either one of this limit is reached, haproxy starts to drop reception of INITIAL packets, preventing any new connection allocation, until the connection excess starts to decrease. This situation may cause browsers to silently downgrade the HTTP versions and switching to TCP.

See also: “maxconn” and the target operating system’s tuning guide.

balance <algorithm> [ <arguments> ]

balance <algorithm> [ <arguments> ]
balance url_param <param> [check_post]

Define the load balancing algorithm to be used in a backend.

May be used in the following contexts: tcp, http, log

May be used in sections: defaults | frontend | listen | backend yes | no | yes | yes

Arguments:

<algorithm> is the algorithm used to select a server when doing load
            balancing. This only applies when no persistence information
            is available, or when a connection is redispatched to another
            server. <algorithm> may be one of the following:

  roundrobin  Each server is used in turns, according to their weights.
              This is the smoothest and fairest algorithm when the server's
              processing time remains equally distributed. This algorithm
              is dynamic, which means that server weights may be adjusted
              on the fly for slow starts for instance. It is limited by
              design to 4095 active servers per backend. Note that in some
              large farms, when a server becomes up after having been down
              for a very short time, it may sometimes take a few hundreds
              requests for it to be re-integrated into the farm and start
              receiving traffic. This is normal, though very rare. It is
              indicated here in case you would have the chance to observe
              it, so that you don't worry. Note: weights are ignored for
              backends in LOG mode.

  static-rr   Each server is used in turns, according to their weights.
              This algorithm is as similar to roundrobin except that it is
              static, which means that changing a server's weight on the
              fly will have no effect. On the other hand, it has no design
              limitation on the number of servers, and when a server goes
              up, it is always immediately reintroduced into the farm, once
              the full map is recomputed. It also uses slightly less CPU to
              run (around -1%). This algorithm is not usable in LOG mode.

  leastconn   The server with the lowest number of connections receives the
              connection. Round-robin is performed within groups of servers
              of the same load to ensure that all servers will be used. Use
              of this algorithm is recommended where very long sessions are
              expected, such as LDAP, SQL, TSE, etc... but is not very well
              suited for protocols using short sessions such as HTTP. This
              algorithm is dynamic, which means that server weights may be
              adjusted on the fly for slow starts for instance. It will
              also consider the number of queued connections in addition to
              the established ones in order to minimize queuing. This
              algorithm is not usable in LOG mode.

  first       The first server with available connection slots receives the
              connection. The servers are chosen from the lowest numeric
              identifier to the highest (see server parameter "id"), which
              defaults to the server's position in the farm. Once a server
              reaches its maxconn value, the next server is used. It does
              not make sense to use this algorithm without setting maxconn.
              The purpose of this algorithm is to always use the smallest
              number of servers so that extra servers can be powered off
              during non-intensive hours. This algorithm ignores the server
              weight, and brings more benefit to long session such as RDP
              or IMAP than HTTP, though it can be useful there too. In
              order to use this algorithm efficiently, it is recommended
              that a cloud controller regularly checks server usage to turn
              them off when unused, and regularly checks backend queue to
              turn new servers on when the queue inflates. Alternatively,
              using "http-check send-state" may inform servers on the load.
              This algorithm is not usable in LOG mode.

  hash        Takes a regular sample expression in argument. The expression
              is evaluated for each request and hashed according to the
              configured hash-type. The result of the hash is divided by
              the total weight of the running servers to designate which
              server will receive the request. This can be used in place of
              "source", "uri", "hdr()", "url_param()", "rdp-cookie" to make
              use of a converter, refine the evaluation, or be used to
              extract data from local variables for example. When the data
              is not available, round robin will apply. This algorithm is
              static by default, which means that changing a server's
              weight on the fly will have no effect, but this can be
              changed using "hash-type". This algorithm is not usable for
              backends in LOG mode, please use "log-hash" instead.

  source      The source IP address is hashed and divided by the total
              weight of the running servers to designate which server will
              receive the request. This ensures that the same client IP
              address will always reach the same server as long as no
              server goes down or up. If the hash result changes due to the
              number of running servers changing, many clients will be
              directed to a different server. This algorithm is generally
              used in TCP mode where no cookie may be inserted. It may also
              be used on the Internet to provide a best-effort stickiness
              to clients which refuse session cookies. This algorithm is
              static by default, which means that changing a server's
              weight on the fly will have no effect, but this can be
              changed using "hash-type". See also the "hash" option above.
              This algorithm is not usable for backends in LOG mode.

  uri         This algorithm hashes either the left part of the URI (before
              the question mark) or the whole URI (if the "whole" parameter
              is present) and divides the hash value by the total weight of
              the running servers. The result designates which server will
              receive the request. This ensures that the same URI will
              always be directed to the same server as long as no server
              goes up or down. This is used with proxy caches and
              anti-virus proxies in order to maximize the cache hit rate.
              Note that this algorithm may only be used in an HTTP backend.
              This algorithm is static by default, which means that
              changing a server's weight on the fly will have no effect,
              but this can be changed using "hash-type".

              This algorithm supports two optional parameters "len" and
              "depth", both followed by a positive integer number. These
              options may be helpful when it is needed to balance servers
              based on the beginning of the URI only. The "len" parameter
              indicates that the algorithm should only consider that many
              characters at the beginning of the URI to compute the hash.
              Note that having "len" set to 1 rarely makes sense since most
              URIs start with a leading "/".

              The "depth" parameter indicates the maximum directory depth
              to be used to compute the hash. One level is counted for each
              slash in the request. If both parameters are specified, the
              evaluation stops when either is reached.

              A "path-only" parameter indicates that the hashing key starts
              at the first '/' of the path. This can be used to ignore the
              authority part of absolute URIs, and to make sure that HTTP/1
              and HTTP/2 URIs will provide the same hash. See also the
              "hash" option above.

  url_param   The URL parameter specified in argument will be looked up in
              the query string of each HTTP GET request.

              If the modifier "check_post" is used, then an HTTP POST
              request entity will be searched for the parameter argument,
              when it is not found in a query string after a question mark
              ('?') in the URL. The message body will only start to be
              analyzed once either the advertised amount of data has been
              received or the request buffer is full. In the unlikely event
              that chunked encoding is used, only the first chunk is
              scanned. Parameter values separated by a chunk boundary, may
              be randomly balanced if at all. This keyword used to support
              an optional <max_wait> parameter which is now ignored.

              If the parameter is found followed by an equal sign ('=') and
              a value, then the value is hashed and divided by the total
              weight of the running servers. The result designates which
              server will receive the request.

              This is used to track user identifiers in requests and ensure
              that a same user ID will always be sent to the same server as
              long as no server goes up or down. If no value is found or if
              the parameter is not found, then a round robin algorithm is
              applied. Note that this algorithm may only be used in an HTTP
              backend. This algorithm is static by default, which means
              that changing a server's weight on the fly will have no
              effect, but this can be changed using "hash-type". See also
              the "hash" option above.

  hdr(<name>) The HTTP header <name> will be looked up in each HTTP
              request. Just as with the equivalent ACL 'hdr()' function,
              the header name in parenthesis is not case sensitive. If the
              header is absent or if it does not contain any value, the
              roundrobin algorithm is applied instead.

              An optional 'use_domain_only' parameter is available, for
              reducing the hash algorithm to the main domain part with some
              specific headers such as 'Host'. For instance, in the Host
              value "haproxy.1wt.eu", only "1wt" will be considered.

              This algorithm is static by default, which means that
              changing a server's weight on the fly will have no effect,
              but this can be changed using "hash-type". See also the
              "hash" option above.

  random
  random(<draws>)
              A random number will be used as the key for the consistent
              hashing function. This means that the servers' weights are
              respected, dynamic weight changes immediately take effect, as
              well as new server additions. Random load balancing can be
              useful with large farms or when servers are frequently added
              or removed as it may avoid the hammering effect that could
              result from roundrobin or leastconn in this situation. The
              hash-balance-factor directive can be used to further improve
              fairness of the load balancing, especially in situations
              where servers show highly variable response times. When an
              argument <draws> is present, it must be an integer value one
              or greater, indicating the number of draws before selecting
              the least loaded of these servers. It was indeed demonstrated
              that picking the least loaded of two servers is enough to
              significantly improve the fairness of the algorithm, by
              always avoiding to pick the most loaded server within a farm
              and getting rid of any bias that could be induced by the
              unfair distribution of the consistent list. Higher values N
              will take away N-1 of the highest loaded servers at the
              expense of performance. With very high values, the algorithm
              will converge towards the leastconn's result but much slower.
              In addition, for large server farms with very low loads (or
              perfect balance), comparing loads will often lead to a tie,
              so in case of equal loads between all measured servers, their
              request rate over the last second are compared, which allows
              to better balance server usage over time in the same spirit
              as roundrobin does, and smooth consistent hash unfairness.
              The default value is 2, which generally shows very good
              distribution and performance. For large farms with low loads
              (less than a few requests per second per server), it may help
              to raise it to 3 or even 4. This algorithm is also known as
              the Power of Two Random Choices and is described here:
              http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

              For backends in LOG mode, the number of draws is ignored and
              a single random is picked since there is no notion of server
              load. Random log balancing can be useful with large farms or
              when servers are frequently added or removed from the pool of
              available servers as it may avoid the hammering effect that
              could result from roundrobin in this situation.

  rdp-cookie
  rdp-cookie(<name>)
              The RDP cookie <name> (or "mstshash" if omitted) will be
              looked up and hashed for each incoming TCP request. Just as
              with the equivalent ACL 'req.rdp_cookie()' function, the name
              is not case-sensitive. This mechanism is useful as a degraded
              persistence mode, as it makes it possible to always send the
              same user (or the same session ID) to the same server. If the
              cookie is not found, the normal roundrobin algorithm is
              used instead.

              Note that for this to work, the frontend must ensure that an
              RDP cookie is already present in the request buffer. For this
              you must use 'tcp-request content accept' rule combined with
              a 'req.rdp_cookie_cnt' ACL.

              This algorithm is static by default, which means that
              changing a server's weight on the fly will have no effect,
              but this can be changed using "hash-type". See also the
              "hash" option above.

  log-hash    Takes a comma-delimited list of converters in argument. These
              converters are applied in sequence to the input log message,
              and the result will be cast as a string then hashed according
              to the configured hash-type. The resulting hash will be used
              to select the destination server among the ones declared in
              the log backend. The goal of this algorithm is to be able to
              extract a key within the final log message using string
              converters and then be able to stick to the same server thanks
              to the hash. Only "map-based" hashes are supported for now.
              This algorithm is only usable for backends in LOG mode, for
              others, please use "hash" instead.

  sticky      Tries to stick to the same server as much as possible. The
              first server in the list of available servers receives all
              the log messages. When the server goes DOWN, the next server
              in the list takes its place. When a previously DOWN server
              goes back UP it is added at the end of the list so that the
              sticky server doesn't change until it becomes DOWN.

<arguments> is an optional list of arguments which may be needed by some
            algorithms. Right now, only "url_param", "uri" and "log-hash"
            support an optional argument.

The load balancing algorithm of a backend is set to “random” when no other algorithm, mode nor option have been set. The algorithm may only be set once for each backend.

With authentication schemes that require the same connection like NTLM, URI based algorithms must not be used, as they would cause subsequent requests to be routed to different backend servers, breaking the invalid assumptions NTLM relies on.

TCP/HTTP Examples:

balance roundrobin
balance url_param userid
balance url_param session_id check_post 64
balance hdr(User-Agent)
balance hdr(host)
balance hdr(Host) use_domain_only
balance hash req.cookie(clientid)
balance hash var(req.client_id)
balance hash req.hdr_ip(x-forwarded-for,-1),ipmask(24)

LOG backend examples:

global
  log backend@mylog-rrb local0 # send all logs to mylog-rrb backend
  log backend@mylog-hash local0 # send all logs to mylog-hash backend

backend mylog-rrb
  mode log
  balance roundrobin

  server s1 udp@127.0.0.1:514 # will receive 50% of log messages
  server s2 udp@127.0.0.1:514

backend mylog-hash
  mode log

  # extract "METHOD URL PROTO" at the end of the log message,
  # and let haproxy hash it so that log messages generated from
  # similar requests get sent to the same syslog server:
  balance log-hash 'field(-2,\")'

  # server list here
  server s1 127.0.0.1:514
  #...

Note: the following caveats and limitations on using the “check_post” extension with “url_param” must be considered:

- all POST requests are eligible for consideration, because there is no way
  to determine if the parameters will be found in the body or entity which
  may contain binary data. Therefore another method may be required to
  restrict consideration of POST requests that have no URL parameters in
  the body. (see acl http_end)

- using a `<max_wait>` value larger than the request buffer size does not
  make sense and is useless. The buffer size is set at build time, and
  defaults to 16 kB.

- Content-Encoding is not supported, the parameter search will probably
  fail; and load balancing will fall back to Round Robin.

- Expect: 100-continue is not supported, load balancing will fall back to
  Round Robin.

- Transfer-Encoding (RFC7230 3.3.1) is only supported in the first chunk.
  If the entire parameter value is not present in the first chunk, the
  selection of server is undefined (actually, defined by how little
  actually appeared in the first chunk).

- This feature does not support generation of a 100, 411 or 501 response.

- In some cases, requesting "check_post" MAY attempt to scan the entire
  contents of a message body. Scanning normally terminates when linear
  white space or control characters are found, indicating the end of what
  might be a URL parameter list. This is probably not a concern with SGML
  type message bodies.

See also: “dispatch”, “cookie”, “transparent”, “hash-type”.

be-unpublished

be-unpublished

Instructs the backend to start in the unpublished state.

May be used in the following contexts: tcp, http, log

May be used in sections: defaults | frontend | listen | backend no | no | yes | yes

In effect, any use_backend and default_backend rules from another proxy which reference the current proxy are ignored and the next content switching rules are evaluated. This can be bypassed though via a “force-be-switch” rule.

This state is similar to the disabled one but with some differences. First, an unpublished backend will still be fully initialized, including the server health checks which remain active. Finally, a backend can be publicly exposed via the command “publish backend” on the CLI. See the management manual.

See also: “force-be-switch”

bind [<address>]:<port_range> [, ...] [param*]

bind [<address>]:<port_range> [, ...] [param*]
bind /<path> [, ...] [param*]

Define one or several listening addresses and/or ports in a frontend.

May be used in the following contexts: tcp, http

May be used in sections: defaults | frontend | listen | backend no | yes | yes | no

Arguments:

<address>     is optional and can be a host name, an IPv4 address, an IPv6
              address, or '*'. It designates the address the frontend will
              listen on. If unset, all IPv4 addresses of the system will be
              listened on. The same will apply for '*' or the system's
              special address "0.0.0.0". The IPv6 equivalent is '::'. Note
              that for UDP, specific OS features are required when binding
              on multiple addresses to ensure the correct network interface
              and source address will be used on response. In other way,
              for QUIC listeners only bind on multiple addresses if running
              with a modern enough systems.

              Optionally, an address family prefix may be used before the
              address to force the family regardless of the address format,
              which can be useful to specify a path to a unix socket with
              no slash ('/'). Currently supported prefixes are:
                - 'ipv4@'  -> address is always IPv4
                - 'ipv6@'  -> address is always IPv6
                - 'udp@'   -> address is resolved as IPv4 or IPv6 and
                  protocol UDP is used. Currently those listeners are
                  supported only in log-forward sections.
                - 'udp4@'  -> address is always IPv4 and protocol UDP
                  is used. Currently those listeners are supported
                  only in log-forward sections.
                - 'udp6@'  -> address is always IPv6 and protocol UDP
                  is used. Currently those listeners are supported
                  only in log-forward sections.
                - 'unix@'  -> address is a path to a local unix socket
                - 'abns@'  -> address is in abstract namespace (Linux only).
                - 'abnsz@'  -> address is in abstract namespace (Linux only)
                   but it is explicitly zero-terminated. This means no \0
                   padding is used to complete sun_path. It is useful to
                   interconnect with programs that don't implement the
                   default abns naming logic that haproxy uses.
                - 'fd@<n>' -> use file descriptor <n> inherited from the
                  parent. The fd must be bound and may or may not already
                  be listening.
                - 'sockpair@<n>'-> like fd@ but you must use the fd of a
                  connected unix socket or of a socketpair. The bind waits
                  to receive a FD over the unix socket and uses it as if it
                  was the FD of an accept(). Should be used carefully.
                - 'quic4@' -> address is resolved as IPv4 and protocol UDP
                  is used. Note that to achieve the best performance with a
                  large traffic you should keep "tune.quic.fe.sock-per-conn
                  default-on". Else QUIC connections will be multiplexed
                  over the listener socket. Another alternative would be to
                  duplicate QUIC listener instances over several threads,
                  for example using "shards" keyword to at least reduce
                  thread contention.
                - 'quic6@' -> address is resolved as IPv6 and protocol UDP
                  is used. The performance note for QUIC over IPv4 applies
                  as well.
                - 'rhttp@' [ EXPERIMENTAL ] -> used for reverse HTTP.
                  Address must be a server with the format
                  '<backend>/<server>'. The server will be used to
                  instantiate connections to a remote address. The listener
                  will try to maintain "nbconn" connections. This is an
                  experimental features which requires
                  "expose-experimental-directives" on a line before this
                  bind.

              You may want to reference some environment variables in the
              address parameter, see section 2.3 about environment
              variables.

<port_range>  is either a unique TCP port, or a port range for which the
              proxy will accept connections for the IP address specified
              above. The port is mandatory for TCP listeners. Note that in
              the case of an IPv6 address, the port is always the number
              after the last colon (':'). A range can either be:
               - a numerical port (ex: '80')
               - a dash-delimited ports range explicitly stating the lower
                 and upper bounds (ex: '2000-2100') which are included in
                 the range.

              Particular care must be taken against port ranges, because
              every <address:port> couple consumes one socket (= a file
              descriptor), so it's easy to consume lots of descriptors
              with a simple range, and to run out of sockets. Also, each
              <address:port> couple must be used only once among all
              instances running on a same system. Please note that binding
              to ports lower than 1024 generally require particular
              privileges to start the program, which are independent of
              the 'uid' parameter.

<path>        is a UNIX socket path beginning with a slash ('/'). This is
              alternative to the TCP listening port. HAProxy will then
              receive UNIX connections on the socket located at this place.
              The path must begin with a slash and by default is absolute.
              It can be relative to the prefix defined by "unix-bind" in
              the global section. Note that the total length of the prefix
              followed by the socket path cannot exceed some system limits
              for UNIX sockets, which commonly are set to 107 characters.

<param*>      is a list of parameters common to all sockets declared on the
              same line. These numerous parameters depend on OS and build
              options and have a complete section dedicated to them. Please
              refer to section 5 to for more details.

It is possible to specify a list of address:port combinations delimited by commas. The frontend will then listen on all of these addresses. There is no fixed limit to the number of addresses and ports which can be listened on in a frontend, as well as there is no limit to the number of “bind” statements in a frontend.

Example:

listen http_proxy
    bind:80,:443
    bind 10.0.0.1:10080,10.0.0.1:10443
    bind /var/run/ssl-frontend.sock user root mode 600 accept-proxy

listen http_https_proxy
    bind:80
    bind:443 ssl crt /etc/haproxy/site.pem

listen http_https_proxy_explicit
    bind ipv6@:80
    bind ipv4@public_ssl:443 ssl crt /etc/haproxy/site.pem
    bind unix@ssl-frontend.sock user root mode 600 accept-proxy

listen external_bind_app1
    bind "fd@${FD_APP1}"

listen h3_quic_proxy
    bind quic4@10.0.0.1:8888 ssl crt /etc/mycrt

Note: regarding Linux’s abstract namespace sockets, “abns” HAProxy sockets uses the whole sun_path length is used for the address length. Some other programs such as socat use the string length only by default. Pass the option “,unix-tightsocklen=0” to any abstract socket definition in socat to make it compatible with HAProxy’s, or use the “abnsz” HAProxy socket family instead.

See also: “source”, “option forwardfor”, “unix-bind” and the PROXY protocol documentation, and section 5 about bind options.

capture cookie <name> len <length>

capture cookie <name> len <length>

Capture and log a cookie in the request and in the response.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend no | yes | yes | no

Arguments:

<name>    is the beginning of the name of the cookie to capture. In order
          to match the exact name, simply suffix the name with an equal
          sign ('='). The full name will appear in the logs, which is
          useful with application servers which adjust both the cookie name
          and value (e.g. ASPSESSIONXXX).

<length>  is the maximum number of characters to report in the logs, which
          include the cookie name, the equal sign and the value, all in the
          standard "name=value" form. The string will be truncated on the
          right if it exceeds <length>.

Only the first cookie is captured. Both the “cookie” request headers and the “set-cookie” response headers are monitored. This is particularly useful to check for application bugs causing session crossing or stealing between users, because generally the user’s cookies can only change on a login page.

When the cookie was not presented by the client, the associated log column will report “-”. When a request does not cause a cookie to be assigned by the server, a “-” is reported in the response column.

The capture is performed in the frontend only because it is necessary that the log format does not change for a given frontend depending on the backends. This may change in the future. Note that there can be only one “capture cookie” statement in a frontend. The maximum capture length is set by the global “tune.http.cookielen” setting and defaults to 63 characters. It is not possible to specify a capture in a “defaults” section.

Example:

capture cookie ASPSESSION len 32

See also: “capture request header”, “capture response header” as well as section 8 about logging.

capture request header <name> len <length>

capture request header <name> len <length>

Capture and log the last occurrence of the specified request header.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend no | yes | yes | no

Arguments:

<name>    is the name of the header to capture. The header names are not
          case-sensitive, but it is a common practice to write them as they
          appear in the requests, with the first letter of each word in
          upper case. The header name will not appear in the logs, only the
          value is reported, but the position in the logs is respected.

<length>  is the maximum number of characters to extract from the value and
          report in the logs. The string will be truncated on the right if
          it exceeds <length>.

The complete value of the last occurrence of the header is captured. The value will be added to the logs between braces (’{}’). If multiple headers are captured, they will be delimited by a vertical bar (’|’) and will appear in the same order they were declared in the configuration. Non-existent headers will be logged just as an empty string. Common uses for request header captures include the “Host” field in virtual hosting environments, the “Content-length” when uploads are supported, “User-agent” to quickly differentiate between real users and robots, and “X-Forwarded-For” in proxied environments to find where the request came from.

Note that when capturing headers such as “User-agent”, some spaces may be logged, making the log analysis more difficult. Thus be careful about what you log if you know your log parser is not smart enough to rely on the braces.

There is no limit to the number of captured request headers nor to their length, though it is wise to keep them low to limit memory usage per stream. In order to keep log format consistent for a same frontend, header captures can only be declared in a frontend. It is not possible to specify a capture in a “defaults” section.

Example:

capture request header Host len 15
capture request header X-Forwarded-For len 15
capture request header Referer len 15

See also: “capture cookie”, “capture response header” as well as section 8 about logging.

capture response header <name> len <length>

capture response header <name> len <length>

Capture and log the last occurrence of the specified response header.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend no | yes | yes | no

Arguments:

<name>    is the name of the header to capture. The header names are not
          case-sensitive, but it is a common practice to write them as they
          appear in the response, with the first letter of each word in
          upper case. The header name will not appear in the logs, only the
          value is reported, but the position in the logs is respected.

<length>  is the maximum number of characters to extract from the value and
          report in the logs. The string will be truncated on the right if
          it exceeds <length>.

The complete value of the last occurrence of the header is captured. The result will be added to the logs between braces (’{}’) after the captured request headers. If multiple headers are captured, they will be delimited by a vertical bar (’|’) and will appear in the same order they were declared in the configuration. Non-existent headers will be logged just as an empty string. Common uses for response header captures include the “Content-length” header which indicates how many bytes are expected to be returned, the “Location” header to track redirections.

There is no limit to the number of captured response headers nor to their length, though it is wise to keep them low to limit memory usage per stream. In order to keep log format consistent for a same frontend, header captures can only be declared in a frontend. It is not possible to specify a capture in a “defaults” section.

Example:

capture response header Content-length len 9
capture response header Location len 15

See also: “capture cookie”, “capture request header” as well as section 8 about logging.

clitcpka-cnt <count>

clitcpka-cnt <count>

Sets the maximum number of keepalive probes TCP should send before dropping the connection on the client side.

May be used in the following contexts: tcp, http, log

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | no

Arguments:

<count>   is the maximum number of keepalive probes.

This keyword corresponds to the socket option TCP_KEEPCNT. If this keyword is not specified, system-wide TCP parameter (tcp_keepalive_probes) is used. The availability of this setting depends on the operating system. It is known to work on Linux.

See also: “option clitcpka”, “clitcpka-idle”, “clitcpka-intvl”.

clitcpka-idle <timeout>

clitcpka-idle <timeout>

Sets the time the connection needs to remain idle before TCP starts sending keepalive probes, if enabled the sending of TCP keepalive packets on the client side.

May be used in the following contexts: tcp, http

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | no

Arguments:

<timeout> is the time the connection needs to remain idle before TCP starts
          sending keepalive probes. It is specified in seconds by default,
          but can be in any other unit if the number is suffixed by the
          unit, as explained at the top of this document.

This keyword corresponds to the socket option TCP_KEEPIDLE. If this keyword is not specified, system-wide TCP parameter (tcp_keepalive_time) is used. The availability of this setting depends on the operating system. It is known to work on Linux.

See also: “option clitcpka”, “clitcpka-cnt”, “clitcpka-intvl”.

clitcpka-intvl <timeout>

clitcpka-intvl <timeout>

Sets the time between individual keepalive probes on the client side.

May be used in the following contexts: tcp, http

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | no

Arguments:

<timeout> is the time between individual keepalive probes. It is specified
          in seconds by default, but can be in any other unit if the number
          is suffixed by the unit, as explained at the top of this
          document.

This keyword corresponds to the socket option TCP_KEEPINTVL. If this keyword is not specified, system-wide TCP parameter (tcp_keepalive_intvl) is used. The availability of this setting depends on the operating system. It is known to work on Linux.

See also: “option clitcpka”, “clitcpka-cnt”, “clitcpka-idle”.

compression algo <algorithm> ...

compression algo <algorithm> ...
compression algo-req <algorithm>
compression algo-res <algorithm>
compression type <mime type> ...

Enable HTTP compression.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | yes

Arguments:

algo     is followed by the list of supported compression algorithms for
         responses (legacy keyword)
algo-req is followed by compression algorithm for request (only one is
  provided).
algo-res is followed by the list of supported compression algorithms for
         responses.
type     is followed by the list of MIME types that will be compressed for
         responses (legacy keyword).
type-req is followed by the list of MIME types that will be compressed for
         requests.
type-res is followed by the list of MIME types that will be compressed for
         responses.

The currently supported algorithms are:

identity     this is mostly for debugging, and it was useful for developing
             the compression feature. Identity does not apply any change on
             data.

gzip         applies gzip compression. This setting is only available when
             support for zlib or libslz was built in.

deflate      same as "gzip", but with deflate algorithm and zlib format.
             Note that this algorithm has ambiguous support on many
             browsers and no support at all from recent ones. It is
             strongly recommended not to use it for anything else than
             experimentation. This setting is only available when support
             for zlib or libslz was built in.

raw-deflate  same as "deflate" without the zlib wrapper, and used as an
             alternative when the browser wants "deflate". All major
             browsers understand it and despite violating the standards,
             it is known to work better than "deflate", at least on MSIE
             and some versions of Safari. Do not use it in conjunction
             with "deflate", use either one or the other since both react
             to the same Accept-Encoding token. This setting is only
             available when support for zlib or libslz was built in.

Compression will be activated depending on the Accept-Encoding request header. With identity, it does not take care of that header. If backend servers support HTTP compression, these directives will be no-op: HAProxy will see the compressed response and will not compress again. If backend servers do not support HTTP compression and there is Accept-Encoding header in request, HAProxy will compress the matching response.

Compression is disabled when: * the request does not advertise a supported compression algorithm in the “Accept-Encoding” header * the response message is not HTTP/1.1 or above * HTTP status code is not one of 200, 201, 202, or 203 * response contain neither a “Content-Length” header nor a “Transfer-Encoding” whose last value is “chunked” * response contains a “Content-Type” header whose first value starts with “multipart” * the response contains the “no-transform” value in the “Cache-control” header * User-Agent matches “Mozilla/4” unless it is MSIE 6 with XP SP2, or MSIE 7 and later * The response contains a “Content-Encoding” header, indicating that the response is already compressed (see compression offload) * The response contains an invalid “ETag” header or multiple ETag headers * The payload size is smaller than the minimum size (see compression minsize-res)

Note: The compression does not emit the Warning header.

Examples:

compression algo gzip
compression type text/html text/plain

See also: “compression offload”, “compression direction”, “compression minsize-req” and “compression minsize-res”

compression minsize-req <size>

compression minsize-req <size>
compression minsize-res <size>

Sets the minimum payload size in bytes for compression to be applied.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend yes | yes | yes | yes

Payloads smaller than this size will not be compressed, avoiding unnecessary CPU overhead for data that would not significantly benefit from compression. “minsize-req” applies on requests and “minsize-res” on responses. The default value is 0.

compression offload

compression offload

Makes HAProxy work as a compression offloader only.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend no | yes | yes | yes

The “offload” setting makes HAProxy remove the Accept-Encoding header to prevent backend servers from compressing responses. It is strongly recommended not to do this because this means that all the compression work will be done on the single point where HAProxy is located. However in some deployment scenarios, HAProxy may be installed in front of a buggy gateway with broken HTTP compression implementation which can’t be turned off. In that case HAProxy can be used to prevent that gateway from emitting invalid payloads. In this case, simply removing the header in the configuration does not work because it applies before the header is parsed, so that prevents HAProxy from compressing. The “offload” setting should then be used for such scenarios.

If this setting is used in a defaults section, a warning is emitted and the option is ignored.

See also: “compression type”, “compression algo”, “compression direction”

compression direction <direction> (deprecated)

compression direction <direction> (deprecated)

Makes haproxy able to compress both requests and responses. Valid values are “request”, to compress only requests, “response”, to compress only responses, or “both”, when you want to compress both. The default value is “response”.

This directive is only relevant when legacy “filter compression” was enabled, as with explicit comp-req and comp-res filters compression direction is redundant.

May be used in the following contexts: http

See also: “compression type”, “compression algo”, “compression offload”

cookie <name> [ rewrite | insert | prefix ] [ indirect ] [ nocache ]

cookie <name> [ rewrite | insert | prefix ] [ indirect ] [ nocache ]
              [ postonly ] [ preserve ] [ httponly ] [ secure ]
              [ domain <domain> ]* [ maxidle <idle> ] [ maxlife <life> ]
              [ dynamic ] [ attr <value> ]*

Enable cookie-based persistence in a backend.

May be used in the following contexts: http

May be used in sections: defaults | frontend | listen | backend yes | no | yes | yes

Arguments:

<name>    is the name of the cookie which will be monitored, modified or
          inserted in order to bring persistence. This cookie is sent to
          the client via a "Set-Cookie" header in the response, and is
          brought back by the client in a "Cookie" header in all requests.
          Special care should be taken to choose a name which does not
          conflict with any likely application cookie. Also, if the same
          backends are subject to be used by the same clients (e.g.
          HTTP/HTTPS), care should be taken to use different cookie names
          between all backends if persistence between them is not desired.

rewrite   This keyword indicates that the cookie will be provided by the
          server and that HAProxy will have to modify its value to set the
          server's identifier in it. This mode is handy when the management
          of complex combinations of "Set-cookie" and "Cache-control"
          headers is left to the application. The application can then
          decide whether or not it is appropriate to emit a persistence
          cookie. Since all responses should be monitored, this mode
          doesn't work in HTTP tunnel mode. Unless the application
          behavior is very complex and/or broken, it is advised not to
          start with this mode for new deployments. This keyword is
          incompatible with "insert" and "prefix".

insert    This keyword indicates that the persistence cookie will have to
          be inserted by HAProxy in server responses if the client did not

          already have a cookie that would have permitted it to access this
          server. When used without the "preserve" option, if the server
          emits a cookie with the same name, it will be removed before
          processing. For this reason, this mode can be used to upgrade
          existing configurations running in the "rewrite" mode. The cookie
          will only be a session cookie and will not be stored on the
          client's disk. By default, unless the "indirect" option is added,
          the server will see the cookies emitted by the client. Due to
          caching effects, it is generally wise to add the "nocache" or
          "postonly" keywords (see below). The "insert" keyword is not
          compatible with "rewrite" and "prefix".

prefix    This keyword indicates that instead of relying on a dedicated
          cookie for the persistence, an existing one will be completed.
          This may be needed in some specific environments where the client
          does not support more than one single cookie and the application
          already needs it. In this case, whenever the server sets a cookie
          named <name>, it will be prefixed with the server's identifier
          and a delimiter. The prefix will be removed from all client
          requests so that the server still finds the cookie it emitted.
          Since all requests and responses are subject to being modified,
          this mode doesn't work with tunnel mode. The "prefix" keyword is
          not compatible with "rewrite" and "insert". Note: it is highly
          recommended not to use "indirect" with "prefix", otherwise server
          cookie updates would not be sent to clients.

indirect  When this option is specified, no cookie will be emitted to a
          client which already has a valid one for the server which has
          processed the request. If the server sets such a cookie itself,
          it will be removed, unless the "preserve" option is also set. In
          "insert" mode, this will additionally remove cookies from the
          requests transmitted to the server, making the persistence
          mechanism totally transparent from an application point of view.
          Note: it is highly recommended not to use "indirect" with
          "prefix", otherwise server cookie updates would not be sent to
          clients.

nocache   This option is recommended in conjunction with the insert mode
          when there is a cache between the client and HAProxy, as it
          ensures that a cacheable response will be tagged non-cacheable if
          a cookie needs to be inserted. This is important because if all
          persistence cookies are added on a cacheable home page for
          instance, then all customers will then fetch the page from an
          outer cache and will all share the same persistence cookie,
          leading to one server receiving much more traffic than others.
          See also the "insert" and "postonly" options.

postonly  This option ensures that cookie insertion will only be performed
          on responses to POST requests. It is an alternative to the
          "nocache" option, because POST responses are not cacheable, so
          this ensures that the persistence cookie will never get cached.
          Since most sites do not need any sort of persistence before the
          first POST which generally is a login request, this is a very
          efficient method to optimize caching without risking to find a
          persistence cookie in the cache.
          See also the "insert" and "nocache" options.

preserve  This option may only be used with "insert" and/or "indirect". It
          allows the server to emit the persistence cookie itself. In this
          case, if a cookie is found in the response, HAProxy will leave it
          untouched. This is useful in order to end persistence after a
          logout request for instance. For this, the server just has to
          emit a cookie with an invalid value (e.g. empty) or with a date in
          the past. By combining this mechanism with the "disable-on-404"
          check option, it is possible to perform a completely graceful
          shutdown because users will definitely leave the server after
          they logout.

httponly  This option tells HAProxy to add an "HttpOnly" cookie attribute
          when a cookie is inserted. This attribute is used so that a
          user agent doesn't share the cookie with non-HTTP components.
          Please check RFC6265 for more information on this attribute.

secure    This option tells HAProxy to add a "Secure" cookie attribute when
          a cookie is inserted. This attribute is used so that a user agent
          never emits this cookie over non-secure channels, which means
          that a cookie learned with this flag will be presented only over
          SSL/TLS connections. Please check RFC6265 for more information on
          this attribute.

domain    This option allows to specify the domain at which a cookie is
          inserted. It requires exactly one parameter: a valid domain
          name. If the domain begins with a dot, the browser is allowed to
          use it for any host ending with that name. It is also possible to
          specify several domain names by invoking this option multiple
          times. Some browsers might have small limits on the number of
          domains, so be careful when doing that. For the record, sending
          10 domains to MSIE 6 or Firefox 2 works as expected.

maxidle   This option allows inserted cookies to be ignored after some idle
          time. It only works with insert-mode cookies. When a cookie is
          sent to the client, the date this cookie was emitted is sent too.
          Upon further presentations of this cookie, if the date is older
          than the delay indicated by the parameter (in seconds), it will
          be ignored. Otherwise, it will be refreshed if needed when the
          response is sent to the client. This is particularly useful to
          prevent users who never close their browsers from remaining for
          too long on the same server (e.g. after a farm size change). When
          this option is set and a cookie has no date, it is always
          accepted, but gets refreshed in the response. This maintains the
          ability for admins to access their sites. Cookies that have a
          date in the future further than 24 hours are ignored. Doing so
          lets admins fix timezone issues without risking kicking users off
          the site.

maxlife   This option allows inserted cookies to be ignored after some life
          time, whether they're in use or not. It only works with insert
          mode cookies. When a cookie is first sent to the client, the date
          this cookie was emitted is sent too. Upon further presentations
          of this cookie, if the date is older than the delay indicated by
          the parameter (in seconds), it will be ignored. If the cookie in
          the request has no date, it is accepted and a date will be set.
          Cookies that have a date in the future further than 24 hours are
          ignored. Doing so lets admins fix timezone issues without risking
          kicking users off the site. Contrary to maxidle, this value is
          not refreshed, only the first visit date counts. Both maxidle and
          maxlife may be used at the time. This is particularly useful to
          prevent users who never close their browsers from remaining for
          too long on the same server (e.g. after a farm size change). This
          is stronger than the maxidle method in that it forces a
          redispatch after some absolute delay.

dynamic   Activate dynamic cookies. When used, a session cookie is
          dynamically created for each server, based on the IP and port
          of the server, and a secret key, specified in the
          "dynamic-cookie-key" backend directive.
          The cookie will be regenerated each time the IP address change,
          and is only generated for IPv4/IPv6.

attr      This option tells HAProxy to add an extra attribute when a
          cookie is inserted. The attribute value can contain any
          characters except control ones or ";". This option may be
          repeated.