HAProxy dynamic backend selection with Lua script

HAProxy is a popular load balancer with extensive configuration options, including the ability to influence balancing and other options via Lua scripts.

In this post i’ll show how it’s possible to influence HAProxy backend selection via a Lua script. The use case for this situation was the necessity to choose a backend server based off of the responses from each possible backend.

First, Lua needs to be installed and HAProxy needs to be installed with Lua support. This involves building HAProxy with USE_LUA=1 environment var set during make.

Here’s a stripped down config example, with relevant lines commented. Not all required config attributes are included for brevity.

global
    # Load custom lua script. I usually put this alongside the haproxy.conf.
    lua-load /etc/haproxy/pick_backend.lua 

# Frontend config, rtmp traffic.
frontend frontendrtmp
    bind *:1935
    mode tcp

    # inspect-delay was required or else was seeing timeouts during lua script run
    tcp-request inspect-delay 1m

    # This line intercepts the incoming tcp request and pipes it through lua function, called "pick backend"
    tcp-request content lua.pick_backend

    # use_backend based off of the "streambackend" response variable we inject via lua script
    use_backend %[var(req.streambackend)]


# Example backends. One server per backend. The Lua script will iterate through all backends
# with "backendrtmp" prefix. 
# HAProxy use_server attribute does not yet support lua scripts, so backends necessary.
backend backendrtmp1
    mode tcp
    server rtmp 123.456.789.0:1935 check

Requests to the “frontendrtmp” frontend are routed through the Lua script, which checks each listed backend and chooses one based off its response.

Here’s the Lua script:

local function pick_backend(txn)
    winner_name = 'backendrtmp1' -- Needs to match available backend.
    winner_count = -1 ---initial count flag

    for backend_name ,v in pairs(core.backends) do
      if (backend_name ~= 'MASTER') then -- Filter out built in backend name
        -- iterate backend servers dict, assuming one server per backend.
        for server_name, server in pairs(v.servers) do
          -- Skip any server that is down.
          if server:get_stats()['status'] ~= 'DOWN' then
            address = string.match(server:get_addr(), '%d+.%d+.%d+.%d+')
            local tcp = core.tcp()
            tcp:settimeout(1)

            -- Connect to rtmp server to get stats counts.
            if tcp:connect(address, 80) then
              if tcp:send('GET /statistics\r\n') then
                local line, _ = tcp:receive('*a')

                -- Do whatever checks you want here with the response.
                -- In this case, i'll just check the number returned
                -- from the statistics endpoint.
                streamers = tonumber(string.match(line, '(%d+)'))

                -- Check and set winner.
                if (winner_count == -1) then
                  print('Set initial backend', backend_name)
                  winner_count = streamers
                  winner_backend = backend_name
                else
                  if (streamers < winner_count) then
                    print('New winner', backend_name)
                    winner_count = streamers
                    winner_backend = backend_name
                  end
                end
              end
              tcp:close()
            else
              print('Socket connection failed')
            end
          end
        end
      end
    end
    print('Winner is:', winner_backend)

    -- Set winner backend name to variable on the request.
    txn:set_var('req.streambackend', winner_backend)
end

core.register_action('pick_backend', {'tcp-req', 'http-req'}, pick_backend)

The Lua script: