Cutting CI running tests from 11 minutes to 3

Ever had one of those issues that dragged on for months and no one knew why it was happening? This is one of those stories. In meeting after meeting, colleagues kept mentioning the same thing: a big gap between running our main API's CI and running the same tests locally. In CI the suite took 10–11 minutes. Locally, it took around 1.5–2 minutes. A gap that large is deeply suspicious - it almost always means something is behaving differently between the two environments.

01How I investigated

I started by ruling things out. It wasn't the Docker image itself: the image we build locally is the very same one we run tests against and push to production. If the container is identical in both places, the container isn't the variable.

That pointed me at the code. My hunch was that the problem was hiding in one of our internal Ruby gems - and that hunch turned out to be right.

The culprit was our internal HTTP client: a thin wrapper over Faraday, built as a middleware stack. Its responsibilities include a circuit breaker, an HTTP cache, and - the key to this story - a retry middleware. Exactly what you want in production, and exactly what you don't want in a test environment.

Digging deeper into the wall-clock time inside CI, it turned out we were doing DNS SRV record lookups on top of those retries. A single unresolved call could cost around 15 seconds; multiply that across thousands of tests and it adds up to eight or nine minutes of pure waiting.

The API was using the WebMock gem to block any outbound connections, but it did not matter - because WebMock intercepts at the HTTP-adapter layer, and the expensive work happened before any HTTP request ever existed. The DNS SRV lookups (four retries with exponential backoff) run earlier, in the service-discovery middleware, while the client is still resolving where to connect. By the time there was a connection for WebMock to block, the seconds had already been spent on DNS.

02How I fixed it

The fix was pretty straightforward: rather than hunting down every unstubbed call, I made the client itself aware that it's running in a test environment. Detection keys off the RACK_ENV or RAILS_ENV variables - our main API is a Rack app, while our Rails services set RAILS_ENV - so the flag lights up automatically in every consumer.

module ServiceClient
  class Configuration
    attr_accessor :user_agent, :test_mode

    def initialize
      @test_mode = %w[RACK_ENV RAILS_ENV].any? { |k| ENV[k] == "test" }
    end

    def test_mode?
      @test_mode
    end
  end
end

On Faraday's side, that single flag flips the defaults: a shorter timeout, a lighter adapter, no HTTP cache, no circuit breaker, and no retries. Every value stays overridable per connection - test mode only changes the defaults.

def initialize(options = {})
  test_mode = ServiceClient.configuration.test_mode?

  # test_mode flips the defaults - callers can still override any of these.
  timeout       = options.fetch(:timeout,       test_mode ? 1 : DEFAULT_TIMEOUT)
  adapter       = options.fetch(:adapter,       test_mode ? :net_http : :httpclient)
  disable_cache = options.fetch(:disable_cache, test_mode)
  retry_enabled = options.fetch(:retry_enabled, !test_mode)

  @faraday = Faraday.new(site) do |builder|
    builder.use ServiceLookup
    builder.use Faraday::HttpCache          unless disable_cache
    builder.use Circuitbox::FaradayMiddleware unless test_mode
    builder.request :retry                   if retry_enabled
    builder.adapter adapter
  end
end

The most expensive piece was the DNS SRV lookups. In test mode there's no service registry to resolve against, so the right thing to do is fail instantly rather than retry into a timeout.

def resolve_service(service_id)
  # No SRV registry in tests - fail fast instead of retrying into a timeout.
  raise DnsLookupError, "SRV lookup disabled in test mode" if test_mode?

  tries   = 4        # attempts before giving up
  backoff = 0.1      # initial delay in seconds; doubles each retry (0.1s, 0.2s, 0.4s...)
end

03The result

The result was striking: CI test execution came down to match local execution. And because this pipeline runs on every push and merge request, the saving compounds. Since 20 March 2026 it has run 152 times, each run handing back eight or nine minutes that used to be spent waiting on a network that would never answer.

Before - the CI run

After - the same jobs

04What I learned

• Don't rely solely on Docker to guarantee your CI matches your local test environment. An identical image still runs inside a very different network.
• Be wary of middleware that does retries and exponential backoff in tests. You almost never want that machinery running against a network that isn't there.
• Production resilience has a hidden cost. The same code that keeps you up in production can quietly make your CI slower - without you realising it.

Code samples above are illustrative reconstructions of the pattern, written for this article - not the original proprietary source. Names and internal details have been generalised.