Running Science Experiments in Production with Scientist

We tried to replace a slow search query with a fast database query. It was 91% faster. It was also wrong 4% of the time.

Scientist caught the mismatch before users saw it. While working with the experiment data, I also found that the gem could report wall time but not CPU time, so I added that measurement.

The experiment that failed

The repo page investigation found a search query adding 198ms to logged-in page loads. A database EXISTS query took about 2ms, so it looked like a straightforward replacement.

We set up a Scientist experiment to run both implementations side by side in production:

science "might-have-packages" do |e|
  e.use   { existing_search_query }   # control: always returned
  e.try   { new_database_query }      # candidate: run & compared
end

Scientist returned the control result to the user and recorded whether the candidate agreed with it. We started at 1% of traffic. The mismatch rate settled at 3-4%.

The database query only checked for one type of package, while the search service knew about all of them. Repositories with non-Maven packages accounted for the mismatches.

We reverted the experiment.

(The eventual fix was even better: remove the check entirely.)

Adding CPU time

The experiment also exposed a gap in the measurements. I could compare wall-clock duration, but I wanted to compare how much CPU work each implementation did.

Wall-clock time includes I/O waits, GC pauses, and time spent waiting for a process to be scheduled. A method can take 200ms to return while using only 5ms of CPU. That distinction matters when the goal is to free CPU capacity across a fleet.

Scientist did not track CPU time, so I added it.

The implementation

The change captures Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID) at the start and end of each observation, alongside the existing wall-clock timing:

science "my-experiment" do |e|
  e.use { control_code }
  e.try { candidate_code }
end

# Each observation now includes:
# observation.duration  -> wall time
# observation.cpu_time  -> CPU time only

Each observation can now report whether a candidate used less CPU, rather than only whether it returned sooner.

Why I contributed it

I did not start with a plan to contribute to Scientist. I was trying to answer a question during an investigation, found that the data was missing, and added it to the tool we were already using.