Methodology

bgbotarena tries to answer one question: in a given format and with given settings, which of two backgammon engines is stronger, and how sure can we be? Each part of the process is set up so that you don't have to take our word for the results.

Rooms

A room is one experiment. It has two bots, each with fixed settings (search depth, network, and so on), playing one format (an N-point match, or money games with or without the Jacoby rule). Settings never change once a room starts. A change means a new room, so every number in a room comes from the same conditions.

Most matches in a room are played at full engine speed. One match at a time is the showcase: it is paced so that you can watch it, and it counts toward the statistics like every other match. The bots alternate colours from match to match, and the opening roll decides who moves first, as in normal play.

Propositions

Some rooms don't start from the opening. They play propositions: every match starts from a set position, such as a deep backgame or a containment game, to show which bot handles that kind of position better. The position fixes the board, who is on roll (and the roll, if it has been rolled), the cube and, in match play, the score. A money proposition is one game; a match proposition is played out until the match ends.

Positions are rarely even, so the bots play them in pairs: bot A takes one side and bot B the other, then they swap, each game with fresh dice. Over a pair, whatever the position itself is worth cancels out, and what remains is the difference between the bots. A room can hold several positions of the same type; each pair uses one of them.

For each position the room also shows how each bot did on each side, for example how often the backgame side wins when each bot plays it.

Dice you can check

Engines never roll their own dice; a referee does. Each match has its own 256-bit seed. The referee turns the seed into rolls with ChaCha20 (RFC 8439):

The seed is the key, the nonce is zero and the block counter starts at zero.
The keystream is read byte by byte. A byte b below 252 gives the die b mod 6 + 1, and other bytes are skipped.

Before the first roll, the SHA-256 hash of the seed is published. When the match ends, the seed itself is revealed (in the match list and in the match record's header). Anyone can then check:

that the hash matches the seed;
that the seed produces every roll in the game record.

Nobody, including us, could choose rolls after seeing a position.

There is no variance reduction such as duplicate dice. Whether a roll is good depends entirely on the position, so every game is a plain, independent playout.

Fair conditions

Each engine runs in its own container, pinned to its own CPU core, with networking disabled.
Every decision has the same time limit for both bots.
Each match records the exact engine version and settings each bot reported when it started.

Sometimes an engine fails: an illegal move, a timeout or a crash. The match is then marked as an error and left out of the statistics, and the error is shown in the room.

Search depth

Engines count their search depth differently. gnubg calls its network's own evaluation 0-ply, while eXtremeGammon, Open Sage and DarkBGHelmet call the same thing 1-ply. So “2-ply” doesn't mean the same search in every engine. The arena labels every bot on one scale, XG's: 1-ply is the network alone, 2-ply looks one roll ahead, 3-ply two rolls, and so on. A room's bot settings also show the engine's own setting.

bgmagic: plies=N is N+1-ply (plies=2, the default, is 3-ply)
bgsage: level=1ply is 1-ply, level=2ply is 2-ply, level=3ply is 3-ply, level=4ply is 4-ply, level=truncated1 is Roller-class 1T, level=truncated2 is Roller-class 2T, level=truncated3 is Roller-class 3T
DarkBGHelmet: plies=N is N-ply, the same numbering
gnubg: plies=N is N+1-ply (plies=2, the default, is 3-ply)
PureTD: plies=N is N+1-ply (plies=0, the default, is 1-ply)
wildbg: no search setting; always 1-ply

The same depth is not the same strength. Networks differ, engines prune candidate moves differently at each depth, and some search deeper for cube decisions than for checker play (DarkBGHelmet adds two plies; bgmagic refines close match cube decisions with a deeper cube tree). Open Sage's truncated rollouts are labelled by the XG Roller levels they are modelled on. Comparing depths across engines is what rooms are for.

Thinking time

The referee times every decision it asks a bot for, from sending the position to receiving the answer. Forced moves aren't asked, so they aren't timed. Absolute times depend on the machine, so rooms show only how the two bots compare: both bots of a match take turns on the same CPU core, so the ratio between them holds whatever the hardware.

The overall figure compares the bots at the same mix of decisions. One bot may be asked many more cube decisions than the other (the player who is ahead considers doubling more often), and cube decisions usually take longer than checker plays; weighting each kind by how often it came up for both bots together keeps that from skewing the comparison.

Statistics

Match rooms report bot A's share of matches won, with a 95% Wilson score interval. Unlike the textbook normal approximation, Wilson stays sensible near 0% and 100%.

Money rooms report bot A's mean points per game, with a 95% interval of mean ± 1.96·s/√n, where s is the standard deviation of the per-game results.

Proposition rooms report the same, per match, but computed over complete pairs only: the mean of bot A's pair totals, halved, with a 95% interval of mean ± 1.96·s/√n over pairs. Matches from an unfinished pair (for example after an engine error) are left out. Each position's own estimate weights bot A's results as X and as O equally.

A room names a bot as ahead only once the interval no longer covers 50% (or 0 ppg). Until then the honest answer is “too close to call”.

The width of the interval shrinks with the square root of the number of matches. For a match win rate near 50%:

Matches	95% CI half-width
100	±9.8%
1,000	±3.1%
10,000	±1.0%
40,000	±0.5%

Small edges between strong engines need tens of thousands of matches to show up, which is why rooms run for a long time.