_{1}

^{*}

Teaching computer programs to play games through machine learning has been an important way to achieve better artificial intelligence (AI) in a variety of real-world applications. Monte Carlo Tree Search (MCTS) is one of the key AI techniques developed recently that enabled AlphaGo to defeat a legendary professional Go player. What makes MCTS particularly attractive is that it only understands the basic rules of the game and does not rely on expert-level knowledge. Researchers thus expect that MCTS can be applied to other complex AI problems where domain-specific expert-level knowledge is not yet available. So far there are very few analytic studies in the literature. In this paper, our goal is to develop analytic studies of MCTS to build a more fundamental understanding of the algorithms and their applicability in complex AI problems. We start with a simple version of MCTS, called random playout search (RPS), to play Tic-Tac-Toe, and find that RPS may fail to discover the correct moves even in a very simple game position of Tic-Tac-Toe. Both the probability analysis and simulation have confirmed our discovery. We continue our studies with the full version of MCTS to play Gomoku and find that while MCTS has shown great success in playing more sophisticated games like Go, it is not effective to address the problem of sudden death/win. The main reason that MCTS often fails to detect sudden death/win lies in the random playout search nature of MCTS, which leads to prediction distortion. Therefore, although MCTS in theory converges to the optimal minimax search, with real world computational resource constraints, MCTS has to rely on RPS as an important step in its search process, therefore suffering from the same fundamental prediction distortion problem as RPS does. By examining the detailed statistics of the scores in MCTS, we investigate a variety of scenarios where MCTS fails to detect sudden death/win. Finally, we propose an improved MCTS algorithm by incorporating minimax search to overcome prediction distortion. Our simulation has confirmed the effectiveness of the proposed algorithm. We provide an estimate of the additional computational costs of this new algorithm to detect sudden death/win and discuss heuristic strategies to further reduce the search complexity.

Teaching computer programs to play games through learning has been an important way to achieve better artificial intelligence (AI) in a variety of real-world applications [

MCTS generates a search tree by running a large number of simulated playouts. Each playout starts at the current game state and simulates a random sequence of legal moves until the game ends. By analyzing the statistics, MCTS estimates which move is more likely to win and takes the statistically best move. MCTS was first proposed in 2006 [

Most studies on MCTS in the literature are based on simulation [

The remaining of this paper is organized as followed. Section 2 investigates RPS playing Tic-Tac-Toe with probability analysis and simulation and highlights the drawback of RPS. Section 3 discusses a simple solution to address the drawback of RPS and extend the discussion to introduce MCTS. Section 4 investigates the problem of sudden death/win, presents two propositions on the capability of detecting sudden death/win with a minimax search and MCTS, and examines the problem of prediction distortion in the context of MCTS playing Gomoku with detailed simulation results. Section 5 proposes an improved MCTS algorithm to enhance the capability to detect sudden death/win, analyzes the additional complexity and presents heuristic strategies to further reduce the search complexity. Appendix A describes the Java implementation of the analysis and simulation codes used in this study.

The Tic-Tac-Toe game is probably the simplest board game. We propose a simplified version of MCTS, Random Playout Search (RPS), to play the Tic-Tac-Toe game in order to build a mathematical model for analysis, and hope that the insight from our analysis is applicable to MCTS playing other more complex games.

To describe RPS, consider the current game position shown in

computer’s turn to move. The computer can move to one of 6 possible positions, labeled 1 - 6. If the computer takes 1, then the opponent can take 6, which blocks the diagonal and opens up a vertical and a horizontal to win.

However, RPS does not have this ability to strategically look beyond a few steps. Instead, RPS employs a large number of simulated playouts t = 1 , 2 , ⋯ and estimates the expected score of move i, for i = 1 - 6 .

Denote M i ( t ) the number of times that move i has been picked up to playout t and N i ( t ) the number of wins minus the number of losses. Initially, M i ( 0 ) = N i ( 0 ) = 0 for all i. Each playout starts with the current game position. RPS randomly picks next move i and increments M i ( t ) by 1. RPS then picks a sequence of random moves for the computer and the opponent until the game ends. If the computer wins, then N i ( t ) increments by 1; if the computer loses, then N i ( t ) decrements by 1; otherwise, N i ( t ) is unchanged. The score is calculated as follows,

S i ( t ) = N i ( t ) M i ( t ) . (1)

When the preset number of playouts T has been reached, RPS selects the move i * with the highest score,

i * = arg max i S i ( T ) . (2)

To assess the performance of RPS, we calculate s P 0 , i , the expected score of move i from the current game position P 0 . To simplify the notation, s P 0 , i is sometimes reduced to s i . Denote P 1 the game position after move i from P 0 , where subscript 1 indicates that P 1 is on level-1 from the starting position P 0 on level-0. There are multiple game positions on level-1 and P 1 is one of them. Denote K P 1 the number of legal moves from position P 1 . Because in a simulated playout RPS uniformly randomly selects the next move of the opponent, we have the following recursive equation

s P 0 , i = s P 1 = 1 K P 1 ∑ j = 1 K P 1 s P 1 , j , (3)

where s P 1 , j is the expected score of move j from P 1 . Equation (3) can be further expanded from every move j of P 1 to the next levels until terminal

positions are reached. A position is terminal if the game ends at the position, and the game result (loss, draw, win) determines the expected score of the terminal position to be − 1,0,1 , respectively.

While Equation (3) starts from the current position P 0 , it is solved by walking backwards from the terminal positions. An example of solving Equation (3) is illustrated in

Specifically, the game position P 1 after move 1 from P 0 is labeled as A, which is expanded into K A = 3 next level positions B , C , D . B is a terminal position (loss), so s B = − 1 . C is further expanded into K C = 2 next level positions E, F. E leads to a terminal position G (draw) where s E = s G = 0 . F leads to a terminal position H (loss), so s F = s H = − 1 . Thus, from Equation (3),

s C = 1 K C ( s E + s F ) = − 1 2 .

Similarly, D is further expanded into K D = 2 next level positions I, J. I leads to a terminal position K (draw) where s I = s K = 0 . J leads to a terminal position L (loss), so s J = s L = − 1 . Thus,

s D = 1 K D ( s I + s J ) = − 1 2 .

Finally, from Equation (3),

s P 0 , 1 = s A = 1 K A ( s B + s C + s D ) = − 2 3 . (4)

Using the same recursive approach, we can obtain

s P 0 , 2 = − 1 3 , s P 0 , 3 = − 2 3 , s P 0 , 4 = − 1 3 . (5)

Because s P 0 , 2 = s P 0 , 4 > s P 0 , 1 = s P 0 , 3 , according to Equation (2) RPS selects either move 2 or 4 with equal probability as the next move from P 0 .

We expect that

s i = l i m t → ∞ S i ( t ) . (6)

Equation (6) states that S i ( t ) given in (1) converges to s i , the expected score calculated with the recursive analysis model (3) as the number of playouts increases. We next validate (6) with simulation.

We have simulated the RPS algorithm in Java to play Tic-Tac-Toe according to

The above analysis and simulation show that for the simple game position in

The reason of RPS failure in those game positions is that in the simulated playouts RPS assumes that the opponent randomly picks its moves. However, in the actual game an intelligent opponent does not move randomly. As a result, the prediction ability of RPS is distorted. In the example of

One way to overcome the prediction distortion is to expand the search depth of RPS. Take

In

the level-2 moves, which are obtained from the simulated playouts. The scores of the level-1 moves are calculated from those of the level-2 moves with the minimax principle. That is, the opponent selects the best move, i.e., with minimum score, instead of a random move on level-2,

S i ( t ) = min j = 1 , 2 , 3 S i , j ( t ) , (7)

for i = 1 , ⋯ , 4 . In comparison, when the search depth is 1, although RPS does not keep track of S i , j ( t ) , in effect S i ( t ) is estimated to be

S i ( t ) = 1 3 ∑ j = 1 , 2 , 3 S i , j ( t ) . (8)

Comparison of Equations (7) and (8) shows how the prediction distortion is corrected for the opponent’s move on level-2 by increasing the search depth to 2.

The idea of expanding the search depth can be generalized to mean that a good algorithm should search beyond the level-1 moves. MCTS takes this idea even further. Starting from the current game position, called the root node, the MCTS algorithm gradually builds a search tree through a number of iterations and stops when some computational budget is met. Similar to the one shown in

Each iteration in MCTS consists of four steps as illustrated in

1) Selection: A child node is selected recursively, according to a tree policy, from the root node to a leaf node, i.e., a node on the boundary of the search tree.

2) Expansion: A new node is added to the search tree as a child node of the leaf node.

3) Simulation: A simulated playout runs from the new node until the game ends, according to a default policy. The default policy is simply RPS, i.e., randomly selecting moves between the computer and the opponent. The simulation result determines a simulation score: +1 for win, −1 for loss, and 0 for draw.

4) Backpropagation: The simulation score (+1, −1, or 0) is backpropagated from the new node to the root node through the search tree to update their scores. A simple algorithm is that the score is added to X j of every node j along the search tree.

Recall that the nodes from the new node to the terminal node are outside the boundary of the search tree. Any node outside the boundary of the search tree does not keep track of any score.

The most popular tree policy in MCTS is based on the Upper Confidence Bounds applied to Trees (UCT) algorithm [

U C B 1 = X j n j + C 2 ln n n j , (9)

where X j is the total score of child node j, n j the number of times that child node j has been tried, and n = ∑ j n j the total number of times that all child nodes have been tried. The first term average score

S j = X j n j (10)

aims to exploit higher-reward moves (exploitation), while the second term

2 ln n n j encourages examining less-visited moves (exploration), and parameter

C balances exploitation and exploration.

After the algorithm stops, the child node of the root node with the highest average score S j is selected and the corresponding move from the root node to the selected child node is taken in the actual game.

An important theoretical result [

In practice, however, MCTS is limited by computational resource and the search tree is a small subset of the entire game position space. Therefore, MCTS has to rely on RPS to explore a large game position space that is outside the search tree. From the study of Tic-Tac-Toe we learn that a major drawback of RPS is that it assumes the opponent is playing randomly when in fact it is possible that the opponent knows exactly what it should do strategically.

To study MCTS, we need more sophisticated games than Tic-Tac-Toe. In this section, we play Gomoku and see how the insight from the study of Tic-Tac-Toe applies here.

computer’s turn to move. In (a), without taking move A, the computer loses in two steps. In (b), without taking move B or C, the computer loses in four steps. On the other hand, if the computer is black and the opponent is green in

Definition 1. Level-k sudden death or sudden win

A level-k sudden death is a game position where the computer will lose the game in k steps if the computer does not take one of some subset of moves. A level-k sudden win is a game position where the computer will win the game in k steps if the computer takes one of some subset of moves.

The subset of moves is called decisive moves. The level k is the number of steps that results in a terminal position from the current game position. k is even for sudden death and odd for sudden win. k is a small number for sudden death or win; otherwise, death or win is not “sudden”. More importantly, because of computational resource constraints, it is impractical to detect definite death or win for a large k. In

One way to detect a level-k sudden death or win is to employ a level-k minimax search, a slightly modified version of minimax search. Specifically, starting from the current game position (root node at level-0), repeatedly expand to the next level by adding the child nodes that represent all the possible next moves from the parent node. Node expansion ends at level-k. A node of terminal position is assigned a score of −1, 0, 1 as in Section 2.2. If a node at level-k is not a terminal position, it is assigned a score of 0, because its state is undetermined. Backpropagate the scores to the root node according to the minimax principle.

principle is that a operation of selecting the minimum or maximum score is applied at every node in alternate levels backwards from level-k. The score of a node at level-4 is the maximum of the scores of its child nodes at level-5 because it is the computer’s turn to move. The score of a node at level-3 is the minimum of the scores of its child nodes at level-4 because it is the opponent’s turn to move, and so on. In

Proposition 2. Any level-k sudden death or win is detectable with the level-k minimax search.

Proof. We focus on sudden win in the proof. The case of sudden death can be proved similarly.

For a level-k sudden win to exist, one of the child nodes of the root node (e.g., A 1 in

At level-k, the score of a node is − 1,0,1 depending on the state of the node (loss, draw/undetermined, or win). From the process described above, for N 1,1 to be a decisive move, a set of level- ( k − 1 ) nodes under N 1,1 must each have at least one child node at level-k being a win, and thus obtain a score of 1 themselves according to the minimax principle, which in turn makes their parent nodes at level- ( k − 2 ) to have a score of 1. Their scores are further backpropagated upwards so that eventually the scores of N 3,1 , N 3,2 , ⋯ are all 1 and those of N 2,1 , N 2,2 , ⋯ are 1 as well, which results in a score of 1 for N 1,1 indicating that a level-k sudden win exists and N 1,1 is a decisive move leading to the sudden win. □

Recall from Section 3.2 that MCTS selects the move on the basis of the average scores of the child nodes of the root node (10). In general, the average scores of the child nodes of the root node are rarely equal to the scores of those nodes in a level-k minimax search precisely. However, prediction distortion is not severe if the discrepancy between the two scores does not make MCTS to select a different child node from what the level-k minimax search would select. It should be pointed out that the level-k minimax search is not necessarily superior to MCTS in all game positions. However, in the presence of sudden death or win, if MCTS selects a move different from the choice of the level-k minimax search, then by definition MCTS fails to make a decisive move. Hence we have the following proposition.

Proposition 3. MCTS fails to detect a level-k sudden death or win if the average scores (10) of the child nodes of the root node are sufficiently deviated from their scores in the level-k minimax search that a decisive move is not selected in MCTS.

Because of computational resource constraints, the search tree in MCTS is a subset of the entire state space. Random moves are applied in the simulation outside the boundary of the search tree, therefore potentially causing prediction distortion as learned in the RPS analysis.

the computer will eventually select the move that results in the computer’s immediate win. This is an example of sudden win for the computer. Furthermore, even for the nodes within the search tree boundary, their scores may not converge to the minimax scores if not enough simulation iterations have been taken.

Clearly as the level increases, it becomes more difficult to discover sudden death/win. In our implementation of MCTS playing Gomoku, we find that MCTS frequently discovers level-1 and level-3 sudden win and level-2 sudden death, but often unable to discovery level-4 sudden death. We will analyze the simulation performance of MCTS next.

We have simulated the MCTS algorithm in Java to play Gomoku on an K × K board. The implementation details are described in Appendix A. We report the simulation results in this section.

We do not model the computational resource constraint in terms of precise memory or runtime. Instead we vary the number of iterations T. We find that the run time increases roughly linearly with T and that the Java program runs into heap memory errors when T exceeds some threshold. For example, for K = 9 , the run time is between 7 to 10 seconds for every move at T = 30000 , and a further increase to T = 40000 causes the out of memory error. In contrast, for K = 11 , the Java program can only run T = 10000 before hitting the memory error. Therefore, we use parameter T to represent the consumed computational resource, and K to represent the complexity of the game.

We are interested in the performance of MCTS in dealing with sudden death/win. To report the win rate of an entire game, it would depend on the strength of the opponent, a quantity difficult to measure consistently. Therefore, we let a human opponent to play the game long enough to encounter various sudden death/win scenarios and record the fraction of times that the program detects sudden death/win. Detecting sudden death means that MCTS anticipates the threat of an upcoming sudden death and takes a move to avoid it. Detecting sudden win means that MCTS anticipates the opportunity of an upcoming sudden win and takes a move to exploit it and win the game.

^{1}For K = 11 , T = 10000 instead of 30,000.

The simulation results are summarized in ^{1}. We make the following observations.

・ Detecting level-4 sudden death is more difficult than detecting level-2 sudden death. The detection rate drops drastically as the game complexity, represented by K, increases. On the other hand, the detection rate drops drastically as the computational resource, represented by T, decrease.

Sudden Death | ||||
---|---|---|---|---|

Level-2 | Level-4 | |||

T = 30,000 | T = 3000 | T = 30,000 | T = 3000 | |

K = 7 | 100% | 100% | 80% | 60% |

K = 9 | 100% | 40% | 50% | 10% |

K = 11 | 60% | <15% | 15% | <5% |

Sudden Win | ||||

Level-1 | Level-3 | |||

T = 30,000 | T = 3000 | T = 30,000 | T = 3000 | |

K = 7 | 100% | 100% | 100% | 100% |

K = 9 | 100% | 100% | 100% | 90% |

K = 11 | 100% | 100% | 80% | 50% |

・ There seems to be a tipping point beyond which the detection rate of sudden death is so low that MCTS is useless. For example, for T = 30,000, the tipping point is around K = 7 to 9.

・ The detection rate is not symmetric between sudden death and sudden win. Indeed, it seems much easier to detect sudden win than to detect sudden death. In particular, MCTS is always able to detect level-1 sudden win. Moreover, at a first glance, one may expect the detection rate of level-2 sudden death to be higher than that of level-3 sudden win, because level-3 sudden win involves more steps to search through. However, it turns out that detecting level-3 sudden win is more robust than detecting level-2 sudden death in a variety of K, T scenarios. We will further investigate this phenomenon by looking deeper into the scores in MCTS iterations.

・ Not shown in

To investigate the reason that MCTS fails to detect level-4 sudden death, we next examine a specific scenario shown in

The moves A , B , C correspond to three child nodes of the root node, represented by three circles labeled as A , B , C in

This observation may be puzzling, because position A leads to level-3 sudden win, after the computer takes A. So, wouldn’t the opponent take B or C therefore resulting in computer’s loss? These two possible moves are represented by two child nodes of node A, labeled as A1, A2 in

To understand why the low, negative scores of A1, A2 fail to bring down the score of node A, we plot in

From the backpropagation step in Section 3.2, we know that a node’s total score is the sum of the total scores of its child nodes. Therefore, we predict that if more computational resource were available (e.g., with greater numbers of

MCTS simulation), the scores of A1 and A2 would eventually dominate the score of their parent node A so that it would be lower than the score of node B or C at which point MCTS would correctly select position B or C and avoid

sudden death. In other words, the practical limitation of MCTS in dealing with sudden death is caused by the computational resource. In a way, this is somewhat expected when we note in Section 2.3 that convergence is quite slow even for a simple game position (Tic-Tac-Toe) and a simple algorithm (RPS).

The example scenario in

The moves A, B correspond to two child nodes of the root node, represented by two circles labeled as A, B in

The difference of the statistics of nodes A and B in

The issue of sudden death/win is a known problem for MCTS [

loss, then play this move; otherwise, play MCTS. Clearly this algorithm only addresses level-1 sudden win or level-2 sudden problems. As mentioned before, level-2 sudden death or level-1 win is not a major issue in MCTS at least for playing Gomoku with reasonable K, T. While it is possible that with sufficient

computational resource MCTS could eventually deal with sudden death/win, next we propose an improved MCTS algorithm that addresses this problem without increasing drastically required computational resource.

To identify a level-k sudden death/win, we have to use a minimax search of depth k starting from the root node [

However, the computational costs of the minimax search would be significant even for a modest k. Carrying out the minimax search for every new root node reduces the computational resource left for MCTS. The key idea of our proposal is to not burden MCTS with the minimax search until it becomes necessary.

iteration; otherwise, run the minimax search from the root node and follow the algorithm of

From Proposition 2 the brute-force algorithm is guaranteed to detect any level-k sudden death or win. However, the improved MCTS algorithm can only guarantee the detectability with probability.

Proposition 4. Any level-k sudden death or win is detectable, with probability, with the improved MCTS algorithm in

The reason of “guarantee with probability” in Proposition 4 is that the improved MCTS algorithm needs to encounter a terminal node within level-k in at least one of the many iterations before a minimax search is triggered. In Gomoku, we find that the probability is close to 1 when the number of iterations is large, the board size is modest, and depth k is a small number, e.g., T = 30,000, K = 9, k = 4.

We have implemented the improved MCTS algorithm in JAVA. The implementation includes a minimax search of depth 4, and we confirm that the program is able to always detect any level-2/4 sudden death and level-1/3 sudden win. To assess the computational resource used in the minimax search of depth 4 and the overall MCTS algorithm, we record the total elapsed time of every MCTS step, which includes T simulations and possibly a minimax search, as well as the elapsed time of every minimax search when it happens.

We observe that the elapsed time of a minimax search is pretty uniformly distributed from 0 to about 1.8 seconds and that the elapsed time of a MCTS step is either small (below 2 seconds) or quite large (above 6 seconds), where the small value corresponds to the scenarios where MCTS can find easy moves to win the game (level-1 or level-3 sudden win). On average, a single minimax search is about 1/6 of the entire MCTS 30,000 simulations. As the depth increases, the minimax search will be exponentially expensive. Therefore, the advantage of the proposed algorithm over the brute-force algorithm could be significant.

In the improved algorithm of

to search over all the child nodes of the root node. Instead, we only need to run the minimax search from a subset of promising child nodes.

Suppose that in one iteration of MCTS a terminal node of loss (or win) is encountered within depth k from the root node, indicating the potential presence of sudden death (or win). Mark the child node of the root node selected in that iteration as a potential sudden death (or win) move.

Consider the first heuristic strategy. Instead of triggering a minimax search immediately as in

1) For the subset of child nodes of the root node whose average scores are in the top x 1 percentile and that have been marked as a potential sudden death move, run a minimax search of depth k − 1 from each of them. If the minimax score of a child node in the subset is equal to −1, then mark it as a sudden death move and refrain from selecting it in subsequent MCTS iterations. Continue to run MCTS.

2) For the subset of child nodes of the root node whose average scores are in the top x 2 percentile and that have been marked as a potential sudden win move, run a minimax search of depth k − 1 from each of them. If the minimax score of any of them is equal to 1, then mark the child node as a sudden win move, which is the best move to select, and stop MCTS. Otherwise, continue to run MCTS.

Rule 1 is to prevent a sudden death move from prevailing in MCTS node selection. For example, move A in

Consider the second heuristic strategy. Instead of running MCTS for some number of iterations, applying the above two rules and then continuing MCTS as in the first strategy, we can run MCTS to the end and then apply the following two similar rules.

1) For the child node of the root node with the highest average score, if it has been marked as a potential sudden death move, run a minimax search of depth k − 1 . If its minimax score is equal to −1, then discard it, proceed to the child node with the second highest average score and repeat this rule. Otherwise, proceed to the next rule.

2) For the subset of child nodes of the root node whose average scores are in the top x 2 percentile and that have been marked as potential sudden win moves, run a minimax search of depth k − 1 from each of them. If the minimax score of any of them is equal to 1, then select the move. Otherwise, select the move according to the MCTS rule.

The performance of the above two heuristic strategy can be assessed with the following proposition.

Proposition 5. Either the first or second heuristic strategy has the same capability to detect a sudden death as the improved MCTS algorithm in

In Gomoku, if x 2 is a small number, the complexity reduction over the improved algorithm of

In this paper we analyze the MCTS algorithm by playing two games, Tic-Tac-Toe and Gomoku. Our study starts with a simple version of MCTS, called RPS. We find that the random playout search is fundamentally different from the optimal minimax search and, as a result, RPS may fail to discover the correct moves even in a very simple game of Tic-Tac-Toe. Both the probability analysis and simulation have confirmed our discovery.

We continue our studies with the full version of MCTS to play Gomoku. We find that while MCTS has shown great success in playing more sophisticated games like Go, it is not effective to address the problem of sudden death/win, which ironically does not often appear in Go, but is quite common on simple games like Tic-Tac-Toe and Gomoku. The main reason that MCTS fails to detect sudden death/win lies in the random playout search nature of MCTS. Therefore, although MCTS in theory converges to the optimal minimax search, with computational resource constraints in reality, MCTS has to rely on RPS as an important step in its simulation search step, therefore suffering from the same fundamental problem as RPS and not necessarily always being a winning strategy.

Our simulation studies use the board size and number of simulations to represent the game complexity and computational resource respectively. By examining the detailed statistics of the scores in MCTS, we investigate a variety of scenarios where MCTS fails to detect level-2 and level-4 sudden death. Finally, we have proposed an improved MCTS algorithm by incorporating minimax search to address the problem of sudden death/win. Our simulation has confirmed the effectiveness of the proposed algorithm. We provide an estimate of the additional computational costs of this new algorithm to detect sudden death/win and present two heuristic strategies to reduce the minimax search complexity and to enhance the MCTS effectiveness.

The importance of sudden death/win is not limited to the board games. The real-world AI applications such as autonomous driving sometimes face similar situations where certain actions lead to drastic consequences. Therefore, we hope to extend the research of the MCTS algorithm to these applications in the future study.

Li, W. (2018) Prediction Distortion in Monte Carlo Tree Search and an Improved Algorithm. Journal of Intelligent Learning Systems and Applications, 10, 46-79. https://doi.org/10.4236/jilsa.2018.102004

The following appendix sections describe the Java implementation of the analysis and simulation codes used in this study.

A1. Tic-Tac-Toe and Gomoku Game GUII downloaded a Java program from the Internet, which provides a graphic user interface (GUI) to play Tic-Tac-Toe. I modified the program to play Gomoku.

The game GUI consists of two panels. The left panel shows two options to play the game: person-to-person or person-to-computer. In this study I am only interested in the latter option. The two colors, black (person) and green (computer), show whose turn it is to move. The right panel is a K × K grid to represent the game board.

The program uses an array currentTable[^{2} to represent the current game position, where element currentTable[i*K+j] is the state of row i and column j of the game board. currentTable[i*K+j] can be in one of three states:

enum TILESTATUS {EMPTY, PERSON, COMPUTER}.

Initially, all elements are EMPTY. When it is the person’s turn to move and the human player clicks on one of the empty grid element, the program changes the corresponding element of currentTable[

int choseTile()

which returns an integer index pointing to one of the elements of currentTable[

element of currentTable[

boolean checkWinner(TILESTATUS[

to check whether the game position current

Function choseTile() is where any AI algorithm is implemented. In the Tic-Tac-Toe Java program downloaded from the Internet, the default implementation is to randomly return any index that corresponds to an EMPTY element. In this study, I have implemented the RPS and MCTS algorithms in choseTile(). I will describe our implementation of RPS and MCTS later.

A2. Implementation of Probability Analysis of RPSI wrote a Java program to calculate the expected score of each move from the current game position of RPS. The Java program is used in the probability analysis in Section 2.2.

The calculation is based on the recursive Equation (3). The current game position is represented by the root node and all the legal next-step moves from the current game position by the computer form its immediately child nodes. The Java problem uses a recursive function

float scoreIteration(TURN turn, int select)

to calculate the expected score of the child node presented by currentTable. Denote K the number of EMPTY elements in the root node and index[k] the k-th EMPTY element. The main() function calls scoreIteration() for each of the K child nodes of the root node, k, with turn set to COMPUTER, select set to index[k], and currentTable[index[k]] set to COMPUTER. currentTable[index[k]] is reset to EMPTY after each function call.

The programming idea of scoreIteration() is that if the node is terminal, then return −1, 0, 1 depending on the game outcome; otherwise, expand to its child nodes and call scoreIteration() recursively. When a node expands to its next level child nodes, the number of EMPTY elements decrements by one. The pseudo of scoreIteration() is given as follows.

A3. MCTS and RPS ImplementationI downloaded a Java program from the Internet (https://code.google.com/archive/p/uct-for-games/source), which implements the MCTS algorithm. I modified the MCTS program to work with the GUI program, to output various statistics to study the simulation results in Sections 2.3 and 4.2, and to allow flexible alrogithm selection, which will be necessary to implement the improved MCTS algorithm of Section 5.

The MCTS Java program defines a class called MCTSNode, which includes a number of variables and methods to implement the algorithm described in Section 3.2. The most important variables are

int totalScore to store the total score of the node,

int timesVisited to store the total number of times the node is visited,

ArrayList nextMoves to store an array list pointing to the child nodes of the node,

int indexFromParentNode to store the move that its parent node takes to reach the node,

TILESTATUS[

TILESTATUS nodeTurn to store whose turn it is to move.

The average score of the node is simply totalScore/timesVisited. The most important methods are described below.

Function choseTile() calls bestMCTSMove(), which returns the move chosen by MCTS.

bestMCTSMove() in turn calls runTrial() in each MCTS iteration, in which the game ends in one of three outcomes

enum TURN {PERSON, COMPUTER, DRAW}.

runTrial() calls simulateFrom() to simulate a random playout beyond the tree boundary.

The RPS algorithm is a simpler version of MCTS. One variable is added to the MCTSNode class:

int levelFromTopNode to store the number of levels from the root node to the node. For a child node of the root node, levelFromTopNode is set to 1.

The only change to implement RPS is that line 5 of function runTrial() is replaced by

if Node is a leaf node AND levelFromTopNode == 0 then

so that only the root node is expanded to add its child nodes, which are never expanded and are kept as the leaf nodes.

A4. Improved MCTS ImplementationI revised the MCTS Java program to implement the improved MCTS algorithm of

int simulationDepth to store the number of steps from the root node to where the game ends in function simulateFrom()

ArrayList nextMinimaxMoves to store an array list pointing to the child nodes of the node that are the optimal moves found in the minimax search. nextMinimaxMoves is a subset of nextMoves.

Besides, a static variable is defined:

int MINIMAXSEARCH to indicate whether the minimax search should be taken and whether has been taken.

The changes to functions bestMCTSMove(), runTrial() and simulateFrom() are summarized as follows.

In bestMCTSMove(), lines 3 to 4 are replaced by the following.

In runTrial(), lines 7 to 11 are replaced by the following. Here, constant NUMSIMULATIONDEPTH is the depth k in

In simulateFrom(), the following is added immediately after line 8.

simulationDepth ← simulationDepth+1

Function minimax() does the minimax search and constructs the array lists nextMinimaxMoves of the root node and the child nodes up to level NUMSIMULATIONDEPTH. I wrote two versions of minimax(). The first version uses recursion.

The recursive version of minimax() can successfully play Tic-Tac-Toe but runs into heap memory errors when playing Gomoku even with modest board sizes. Although the code is simple and clean, the recursive function calls consume a significant amount of memory. To overcome this problem, I wrote the second version using iteration. The second version does not run into any errors but is not general enough for any NUMSIMULATIONDEPTH. NUMSIMULATIONDEPTH has to be set to 4.