4Hammer

Our flagship example is 4Hammer (source code), a reinforcement-learning environment with extreamly complex interactive sequences into ~5 k lines of code (graphics included). Thanks to Rulebook, 4Hammer automatically gains:

• reinforcement-learning support

• fuzzer-based testing

• textual state encoding

• trace replay

• remote execution

• self-configuring UIs

—all in both browser and desktop builds.

Depending on the innate complexity of the problem 4hammer is trying to solve, 5k lines can be a lot, or not.

The size of the problem.

4Hammer implements a digital version of Warhammer 40,000, for the purpose of validating reinforcement learning environments. Warhammer 40,000 is the most played miniature board game in the world, with hundred of thousands of games being played at the tournament level each year. The community of players has interest in developing tools to understand the game better, and has done so in limited domains, such as building dice rolling applications that evaluate the probability of a cercain event happening, but the general problem is untractable.

• The game includes hundreds of slightly different game pieces, all with their own quirks.

• The game includes components that would require a physic engine to evaluate.

• The game game has very convoluted interactive sequences, where small difference in game state can lead to compleatly different interactive sequences. (example, attacching the special rules Torrent and Devastating wounds to a game pieces almost entirelly modifies how that model behaves in the Single Attack game sequence).

• The rules of the game are updated by the parent company at high speed, making the job of mantaining a digital implementation unsustainable.

Scope

4Hammer does not implement every single profile of every unit, which are hundreds and not particularly interesting unless one wants a perfect digital twin to sell. Similarly it does not implement the physic system required to fully simulate the game.

Instead we focus on implenting all rules needed to play 6 factions of the 35 factions available in combact patrol mode so that we can validate neural networks by seeing how well they can play with convoluted game mechanics.

This design decisions are acceptable because what we have implemented allows the network to play out the strategic level, where network decide where to allocate their game resources while playing, and this is still an unsolved problem. When this will be solved we can start implementing all the minute physical rules that affect game tactics instead of game strategy, so that we can obtain networks that are superhuman at playing the full rules or warhammer 40,000.

The following image contains are all possible interactive actions the player game execute in the 4Hammer implementation. Edges means that RLC thinks a the target of the arrow can sometimes(or always) be executed after the previous action. Diamonds rappresent choise points, where the player can decide among the successors of the diamon to select which action they want to perform. Solid boxes are atomic choises, such as deciding to use a optional rule or not. Dashes boxes are the invocation of a sub interactive sequence.

Rolling a dice

So that we can understand the complexity of the issue ad hand, let us focus on the simplest Warhammer 40,000 interactive sequence: Rolling a dice. Here are the rules

• Dice have a number of faces, typically 3 or 6.

• Non-rerollable rolls are atomic, a random number between 1 and the number of faces of the dice is extracted, then game sequence invoking the non-rerollable dice resumes.

• Rerollable dices trigger a non-rerollable roll, then depending on the state of the game, may offer to the player that rolled the dice the decision of rerolling it. The second result must be kept. (making it distict from rules that say “roll 2 dices and pick the best”).

  • sometimes the dice can be rerolled for free

  • sometimes the player can reroll for free, but only if the roll resulted int a 1.

  • sometimes the player can reroll the dice, but only by paying a command point (a resource of the game).

  • if the player has already used a command point to reroll a dice, theyn cannot do so for the rest of the phase.

  • the player knows the the result of the dice before deciding if they wish to reroll or not, so that information must be exposed to the user.

  • the player knows if rerolling the dice costs command points or not, so that information must be exposed to the user.

Furthermore we have the following requirements imposed by the objective of 4Hammer.

• Rolling a dice must be suspendable to wait for user input.

• Rolling a dice must ensure not wrong arguments are received.

• Rolling a dice must be convertible to RL encoding for training.

• Rolling a dice must convertible to a textual rappresentation.

• Rolling a dice must contain the value of the rolled dice in a way that can be accessed by the graphical engine.

Here is the graph showing all possible ways of navigating the roll dice sequence.

This is the simplest interactive sequence of the game, which is then reused in multiple points across the program.

If you are new to Rulebook, stop for a second thinking how you would implement a program that meets all requirements.

We have omitted all serialization methods that would require to be hand written in cpp, but the cpp encoding that achieves all requirements is:

class RerrolableRoll {
    // data to require to serialize and restore the state
    public:

    int suspensionPoint; // keeps track of what is the current point of the sequence
    Dice result; // the final result of the roll
    bool can_reroll; // can reroll for free
    bool can_reroll_1s; // can reroll 1s for free
    bool can_cp_reroll; // can reroll by paying a command point
    bool is_non_cp_rerollable; //it is rerollable in a way that does not cost a cp
    Bool current_player; // the player that is performing the roll, bool is fine since it is a two player game

    // constructor that starts the sequence
    RerollebleRoll(
        bool can_reroll,
        bool can_reroll_1s,
        bool can_cp_reroll,
        bool is_non_cp_rerollable,
        Bool current_player,
        ):
        suspension_point(0),
        can_reroll(can_reroll),
        can_reroll_1s(can_reroll_1s),
        can_cp_reroll(can_cp_reroll),
        is_non_cp_rerollable(is_non_cp_rerollable),
        current_player(current_player) {}

    void roll(GameState& state, Dice dice) {
        assert(can_roll(state) );
        result = dice;

        if (suspensionPoint == 0){ // if this is the first time we roll the dice
            // figure out if we can reroll for free
            is_non_cp_rerollable = can_reroll or (can_reroll_1s and result == 1)
            // figure out if the player has already rerolled a dice this phase
            is_cp_rerollable = cp_rerollable and board.can_use_strat(current_player, Stratagem::reroll)


            if (is_non_cp_rerollable or is_cp_rerollable) { // if we can reroll in any way
                // the next state is waiting for the user decision
                suspensionPoint = 1;
            }
            else {
                // otherwise we are done
                suspensionPoint = -1;
            }

        } else if (suspensionPoint == 2) { // if it is the second roll
            // if the roll was not free
            if (!is_non_cp_rerollable) {
                board.consume_command_point();
            }
            result = dice;
        }
    }
    bool can_roll(GameState& state, Dice dice) {
        // actions have no precondition, so we just need to check
        // if the resume index is the associated to the first or second roll
        return suspensionPoint == 0 or suspensionPoint == 2;
    }

    void keep_it(GameState& state, Bool do_it) {
        assert(can_keep_it(state, dice));
        if (do_it)
            suspensionPoint = 2; // goes to the second roll
        else
            suspensionPoint = -1; // goes to the end
    }

    bool can_keep_it(GameState& state, Bool do_it) {
        return suspensionPoint == 1;
    }

    // a class that rappresents a action, in a way that can be stored and executed later
    struct ActionRoll {
        Dice roll;

        void apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           rollBeingResolved.roll(state, roll);
        }

        bool can_apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           return rollBeingResolved.can_roll(state, roll);
        }

    };

    struct ActionKeepIt {
        bool do_it;

        void apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           rollBeingResolved.keep_it(state, do_it);
        }

        bool can_apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           return rollBeingResolved.do_it(state, do_it);
       }

    };

    // a union between all possible actions so that they stored in a single cpp vector without inheritance
    struct AnyRerollableDiceRollAction  {
        union PayLoad{
            ActionRoll first;
            ActionKeepIt second;
        };
        PayLoad content;
        Int currentActiveIndex;

        void apply(ReroolableDiceRoll rollBeingResolved, GameState& staet){
          if (currentActiveIndex == 0) {
            content.first.apply(rollBeingResolved, state);
          } else {
            content.second.apply(rollBeingResolved, state);
          }
        }

        void can_apply(ReroolableDiceRoll rollBeingResolved, GameState& staet){
          if (currentActiveIndex == 0) {
            return content.first.can_apply(rollBeingResolved, state);
          } else {
            return content.second.can_apply(rollBeingResolved, state);
          }
        };
    };

};

The code stores all the variables required to suspend the execution in the class body, then exposes 2 function for each action that can be performed. One to check if the action is valid, one to execute it. The functions that performs the actions contain a handrolled state machine. Finally, to be able to delay execution of actions, they are wrapped as well into a class. Even without all the serialization functions that have been omitted and can sometimes be generated in other languages, there is strong coupling between various components of the system, and modifying this implementation without breaking it is hard. Languages like python may save you the union at the cost of runtime performance, but most of the code just shown is mandatory in almost every imperative language.

Here is the implementation in Rulebook instead

act rerollable_dice_roll(ctx Board board,
                         frm Bool reroll,
                         frm Bool reroll_1s,
                         frm Bool cp_rerollable,
                         frm Bool current_player) -> RerollableDice:
    # sets the value for the rolled dice
    act roll(frm Dice result)
    frm is_non_cp_rerollable = reroll or (reroll_1s and result == 1)
    let is_cp_rerollable = cp_rerollable and board.can_use_strat(current_player, Stratagem::reroll)
    if is_non_cp_rerollable or is_cp_rerollable:
        # reject the possibility of rerolling the dice and keep the current result
        act keep_it(Bool do_it)
        if do_it:
            return
        # resets the rolled value
        act reroll(Dice new_value)
        if !is_non_cp_rerollable:
            board.spend_command_point()
        result = new_value

This implementation generates a series of classes that are equivalent to the cpp ones, including all serializers that have been omitted before. You can notice there is no coupling between any part of the system, and modifying the code to add another action or another condition would only require local changes.

Furthermore, the cpp code would still need glue before it can be run in a fuzzer or a machine learning infrastructure, we get it for free.

Remember this is the simplest interactive sequence of the game.

Software architecture

Here is the architecture of our example project 4hammer. 4Hammer is a godot based reinforcement learning environments. What is relevant to this section is that:

• 4Hammer has a architecture and usecase similar to other graphical engine programs, such as videogames.

• Rulebook code is invoked by godot scripts, so that godot can decide what to render on screen.

• Rulebook code is used by python reinforcement learning scripts to maximize arbitrary objectives.

Rulebook code(red) describing the rules of the environment is compiled into a shared library(lib.so) using RLC. RLC emits as well a wrapper for Python(wrapper.py) and for Godot(godot_header.h).

4Hammer graphical engine

The Godot wrapper and simulation rules are used to compile a Godot plugin, which exposes Rulebook typesafe rules, types and functions to godot scripts.

The godot plugin and the godot graphical elements are exported by the godot editor to produce 4hammer_graphical_engine which is the final product that can be used by users as a standalone application. Since godot can run on the web, and rlc exports webassembly files too, you can use 4hammer_graphical_engine on the web too.

Python drivers

The rulebook code is as well executed by python scripts that load the wrapper and the library. Since the rules are the same between the engine and python, python can connect over network to the engine and issue commands related to rulebook components.

Fuzzer

The rulebook libraries have been designed to expose a finite interactive program. This allows to automatically compile a fuzzer, which stresses the controller of the environment for free.