4Hammer

Our flagship example is 4Hammer (source code), a reinforcement learning environment that encodes extremely complex interactive sequences in just ~5,000 lines of code—including the graphics.

Thanks to Rulebook, 4Hammer automatically gains:

• Reinforcement learning support

• Fuzzer-based testing

• Textual state encoding

• Trace replay

• Remote execution

• Self-configuring UIs

—all available in both browser and desktop builds.

Depending on the complexity of the problem 4Hammer is addressing, 5,000 lines of code might seem like a lot—or not. It’s difficult to make a direct comparison, but at feature parity, we believe that using another language would have required 5 to 20 times more code, depending on the language and tooling.

The project rationale is explained in detail in the related paper

---

The Size of the Problem

4Hammer implements a digital version of Warhammer 40,000, designed specifically for validating reinforcement learning environments.

Warhammer 40,000 is the most-played miniature board game in the world, with hundreds of thousands of games played at a competitive level each year. The community is deeply invested in understanding the game better and has already built tools in narrow domains (like dice probability calculators). But general simulation of the full game is currently intractable.

Challenges include:

• Hundreds of unique game units, each with slight rule variations and quirks.

• Rule interactions that require physics simulation to resolve spatial mechanics.

• Highly non-linear interactive sequences, where small state differences can completely change the legal set of actions. (Example: applying the special rules Torrent and Devastating Wounds to a unit fundamentally alters its behavior during the Single Attack sequence.)

• Frequent updates from the publisher, making full digital replication difficult to maintain.

---

Scope

4Hammer does not aim to fully implement:

• Every single unit profile (there are hundreds).

• A full physics simulation of the game.

These features aren’t necessary unless you’re building a perfect digital twin for resale or official use.

Instead, 4Hammer focuses on:

• Implementing all core rules required to play 6 out of the 35 factions.

• Using Combat Patrol mode as the scenario baseline.

• Enabling neural networks to interact with complex strategic mechanics.

This scope is sufficient for training and validating learning agents at the strategic level—deciding how to allocate resources and respond to game events. Once this is solved, we can begin layering in more detailed tactical rules (e.g. unit placement, collisions, line of sight) and move toward superhuman play under the full rule set of Warhammer 40,000.

---

Visualizing the Game’s Complexity

The image below shows all possible interactive actions available to the player in the 4Hammer implementation.

• Edges indicate that RLC believes the action at the target of the arrow can sometimes—or always—follow the source action.

• Diamonds represent decision points, where the player must choose between multiple next actions.

• Solid boxes represent atomic choices, such as deciding whether to use an optional rule.

• Dashed boxes denote the invocation of a sub-interactive sequence.

Rolling a Die

To better understand the complexity of interactive sequences in Warhammer 40,000, let’s focus on the simplest one: rolling a die. Even this basic action involves a surprising amount of logic and variability.

Core Rules

• Dice typically have 3 or 6 faces.

• A non-rerollable roll is atomic: A random number between 1 and the number of faces is generated, and the game sequence that triggered the roll resumes.

• A rerollable die involves additional logic:

  • It begins with a non-rerollable roll.

  • Depending on the current game state, the player may be offered the option to reroll.

  • If rerolled, the second result must be kept. (This distinguishes it from rules like “roll two dice and pick the best.”)

Variants of Rerolling

• Sometimes the die may be rerolled for free.

• Sometimes the player may reroll for free, but only if the result is a 1.

• Sometimes the player may reroll by spending a command point (a limited in-game resource).

• Only one command-point-based reroll is allowed per phase—after it’s used, no more rerolls of that type are allowed.

• The player must know:

  • The result of the initial roll before deciding to reroll.

  • Whether the reroll costs a command point.

  • Whether the command point has already been spent.

This information must be visible to the user during interaction.

Additional Requirements from 4Hammer

Because of 4Hammer’s goals, the dice roll sequence must satisfy additional constraints:

• ✅ Suspendable: Can wait for user input (e.g., for reroll decisions).

• ✅ Validated: Rejects invalid arguments at runtime.

• ✅ Reinforcement Learning-Compatible: Must be convertible to a format suitable for RL training.

• ✅ Text-Encodable: Can be converted to a readable textual representation.

• ✅ Inspectable: The final result of the roll must be stored and accessible (e.g., for UI rendering).

The following graph shows the how the control flow may evolve when rolling a die.

This is the simplest interactive sequence of the game, which is then reused in multiple points across the program.

If you are new to Rulebook, stop for a second thinking how you would implement a program that meets all requirements.

CPP Implementation

The following code is the cpp implementation that achieves all requirements stated in the previous section.

To make the comparison more fair we have omitted all serialization methods that would require to be hand written in cpp, which could be generated in other languages such as python. Similarly we have removed all empty lines and put comments next to each line of code, so that we don’t get accused of trying to inflate the code size. The code has been written in a style similar to automata base programming which is a well know way of writing state machines in c and cpp. The implementation is mandatory, you cannot use CPP20 coroutines because they have no mechanism for precondition checking, or state inspection.

class RerrolableRoll {
    public: // data to require to serialize and restore the state
    int suspensionPoint; // keeps track of what is the current point of the sequence
    Dice result; // the final result of the roll
    bool can_reroll; // can reroll for free
    bool can_reroll_1s; // can reroll 1s for free
    bool can_cp_reroll; // can reroll by paying a command point
    bool is_non_cp_rerollable; //it is rerollable in a way that does not cost a cp
    Bool current_player; // the player that is performing the roll, bool is fine since it is a two player game
    RerollebleRoll( // constructor that starts the sequence
        bool can_reroll,
        bool can_reroll_1s,
        bool can_cp_reroll,
        bool is_non_cp_rerollable,
        Bool current_player,
        ):
        suspension_point(0),
        can_reroll(can_reroll),
        can_reroll_1s(can_reroll_1s),
        can_cp_reroll(can_cp_reroll),
        is_non_cp_rerollable(is_non_cp_rerollable),
        current_player(current_player) {}
    void roll(GameState& state, Dice dice) {
        assert(can_roll(state), state);
        result = dice;
        if (suspensionPoint == 0){ // if this is the first time we roll the dice
            is_non_cp_rerollable = can_reroll or (can_reroll_1s and result == 1) // figure out if we can reroll for free
            is_cp_rerollable = cp_rerollable and board.can_use_strat(current_player, Stratagem::reroll) // figure out if the player has already rerolled a dice this phase
            if (is_non_cp_rerollable or is_cp_rerollable) { // if we can reroll in any way
                suspensionPoint = 1; // the next state is waiting for the user decision
            } else {
                suspensionPoint = -1; // otherwise we are done
            }
        } else if (suspensionPoint == 2) { // if it is the second roll
            if (!is_non_cp_rerollable) { // if the roll was not free
                board.consume_command_point();
            }
            result = dice;
        }
    }
    bool can_roll(GameState& state, Dice dice) {
        return suspensionPoint == 0 or suspensionPoint == 2; // actions have no precondition, so we just need to check if the resume index is the associated to the first or second roll
    }
    void keep_it(GameState& state, Bool do_it) {
        assert(can_keep_it(state, dice));
        if (do_it)
            suspensionPoint = 2; // goes to the second roll
        else
            suspensionPoint = -1; // goes to the end
    }
    bool can_keep_it(GameState& state, Bool do_it) {
        return suspensionPoint == 1;
    }
    struct ActionRoll { // a class that rappresents an action, in a way that can be stored and executed later
        Dice roll;
        void apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           rollBeingResolved.roll(state, roll);
        }
        bool can_apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           return rollBeingResolved.can_roll(state, roll);
        }
    };
    struct ActionKeepIt {
        bool do_it;
        void apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           rollBeingResolved.keep_it(state, do_it);
        }
        bool can_apply(RerollableDiceRoll rollBeingResolved, GameState& state) {
           return rollBeingResolved.do_it(state, do_it);
       }
    };
    using AnyRerollableDiceRollAction = std::variant<ActionRoll, ActionKeepIt> // a union between all possible actions so that they stored in a single cpp vector without inheritance
    void apply(AnyRerollableDiceRollAction& self, RerollableDiceRoll rollBeingResolved, GameState& state){
        std::apply([&](auto& activeAlternative){ activeAlternative.apply(rollBeingResolved, state); }, self);
    }
    bool can_apply(AnyRerollableDiceRollAction& self, RerollableDiceRoll rollBeingResolved, GameState& state){
        return std::apply([&](auto& activeAlternative){ activeAlternative.can_apply(rollBeingResolved, state); }, self);
    }
};

The code stores all the variables required to suspend the execution in the class body, then exposes 2 function for each action that can be performed. One to check if the action is valid, one to execute it. The functions that performs the actions contain a handrolled state machine. Finally, to be able to delay execution of actions, they are wrapped as well into a class. Even without all the serialization functions that have been omitted and can sometimes be generated in other languages, there is strong coupling between various components of the system, and modifying this implementation without breaking it is hard. Languages like python may save you the union at the cost of runtime performance, but most of the code just shown is mandatory in almost every imperative language. There is a logical error in the code. Did you spot it?

Here is the implementation in Rulebook instead

Rulebook Implementation

act rerollable_dice_roll(ctx Board board,
                         frm Bool reroll,
                         frm Bool reroll_1s,
                         frm Bool cp_rerollable,
                         frm Bool current_player) -> RerollableDice:
    # sets the value for the rolled dice
    act roll(frm Dice result)
    frm is_non_cp_rerollable = reroll or (reroll_1s and result == 1)
    let is_cp_rerollable = cp_rerollable and board.can_use_strat(current_player, Stratagem::reroll)
    if !is_non_cp_rerollable and !is_cp_rerollable:
        return
    # reject the possibility of rerolling the dice and keep the current result
    act keep_it(Bool do_it)
    if do_it:
        return
    # resets the rolled value
    act reroll(Dice new_value)
    if !is_non_cp_rerollable:
        board.spend_command_point()
    result = new_value

This implementation generates a series of classes that are equivalent to the cpp ones, including all serializers that have been omitted before. You can notice there is no coupling between any part of the system, and modifying the code to add another action or another condition would only require local changes.

Furthermore, the cpp code would still need glue before it can be run in a fuzzer or a machine learning infrastructure, we get it for free.

Remember this is the simplest interactive sequence of the game. Sequences such as performing a single attack resolution from one of the models explode in complexity and I don’t think i would be able to write the cpp code by hand without committing multiple mistakes.

Software architecture

Here is the architecture of our example project 4hammer. 4Hammer is a godot based reinforcement learning environments. What is relevant to this section is that:

• 4Hammer has a architecture and usecase similar to other graphical engine programs, such as videogames.

• Rulebook code is invoked by godot scripts, so that godot can decide what to render on screen.

• Rulebook code is used by python reinforcement learning scripts to maximize arbitrary objectives.

Rulebook code(red) describing the rules of the environment is compiled into a shared library(lib.so) using RLC. RLC emits as well a wrapper for Python(wrapper.py) and for Godot(godot_header.h).

4Hammer graphical engine

The Godot wrapper and simulation rules are used to compile a Godot plugin, which exposes Rulebook typesafe rules, types and functions to godot scripts.

The godot plugin and the godot graphical elements are exported by the godot editor to produce 4hammer_graphical_engine which is the final product that can be used by users as a standalone application. Since godot can run on the web, and rlc exports webassembly files too, you can use 4hammer_graphical_engine on the web too.

Python drivers

The rulebook code is as well executed by python scripts that load the wrapper and the library. Since the rules are the same between the engine and python, python can connect over network to the engine and issue commands related to rulebook components.

Fuzzer

The rulebook libraries have been designed to expose a finite interactive program. This allows to automatically compile a fuzzer, which stresses the controller of the environment for free.

Self configuring UI

One of the objective of the Rulebook is to be able to modify rules and have the UI automatically adapt to them. We achieve this using the language action classes. Here is a example, the godot script that handles the game board so that whenever an action related to selecting the board is possible, the eligible locations glow yellow.

# triggers after every action
func on_state_changed():
	for quad in quads.values():
		quad.visible = false
    # for all actions that can be taken
	for action in GlobalRules.valid_actions:
		var unwrapped = action.unwrap() # the the raw valid action
		if unwrapped.members_count() != 1: # if has not exactly one argument we don't know what the semantics are, we will ignore it.
			continue
		if not unwrapped.get_member(0) is RLCBoardPosition: # if the only argument is not a RLCBoardPosition we don't know the semantics
			continue
		var board_position = unwrapped.get_member(0) as RLCBoardPosition # otherwise it is a action that accepts a board position
		if not board_position:
			continue
        # highlight that board position for the user to click
		quads[board_position.get_x().get_value() * 100 + board_position.get_y().get_value()].visible = true

The UI only can introspect valid actions, make sure that they have the expected shape of only requiring a single BoardPosition argument, and then highlight that particular board location.

This means that the rule writer can write rules such as the current implementation of moving a game piece on the board

# core rules movement rules, including the possibility
# of advancing and the triggering of overwatches
act move(ctx Board board, ctx UnitID unit, frm StatModifier additional_movement) -> Move:
    if board[unit.get()].models.size() == 0:
        return
    board.current_decision_maker = board.current_player
    act move_to(BoardPosition position) {
        (position.as_vector() - board[unit.get()][0].position.as_vector()).length() <= float(board[unit.get()][0].profile.movement() + additional_movement.value)
    }
    ...

Without knowing anything related to the UI, the user declared a move_to action, never referenced by any Godot-specific code. Since it accepts a BoardPosition, it is auto detected by the UI code.

Of course, this is very inefficient. In practice it does not matter since the objective of 4hammer is not to draw billions of polygons on screen, but if one wished to make it more efficient it could have enumerated all possible types of valid actions instead of introspecting actions at run time, and then checked if those particular actions were valid.

An even more efficient, but less flexible implementation would have been to directly invoke a callback from RLC code yielding the most optimal implementation, but that would require some extra trickery to make sure machine learning stuff still works.

Supported autoconfigurations

• all actions that accepts a single BoardPosition are interpreted as the user being allowed to click on the board.

• all actions that accept a UnitID are interpreted as the user wishing to click on a unit. All valid units glow and can be clicked.

• all actions that accept a ModelID display the model glowing and allow the user to click on it. The rule writer must specify which unit that model belongs to, to be able to tell them apart.

• all actions that accept 2 UnitID are interpreted as the user wishing to make the first unit interact with the second unit. For example shooting with a unit onto another. The UI allows you to drag an arrow from the start unit toward the target unit.

• actions that accept a single bool display to the user the name of the action phrased as a question.

• actions that accept an enum display the valid choices on screen for the player to select.

Remote execution

Since the entirety of the game rules are written into Rulebook action functions, we obtain for free remote execution. Here is the trivial implementation of the server component on the side of Godot, that waits for connections that then issue server commands. Some special server commands are handled natively by Godot, such as exfiltrating the current image on screen and sending it over the network or stopping the rendering of the screen entirely.

Every other action is simply a rulebook action implied by the rules of the game. Every time a rule programmer changes one of those rules, the remote execution mechanism updates itself automatically. Furthermore, since the network is issuing commands exactly like in process graphical elements are, the only thing that cannot be tested over network is the act on interacting on ui elements themselves.

Python drivers

Since we have remote execution, and Rulebook emits wrappers for other languages, we have the ability to use python to perform typical activities that you may wish to do in reinforcement learning or game programming, beside of course all those that we get off-the-shelf with RLC.

Here is a possible implementation of the remote execution mechanism on the side of the client, say python, that knows the full rules too! You can see it used in practice in a example that plays out at random the game over network.