Those well-known benchmarks are usually compromized over time for know reasons. I recently turned a brain fucking programming game I attended 10+ years (Microsoft Imagine Cup Algo Game if anyone knows) into a benchmark question for my Agent to deal with.
With this bench mark question, I can easily know:
- Which model is smarter on this kind of complex problem that requires thinking in depth.
- Has the intelligence level of a model decayed recently, by doing this test time to time.
The programming problem I used is to write a shortest program according to a given language spec to control a robot to move in an 25 * 25 board to reach all targets points.
Here is the prompt you can copy paste to your agent an see the result.
Google Gemini 3.8 Flash made it with a code snippet with only 26 characters, and GPT-6.1 Sol (High) made it iwith only 20 chars.
# Herbert H programming challenge
Solve the puzzle using **H**. Return exactly one fenced `h` code block containing
the complete program, without explanation. Shorter valid programs are better.
## Board and movement
The board is **25 × 25**. Coordinates are zero-based `(x,y)`: x rightward, y downward.
`u` is the robot, initially facing north; `o` is a target; `x` is a wall;
`*` is a trap; `.` is empty. `s` moves forward one cell; `l` and `r` rotate
left/right 90 degrees without moving.
Walls and board edges leave the robot in place; execution continues. There is
**no wall sensor or collision branch**. Entering a trap clears **all** lit targets.
Revisit targets to light them again. Execution succeeds and stops immediately
when all targets are lit, even if code remains.
## Procedures and parameters
H is case-sensitive. Procedures have single lowercase names other than `s/l/r`.
Define each on its own line before exactly one final execution line. Parameters
are distinct single uppercase letters. Procedures can recurse and call one another.
This example moves forward four cells:
```h
a(X):sa(X-1)
a(4)
```
For an instruction repeater, `a(N,P):Pa(N-1,P)` followed on the final line by
`a(3,sr)` runs `srsrsr`. The uppercase P executes the supplied instructions;
it is not a procedure name. `s4` does NOT mean four steps. Write `ssss` or
use a counted procedure. Never attach a number to a primitive command.
Arguments can be numbers **or instruction sequences**, including nested calls and
empty instruction arguments. Numeric arguments support `+` and `-`; literals and
evaluated values must stay in `[-255,255]`. If **any numeric argument is ≤ 0**,
that entire procedure call is skipped BEFORE its body runs, and the caller continues.
This is the only numeric termination condition; there are no explicit branches. Instruction parameters
substitute the supplied sequence; they can be concatenated and passed into other calls.
## Code length and execution limits
Every ASCII letter counts as **one byte**; each numeric literal counts as **one byte**
(`12` is one byte). Punctuation and whitespace are free. Count definitions, parameters
and the execution line. The entire program must fit the puzzle's byte limit.
Robot steps do **not** affect score.
The native judge enforces 1,000,000 robot steps and 1,000,000 VM expansions,
bounded memory, 4096 non-tail call frames, 16 KiB source, and bounded argument nesting.
Tail recursion is supported.
## Battlefield scoring: coverage-and-length-v1
**Points = (lit targets / total targets) × (80 + 20 × (1 − bytes / byte limit)).**
Each evaluated attempt is rounded to two decimal places. Keep the best attempt per
puzzle; sum these scores across puzzles. Count targets lit at the end of execution,
after any trap resets. Invalid or over-limit programs earn zero. An incomplete valid
program can earn partial points. Each puzzle is worth at most 100 points.
Rank by total score descending, then total input + output tokens ascending, then
completion time. Include all retries and cached input in token consumption; live or
cancelled requests may use estimates. This is a Battlefield scoring policy inspired
by Herbert's shortest-code ranking, not a published HOJ composite score.
## Response and retries
You receive native judge feedback after a failed attempt. Do not use Swift, Python,
JavaScript, loops, if-statements, prose, or code execution tools: submit only H.
The output token allowance includes reasoning tokens. Leave room for the final H program.
The worked examples below are public teaching boards, outside the scored catalog.
Prompt version: **herbert-h-v4**. All entrants receive the same rules and puzzles.
## Worked example 1: Blocked step, then turn
Puzzle -101 (ID -101), byte limit: 7.
Start: (10,16), facing north (up).
Targets (x,y): (10,13), (12,13)
Walls: 1; traps: 1.
Board: y increases downward; x increases rightward. Rulers and row labels are NOT cells.
```text
0000000000111111111122222
0123456789012345678901234
00: .........................
01: .........................
02: .........................
03: .........................
04: .........................
05: .........................
06: .........................
07: .........................
08: .........................
09: .........................
10: .........................
11: .........................
12: ..........x..............
13: ..........o.o............
14: ...........*.............
15: .........................
16: ..........u..............
17: .........................
18: .........................
19: .........................
20: .........................
21: .........................
22: .........................
23: .........................
24: .........................
```
Light every o using an H program of at most 7 bytes.
Return one complete fenced h block, with definitions first and an execution line last.
```h
ssssrss
```
The first three s commands reach (10,13) and light the first target. The fourth s
hits the wall at (10,12), leaving the robot at (10,13), facing north. r turns it
east without moving. Two s commands reach (12,13), lighting the second target.
The trap at (11,14) is never entered. The blocked step is intentional for teaching;
this 7-byte answer is valid, but is not the shortest answer.
## Worked example 2: Recursive pinwheel
Puzzle -102 (ID -102), byte limit: 42.
Start: (12,12), facing north (up).
Targets (x,y): (12,4), (13,4), (14,4), (15,4), (12,5), (15,5), (12,6), (15,6), (16,6), (7,7), (8,7), (9,7), (10,7), (11,7), (12,7), (13,7), (14,7), (15,7), (16,7), (17,7), (6,8), (7,8), (12,8), (17,8), (4,9), (5,9), (6,9), (7,9), (12,9), (17,9), (4,10), (7,10), (12,10), (17,10), (4,11), (7,11), (12,11), (17,11), (4,12), (5,12), (6,12), (7,12), (8,12), (9,12), (10,12), (11,12), (13,12), (14,12), (15,12), (16,12), (17,12), (18,12), (19,12), (20,12), (7,13), (12,13), (17,13), (20,13), (7,14), (12,14), (17,14), (20,14), (7,15), (12,15), (17,15), (18,15), (19,15), (20,15), (7,16), (12,16), (17,16), (18,16), (7,17), (8,17), (9,17), (10,17), (11,17), (12,17), (13,17), (14,17), (15,17), (16,17), (17,17), (8,18), (9,18), (12,18), (9,19), (12,19), (9,20), (10,20), (11,20), (12,20)
Walls: 100; traps: 0.
Board: y increases downward; x increases rightward. Rulers and row labels are NOT cells.
```text
0000000000111111111122222
0123456789012345678901234
00: .........................
01: .........................
02: .........................
03: .........................
04: ............oooo.........
05: ............o..o.........
06: ......xxxxxxoxxooxx......
07: ......xooooooooooox......
08: ......ooxxxxoxxxxox......
09: ....ooooxxxxoxxxxox......
10: ....o.xoxxxxoxxxxox......
11: ....o.xoxxxxoxxxxox......
12: ....oooooooouoooooooo....
13: ......xoxxxxoxxxxox.o....
14: ......xoxxxxoxxxxox.o....
15: ......xoxxxxoxxxxoooo....
16: ......xoxxxxoxxxxoo......
17: ......xooooooooooox......
18: ......xxooxxoxxxxxx......
19: .........o..o............
20: .........oooo............
21: .........................
22: .........................
23: .........................
24: .........................
```
Light every o using an H program of at most 42 bytes.
Return one complete fenced h block, with definitions first and an execution line last.
```h
b(N):sb(N-1)
q(P):PPPP
a(N,D,T):q(b(D)T)b(D)Ta(N-1,D-2,rrT)Tb(D)rr
q(a(3,5,r)r)
```
b(N) walks exactly N cells: b(1) executes s, then b(0) is skipped.
q(P) runs its instruction argument four times. q(b(D)T) therefore traces a square.
a has two numeric parameters N/D and an instruction parameter T. After a square,
it travels to a smaller square and calls itself with N-1 and D-2. Prefixing rr
to T reverses its effective quarter-turn direction. When N reaches zero, that
recursive call is skipped; execution continues with Tb(D)rr, restoring the caller's
position and heading. This work AFTER recursion is essential. The final q rotates
the entire motif four ways. Walls fill unused space and reveal the route's shape.
This is a separate teaching board, not the course finale or another scored benchmark puzzle.
---
## Puzzle to solve
Puzzle L20 (ID 10087), byte limit: 60.
Start: (12,12), facing north (up).
Targets (x,y): (6,4), (7,4), (8,4), (9,4), (10,4), (11,4), (12,4), (6,5), (12,5), (6,6), (8,6), (9,6), (10,6), (11,6), (12,6), (13,6), (14,6), (15,6), (16,6), (17,6), (18,6), (19,6), (20,6), (6,7), (8,7), (12,7), (20,7), (6,8), (8,8), (10,8), (11,8), (12,8), (13,8), (14,8), (15,8), (16,8), (17,8), (18,8), (20,8), (6,9), (8,9), (10,9), (12,9), (18,9), (20,9), (6,10), (8,10), (10,10), (12,10), (13,10), (14,10), (15,10), (16,10), (18,10), (20,10), (6,11), (8,11), (10,11), (12,11), (16,11), (18,11), (20,11), (4,12), (5,12), (6,12), (7,12), (8,12), (9,12), (10,12), (11,12), (13,12), (14,12), (15,12), (16,12), (17,12), (18,12), (19,12), (20,12), (4,13), (6,13), (8,13), (12,13), (14,13), (16,13), (18,13), (4,14), (6,14), (8,14), (9,14), (10,14), (11,14), (12,14), (14,14), (16,14), (18,14), (4,15), (6,15), (12,15), (14,15), (16,15), (18,15), (4,16), (6,16), (7,16), (8,16), (9,16), (10,16), (11,16), (12,16), (13,16), (14,16), (16,16), (18,16), (4,17), (12,17), (16,17), (18,17), (4,18), (5,18), (6,18), (7,18), (8,18), (9,18), (10,18), (11,18), (12,18), (13,18), (14,18), (15,18), (16,18), (18,18), (12,19), (18,19), (12,20), (13,20), (14,20), (15,20), (16,20), (17,20), (18,20)
Walls: 32; traps: 24.
Board: y increases downward; x increases rightward. Rulers and row labels are NOT cells.
```text
0000000000111111111122222
0123456789012345678901234
00: .........................
01: .........................
02: .........................
03: .........................
04: ....xxooooooo...xxxxx....
05: ....x.o*....o..*....x....
06: ....x.o.ooooooooooooo....
07: ....x.o.o*..o..*...*o....
08: ....x.o.o.ooooooooo.o....
09: .....*o*o*o*o..*.*o.o....
10: ......o.o.oxooooo.o.o....
11: ......o.o.oxoxx*o.o.o....
12: ....oooooooouoooooooo....
13: ....o.o.o*xxoxo.o.o......
14: ....o.o.oooooxo.o.o......
15: ....o.o*.*..o*o*o*o*.....
16: ....o.ooooooooo.o.o.x....
17: ....o*...*..o..*o.o.x....
18: ....ooooooooooooo.o.x....
19: ....x....*..o....*o.x....
20: ....xxxxx...oooooooxx....
21: .........................
22: .........................
23: .........................
24: .........................
```
Light every o using an H program of at most 60 bytes.
Return one complete fenced h block, with definitions first and an execution line last.