How do you reach that conclusion?

I don't think we are talking about raw strength here, are we?

Thanks Uberdude for the very clear and detailed explanation! In the light of that, the other comments are making a lot more sense now :-) To summarise:

• "Visits" is often used to mean total number of visits to the root node.
• If analysing a single position in isolation, visits (in this sense) and playouts are the same thing.
• When playing a game, the convention is that "visits" includes tree reuse from previous moves but "playouts" doesn't.
• Limiting an engine to "n visits per move" means that if the visit count (including tree reuse) is less than n then you keep adding playouts until you get to n.
• Limiting an engine to "n playouts per move" means that you do n playouts every move in addition to any tree reuse.
• With tree reuse, visits/second is a higher number than playouts/second because you're counting the re-used visits again.

This one I'm still finding hard to imagine. Tree reuse happens when the opponent plays a move that you've already explored, so you can reuse that part of the tree. Tree reuse is maximised when the opponent plays the most explored move, which is often the move that you assess as best. So if the "weaker" net is getting a lot of tree reuse, that means the "stronger" net is consistently choosing the same move that the "weaker" net would have picked. It looks to me like the so-called "stronger" net in this scenario isn't actually that much stronger?

The point here that the less of a branching factor a policy has, the narrower tree it builds in its memory, the more the potential for reuse. So if the weaker net only looks at 2 moves everywhere (vs, say, 3 for the stronger one), it may be weaker with blind spots, but it will benefit from tree reuse more.

As mentioned earlier, this may account for a, say, 1.5x search speed advantage. Then if you don't know this specifically, and only have the result of a test match at 1000 playouts, you are less likely to correctly predict the result of 10000 playouts match (your actual use case). Basically you are in same situation like if you did time based test on unknown hardware - there is an unclear speed related factor that affects your results in unknown ways and extent, and not necessarily the same way during test than during later usage (speed vs strength works very differently at low search than at high search).

The wider your test is, the more factors you allow to affect its result, the more tests you need to perform to get the same knowledge/confidence (because first you need to guess each individual factor from the results). Again this is for the case where your use case / conditon is significantly different and you cannot test on it directly (otherwise you don't need to know individual factors and are fine with a single test there, since you can be sure all factors will work the same way during test than during later usage).


Look above where we talked about the linked scalability graph. Same winner at 1000 visits as at 10000 visits = curves don't cross the 1.0 line. Unknown bonus from tree reuse = different winner at 1000 vs 2000 visits than at 10000 vs 20000 visits = some curves cross the 2.0 line etc.

Will future versions have a way to set minimum playouts per move?

new 20b 800v against elf2 800v on cgos: 30-1 wow

http://www.yss-aya.com/cgos/19x19/cross ... 3v800.html

s191 had 75% win rate

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

50 game test
KataGo 1.3.3 (g170e-b20c256x2-s2430231552-d525879064) v. LZ017 (#268)
1600 visits for both (~ time parity)
Katago wins 37-13 (74%)
twogtp 1.5.1, no error, no duplicate game, all games by resignation.


Stats : (Katago always shows as W, because of the command -alternate)

Vargo games lost from the ladder, do you consider? are there any statistics, with the promotion LZ the number of games lost due to the ladder decreases? and how does playout affect it?

No, I haven't looked at the games. If you want, you can see for yourself. KataGo is W in every even-numbered game, and the 13 games lost by KataGo are n° 1,12,15,16,20,21,23,24,25,32,34,41,47.

When a bot knows ladder, many interesting things happen, like this one.

Another 50 game test with the exact same commands
KataGo 1.3.3 (g170e-b20c256x2-s2430231552-d525879064) v. LZ017 (#268)
1600 visits for both (~ time parity)
Katago wins 34-16 = 68% ( Last time it was 37-13 = 74%)
twogtp 1.5.1, no error, no duplicate game, all games by resignation.


Stats : (Katago always shows as W, because of the command -alternate, so, B+R means Katago lost)The games :

with the ladder code version of leela zero, it's much more even.

https://github.com/yssaya/leela-zero-ladder/releases

katago s243 vs lz 270 ladder version, both bots 1 thread 512 playouts:

lz wins 3 games kata wins 1. Katago lost the first three but won the last easily

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

Some new nets have been uploaded:

Roughly 50 Elo gain for the 20-block extended training net, and slightly more than 100 Elo gain for the 30 and 40-block nets. Enjoy!

https://github.com/lightvector/KataGo/releases

A last 50 game test for the 20b net : g170e-b20c256x2-s2430231552-d525879064
KataGo 1.3.3 g170e-b20c256x2-s2430231552-d525879064 v. LZ017#268 LADDER VERSION
Katago wins 35-15 = 70% (against the "regular" LZ017#268, KG won 74% and 68%)
Twogtp 1.5.1, 1600 visits for both (~ time parity), no error, no duplicate game, all games by resignation.

Details : (KG always appears as W, because of the command -alternate)
LADDER VERSION : https://github.com/yssaya/leela-zero-ladder/releases

Thx for the big nets

b30c320x2
b40c256x2

From what I have seen so far, if the second number is bigger (320), then the net has a very good chance to be stronger and if the first number is bigger (40), then the net has a medium chance to be stronger.

Some ideas which could be tried:

30 and 320 vs 30 and 384
30 and 320 vs 30 and 512

40 and 256 vs 40 and 320
40 and 256 vs 40 and 384
40 and 256 vs 40 and 512

50 game test between the two newest 20b nets : g170e-b20c256x2-s2430231552-d525879064 v. g170e-b20c256x2-s2971705856-d633407024
The newest , g170e-b20c256x2-s2971705856-d633407024 wins 27-23 (54%)
Twogtp 1.5.1, 1600 visits for both (~ time parity), no error, no duplicate game, all games by resignation.

Details : (s2971 always appears as W, because of the command -alternate)

50 game test KG 1.3.3 g170e-b20c256x2-s2971705856-d633407024 v. LZ#270
KG wins 30-20 (60%, "only" , maybe 50 games is not enough, or maybe s2971 doesn't like #270 )
Again, Twogtp 1.5.1, 1600 visits for both (~ time parity), no error, no duplicate game, all games by resignation.

Details :

Vargo in this case, maybe another 50?

katago 1.3.3 20b s297 1 thread 12 playouts vs gtp4zen7 9d setting; 4-0

Wow I can't beieve it is possibe to have a bot this strong and fast on so few playouts.

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

50 more games KG 1.3.3 g170e-b20c256x2-s2971705856-d633407024 v. LZ#270
KataGo wins 37-13 = 74% (last test, it was 60%)

details : (KG always appears as W, because of the command -alternate)

LZ will never catch up with the KataGo. Does anyone see the point in the further development of LZ? what are its advantages? different game style?