Post by steadyeddie on Dec 18, 2016 6:20:19 GMT -8
Adding the results of a rollout back up the selection path is pretty simple, there is no secret there.
What is more surprising, is that propagating the information to parent nodes which are not the selection path seems to be actively unhelpful. Why is that? Well, if you try and back propagate immediately then because you are expanding down better lines, you'll be pushing "good" rollouts up to poorer first moves. This results in the tree getting excited about these poor moves.
So instead, what about storing the rollout, and then returning it if a selection comes down to that node via a different parent? I couldn't make this work, but I don't know why. There's an article on the forum somewhere with Steve and I discussing this- he's probably got better ideas than me, I'd look that up.
I don't think expansion is that complex. On Steve's advice (there's a forum post somewhere), I fake a rollout result if I hit a terminal node that the parent likes. So if I can choose a 100 score, then I back propagate it. Or if all the children are now confident (terminal). But not otherwise.
What is slightly more interesting is back propagation of confidence. If I have a non-confident node, and I discover, following a back expansion notification from a child, that all my children are now confident, then I change my confidence to 100, and nothing all my parents for them to do the same. Note that is different to back propagation rollouts.
What is more surprising, is that propagating the information to parent nodes which are not the selection path seems to be actively unhelpful. Why is that? Well, if you try and back propagate immediately then because you are expanding down better lines, you'll be pushing "good" rollouts up to poorer first moves. This results in the tree getting excited about these poor moves.
So instead, what about storing the rollout, and then returning it if a selection comes down to that node via a different parent? I couldn't make this work, but I don't know why. There's an article on the forum somewhere with Steve and I discussing this- he's probably got better ideas than me, I'd look that up.
I don't think expansion is that complex. On Steve's advice (there's a forum post somewhere), I fake a rollout result if I hit a terminal node that the parent likes. So if I can choose a 100 score, then I back propagate it. Or if all the children are now confident (terminal). But not otherwise.
What is slightly more interesting is back propagation of confidence. If I have a non-confident node, and I discover, following a back expansion notification from a child, that all my children are now confident, then I change my confidence to 100, and nothing all my parents for them to do the same. Note that is different to back propagation rollouts.
