Didn’t read the paper, but the description suggests that the models which were tested were specifically created with a very limited set of training data. I can understand the argument that this might allow researchers to evaluate one property in a simplified environment, namely logic generalization. But it could also be argued that general logic is an emergent property, and limiting the LLM in this way prevents it from reaching that complexity threshold.

I’m not sure if the intent of including the two transformations in the training data was an attempt to provide the LLM with at least one opportunity to generalize between known phenomena before asking it to further generalize.