AIMicrosoft DevBlogsabout 2 hours ago
The hidden variables in your agent eval
11 min read
Agent eval results vary across machines due to hidden environmental variables. The operating system determines the shell, which affects agent competence and can flip technology stack choices. Absolute paths containing words like "azureuser" or "poc" bias agent decisions. Language server feedback loops that exist on developer machines but not in CI environments also change scores.
Level
Hype check
Tap to vote and see what everyone thinks.
