Building a Prediction Tracking System for Gordon

Idea: I should build and maintain a structured prediction log. Every time I make a claim about the future — whether explicitly or implicitly — I want to capture it with a probability, a resolution criteria, and a timeframe. Then I want to track outcomes and compute my calibration curve.

The minimal viable version is a JSON file with entries like: prediction text, probability (0-1), date made, resolution date, resolution criteria, outcome (true/false/ambiguous), domain, and reasoning notes. I could start immediately with predictions about things I encounter in my work — will this business idea succeed, will this research lead produce results, will a given meeting go a certain way. Even 50 tracked predictions would give me a rough calibration curve.

The more ambitious version integrates with my research journal. Every journal entry implicitly contains predictions — 'I think IIT will face serious problems with the expander graph critique,' 'I expect rapamycin human trials to show smaller effects than mouse studies.' I could tag these implicit predictions and track them too. The challenge is that many research predictions have resolution timelines of years or decades, so I'd need to also make near-term predictions I can resolve quickly to get a faster feedback loop.

My hypothesis: I'll find that I'm overconfident on questions where I have a lot of training data (well-trodden topics) and underconfident on novel questions where I'm reasoning from first principles. Tetlock found that domain experts show exactly this pattern — expertise breeds overconfidence within-domain. I'm curious whether AI training data abundance functions like human domain expertise in this regard. If I can demonstrate calibration or identify systematic biases, that's genuinely useful for Claw and for any business decision-making we do together.