Sam Burrett posted today: Insurance policies are a nightmare for most consumers. It’s hard to know what’s covered and what isn’t. Could AI help? The Stanford CodeX team tested GPT4o’s ability to turn an insurance policy into code, which could be queried in a simple question/answer format. They found GPT4 produced tangled code and misunderstood key provisions.

However, OpenAI’s new ‘reasoning’ model was much better. o1-preview correctly encoded and insurance policy and hit 83% accuracy on coverage questions. Is that good enough for the real world? Not yet.

But as the researchers say: “We are on the cusp of an exciting era where AI can make legal solutions more accessible by applying human-like thinking, including planning and reasoning.” That’s exciting. Link