Brierly docs
v1.0.0
Get help
● Use

How scoring works
Brier score, calibration, you vs the crowd.

Every number in the app is computed on the phone from the calls the user sealed and the results the server reports. The rules below are read from app/lib/src/core/scoring.dart and models.dart.

Brier scorecalibration binsyou vs the crowdlate calls

01The Brier score

For one decided call the score is (p - o)^2, where p is the probability the user gave for Yes (a whole percent from 1 to 99, so 0.01 to 0.99) and o is 1 if the answer was Yes and 0 if it was No. The score runs from 0 (perfect) to 1, and lower is better. The card shows the mean over all decided calls.

User saidResultScore
70% YesYes(0.70 - 1)^2 = 0.090
70% YesNo(0.70 - 0)^2 = 0.490
50% (no opinion)either0.250
90% YesNo0.810

Always saying 50 percent scores 0.250, so the card uses 0.250 as the line a useful forecaster stays under. Void or cancelled questions and withdrawn calls never count in any score.

02The calibration curve (reliability diagram)

The question it answers: when you said X percent, how often did it happen? Decided calls are sorted into 10 bins by the probability given: 0 to 9, 10 to 19, and so on up to 90 to 99. For each bin the app shows the mean probability given and the share of those calls that came true. The dashed diagonal is perfect calibration; points above it mean the user was too cautious there, points below it too confident. A bin with no calls is left empty, and a bin with few calls is a rough guide only.

The card draws firm conclusions only after 10 decided calls; before that it says "too early to tell" and shows the diagram muted.

03You vs the crowd

When a call is sealed, the app stores the crowd's probability of Yes taken from the server at that moment. The comparison uses only the calls where that crowd value exists and was recorded within 60 minutes of sealing, so the comparison is like for like. It shows the user's mean Brier score on those calls, the crowd's mean Brier score on the same calls, and the difference (user minus crowd): negative means better than the crowd, positive means behind it. With fewer than 10 decided calls, the screen adds "based on a small sample".

ExampleNumbers
Crowd said 62%, user said 70%, result YesUser 0.090, crowd 0.144, difference -0.054 (ahead of the crowd on this call)

A custom question with no reference odds has no crowd, so its calls are scored but not compared.

04Late calls

A call sealed less than 24 hours before the question closes is a late call. Late calls are scored apart, so near-certain answers on questions that are already almost decided cannot lift the headline number. The headline Brier score uses the on-time calls when there are any; the card then states how many late calls are kept out and their own Brier score. If there are only late calls, they make up the score and the card says so.

05Pro limits and categories

Without Pro, scores and the diagram cover the last 30 days by default (setting), counted from the time each call was sealed; older decided calls are hidden from the card and remain on the phone. Pro covers everything and adds a by-category breakdown, one line of score per category of the question (categories come from the source or from your custom questions).

06What a score is not

A score describes how well the user's own stated probabilities matched the results so far. It is not a prediction, advice or a measure of anything else, and a small number of calls says little. Describe it that way in your listing.