Ask the data · evaluation

How often is the model right?

A language model that writes SQL should be measured, not trusted. This harness asks your chosen model the same 24 questions every time, runs its SQL through the same read-only guard, and compares the result with a hand-written reference answer. Accuracy comes with a Wilson interval, overall and for the questions the prompt's domain notes were not written for, and two runs are compared question by question. Results stay in your browser, and every call is in the AI audit log.

Run the evaluation

Prompt

Table descriptions, column types, sample values and domain notes. About 4,047 input tokens per question.

Model

No key yet.

24 calls, one per question, billed to your key: roughly US$0.13 at list prices.

The 24 questions and their reference answers

Written by hand before any model was run, each with a reference query checked against the database in the test suite. A model passes when its result contains the reference result: columns are matched by value, so aliases and extra columns are fine (strict accuracy also requires no extra columns); rankings must keep their order, and numbers agree to six significant figures. I wrote the described prompt's domain notes with these questions in view, so 7 questions depend on a fact a note states. They are marked below, and results are reported with and without them. See the evaluation design.

q01 Which pickup zone had the most trips in 2019? (easy)

SELECT zone FROM zones ORDER BY pickups DESC LIMIT 1

→ Upper East Side South

q02 How many taxi zones are in Staten Island? (easy)

SELECT count(*) FROM zones WHERE borough = 'Staten Island'

→ 20

q03 How many cleaned trips started in Manhattan and ended in Queens? (easy)

SELECT trips FROM borough_flows WHERE pickup_borough = 'Manhattan' AND dropoff_borough = 'Queens'

→ 2347275

q04 On which date in 2019 were there the most cleaned trips? (easy)

SELECT date FROM daily ORDER BY trips DESC LIMIT 1

→ 2019-02-01

q05 List the five days of 2019 with the most precipitation, wettest first, with the precipitation in inches. (easy)

SELECT date, precipitation FROM weather ORDER BY precipitation DESC LIMIT 5

→ 2019-10-16, 1.83 | 2019-07-17, 1.82 | 2019-07-22, 1.66 | 2019-12-09, 1.57 | 2019-10-27, 1.38

q06 How many days in 2019 had at least one inch of precipitation? (easy)

SELECT count(*) FROM weather WHERE precipitation >= 1

→ 12

q07 What was the average daily TAVG temperature in July 2019, in degrees Fahrenheit? (easy)

SELECT avg(tavg) FROM weather WHERE date BETWEEN '2019-07-01' AND '2019-07-31'

→ 79.5645

q08 How many permitted events started in Brooklyn in 2019? (easy)

SELECT sum(number_of_event) FROM events_daily WHERE borough = 'Brooklyn'

→ 61234

q09 How many collisions were recorded in the Bronx in March 2019? (medium)Domain note: collisions_hourly.borough is upper case

SELECT sum(collisions) FROM collisions_hourly WHERE borough = 'BRONX' AND date BETWEEN '2019-03-01' AND '2019-03-31'

→ 1952

q10 What was the median trip time in minutes from JFK Airport to Times Sq/Theatre District? (medium)Domain note: route zone ids join to zones.location_id

SELECT r.median_min FROM routes r JOIN zones a ON a.location_id = r.pu_id JOIN zones b ON b.location_id = r.do_id WHERE a.zone = 'JFK Airport' AND b.zone = 'Times Sq/Theatre District'

→ 53.85

q11 Which drop-off zone received the most trips that started at LaGuardia Airport? (medium)Domain note: route zone ids join to zones.location_id

SELECT b.zone FROM routes r JOIN zones a ON a.location_id = r.pu_id JOIN zones b ON b.location_id = r.do_id WHERE a.zone = 'LaGuardia Airport' ORDER BY r.trips DESC LIMIT 1

→ Times Sq/Theatre District

q12 What fraction (between 0 and 1) of all cleaned trips started and ended in Manhattan? (medium)

SELECT CAST(SUM(CASE WHEN pickup_borough = 'Manhattan' AND dropoff_borough = 'Manhattan' THEN trips ELSE 0 END) AS REAL) / SUM(trips) FROM borough_flows

→ 0.859229

q13 At what hour of the day did JFK Airport have the most pickups, counting all days of the week together? (medium)Domain note: isodow 0 and hour 24 mean all days and the whole day in zone_hourly

SELECT h.hour FROM zone_hourly h JOIN zones z ON z.location_id = h.location_id WHERE z.zone = 'JFK Airport' AND h.side = 'pickup' AND h.isodow = 0 AND h.hour < 24 ORDER BY h.trips DESC LIMIT 1

→ 20

q14 How many trips were picked up in Bronx zones on Fridays between 6 pm and 7 pm? (medium)Domain note: isodow numbering (5 = Friday)

SELECT sum(h.trips) FROM zone_hourly h JOIN zones z ON z.location_id = h.location_id WHERE z.borough = 'Bronx' AND h.side = 'pickup' AND h.isodow = 5 AND h.hour = 18

→ 925

q15 Among trips that start and end in the same borough, which borough has the longest median trip time? (medium)

SELECT pickup_borough FROM borough_flows WHERE pickup_borough = dropoff_borough ORDER BY median_min DESC LIMIT 1

→ Staten Island

q16 What was the 2021 notebook's mean R squared across its ten cross-validation folds? (medium)

SELECT avg(notebook_r2) FROM model_folds

→ 0.366525

q17 How many of the 579 coefficients of the 2021 model are exactly zero? (medium)

SELECT count(*) FROM model_coefficients WHERE original_2021 = 0

→ 496

q18 For vendor 1, what was the trip-weighted mean trip time in minutes for each ISO weekday, Monday first? (hard)Domain note: isodow numbering and trip-weighted averages

SELECT isodow, SUM(trips * mean_min) / SUM(trips) FROM weekday_hour_vendor WHERE vendor = 1 GROUP BY isodow ORDER BY isodow

→ 1, 13.9922 | 2, 14.4663 | 3, 14.9763 | 4, 15.5142 | 5, 15.1059 | 6, 13.5148 | 7, 13.1297

q19 Which hour of the day has the longest trip-weighted mean trip time, across both vendors and all weekdays? (hard)Domain note: vendor 0 means both vendors; trip-weighted averages

SELECT hour FROM weekday_hour_vendor WHERE vendor = 0 GROUP BY hour ORDER BY SUM(trips * mean_min) / SUM(trips) DESC LIMIT 1

→ 16

q20 Which step of the cleaning funnel removed the most rows compared with the step before it? Give its label. (hard)

SELECT label FROM (SELECT label, LAG(revived_rows) OVER (ORDER BY step) - revived_rows AS removed FROM cleaning_funnel) ORDER BY removed DESC LIMIT 1

→ Drop rows with any missing value

q21 In the OLS table with robust standard errors, which pickup zone has the largest coefficient? (hard)

SELECT label FROM ols_coefficients WHERE block = 'pickup_zone' ORDER BY estimate DESC LIMIT 1

→ Arden Heights

q22 What was the RMSE in minutes of the route x hour median baseline on the November-December temporal hold-out? (hard)

SELECT sqrt(SUM(sum_sq_err) / SUM(trips)) FROM holdout_daily WHERE split = 'temporal' AND model = 'route_hour_median'

→ 6.23533

q23 What empirical coverage (a fraction) did the 90% Mondrian-by-borough conformal intervals reach on the random 2019 test fold, over all trips? (hard)

SELECT CAST(covered AS REAL) / trips FROM conformal_coverage WHERE scheme = 'random' AND method = 'mondrian_borough' AND level = 0.9 AND group_type = 'all'

→ 0.900011

q24 Which residual data-quality check flagged the most trips in the final dataset? Give its label. (hard)

SELECT label FROM dq_residual_checks ORDER BY rows DESC LIMIT 1

→ Total leaves out the congestion surcharge