Evaluating usage-based billing software: a seven-test scorecard
Two usage-based billing vendors, two good demos, two similar prices. After three weeks of evaluation your team has a 60-row spreadsheet with "yes" next to almost every line for both. The choice ends up being made on rapport or list price, and the real differences only show in month three after go-live, when a customer disputes an invoice for 48,120 API calls.
This article is a scorecard for evaluating usage-based billing software: seven weighted criteria, each with a test the vendor has to run in front of you, scored from 0 to 3. General requirements for billing software in the EU, from SEPA to accounting exports, are in our requirements list for billing software. This piece is about what's different with usage: events, corrections, pricing logic and traceability.
Score vendors on your own data
Before comparing anyone, export a week of real usage events from your product, anonymised and with all their mess intact: duplicates from retries, events timestamped in the previous week, events from test accounts. Add three real contracts, including the most complicated one sales has ever signed.
Every vendor gets the same data and the same contracts. You score what happens in the sandbox, not what's on the slides. A vendor who won't provide a sandbox for this has already answered one question.
The scorecard: seven criteria, weighted
The weights below suit a typical B2B SaaS company with API or volume pricing and customers in several EU countries. Shift them for your business, but fix them before you see the first demo.
| Criterion | Weight | Test in the sandbox |
|---|---|---|
| Event ingestion and idempotency | 20 | Send the same week of events twice. Does the invoice change? |
| Corrections and late events | 15 | Send an event timestamped in a month that's already been invoiced |
| Pricing models | 15 | Have them model your most complex contract without custom development |
| Invoice line traceability | 15 | Drill down from an invoice line to the individual events |
| EU tax and e-invoicing | 15 | Produce a usage invoice for a B2B customer in another EU country as an EN 16931 e-invoice |
| Customer visibility and limits | 10 | See usage in the customer portal and trigger an alert at 80% of an allowance |
| Export and data access | 10 | Export raw events and billed quantities for one month |
Scoring per criterion: 0 = not available, 1 = only with custom code or a manual step, 2 = available with limitations, 3 = available and passed the test.
What each criterion is really testing
Event ingestion and idempotency: each event needs a unique key, and the software must discard a second event with the same key, without an error and without double-counting. The IETF Idempotency-Key header draft describes the pattern. Ask about throughput too, and what happens under overload: are events buffered or rejected? How to get idempotency right on your side is covered in our post on idempotent usage event processing.
Corrections and late events: an event from 30 April arrives on 3 May, after April's invoice has been finalised. Good software has a fixed rule for this, such as billing it on the May invoice with a reference to the April service period. Poor software edits the finalised invoice or drops the event. Either one is a knock-out.
Pricing models: free allowances, graduated tiers, volume pricing, minimum commitments, prepaid credits and a per-seat base fee. The distinction that trips people up is graduated versus volume. At 12,000 calls with a break at 10,000, graduated pricing bills the first 10,000 at the first rate and 2,000 at the second, while volume pricing bills all 12,000 at the second. Have the vendor calculate both with the same numbers.
Invoice line traceability: a customer disputes "48,120 API calls". Your support team needs to show within minutes which events sit behind that number, which were discarded as duplicates and which correction was applied. In Germany invoices must be kept for eight years, and the derivation of the quantity should be reconstructable for as long.
EU tax and e-invoicing: usage invoices are invoices. A customer in France with a valid VAT ID needs the reverse-charge wording, and German business customers need e-invoices for domestic B2B from 2028. Check the output with the KoSIT validator, not by eye. Also ask whether tax is determined per line or per invoice when a contract mixes services with different tax treatment.
Customer visibility and limits: usage pricing produces invoices customers didn't see coming. A portal showing current usage and alerting before an allowance runs out saves support tickets and credit notes.
Export and data access: you need raw events and billed quantities in your data warehouse, or you'll have two versions of the truth about revenue and usage. Check whether the export is complete or only gives you aggregates.
A worked example with two vendors
Made-up results for two vendors after testing with your data:
| Criterion | Weight | Vendor A | Vendor B |
|---|---|---|---|
| Event ingestion and idempotency | 20 | 3 | 3 |
| Corrections and late events | 15 | 1 | 3 |
| Pricing models | 15 | 3 | 2 |
| Invoice line traceability | 15 | 2 | 3 |
| EU tax and e-invoicing | 15 | 1 | 3 |
| Customer visibility and limits | 10 | 3 | 1 |
| Export and data access | 10 | 2 | 2 |
| Total (weight × score, max 300) | 215 | 255 |
Vendor A had the more impressive demo: flexible pricing and a polished customer portal. Vendor B still wins clearly because it passes on late events and EU tax, where A needs custom work. Those gaps cost effort every month, whereas a weaker portal can be built around.
Set knock-out criteria that apply regardless of the total as well. A 0 on idempotency or corrections should rule a vendor out however high it scores elsewhere.
When a simpler tool is enough
Not every company with a usage component needs a specialist system. An invoicing tool with simple quantity lines often does the job if all three of these hold:
- There's a single usage metric, aggregated monthly, with a linear price.
- The quantity comes from a report your team reviews before the billing run.
- Corrections are rare enough that a manual credit note is acceptable.
Once you need several meters, tiers, prepaid credits or real-time limits, or once customers regularly dispute their bills, it stops being enough.
The order to run the evaluation in
- Write down weights and knock-out criteria.
- Prepare a week of real events and three contracts.
- Have each vendor run the seven tests in a sandbox with finance and engineering in the room.
- Score straight after each session, not at the end.
- With the winner, bill a full month in the sandbox and compare it with your current invoices.
Kontier combines usage metering, tiered pricing, subscription billing and EU tax logic in one system; the usage-based billing page has the details.
Not tax or legal advice. This post describes the law and common practice in general terms. For your own case, ask your tax adviser. As of: