LAKSPIRE
LAKSPIRE

Proof of Work · Portfolio

Data annotation, documented end to end.

Lakspire provides AI data annotation services: image and video annotation, object tracking, segmentation and dataset quality validation, with text and speech labelling alongside. This portfolio sets out each service, six pilot projects with sample outputs, three case studies and the QA standard applied to every delivery.

View pilot projectsRequest a pilot batch
6Pilot projects4Data types3Case studies7QA steps

Service catalogue

6 services

Offerings

Four core services.
Two more for text and speech.

Image and video annotation, object tracking, segmentation and dataset quality validation, each delivered against a written guideline and a measured QA threshold.

01

Image & video annotation

Bounding boxes, polygons, keypoints, cuboids and classification tags, drawn to written guidelines and checked frame by frame.

COCOYOLOPascal VOC
02

Object tracking

Persistent IDs across video frames, kept through occlusion, crossings and frame exits, with interpolation checked and drift corrected.

MOTChallengeCVAT XML
03

Segmentation

Semantic, instance and panoptic masks with pixel-level boundaries. Model-assisted pre-masks, always corrected by a person.

COCO RLEPNG masks
04

Dataset quality validation

Audits of existing datasets: label-error detection, agreement scoring, class balance and duplicate checks, with a per-item change log.

Audit reportJSONL log

Text & NLP labelling

Intent, entity, sentiment and PII tagging on conversations, documents and search queries.

JSONLCoNLL / IOB

Audio & speech

Verbatim transcription, speaker turns and sound-event tags, time-aligned to the waveform.

RTTMWebVTTJSON

Pilot projects

Six pilots, six data types.
Drag to see raw become labelled.

Why pilots, not logos.

Lakspire is a new company, so we do not show borrowed logos or unverifiable client numbers. Instead we run pilots on public datasets that come with published ground truth. You can download the same data, check our labels against it, and see exactly how we work before you send us a single file.

ID 12 · carID 3 · personID 4 · personID 7 · car4 tracks · IDs held through crossing · 0 switchesRAWANNOTATED
Internal pilot · acceptance targetsBounding boxes + persistent track IDs

Multi-object tracking in street video

Pedestrians and vehicles tracked across crowded street scenes. The hard part is keeping one identity per object through occlusion and crossings. That is what the QA pass is built around.

≥ 0.80

Box IoU vs ground truth

Target

≥ 0.95

IDF1

Target

≤ 2

ID switches / 1k track-frames

Target

Volume
3 sequences · 1,800 frames
QA sample
180 frames (10%) blind re-review
Automated checks
100% of tracks
Ground truth
MOT17 published labels
CVATTrackEvalFiftyOne

Guideline excerpt · v1.0

  1. 01Box the visible extent only. Over 50% occluded → flag “occluded”, keep the track alive.
  2. 02A track ID is never reused after its object leaves the frame.
  3. 03Keyframe every 10 frames, interpolate, then correct drift on every frame.
  4. 04Objects cut by the frame edge are boxed to the edge and flagged “truncated”.

Output sample

MOTChallenge gt.txt

# frame,id,x,y,w,h,conf,class,visibility
1,7,912,484,97,109,1,1,1.00
2,7,915,483,97,110,1,1,0.92
2,12,1338,418,167,379,1,1,0.40
3,12,1341,417,166,380,1,1,0.35

Source data:MOT17 (MOTChallenge)· CC BY-NC-SA 3.0 · scene above is an illustration, not dataset imagery

Case studies

Three problems every annotation team hits.
And the process we use to fix each one.

Each case starts with a real, documented failure mode, explains why it happens, and lays out the steps, checks and deliverables we use to prevent it.

ACase A · Object tracking

Track IDs that swap when objects cross

The problem

In crowded video, two pedestrians walk past each other or a car disappears behind a bus. When they reappear, the old ID ends up on the wrong object, or a fresh ID is started. Every frame still “looks” labelled, but the model is now learning broken identities.

Identity switches are a core error type in the standard tracking metrics (CLEAR MOT, IDF1, HOTA). One swap mid-sequence damages every frame after it, not just one frame.
Luiten et al., “HOTA”, IJCV 2021; Ristani et al., ECCV-W 2016

Why it happens

  • Linear interpolation between keyframes cannot follow two objects that cross paths.
  • Annotators work frame by frame and rarely re-check identity after an occlusion.
  • QA that samples random frames almost never lands on the few frames where the swap happens.
INTERPOLATION ONLYID 7ID 12ID switch ×2WITH CROSSING-EVENT REVIEWID 7ID 12event reviewed ✓ · IDs held

How Lakspire handles it

Step 01

Occlusion-aware guideline

IDs persist through occlusion for up to 60 frames. Re-identification uses both appearance and trajectory, and that rule is written down.

Step 02

Denser keyframes where it is crowded

Segments with many overlapping boxes switch from 10-frame to 2-frame keyframing, so interpolation is never trusted across a crossing.

Step 03

Crossing-event review queue

A script flags every frame where two tracks overlap (IoU > 0.3). A senior reviewer confirms identities before and after each event.

Step 04

Trajectory continuity check

Sudden jumps in velocity or box size are flagged automatically. That is the usual sign of an ID landing on the wrong object.

Step 05

Score before delivery

A gold sequence is scored with TrackEval (IDF1, HOTA, ID switches). The batch ships only if it meets the agreed threshold.

≤ 2

ID switches / 1k track-frames

≥ 0.95

IDF1 on gold sequence

100%

Crossing events reviewed

What you receive

MOTChallenge-format tracks, a crossing-event log, and a TrackEval score sheet per batch.

BCase B · Label consistency

Annotators disagree on edge cases

The problem

Two trained annotators look at the same item and label it differently. Is a half-visible bag a “handbag” or a “backpack”? Does the polyp boundary sit at the shadow or at the fold? Each label is defensible on its own, but together they are noise, and the model learns the disagreement.

The standard reference on content analysis recommends reliability of α ≥ 0.800 before relying on labelled data, and treats 0.667–0.800 as suitable for tentative conclusions only.
Krippendorff, Content Analysis, 2nd ed., 2004 (summary on Wikipedia)

Why it happens

  • Guidelines describe the typical case, but edge cases are where the disagreement lives.
  • A single overall agreement score hides the two or three classes that are inconsistent.
  • When an adjudicator settles a dispute in chat, the ruling is never written down, so the next annotator makes the same call differently.
1Overlap slice2Agreement / class3Adjudicate4Guideline v+15Gold quizκ / α per class≥ 0.80guideline v1.0guideline v1.1guideline v1.2

How Lakspire handles it

Step 01

Overlap slice

10–15% of every batch is labelled independently by two annotators, without either seeing the other’s work.

Step 02

Agreement per class

Cohen’s κ or Krippendorff’s α is computed for each class, not just overall, so the weak classes are visible.

Step 03

Adjudicate and record the ruling

An adjudicator resolves every disagreement. Each ruling becomes a do / don’t example in the guideline.

Step 04

Version the guideline

v1.0 → v1.1 → v1.2, with a changelog. Every label records the guideline version it was made under.

Step 05

Recalibrate with gold questions

Annotators pass a gold quiz on the new version, and about 5% hidden gold items stay in the live queue.

≥ 0.80

Agreement per class (κ / α)

≥ 95%

Gold-question accuracy

100%

Rulings captured in guideline

What you receive

A versioned guideline with a decision log, per-class agreement reports, and labels tagged with the guideline version they were made under.

CCase C · Dataset quality validation

A client’s existing dataset is noisy

The problem

The client already has a labelled dataset, from an in-house team, a crowd platform or a previous vendor. The model has plateaued and nobody knows whether the cause is the model or the labels. Relabelling everything is slow and expensive.

A study of ten widely used benchmark test sets found an average of at least 3.3% label errors, and around 6% in the ImageNet validation set. Test sets are usually the most carefully checked data a team has.
Northcutt, Athalye & Mueller, NeurIPS 2021 Datasets & Benchmarks

Why it happens

  • Errors are rare (a few percent), so random spot checks miss most of them.
  • Crowd labels drift as guidelines change between batches.
  • Duplicates across train and test inflate scores and hide the real problem.
DATASETcleanlab scanhumanreview100% of flagsrelabelkeepremoveambiguouschange logbefore/after

How Lakspire handles it

Step 01

Intake & schema audit

Class list, distribution, near-duplicates and train/test leakage checks, before any labels are touched.

Step 02

Model-assisted triage

Confident learning (cleanlab) on out-of-sample predictions ranks every item by how likely its label is wrong.

Step 03

Human review of suspects

Each flagged item gets one decision: keep, relabel, remove or ambiguous. The reason is recorded every time.

Step 04

Estimate what was missed

A random sample of unflagged items is reviewed too, so the leftover error rate is measured, not assumed.

Step 05

Deliver with a change log

The corrected dataset ships with a per-item log and a before/after error-rate report for each class.

100%

Flagged items human-reviewed

Reported ± CI

Residual error estimate

100%

Decisions with reason

What you receive

Corrected dataset, a JSONL change log, a per-class error report and a duplicate/leakage report.

Our QA standard

Seven steps, every engagement.
Each one leaves a document behind.

This is the process a serious annotation vendor should follow. Every step produces something you can inspect, so quality never rests on our word alone.

01

Scope & data audit

Agree on classes, edge cases, volumes and acceptance metrics before anything is labelled.

→ Statement of work + metric thresholds

02

Guideline v1.0

A written taxonomy with visual do / don’t examples for every class and every known edge case.

→ Versioned guideline PDF

03

Calibration pilot

A small batch labelled by the full team, scored against a gold set and reviewed with you.

→ Pilot report + guideline v1.1

04

Production labelling

Trained annotators work in the agreed tool. Model-assisted pre-labels are always corrected by a person.

→ Labelled batches

05

Multi-layer QA

Self-check, peer review, hidden gold questions, an overlap slice for agreement, and automated geometry and schema checks.

→ QA scorecard per batch

06

Versioned delivery

Exports in your format, with checksums, the guideline version and a per-batch QA report.

→ Dataset release vX.Y

07

Feedback loop

Your model’s failure cases come back to us and become new guideline examples and targeted re-reviews.

→ Guideline changelog

What we measure

Boxes
IoU vs gold
Masks
Dice / boundary error
Tracking
IDF1 · HOTA · ID switches
Classes / tags
Accuracy vs gold
Consistency
Cohen’s κ · Krippendorff’s α
Text spans
Entity-level F1
Speech
Word error rate
Audits
Error rate before / after

Thresholds are agreed per project in the statement of work, and reported on every batch.

Tools & formats

Open, auditable tooling.
Delivered in the format your pipeline expects.

We work in established open-source tools, so there is no lock-in. Your team can open our project files, re-run our checks, or host the same tools yourselves.

CVATImage & video annotation, trackingMIT
Label StudioImage, text & audio labellingApache-2.0
FiftyOneDataset review & QA viewsApache-2.0
cleanlabLabel-error detectionAGPL-3.0
Segment AnythingMask pre-labelsApache-2.0
TrackEvalHOTA / IDF1 scoringMIT
doccanoText span labellingMIT
spaCyText export & validationMIT
WhisperSpeech pre-transcriptsMIT
pyannote.audioSpeaker pre-diarizationMIT

Export formats

COCO JSONYOLOPascal VOCMOTChallengeCVAT XMLPNG masksJSONLCoNLL / IOBRTTMWebVTTCSV

Sources

Every dataset and figure, cited.
So you can check our work.

Pilot scenes on this page are illustrations drawn by our team, not dataset imagery. The pilots themselves run on the public datasets below, under their published licences.

MOT17Milan et al., “MOT16: A Benchmark for Multi-Object Tracking”, arXiv:1603.00831Tracking pilotCC BY-NC-SA 3.0Kvasir-SEGJha et al., “Kvasir-SEG: A Segmented Polyp Dataset”, MMM 2020 (1,000 images + masks)Segmentation pilotResearch & educationOpen Images V7Google, boxes for Footwear, Handbag, Watch, Sunglasses, HatE-commerce pilotCC BY 4.0 / CC BY 2.0Quick, Draw! + label-error listNorthcutt, Athalye & Mueller, “Pervasive Label Errors in Test Sets…”, NeurIPS 2021Audit pilot · Case study CCC BY 4.0CLINC150Larson et al., “An Evaluation Dataset for Intent Classification…”, EMNLP 2019Text pilotCC BY 3.0Common Voice · AMI Meeting CorpusMozilla Common Voice; Carletta et al., AMI Meeting CorpusAudio pilotCC0 · CC BY 4.0HOTA · IDF1 · CLEAR MOTLuiten et al., IJCV 2021; Ristani et al., ECCV-W 2016; Bernardin & Stiefelhagen, 2008Case study APapersKrippendorff’s α thresholdsKrippendorff, Content Analysis: An Introduction to Its Methodology, 2004Case study BBookConfident learningNorthcutt, Jiang & Chuang, “Confident Learning…”, JAIR 2021Case study CPaper

Send us a sample of your data. Judge the labels yourself.

We will agree on a guideline with you, label a pilot batch, and return it with a QA scorecard against the thresholds you set.

Request a pilot batch