Denali-AI Garment VLM Benchmark
Submit your Vision-Language Model for evaluation on our 3,500-sample hard garment classification benchmark.
Your model will be automatically downloaded, served via vLLM on our GPU cluster (2x RTX PRO 6000 Blackwell, 196 GB VRAM), and evaluated across 6 metrics on 9 garment attribute fields.
Supported models: Any VLM on HuggingFace compatible with vLLM — public, private, or gated.
Benchmark Results
Ranked by Weighted Field Score (SBERT+NLI combined, field weights: type=2.5x, defect=2.0x, brand/size=1.5x, others=1.0x)
Submit Your Model for Benchmarking
Fill in the form below. Your model will be queued and automatically evaluated. Results typically appear within 30-60 minutes depending on model size and queue depth.
Evaluation Queue
Models currently waiting to be or being evaluated.
Check Submission Progress
Paste your submission ID (8-character code shown when you submitted) or your model ID (org/name) to see live status, queue position, current step, and progress.
Benchmark Details
Dataset
- 3,500 challenging garment images from diverse sources
- Covers shoes, bags, caps, swimwear, tops, bottoms, dresses, outerwear, and more
- Includes edge cases: unusual colors, mixed patterns, hard-to-read labels, subtle defects
Task
Extract 9 structured attributes from each garment image as a JSON object:
{
"type": "Dress",
"color": "Navy Blue",
"pattern": "Floral",
"neckline": "V-Neck",
"sleeve_length": "Short Sleeve",
"closure": "Zipper",
"brand": "N/A",
"size": "M",
"defect_type": ""
}
Metrics (6 total)
| Metric | Description |
|---|---|
| SBERT Cosine | Semantic similarity via all-MiniLM-L6-v2 embeddings |
| NLI Score | Entailment/contradiction via cross-encoder/nli-MiniLM2-L6-H768 |
| Levenshtein Ratio | Character-level edit distance (0.0-1.0) |
| Token F1 | Token-level precision, recall, F1 for multi-word fields |
| SBERT+NLI Combined | Primary metric — cascaded SBERT base + NLI adjustment |
| Weighted Field Score | Headline metric — field-weighted SBERT+NLI |
Field Weights
| Field | Weight | Rationale |
|---|---|---|
| type | 2.5x | Most critical for classification |
| defect_type | 2.0x | Quality control critical |
| brand | 1.5x | Authentication value |
| size | 1.5x | Inventory management |
| color, pattern, neckline, sleeve, closure | 1.0x | Standard attributes |
Private & Gated Models
You can submit private or gated models by providing a HuggingFace access token in the submission form. The token is stored securely in a private dataset repo and used only to download the model weights during evaluation. Read-access tokens are sufficient.
Infrastructure
- 2x NVIDIA RTX PRO 6000 Blackwell (98 GB VRAM each, 196 GB total)
- Models served via vLLM with OpenAI-compatible API
- 8 concurrent evaluation workers
- Typical eval time: 15-45 minutes per model (depending on size)
Supported Model Formats
Any VLM that works with vLLM's OpenAI-compatible serving:
- Qwen-VL / Qwen2-VL / Qwen2.5-VL / Qwen3-VL
- InternVL / InternVL2 / InternVL3
- Florence-2
- LLaVA / LLaVA-NeXT
- Molmo, Pixtral, Phi-Vision, MiniCPM-V, DeepSeek-VL
- Any model with a supported
chat_templatefor vision
Contact
Questions or issues? Open a discussion on the Denali-AI org.