# VCD2 Dataset Metadata

This directory contains metadata for the VCD2 dataset, a camera-native talking-head video dataset
for video compression, quality assessment, super-resolution, and enhancement research.

For more information, see the [VCD repository](https://github.com/microsoft/VCD).

## Files

| File | Rows | Description |
|------|-----:|-------------|
| `vcd2_metadata.csv` | 838 | Full dataset: all published clips |
| `benchmarking-set_metadata.csv` | 120 | Benchmarking subset: stratified selection for codec and SR evaluation |

## Dataset Overview

- **838 clips**, each approximately 15 seconds, captured from 799 participants using 442 unique consumer webcams
- All recordings are stored using the **FFV1 lossless codec** in MKV containers
- Camera-native signal preserved: uncompressed YUYV422/NV12 (24.7%) or MJPEG-encoded (75.3%)
- Each clip is annotated with a **Mean Opinion Score (MOS)** and **10 perceptual quality tokens**

## Container Structure

Files are hosted at `https://vcdpublic.blob.core.windows.net/vcd2/` with the following layout:

```
vcd2/
├── raw/
│   └── mkv/                 # 838 full clips (FFV1 lossless)
├── benchmarking/
│   └── mkv/
│       ├── TH/              # 40 clips — Talking Head (original)
│       ├── TH-BB/           # 40 clips — Talking Head, Background Blur
│       └── TH-BR/           # 40 clips — Talking Head, Background Replacement
└── metadata/
    ├── vcd2_metadata.csv
    └── benchmarking-set_metadata.csv
```

## Variable Frame Rate (VFR) Note

The raw clips (`raw/mkv/`) preserve the original camera-native timestamps, which may include
**Variable Frame Rate (VFR)** irregularities. This is common with USB webcam captures, where the
camera or USB bus occasionally delivers frames unevenly (e.g., 0 ms burst gaps followed by
67–100 ms gaps). Approximately 40% of the TH-group source clips exhibit VFR timestamps.

The **benchmarking subset** (`benchmarking/mkv/`) has been corrected to **Constant Frame Rate (CFR)**
using a lossless YUV roundtrip that guarantees pixel-perfect output with no frame duplication,
dropping, or interpolation:

1. Extract raw YUV pixels from the source MKV using `ffmpeg -fps_mode passthrough` (preserves all
   frames without duplication)
2. Re-wrap the raw pixels into a new FFV1 MKV container with evenly-spaced CFR timestamps at the
   rounded real frame rate

This process preserves the exact pixel data while producing clean, constant-rate timestamps
suitable for codec benchmarking and downstream encoding.
---

## Column Descriptions

### `vcd2_metadata.csv`

| Column | Type | Description |
|--------|------|-------------|
| `relative_path` | string | Path to the MKV file relative to the container root (e.g., `raw/mkv/{hash}.mkv`) |
| `video_resolution` | string | Video resolution as `WxH` (e.g., `1920x1080`, `1280x720`, `2560x1440`) |
| `video_real_fps` | float | Measured frame rate of the recording (may differ from nominal 30 fps) |
| `video_frames` | int | Total number of frames in the recording |
| `video_duration` | string | Duration of the recording (e.g., `15.00s`) |
| `video_pixel_format` | string | Pixel format of the stored video (e.g., `yuv422p`) |
| `log_recording_camera_code` | string | USB vendor and product ID of the webcam (e.g., `vid_046d&pid_08e5`) |
| `log_recording_input_fmt` | string | Raw input pixel format from the camera: `mjpeg`, `yuyv422`, or `nv12` |
| `log_scenario_id` | string | Recording scenario: `S01` (talking head), `S02` (gestures/movement), `S03` (reading), `S04` (mixed) |
| `n` | int | Number of raters who scored this clip |
| `MOS` | float | Mean Opinion Score (1–5 scale; higher is better quality) |
| `std` | float | Standard deviation of opinion scores |
| `95%CI` | float | 95% confidence interval half-width of the MOS |
| `mos_category` | string | Quality category derived from MOS: `High`, `Medium`, or `Low` |
| `si_mean` | float | Mean Spatial Information (SI) — higher values indicate more spatial detail |
| `si_95ci` | float | 95% confidence interval half-width of SI |
| `ti_mean` | float | Mean Temporal Information (TI) — higher values indicate more motion |
| `ti_95ci` | float | 95% confidence interval half-width of TI |
| `si_ti_group` | string | SI/TI quadrant: `High SI - High TI`, `High SI - Low TI`, `Low SI - High TI`, or `Low SI - Low TI` |
| `num_raters_problem_tokens` | int | Number of raters who provided perceptual quality tokens |
| `token_blur_percent` | float | Fraction of raters who identified blur (0.0–1.0) |
| `token_blockiness_percent` | float | Fraction of raters who identified blockiness |
| `token_light_percent` | float | Fraction of raters who identified lighting issues |
| `token_banding_percent` | float | Fraction of raters who identified banding artifacts |
| `token_no-issue_percent` | float | Fraction of raters who reported no quality issues |
| `token_framerate_percent` | float | Fraction of raters who identified frame rate issues |
| `token_noise_percent` | float | Fraction of raters who identified noise |
| `token_motion_percent` | float | Fraction of raters who identified motion artifacts |
| `token_low-res_percent` | float | Fraction of raters who identified low resolution |
| `token_others_percent` | float | Fraction of raters who identified other issues |
| `gender` | string | Self-reported gender of the participant |
| `ethnicity_simplified` | string | Self-reported ethnicity (simplified categories) |
| `vcd2_benchmarking` | string | Benchmarking group assignment: `TH`, `TH-BB`, `TH-BR`, or `none` if not in the benchmarking subset |


### `benchmarking-set_metadata.csv`

This file contains the 120 clips selected for the benchmarking subset. Columns are the same as
`vcd2_metadata.csv` with the following differences:

| Column | Type | Description |
|--------|------|-------------|
| `relative_path` | string | Path relative to container root (e.g., `benchmarking/mkv/TH/{hash}.mkv`) |
| `group` | string | Benchmarking group: `TH` (Talking Head), `TH-BB` (Background Blur), or `TH-BR` (Background Replacement) |

The columns `mos_category`, `si_ti_group`, `vcd2_benchmarking`, and `index` are not present in this file.

---

## Benchmarking Groups

| Group | Full Name | Clips | Description |
|-------|-----------|------:|-------------|
| **TH** | Talking Head | 40 | Original clips without post-processing |
| **TH-BB** | Talking Head — Background Blur | 40 | Processed with commercial-grade background blur |
| **TH-BR** | Talking Head — Background Replacement | 40 | Processed with commercial-grade background replacement using four virtual backgrounds |

## Citation

```BibTeX
@inproceedings{naderi2026vcd2,
  title={A Camera-Native Talking-Head Video Dataset for Various Computer Vision Tasks},
  author={Naderi, Babak and Cutler, Ross and Khongbantabam, Nabakumar Singh},
  booktitle={arXiv},
  year={2026}
}
```