Finetuned SigLIP for Person Visual Descriptions and Reidentification

This model is part of the family of SigLIP models finetuned for person visual description and retrieval.

Model Details

• Base model: google/siglip2-base-patch16-224
• Architecture modifications: none

Intended Uses & Limitations

Example Applications

• Person retrieval based on textual or visual descriptions of the person
• Person image re-identification
• Embedding extraction for retrieval systems

Limitations and Bias

• May inherit biases from the base SigLIP model and training data
• Not suitable for tasks requiring detailed fine-grained recognition without further training
• Trained on surveillance data; suitable for tasks where a substantial portion of the person is visible

Training

Loss function

• Type: Soft contrastive loss with label smoothing; image-to-image contrastive loss for same-identity image pairs.
• Description: The model is trained to align text and image embeddings using a modified contrastive objective. Instead of relying on hard one-hot targets, label smoothing allocates a small probability mass to all other samples in the batch. All embeddings are normalized prior to similarity computation, and the loss is applied symmetrically in both the image-to-text and text-to-image directions. To encourage re-identification and emphasize clothing features rather than pose or background, an additional image-to-image contrastive loss is incorporated.

Evaluation Metrics

• Truncated Cumulative Matching Characteristic (CMC) AUC
  • Measures the fraction of queries where the correct match appears within the top K ranks (e.g., top-10). Unlike MRR or strict top-1 accuracy, this metric rewards consistent retrieval of all relevant matches near the top ranks, rather than a few perfect hits with others ranked very low.

Datasets

• Sources: CUHK-PEDES, ICFG-PEDES, IIITD, ITCPR, PRW-TBPS, PETA, (part of SYNTH and RSTPReid for testing)
• Processing: Text descriptions were processed with the Mistral LLM to remove ambiguous information about pose or context, leaving only clear visual characteristics in a structured format, where key features are separated with comma. During train/val/test the original splits are respected as much as possible.

The splits are created as follows:

• CUHK-PEDES: test: 1000 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• ICFG-PEDES: test: 946 identities following the original split; validation: 150 identities sampled from the original training split; train: all remaining identities
• IIITD: test: 2500 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• ITCPR: Originally designed for zero-shot evaluation, so a new split is created:test: 1000 randomly selected identities; validation: 150 randomly selected identities; train: all remaining identities
• PRW: test: 450 identities following the original split; validation: 150 identities sampled from the original training split; train: all remaining identities
• PETA: The original random image-level split can cause identity leakage. Therefore, the identity-based split from PETA-ZS is adopted: test: 1706 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• RSTPReid: Only the 200 test images are used for evaluation due to annotation errors in the rest of the dataset.
• SYNTH: 1000 random images are used for the test set because the generated captions are noisy and sometimes inaccurate.

Training Setup

• Frozen model head initially; only heads were trained.
• Fine-tuning: later trained the whole model.
• Optimizer: AdamW
• Learning rate: 1e-4 for warm-up, 5e-6 for the rest
• Epochs: 4 epochs for model head fine-tuning, and 50 epochs for full model fine-tuning
• Visual Augmentations: random operations including small rotations, scale, hue variations, horizontal flip, and color jitter
• Text Augmentations: random subsets of key features are selected and removed from the comma-separated description strings to create augmented training samples
• Training Codes: codes are available on [github] (https://github.com/MarketaJu/ReducedSiglipForPersonDescription)
• Package Versions: torch==2.9.1, transformers==4.57.3, pillow==12.0.0, torchvision==0.24.1

Results

The model is evaluated on data mentioned above. The following table summarizes the number of identities, images, and queries for each subset.
The final test set is a merge of all subsets.

Evaluation of the Model on Text-based Image Retrieval Task

Since the task is focused on retrieving the correct person identity rather than the exact matching image, the evaluation is performed as follows:

• For each text query, the goal is to retrieve the correct identity, not the exact corresponding image.
• During evaluation, scores are computed over all images belonging to the same identity, and the maximum score is taken to represent that identity.
• These identity-level scores are then ranked, and the following retrieval metrics are calculated:
  • Top-k: standard top-1, top-5, top-10 accuracy
  • MRR: Mean Reciprocal Rank
  • CMC AUC: Truncated CMC AUC evaluated up to rank 20

Cross-Model Comparison

Evaluation of the Model on Image Reidentification Task

Since some images from Market-1501 are also included in the aforementioned datasets used for training, the reported results on this dataset are not objective. Reported metrics are Top-1/5/10 and mAP.

Usage

The usage is identical to SigLIP.

# Import custom model code from repository
from modeling_resipvd import ReSiPVDModel

# Load the model from Hugging Face Hub
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224")
model = AutoModel.from_pretrained("MarketaJu/siglip2-person-description-reid")

# Example: get embeddings
from skimage.io import imread
image = imread("test.jpg")
text_inputs = processor(text=["random person description"], return_tensors="pt", padding="max_length", max_length=64, truncation=True)
image_inputs = processor(images=image, return_tensors="pt", padding="max_length", max_length=64, truncation=True)
text_embeds = model.get_text_features(**text_inputs)
image_embeds = model.get_image_features(**image_inputs)


---

## Citation

If you use this model, please cite:

```bibtex
@misc{reduced-siglip-visualdescription,
  title={Reduced SigLIP for Visual Descriptions},
  author={Marketa Jurankova},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/collections/MarketaJu/reduced-siglip-for-person-visual-description}}
}