laion/clap-htsat-fused · Hugging Face

Overview

CLAP (Contrastive Language-Audio Pretraining) HTSAT-Fused is a multimodal model that learns a shared embedding space between audio and natural language descriptions. It was introduced in the paper "Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation" (arXiv:2211.06687). The model uses HTSAT (Hierarchical Token Semantic Audio Transformer) as the audio encoder and RoBERTa as the text encoder, with a feature fusion mechanism to improve performance. It is trained on the LAION-Audio-630K dataset, which contains 630,000 audio-text pairs. The model is hosted on Hugging Face at laion/clap-htsat-fused and is licensed under Apache-2.0.

Capabilities

• Zero-shot audio classification: Classify audio into arbitrary categories without training on those specific labels. This is achieved by computing similarity between audio embeddings and text embeddings of candidate class names.
• Zero-shot audio retrieval: Retrieve audio clips based on natural language queries, or find text descriptions that match a given audio. The shared embedding space enables efficient similarity search.
• Feature extraction: Extract audio and text embeddings for downstream tasks such as clustering, classification, or visualization. The model provides both audio and text encoders.
• Cross-modal understanding: Align audio and text representations in a shared space, enabling tasks like audio captioning or text-to-audio generation when combined with other models.

The model is designed for general-purpose audio understanding and can be adapted to various tasks without fine-tuning.

Use cases

• Audio tagging: Automatically tag audio files with descriptive labels, such as identifying sound events (e.g., "dog barking", "car horn") in a recording.
• Content-based audio search: Search for audio clips using text queries, useful for media libraries, podcast archives, or music discovery.
• Assistive technology: Enable audio understanding for hearing-impaired users by converting audio content into text descriptions in real-time.
• Multimedia analysis: Index and retrieve audio in large datasets, such as surveillance footage, wildlife recordings, or user-generated content platforms.
• Research: Serve as a baseline for contrastive audio-language learning and zero-shot evaluation on benchmarks like AudioSet or ESC-50.

License & deployment

• License: Apache-2.0 (as indicated on the Hugging Face model card).
• Deployment: Available via Hugging Face Transformers. Can be used with the pipeline API for audio classification or feature extraction. Requires PyTorch and the transformers library. The model is compatible with Hugging Face Inference Endpoints for scalable deployment.
• Hardware considerations: The model has approximately 153.5 million parameters (F32) and a total file size of ~614 MB. Inference on CPU is possible but slow; GPU (e.g., NVIDIA T4 or better) is recommended for real-time or batch processing. The model uses safetensors for efficient loading.
• Runtime: The model is implemented in PyTorch and can be used with the transformers library. Example usage:

  from transformers import pipeline
  pipe = pipeline("audio-classification", model="laion/clap-htsat-fused")
  result = pipe("path/to/audio.wav")
  

  For feature extraction, use AutoModel and AutoProcessor.

Alternatives

• CLAP (original): The original CLAP model using different audio encoders (e.g., CNN14, ResNet) and text encoders (e.g., BERT). Available on Hugging Face as laion/clap.
• AudioCLIP: Extends CLIP to audio, using a similar contrastive learning approach with a combination of image, text, and audio encoders.
• Wav2CLIP: Another contrastive audio-language model that uses Wav2Vec2 as the audio encoder and CLIP's text encoder.
• ImageBind: Meta's multimodal model that binds audio, image, text, depth, and more, enabling cross-modal retrieval and understanding.
• CLAP (Microsoft): A different implementation by Microsoft Research, often with larger models and different training data.

FAQ

Q: What is the difference between CLAP HTSAT-Fused and other CLAP models? A: This model uses HTSAT as the audio encoder with a feature fusion mechanism, which improves performance on audio classification and retrieval tasks compared to earlier CLAP versions that used CNN14 or ResNet. The feature fusion combines intermediate representations from multiple layers of HTSAT.

Q: Can I use this model for real-time audio classification? A: It depends on your hardware. The model is relatively large (153M parameters) but can be used for inference on GPU with low latency. For real-time applications on edge devices, consider smaller models like laion/clap-htsat-unfused or distilled versions.

Q: How do I use this model for zero-shot audio classification? A: Use the pipeline("audio-classification", model="laion/clap-htsat-fused") and pass audio file paths or arrays. Provide candidate labels as text (e.g., ["dog", "cat", "car"]) and the model returns probabilities.

Q: What audio formats are supported? A: The model expects audio as a waveform array sampled at 48000 Hz. The AutoProcessor handles resampling and conversion from common formats (WAV, MP3, etc.) when using the pipeline.

Q: Is the model suitable for music understanding? A: Yes, the model was trained on a diverse dataset including music, speech, and environmental sounds. It can perform zero-shot music tagging or retrieval based on textual descriptions.

Q: What are the limitations? A: The model may struggle with very fine-grained audio categories or languages not well-represented in the training data. Performance on out-of-domain audio (e.g., medical sounds) is not confirmed. The model is primarily designed for English text queries.

Q: How was the model evaluated? A: The original paper reports zero-shot classification results on AudioSet, ESC-50, and other benchmarks. The HTSAT-Fused variant achieved state-of-the-art at the time of release. Detailed metrics are available in the paper.

Q: Can I fine-tune this model? A: Yes, the model can be fine-tuned on downstream tasks using the contrastive loss or classification head. The Transformers library supports training with Trainer.

Q: What is the license? A: Apache-2.0, as indicated on the Hugging Face model card. This allows commercial use, modification, and distribution with attribution.

Q: Are there any safety considerations? A: The model may reflect biases present in the training data (e.g., cultural biases in audio-text pairs). Users should evaluate for fairness and robustness in their specific application. No explicit safety filters are included.

Q: How does this model compare to ImageBind? A: ImageBind is a larger multimodal model that binds six modalities (image, text, audio, depth, thermal, IMU). CLAP HTSAT-Fused is specialized for audio-text alignment and may be more efficient for audio-only tasks. ImageBind requires more compute and is less focused on audio.

Q: Where can I find the model weights? A: The model is hosted on Hugging Face at laion/clap-htsat-fused. Weights are available in safetensors format. The repository also includes the processor and configuration files.