CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binaryMultilingual Embeddings for Wikipedia in 300+ Languages (int8 & binary embeddings) This dataset contains the wikimedia/wikipedia dataset dump from 2023 11 01 from Wikipedia in all 300+ languages. The embeddings are provided as int8 and ubinary that allow quick search and reduction of your vector index size up to 32. For more details, see Cohere int8 & binary Embeddings The individual articles have been chunked and embedded with the state of the art multilingual Cohere Embed V3 embedding model. This enables an easy way to semantically search across all of Wikipedia or to use it as a knowledge source for your RAG application. In total is it close to 250M paragraphs / embeddings. You can also use the model to perform cross lingual search: Enter your search query in any language and get the most relevant results back. Loading the dataset Loading the document embeddings You can either load the dataset like this: Or you can also stream it without downloading it before: Note, depending on the language, the download can be quite large. Search int8 embeddings int8 embeddings keep 99.7 100% of the search quality, while reducing your vector index 4x and speeding up vector search up to 30%. Be…

← Back to CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binary

SQL Console

Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.

Loading dataset into an in-browser SQL engine…

We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy