Dataset Summary KoPI CC (Korpus Perayapan Indonesia) CC is Indonesian only extract from Common Crawl snapshots using ungoliant, each snapshot also filtered using some some deduplicate technique such as exact hash(md5) dedup technique and minhash LSH neardup Preprocessing Each folder name inside snapshots folder denoted preprocessing technique that has been applied . Raw this processed directly from cc snapshot using ungoliant without any addition filter ,you can read it in their paper (citation below) use same "raw cc snapshot" for 2021 10 and 2021 49 which can be found in oscar dataset (2109 and 2201) Dedup use data from raw folder apply cleaning techniques for every text in documents such as fix html remove noisy unicode fix news tag remove control char filter by removing short text (20 words) filter by character ratio occurred inside text such as min alphabet ratio (0.75) max upper ratio (0.10) max number ratio(0.05) filter by exact dedup technique hash all text with md5 hashlib remove non unique hash full code about dedup step adapted from here Neardup use data from dedup folder create index cluster using neardup Minhash and LSH with following config : use 128 permuation 6 n gr…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy