- #Online duplicate detector update#
- #Online duplicate detector mods#
- #Online duplicate detector Offline#
Our online system presents state-of-the-art F1-scores, and can be tuned to trade precision for recall and vice-versa. This system adapts the shingling algorithm proposed by Broder (1997), and we test it on a challenging dataset of web-based news articles. We propose an online system which flags a near-duplicate document by finding its most likely original.
#Online duplicate detector Offline#
Previous near-duplicate detection methods typically work offline to identify all near-duplicate pairs in a set of documents. Filtering near-duplicates out of a collection is thus important, and is particularly challenging in applications that require them to be filtered out in real-time with high precision. Near-duplicate documents have potentially significant costs, including bloating corpora with redundant information (biasing techniques built upon such corpora) and requiring additional human and computational analytic resources for marginal benefit.
#Online duplicate detector update#
Editors often update wirefeed articles to address space constraints in print editions or to add local context journalists often lightly modify previous articles with new information or minor corrections. Publisher = "European Language Resources Association",Ībstract = "Near-duplicate documents are particularly common in news media corpora.
#Online duplicate detector mods#
Cite (Informal): Online Near-Duplicate Detection of News Articles (Rodier & Carter, LREC 2020) Copy Citation: BibTeX Markdown MODS XML Endnote More options… PDF: = "Online Near-Duplicate Detection of News Articles",īooktitle = "Proceedings of the Twelfth Language Resources and Evaluation Conference", In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1242–1249, Marseille, France. Online Near-Duplicate Detection of News Articles. Anthology ID: 2020.lrec-1.156 Volume: Proceedings of the Twelfth Language Resources and Evaluation Conference Month: May Year: 2020 Address: Marseille, France Venue: LREC SIG: Publisher: European Language Resources Association Note: Pages: 1242–1249 Language: English URL: DOI: Bibkey: rodier-carter-2020-online Cite (ACL): Simon Rodier and Dave Carter. We present one such application, filtering near-duplicates to improve productivity of human analysts in a situational awareness tool. Given its performance and online nature, our method can be used in many real-world applications.
Abstract Near-duplicate documents are particularly common in news media corpora.
0 kommentar(er)
