{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T20:41:14Z","timestamp":1767991274573,"version":"3.49.0"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2013,4,1]],"date-time":"2013-04-01T00:00:00Z","timestamp":1364774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2013,4]]},"abstract":"<jats:p>\n            We are witnessing a paradigm shift in Human Language Technology (HLT) that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this new approach is\n            <jats:italic>Phrase Detectives<\/jats:italic>\n            , an interactive online\n            <jats:italic>game with a purpose<\/jats:italic>\n            for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise.\n          <\/jats:p>\n          <jats:p>The purpose of this article is to first of all give an overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods we used to the results we have obtained so far. It furthermore summarizes the lessons that we have learned in developing this game which should help other researchers to design and implement similar games.<\/jats:p>","DOI":"10.1145\/2448116.2448119","type":"journal-article","created":{"date-parts":[[2013,4,23]],"date-time":"2013-04-23T13:04:26Z","timestamp":1366722266000},"page":"1-44","update-policy":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":52,"title":["Phrase detectives"],"prefix":"10.1145","volume":"3","author":[{"given":"Massimo","family":"Poesio","sequence":"first","affiliation":[{"name":"University of Essex, UK"}]},{"given":"Jon","family":"Chamberlain","sequence":"additional","affiliation":[{"name":"University of Essex, UK"}]},{"given":"Udo","family":"Kruschwitz","sequence":"additional","affiliation":[{"name":"University of Essex, UK"}]},{"given":"Livio","family":"Robaldo","sequence":"additional","affiliation":[{"name":"University of Turin, Italy"}]},{"given":"Luca","family":"Ducceschi","sequence":"additional","affiliation":[{"name":"University of Utrecht\/Verona, The Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2013,4,24]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13084-7_2"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Workshop on the Future of Information Retrieval Evaluation, Collocated at Special Interest Group on Information Retrieval Conference (SIGIR).","author":"Alonso O."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_2_1_5_1","volume-title":"Phratris. In Proceedings of the INSEMTIVES'10 (Demo).","author":"Attardi G."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-009-9081-4"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Workshop \u201cTowards Standards and Tools for Discourse Tagging\u201d. Association for Computational Linguistics, 1--10","author":"Bird S."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Semantic Evaluation Workshop (SEMEVAL).","author":"Broscheit S."},{"key":"e_1_2_1_9_1","volume-title":"Computational Lexicography: Methods and Applications","author":"Burchardt A.","year":"2009"},{"key":"e_1_2_1_10_1","unstructured":"Burnard L. 2000. The british national corpus reference guide. Tech. rep. Oxford University Computing Services Oxford UK.  Burnard L. 2000. The british national corpus reference guide. Tech. rep. Oxford University Computing Services Oxford UK."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/1699510.1699548"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Chamberlain J. Kruschwitz U. and Poesio M. 2009a. Constructing an anaphorically annotated corpus with non-experts: Assessing the quality of collaborative annotations. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing Workshop on the People's Web Meets NLP: Collaboratively Constructed Semantic Resources.   Chamberlain J. Kruschwitz U. and Poesio M. 2009a. Constructing an anaphorically annotated corpus with non-experts: Assessing the quality of collaborative annotations. In Proceedings of the Joint Conference of the 47 th Annual Meeting of the Association for Computational Linguistics and the 4 th International Joint Conference on Natural Language Processing Workshop on the People's Web Meets NLP: Collaboratively Constructed Semantic Resources.","DOI":"10.3115\/1699765.1699774"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the Conference on Collective Intelligence (CI'12)","author":"Chamberlain J."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Symposium on Semantics in Systems for Text Processing (STEP).","author":"Chamberlain J."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the International Conference on Semantic Systems (ISemantics'08)","author":"Chamberlain J."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the WWW Workshop on Web Incentives (WEBCENTIVES'09)","author":"Chamberlain J."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1088622.1088630"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1088622.1088644"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2008.86"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Workshop on the People's Web Meets NLP: Collaboratively Constructed Semantic Resources. Association for Computational Linguistics, 51--56","author":"Feng D."},{"key":"e_1_2_1_21_1","unstructured":"Garnham A. 2001. Mental Models and the Interpretation of Anaphora. Psychology Press.  Garnham A. 2001. Mental Models and the Interpretation of Anaphora. Psychology Press."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.3115\/980845.980937"},{"key":"e_1_2_1_23_1","unstructured":"Hladka B. Mirovsky J. and Schlesinger P. 2009. Play the language: Play coreference. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing. Association for Computational Linguistics 209--212.   Hladka B. Mirovsky J. and Schlesinger P. 2009. Play the language: Play coreference. In Proceedings of the Joint Conference of the 47 th Annual Meeting of the Association for Computational Linguistics and the 4 th International Joint Conference on Natural Language Processing. Association for Computational Linguistics 209--212."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/21922.24343"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 57--60","author":"Hovy E."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 6th International Conference on Artificial Life. MIT Press.","author":"Johnson N. L."},{"key":"e_1_2_1_27_1","unstructured":"Kabadjov M. A. 2007. Task-Oriented evaluation of anaphora resolution. Ph.D. thesis University of Essex Colchester UK.  Kabadjov M. A. 2007. Task-Oriented evaluation of anaphora resolution. Ph.D. thesis University of Essex Colchester UK."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Kamp H. and Reyle U. 1993. From Discourse to Logic. D. Reidel Dordrecht.  Kamp H. and Reyle U. 1993. From Discourse to Logic. D. Reidel Dordrecht.","DOI":"10.1007\/978-94-017-1616-1"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996911"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572019"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 6th International Natural Language Generation Conference.","author":"Koller A."},{"key":"e_1_2_1_32_1","unstructured":"Koster R. 2005. A Theory of Fun for Game Design. Paraglyph.   Koster R. 2005. A Theory of Fun for Game Design. Paraglyph."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2007.09.001"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the International Conference on Web Science (WebSci'09)","author":"Kruschwitz U."},{"key":"e_1_2_1_35_1","unstructured":"Kucera H. and Francis W. N. 1967. Computational Analysis of Present-Day American English. Brown University Press.  Kucera H. and Francis W. N. 1967. Computational Analysis of Present-Day American English. Brown University Press."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 3rd International Conference on Language Resources and Evaluation. 512--519","author":"Lesmo L."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the Workshop on Common Sense and Intelligent User Interfaces held in Conjunction with the International Conference on Intelligent UserInterfaces (IUI'07)","author":"Lieberman H."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/972470.972475"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v58:8"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809400.1809422"},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Mintz M. Bills S. Snow R. and Jurafsky D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language. 1003--1011.   Mintz M. Bills S. Snow R. and Jurafsky D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47 th Annual Meeting of the Association for Computational Linguistics and the 4 th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language. 1003--1011.","DOI":"10.3115\/1690219.1690287"},{"key":"e_1_2_1_42_1","unstructured":"Mitkov R. 2002. Anaphora Resolution. Longman.  Mitkov R. 2002. Anaphora Resolution. Longman."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. 443--451","author":"Mrozinski J."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613795"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Nivre J. 2005. Dependency grammar and dependency parsing. Tech. rep. Vaxjo University.  Nivre J. 2005. Dependency grammar and dependency parsing. Tech. rep. Vaxjo University.","DOI":"10.1007\/1-4020-4889-0"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220230"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/1608938.1608948"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL).","author":"Poesio M.","year":"2004"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 6th International Conference on Language Resources and Evaluation.","author":"Poesio M."},{"key":"e_1_2_1_50_1","first-page":"175","article-title":"Markup infrastructure for the anaphoric bank: Supporting web collaboration. In Modeling, Learning, and Processing of Text Technological Data Structures, A. Mehler, K.-U. Kuhnberger, H. Lobin, H. Lungen, A. Storrer, and A. Witt, Eds","volume":"370","author":"Poesio M.","year":"2011","journal-title":"Studies in Computational Intelligence"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation.","author":"Poesio M."},{"key":"e_1_2_1_52_1","volume-title":"Anaphora Resolution: Algorithms, Resources and Applications","author":"Poesio M.","year":"2011"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15326950dp4202_4"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/972732.972733"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622637.1622642"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 15th Computational Natural Language Learning Conference (CoNLL). Association for Computational Linguistics, 1--27","author":"Pradhan S."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSC.2007.106"},{"key":"e_1_2_1_58_1","volume-title":"Discourse Description: Diverse Analyses of a Fund-Raising Text","author":"Prince E. F."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557914.1557948"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the Semantic Evaluation Workshop (SEMEVAL).","author":"Recasens M."},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 12th Congress of the Italian Association for Artificial Intelligence. Lecture Notes in Computer Science","volume":"6934","author":"Robaldo L."},{"key":"e_1_2_1_62_1","unstructured":"Settles B. 2009. Active learning literature survey. Tech. rep. 1648 Department of Computer Science University of Wisconsin at Madison.  Settles B. 2009. Active learning literature survey. Tech. rep. 1648 Department of Computer Science University of Wisconsin at Madison."},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the AAAI Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for Information Access.","author":"Singh P.","year":"2002"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2008.45"},{"key":"e_1_2_1_65_1","unstructured":"Smadja F. 2009. Mixing financial social and fun incentives for social voting. In World Wide Web Internet and Web Information Systems.  Smadja F. 2009. Mixing financial social and fun incentives for social voting. In World Wide Web Internet and Web Information Systems."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'08)","author":"Snow R."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.5555\/972597.972602"},{"key":"e_1_2_1_68_1","volume-title":"Balisage: The Markup Conference.","author":"Stuhrenberg M."},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the Association for Computational Linguistics, Linguistic Annotation Workshop. 140--147","author":"Stuhrenberg M."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12275-0_18"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120100750105948"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.3115\/1072399.1072405"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the Workshop on Adaptive Text Extraction and Mining, Collocated at the European Chapter of the Association for Computational Linguistics.","author":"Vlachos A.","year":"2006"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378719"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124782"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2010.04.011"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2006.32.4.577"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/2448116.2448119","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/2448116.2448119","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:35:24Z","timestamp":1750235724000},"score":1,"resource":{"primary":{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/2448116.2448119"}},"subtitle":["Utilizing collective intelligence for internet-scale language resource creation"],"short-title":[],"issued":{"date-parts":[[2013,4]]},"references-count":79,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2013,4]]}},"alternative-id":["10.1145\/2448116.2448119"],"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1145\/2448116.2448119","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,4]]},"assertion":[{"value":"2011-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-04-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}