zbMATH — the first resource for mathematics

Crowdsourcing contests. (English) Zbl 1431.91180
Summary: In a crowdsourcing contest, a requester posts a task (e.g., logo design, programming task) on a platform and announces a monetary reward that he is willing to pay for a winning solution. Contestants (e.g., designers or programmers) submit solutions on the platform and the requester chooses the best solution (possibly more than one) and awards the prize. On-line platforms for crowdsourcing contests are already abundant and growing rapidly in market size. In this survey, we present two streams of literature that study crowdsourcing contests. The first is theoretical research, which tries to capture the characteristics of these contests, describe them as a game and then analyze the equilibrium behavior of contestants. The second is the empirical research which collects crowdsourcing data and analyzes the behavior of the contestants in these platforms. The aim of this survey is to clarify the current status of the research of incentives and behavior of contestants, organizers and the platform in crowdsourcing contests and to highlight the many questions that are still open.
91B26 Auctions, bargaining, bidding and selling, and other market models
91A80 Applications of game theory
Full Text: DOI
[1] Working paper.
[2] Ales, L.; Cho, S.; Korpeoğlu, E., Optimal award scheme in innovation tournaments, Operations Research, 65, 3, 693-702 (2017) · Zbl 1407.91123
[3] Araujo, R., 99designs: An analysis of creative competition in crowdsourced design, Proceedings of the first AAAI conference on human computation and crowdsourcing (2013)
[4] WWW 2010.
[5] Archak, N.; Sundararajan, A., Optimal design of crowdsourcing contests, Proceedings of the thirtieth international conference on information system, ICIS 2009 (2009)
[6] Azmat, G.; Moller, M., Competition among contests, RAND Journal of Economics, 40, 4, 743-768 (2009)
[7] Baba, Y.; Kinoshita, K.; Kashima, H., Participation recommendation system for crowdsourcing contests, Expert Systems With Applications, 58, 174-183 (2016)
[8] Bockstedt, J.; Druehl, C.; Mishra, A., Heterogeneous submission behavior and its implications for success in innovation contests with public submissions, Production and Operations Management, 25, 7, 1157-1176 (2016)
[9] Boudreau, K.; Lacetera, N.; Lakhani, K., Incentives and problem uncertainty in innovation contests: An empirical analysis, Management Science, 57, 5, 843-863 (2011)
[10] Boudreau, K.; Lakhani, K., “open” disclosure of innovations, incentives and follow-on reuse: Theory on processes of cumulative innovation and a field experiment in computational biology, Research Policy, 44, 4-19 (2015)
[11] Boudreau, K.; Lakhani, K.; Menietti, M., Performance responses to competition across skill levels in rank-order tournaments: field evidence and implications for tournament design, The RAND Journal of Economics, 47, 140-165 (2016)
[12] Cavallo, R.; Jain, S., Efficient crowdsourcing contests, Proceedings of the 11th international conference on autonomous agents and multi-agent systems (AAMAS 2012) (2012)
[13] Cavallo, R.; Jain, S., Winner-take-all crowdsourcing contests with stochastic production, Proceedings of the first AAAI conference on human computation and crowdsourcing, 34-41 (2013)
[14] Che, Y. K.; Gale, I., Caps on political lobbying, American Economic Review, 88, 3, 643-651 (1998)
[15] Working paper
[16] Choi, J. P., Dynamic r&d competition under ‘hazard rate’ uncertainty, RAND Journal of Economics, 22, 4, 596-610 (1991)
[17] Clark, D.; Riis, C., Contest success functions: an extension, Economic Theory, 11, 1, 201-204 (1998) · Zbl 0901.90186
[18] Corchón, L., The theory of contests: a survey, Review of Economic Design, 11, 69-100 (2007) · Zbl 1274.91345
[19] Deck, C.; Kimbrough, E., Experimenting with contests for experimentation, Southern Economic Journal, 84, 2, 391-406 (2017)
[20] DiPalantino, D.; Karagiannis, T.; Vojnovic, M., Individual and collective user behavior in crowdsourcing services, Technical report MSR-TR-2010-59, microsoft research (2011)
[21] DiPalantino, D.; Vojnovic, M., Crowdsourcing and all-pay auctions, Proceedings of the 10th ACM conference on Electronic commerce (EC 09), 119-128 (2009)
[22] Ewerhart, C., Mixed equilibria in tullock contests, Economic Theory, 60, 1, 59-71 (2015) · Zbl 1367.91013
[23] Gallice, A., An approximate solution to rent-seeking contests with private information, European Journal of Operational Research, 256, 673-684 (2017) · Zbl 1394.91159
[24] Gao, A.; Bachrach, Y.; Key, P.; Graepel, T., Quality expectation-variance trade-offs in crowdsourcing contests, Proceedings of the twenty-sixth AAAI conference on artificial intelligence (2012)
[25] WWW 2012.
[26] WWW 2015.
[27] Ghosh, A.; Kleinberg, R., Optimal contest design for simple agents, Proceedings of the fifteenth ACM conference on Economics and computation (EC 14), 913-930 (2014)
[28] Giebe, T., Innovation contests with entry auction, Journal of Mathematical Economics, 55, 165-176 (2014) · Zbl 1304.91085
[29] Gross, D., Performance feedback in competitive product development, RAND Journal of Economics, 48, 2, 438-466 (2017)
[30] NBER Working Paper No. 25057.
[31] Halac, M.; Kartik, N.; Liu, Q., Contests for experimentation, Journal of Political Economy, 125, 5, 1523-1569 (2017)
[32] Hillman, A.; Riley, J., Politically contestable rents and transfers, Journal of Public Economics, 34, 2, 129-142 (1989)
[33] Hillman, A.; Samet, D., Dissipation of contestable rents by small numbers of contestants, Public Choice, 54, 1, 63-82 (1987)
[34] Working paper
[35] Huang, Y.; Vir Singh, P.; Mukhopadhyay, T., How to design crowdsourcing contest: A structural empirical analysis, Proceedings of the conference on information systems and technology, CIST 2012 (2013)
[36] Working paper
[37] Working paper.
[38] Working paper.
[39] Khasraghi, H. J.; Aghaie, A., Crowdsourcing contests: understanding the effect of competitors’ participation history on their performance, Behaviour & Information Technology, 33, 12, 1383-1395 (2014)
[40] Konrad, K., Strategy and dynamic in contests (2009), Oxford University press
[41] Korpeoğlu, E.; Cho, S. H., Incentives in contests with heterogeneous solvers, Management Science, 64, 6, 2709-2715 (2018)
[42] Working paper
[43] Working paper
[44] Liu, T. X.; Yang, J.; Adamic, L. A.; Chen, Y., Crowdsourcing with all-pay auctions: A field experiment on taskcn, Management Science, 60, 8, 2020-2037 (2014)
[45] Working paper.
[46] Mao, K.; Capra, L.; Harman, M.; Jia, Y., A survey of the use of crowdsourcing in software engineering, The Journal of Systems and Software, 126, 57-84 (2017)
[47] Mihm, J.; Schlapp, J., Sourcing innovation: On feedback in contests, Management Science, 65, 2, 559-576 (2018)
[48] Moldovanu, B.; Sela, A., The optimal allocation of prizes in contests, American Economic Review, 91, 542-558 (2001)
[49] Moldovanu, B.; Sela, A., Contest architecture, Journal of Economic Theory, 126, 70-96 (2006) · Zbl 1108.91008
[50] Morgan, J., Sequential contests, Public Choice,, 116, 1-18 (2003)
[51] Nitzan, S., Modeling rent seeking contests, European Journal of Political Economy, 10, 1, 41-60 (1994)
[52] Segev, E.; Sela, A., Multi-stage sequential all-pay auction, European Economic Review, 70, 371-382 (2014)
[53] Shao, B.; Shi, L.; Xu, B.; Liu, L., Factors affecting participation of solvers in crowdsourcing: an empirical study from china, Electronic Markets, 22, 73-82 (2012)
[54] Sisak, D., Multiple prize contests – the optimal allocation of prizes, Journal of Economic Surveys, 23, 1, 82-114 (2009)
[55] Working paper.
[56] Working paper.
[57] Taylor, C. R., Digging for golden carrots: An analysis of research tournaments, American Economic Review, 85, 4, 872-890 (1995)
[58] Terwiesch, C.; Xu, Y., Innovation contests, open innovation, and multiagent problem solving, Management Science, 54, 9, 1529-1543 (2008)
[59] Tullock, G., Efficient rent seeking, (Buchanan, J. M.; Tollison, R. D.; Tullock, G., Toward a theory of the rent-seeking society (1980), Texas A & M University Press: Texas A & M University Press Dallas)
[60] Vojnovic, M., Contest theory (2015), Cambridge University press
[61] Working paper.
[62] Wang, L.; Tian, J.; Xu, Y., Relationship between design elements and performance in online innovation contests: Contest sequence is moderator?” WHICEB 2015 proceedings (2015)
[63] Wen, Z.; Lin, L., Optimal fee structures of crowdsourcing platforms, Decision Sciences, 47, 5, 820-850 (2016)
[64] Wooten, J.; Ulrich, K., Idea generation and the role of feedback: Evidence from field experiments with innovation tournaments, Production and Operations management, 26, 1, 80-99 (2017)
[65] Xu, H.; Larson, K., Improving the efficiency of crowdsourcing contests, Proceedings of the 13th international conference on autonomous agents and multiagent systems (AAMAS 2014) (2014)
[66] Yang, J.; Adamic, L.; Ackerman, M., Crowdsourcing and knowledge sharing: Strategic user behavior on taskcn, Proceedings of the EC 2008 (2008)
[67] Yang, Y.; Chen, P.; Banker, R., Impact of past performance and strategic bidding on winner determination of open innovation contest, Proceedings of the workshop on information systems and economics, St. Louis 2010 (2010)
[68] Yang, Y.; Chen, P.; Pavlou, P., Open innovation: An empirical study of online contests, Proceedings of the thirtieth international conference on information system, ICIS 2009 (2009)
[69] Yuen, M.; King, I.; Leung, K., A survey of crowdsourcing systems, Proceedings of the IEEE international conference on privacy, security, risk, and trust, and IEEE international conference on social computing (2011)
[70] Working paper.
[71] Zhao, Y.; Zhu, Q., Evaluation on crowdsourcing research: Current status and future direction, Information Systems Frontiers, 16, 417-434 (2014)
[72] Zheng, H.; Xie, Z.; Hou, W.; Li, D., Antecedents of solution quality in crowdsourcing: The sponsor’s perspective, Journal of Electronic Commerce Research, 15, 3, 212-224 (2014)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.