2022
Haochen, L; Thekinen, J; Mollaoglu, S; Tang, D; Yang, J; Cheng, Y; Liu, H; Tang, J
Toward Annotator Group Bias in Crowdsourcing Conference Forthcoming
60th Annual Meeting of the Association for Computational Linguistics (ACL), Dublin, Forthcoming.
Abstract | BibTeX | Tags: Information Systems, Machine Learning | Links:
@conference{nokey,
title = {Toward Annotator Group Bias in Crowdsourcing},
author = {Haochen, L and J Thekinen and Mollaoglu, S and Tang, D and Yang, J and Cheng, Y and Liu, H and Tang, J},
url = {https://josephdthekinen.com/wp-content/uploads/2022/03/Toward_Annotator_Group_Bias_in_Crowdsourcing____ACL_camera_ready.pdf},
year = {2022},
date = {2022-05-23},
urldate = {2022-05-23},
booktitle = {60th Annual Meeting of the Association for Computational Linguistics (ACL)},
address = {Dublin},
abstract = {Crowdsourcing has emerged as a popular approach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating individual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same demographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilistic graphical framework GroupAnno to capture annotator group bias with an extended Expectation Maximization (EM) algorithm. We conduct experiments on both synthetic and realworld datasets. Experimental results demonstrate the effectiveness of our model in modeling annotator group bias in label aggregation and model learning over competitive baselines.},
keywords = {Information Systems, Machine Learning},
pubstate = {forthcoming},
tppubtype = {conference}
}
Crowdsourcing has emerged as a popular approach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating individual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same demographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilistic graphical framework GroupAnno to capture annotator group bias with an extended Expectation Maximization (EM) algorithm. We conduct experiments on both synthetic and realworld datasets. Experimental results demonstrate the effectiveness of our model in modeling annotator group bias in label aggregation and model learning over competitive baselines.
2021
Thekinen, J; Grogan, P
Information Exchange Patterns in Digital Engineering: An Observational Study Using Web-Based Virtual Design Studio Journal Article
In: Journal of Computing and Information Science in Engineering, 21 (4), pp. 041012, 2021.
Abstract | BibTeX | Tags: Decentralized Design, Digital Engineering, Information Systems | Links:
@article{thekinen2021information,
title = {Information Exchange Patterns in Digital Engineering: An Observational Study Using Web-Based Virtual Design Studio},
author = {Thekinen, J and Grogan, P},
url = {https://josephdthekinen.com/wp-content/uploads/2021/11/2021_JCISE_Information-Exchange-Patterns-in-Digital-Engineering-An-Observational-Study-Using-Web-Based-Virtual-Design-Studio.pdf},
doi = {10.1115/1.4050087},
year = {2021},
date = {2021-02-01},
urldate = {2021-02-01},
journal = {Journal of Computing and Information Science in Engineering},
volume = {21},
number = {4},
pages = {041012},
publisher = {American Society of Mechanical Engineers},
abstract = {This paper performs an observational human subjects study to investigate how design teams use an information system to exchange, store, and synthesize information in an engineering design task. Framed through the lens of decision-based design, a surrogate design task models an aircraft design problem with 12 design parameters across four roles and six system-level functional requirements. A virtual design studio provides a browser-based interface for four participants in a 30-min design session. Data collected across 10 design sessions provide process factors about communication patterns and outcome factors about the resulting design. Correlation analysis shows a positive relationship between design iteration and outcome performance but a negative relationship between chat messages and outcome performance. Discussion explains how advances in information exchange, storage, and synthesis can support future design activities.},
keywords = {Decentralized Design, Digital Engineering, Information Systems},
pubstate = {published},
tppubtype = {article}
}
This paper performs an observational human subjects study to investigate how design teams use an information system to exchange, store, and synthesize information in an engineering design task. Framed through the lens of decision-based design, a surrogate design task models an aircraft design problem with 12 design parameters across four roles and six system-level functional requirements. A virtual design studio provides a browser-based interface for four participants in a 30-min design session. Data collected across 10 design sessions provide process factors about communication patterns and outcome factors about the resulting design. Correlation analysis shows a positive relationship between design iteration and outcome performance but a negative relationship between chat messages and outcome performance. Discussion explains how advances in information exchange, storage, and synthesis can support future design activities.