A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation
Crowdsourcing is a new mode of value creation in which organizations leverage numerous Internet users to accomplish tasks. However, because these workers have different backgrounds and intentions, crowdsourcing suffers from quality concerns. In the literature, tracing the behavior of workers is pref...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2021-07-01
|
Series: | Sensors |
Subjects: | |
Online Access: | https://www.mdpi.com/1424-8220/21/15/5007 |
id |
doaj-2eb198a34a7741b592ea736d0f96c469 |
---|---|
record_format |
Article |
spelling |
doaj-2eb198a34a7741b592ea736d0f96c4692021-08-06T15:31:09ZengMDPI AGSensors1424-82202021-07-01215007500710.3390/s21155007A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality AnnotationFattoh Al-Qershi0Muhammad Al-Qurishi1Mehmet Sabih Aksoy2Mohammed Faisal3Mohammed Algabri4Department of Information Systems, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi ArabiaCollege of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi ArabiaDepartment of Information Systems, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi ArabiaCollege of Applied Computer Sciences, King Saud University, Riyadh 145111, Saudi ArabiaDepartment of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi ArabiaCrowdsourcing is a new mode of value creation in which organizations leverage numerous Internet users to accomplish tasks. However, because these workers have different backgrounds and intentions, crowdsourcing suffers from quality concerns. In the literature, tracing the behavior of workers is preferred over other methodologies such as consensus methods and gold standard approaches. This paper proposes two novel models based on workers’ behavior for task classification. These models newly benefit from time-series features and characteristics. The first model uses multiple time-series features with a machine learning classifier. The second model converts time series into images using the recurrent characteristic and applies a convolutional neural network classifier. The proposed models surpass the current state of-the-art baselines in terms of performance. In terms of accuracy, our feature-based model achieved 83.8%, whereas our convolutional neural network model achieved 76.6%.https://www.mdpi.com/1424-8220/21/15/5007annotationcrowdsourcingclassificationneural networksquality controltime-series |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Fattoh Al-Qershi Muhammad Al-Qurishi Mehmet Sabih Aksoy Mohammed Faisal Mohammed Algabri |
spellingShingle |
Fattoh Al-Qershi Muhammad Al-Qurishi Mehmet Sabih Aksoy Mohammed Faisal Mohammed Algabri A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation Sensors annotation crowdsourcing classification neural networks quality control time-series |
author_facet |
Fattoh Al-Qershi Muhammad Al-Qurishi Mehmet Sabih Aksoy Mohammed Faisal Mohammed Algabri |
author_sort |
Fattoh Al-Qershi |
title |
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation |
title_short |
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation |
title_full |
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation |
title_fullStr |
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation |
title_full_unstemmed |
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation |
title_sort |
time-series-based new behavior trace model for crowd workers that ensures quality annotation |
publisher |
MDPI AG |
series |
Sensors |
issn |
1424-8220 |
publishDate |
2021-07-01 |
description |
Crowdsourcing is a new mode of value creation in which organizations leverage numerous Internet users to accomplish tasks. However, because these workers have different backgrounds and intentions, crowdsourcing suffers from quality concerns. In the literature, tracing the behavior of workers is preferred over other methodologies such as consensus methods and gold standard approaches. This paper proposes two novel models based on workers’ behavior for task classification. These models newly benefit from time-series features and characteristics. The first model uses multiple time-series features with a machine learning classifier. The second model converts time series into images using the recurrent characteristic and applies a convolutional neural network classifier. The proposed models surpass the current state of-the-art baselines in terms of performance. In terms of accuracy, our feature-based model achieved 83.8%, whereas our convolutional neural network model achieved 76.6%. |
topic |
annotation crowdsourcing classification neural networks quality control time-series |
url |
https://www.mdpi.com/1424-8220/21/15/5007 |
work_keys_str_mv |
AT fattohalqershi atimeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT muhammadalqurishi atimeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mehmetsabihaksoy atimeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mohammedfaisal atimeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mohammedalgabri atimeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT fattohalqershi timeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT muhammadalqurishi timeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mehmetsabihaksoy timeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mohammedfaisal timeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation AT mohammedalgabri timeseriesbasednewbehaviortracemodelforcrowdworkersthatensuresqualityannotation |
_version_ |
1721217566511202304 |