BACKGROUND: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality.
OBJECTIVE: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data.
METHODS: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips.
RESULTS: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts\' opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance.
CONCLUSIONS: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.

Every year, 11% of infants are born preterm with significant health consequences, with the vaginal microbiome a risk factor for preterm birth. We crowdsource models to predict (1) preterm birth (PTB; <37 weeks) or (2) early preterm birth (ePTB; <32 weeks) from 9 vaginal microbiome studies representing 3,578 samples from 1,268 pregnant individuals, aggregated from public raw data via phylogenetic harmonization. The predictive models are validated on two independent unpublished datasets representing 331 samples from 148 pregnant individuals. The top-performing models (among 148 and 121 submissions from 318 teams) achieve area under the receiver operator characteristic (AUROC) curve scores of 0.69 and 0.87 predicting PTB and ePTB, respectively. Alpha diversity, VALENCIA community state types, and composition are important features in the top-performing models, most of which are tree-based methods. This work is a model for translation of microbiome data into clinically relevant predictive models and to better understand preterm birth.

BACKGROUND: Dermoscopy is commonly used for the evaluation of pigmented lesions, but agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labeling of medical data is a bottleneck in the development of machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images.
OBJECTIVE: The aim of this study is to demonstrate that crowdsourcing can be used to label basic dermoscopic structures from images of pigmented lesions with similar reliability to a group of experts.
METHODS: First, we obtained labels of 248 images of melanocytic lesions with 31 dermoscopic \"subfeatures\" labeled by 20 dermoscopy experts. These were then collapsed into 6 dermoscopic \"superfeatures\" based on structural similarity, due to low interrater reliability (IRR): dots, globules, lines, network structures, regression structures, and vessels. These images were then used as the gold standard for the crowd study. The commercial platform DiagnosUs was used to obtain annotations from a nonexpert crowd for the presence or absence of the 6 superfeatures in each of the 248 images. We replicated this methodology with a group of 7 dermatologists to allow direct comparison with the nonexpert crowd. The Cohen κ value was used to measure agreement across raters.
RESULTS: In total, we obtained 139,731 ratings of the 6 dermoscopic superfeatures from the crowd. There was relatively lower agreement for the identification of dots and globules (the median κ values were 0.526 and 0.395, respectively), whereas network structures and vessels showed the highest agreement (the median κ values were 0.581 and 0.798, respectively). This pattern was also seen among the expert raters, who had median κ values of 0.483 and 0.517 for dots and globules, respectively, and 0.758 and 0.790 for network structures and vessels. The median κ values between nonexperts and thresholded average-expert readers were 0.709 for dots, 0.719 for globules, 0.714 for lines, 0.838 for network structures, 0.818 for regression structures, and 0.728 for vessels.
CONCLUSIONS: This study confirmed that IRR for different dermoscopic features varied among a group of experts; a similar pattern was observed in a nonexpert crowd. There was good or excellent agreement for each of the 6 superfeatures between the crowd and the experts, highlighting the similar reliability of the crowd for labeling dermoscopic images. This confirms the feasibility and dependability of using crowdsourcing as a scalable solution to annotate large sets of dermoscopic images, with several potential clinical and educational applications, including the development of novel, explainable ML tools.

BACKGROUND: Chronic pain is a prolonged condition that deteriorates one\'s quality of life. Treating chronic pain requires a multicomponent approach, and in many cases, there are no \"silver bullet\" solutions. Mobile health (mHealth) is a rapidly expanding category of solutions in digital health with proven potential in chronic pain management.
OBJECTIVE: This study aims to contrast the viewpoints of 2 groups of people with chronic pain concerning mHealth: people who have adopted the use of mHealth and those who have not. We highlight the benefits of mHealth solutions for people with chronic pain and the perceived obstacles to their increased adoption. We also provide recommendations to encourage people to try mHealth solutions as part of their self-care.
METHODS: The Prolific crowdsourcing platform was used to collect crowdsourced data. A prescreening questionnaire was released to determine what type of pain potential participants have and whether they are currently using mHealth solutions for chronic pain. The participants were invited based on their experience using mHealth to manage their pain. Similar questions were presented to mHealth users and nonusers. Qualitative and quantitative analyses were performed to determine the outcomes of this study.
RESULTS: In total, 31 responses were collected from people (aged 19-63 years, mean 31.4, SD 12.1) with chronic pain who use mHealth solutions. Two-thirds (n=20, 65%) of the users identified as female and 11 (35%) as male. We matched these mHealth users with an equal number of nonusers: 31 responses from the pool of 361 participants in the prescreening questionnaire. The nonusers\' ages ranged from 18 to 58 years (mean 30.8, SD 11.09), with 15 (50%) identifying as female and 15 (50%) as male. Likert-scale questions were analyzed using the Mann-Whitney-Wilcoxon (MWW) test. Results showed that the 2 groups differed significantly on 10 (43%) of 23 questions and shared similar views in the remaining 13 (57%). The most significant differences were related to privacy and interactions with health professionals. Of the 31 mHealth users, 12 (39%) declared that using mHealth solutions has made interacting with health or social care professionals easier (vs n=22, 71%, of nonusers). The majority of the nonusers (n=26, 84%) compared with about half of the users (n=15, 48%) expressed concern about sharing their data with, for example, third parties.
CONCLUSIONS: This study investigated how mHealth is currently used in the context of chronic pain and what expectations mHealth nonusers have for mHealth as a future chronic pain management tool. Analysis revealed contrasts between mHealth use expectations and actual usage experiences, highlighting privacy concerns toward mHealth solutions. Generally, the results showed that nonusers are more concerned about data privacy and expect mHealth to facilitate interacting with health professionals. The users, in contrast, feel that such connections do not exist.

COVID-19 prompted a bike boom and cities around the world responded to increased demand for space to ride with street reallocations. Evaluating these interventions has been limited by a lack of spatially-temporally continuous ridership data. Our paper aims to address this gap using crowdsourced data on bicycle ridership. We evaluate changes in spatial patterns of bicycling during the first wave of the COVID-19 pandemic (Apr - Oct 2020) in Vancouver, Canada using Strava data and a local indicator of spatial autocorrelation. We map statistically significant change in ridership and reference clusters of change to a high-resolution base map. Amongst streets where bicycling increased, we measured the proportion of increase occurring on pre-existing bicycle facilities or street reallocations compared to streets without. In all our analyses, we evaluate patterns across subsets of Strava data representing recreation, commuting, and ridership generated by women and older adults (55 + ). We found consistent and unique patterns by trip purpose and demographics: samples generated by women and older adults showed increases near green and blue spaces and on street reallocations that increased access to parks, and these patterns were also mirrored in the recreation sample. Commute ridership highlighted distinct patterns of increase around the hospital district. Across all subsets most increases occurred on bicycle facilities (pre-existing or provisional), with a strong preference for high-comfort facilities. We demonstrate that changes in spatial patterns of bicycle ridership can be monitored using Strava data, and that nuanced patterns can be identified using trip and demographic labels in the data.

After 43 years of dormancy, Taal Volcano violently erupted in January 2020 forming a towering eruption plume. The fall deposits covered an area of 8605 km2, which includes Metro Manila of the National Capital Region of the Philippines. The tephra fall caused damage to crops, traffic congestion, roof collapse, and changes in air quality in the affected areas. In a tropical region where heavy rains are frequent, immediate collection of data is crucial in order to preserve the tephra fall deposit record, which is readily washed away by surface water runoff and prevailing winds. Crowdsourcing, field surveys, and laboratory analysis of the tephra fall deposits were conducted to document and characterize the tephra fall deposits of the 2020 Taal Volcano eruption and their impacts. Results show that the tephra fall deposit thins downwind exponentially with a thickness half distance of about 1.40 km and 9.49 km for the proximal and distal exponential segments, respectively. The total calculated volume of erupted fallout deposit is 0.057 km3, 0.042 km3, or 0.090 km3 using the exponential, power-law, and Weibull models, respectively, and all translate to a VEI of 3. However, using a probabilistic approach (Weibull method) with 90% confidence interval, the volume estimate is as high as 0.097 km3. With the addition of the base surge deposits amounting to 0.019 km3, the volume translates to a VEI of 4, consistent with the classification for the observed height and umbrella radius of the 2020 main eruption plume. VEI 4 is also consistent with the calculated median eruption plume height of 17.8 km and sub-plinian classification based on combined analysis of isopleth and isopach data. Phreatomagmatic activity originated from a vent located in Taal Volcano\'s Main Crater Lake (MCL), which contained 42 million m3 of water. This eruptive style is further supported by the characteristics of the ash grain components of the distal 12 January 2020 tephra fall deposits, consisting dominantly of andesitic vitric fragments (83-90%). Other components of the fall deposits are lithic (7-11%) and crystal (less than 6%) grains. Further textural and geochemical analysis of these tephra fall deposits contributes to better understand the volcanic processes that occurred at Taal Volcano, one of the 16 Decade Volcanoes identified by the International Association of Volcanology and Chemistry of the Earth\'s Interior (IAVCEI) because of its destructive nature and proximity to densely populated areas. The crowdsourcing initiative provided a significant portion of the data used for this study while at the same time educating and empowering the community to build resilience.
BACKGROUND: The online version contains supplementary material available at 10.1007/s00445-022-01534-y.

BACKGROUND: The ubiquity of mobile phones and increasing use of wearable fitness trackers offer a wide-ranging window into people\'s health and well-being. There are clear advantages in using remote monitoring technologies to gain an insight into health, particularly under the shadow of the COVID-19 pandemic.
OBJECTIVE: Covid Collab is a crowdsourced study that was set up to investigate the feasibility of identifying, monitoring, and understanding the stratification of SARS-CoV-2 infection and recovery through remote monitoring technologies. Additionally, we will assess the impacts of the COVID-19 pandemic and associated social measures on people\'s behavior, physical health, and mental well-being.
METHODS: Participants will remotely enroll in the study through the Mass Science app to donate historic and prospective mobile phone data, fitness tracking wearable data, and regular COVID-19-related and mental health-related survey data. The data collection period will cover a continuous period (ie, both before and after any reported infections), so that comparisons to a participant\'s own baseline can be made. We plan to carry out analyses in several areas, which will cover symptomatology; risk factors; the machine learning-based classification of illness; and trajectories of recovery, mental well-being, and activity.
RESULTS: As of June 2021, there are over 17,000 participants-largely from the United Kingdom-and enrollment is ongoing.
CONCLUSIONS: This paper introduces a crowdsourced study that will include remotely enrolled participants to record mobile health data throughout the COVID-19 pandemic. The data collected may help researchers investigate a variety of areas, including COVID-19 progression; mental well-being during the pandemic; and the adherence of remote, digitally enrolled participants.
UNASSIGNED: DERR1-10.2196/32587.

BACKGROUND: Sentinel influenza-like illness (ILI) surveillance is an essential component of a comprehensive influenza surveillance program. Community-based ILI surveillance systems that rely solely on sentinel healthcare practices omit important segments of the population, including those who do not seek medical care. Participatory surveillance, which relies on community participation in surveillance, may address some limitations of traditional ILI systems.
OBJECTIVE: We aimed to evaluate FluWatchers, a crowdsourced ILI application developed to complement and complete ILI surveillance in Canada.
METHODS: Using established frameworks for surveillance evaluations, we assessed the acceptability, reliability, accuracy and usefulness of the FluWatchers system 2015-2016, through 2018-2019. Evaluation indicators were compared against national surveillance indicators of ILI and of laboratory confirmed respiratory virus infections.
RESULTS: The acceptability of FluWatchers was demonstrated by growth of 50%-100% in season-over-season participation, and a consistent season-over-season retention of 80%. Reliability was greater for FluWatchers than for our traditional ILI system, although both systems had week-over-week fluctuations in the number of participants responding. FluWatchers\' ILI rates had moderate correlation with weekly influenza laboratory detection rates and other winter seasonal respiratory virus detections including respiratory syncytial virus and seasonal coronaviruses. Finally, FluWatchers has demonstrated its usefulness as a source of core FluWatch surveillance information and has the potential to fill data gaps in current programs for influenza surveillance and control.
CONCLUSIONS: FluWatchers is an example of an innovative digital participatory surveillance program that was created to address limitations of traditional ILI surveillance in Canada. It fulfills the surveillance system evaluation criteria of acceptability, reliability, accuracy and usefulness.

暂无摘要。

With only ∼20 % of bicycling crashes captured in official databases, studies on bicycling safety can be limited. New datasets on bicycling incidents are available via crowdsourcing applications, with opportunity for analyses that characterize reporting patterns. Our goal was to characterize patterns of injury in crowdsourced bicycle incident reports from BikeMaps.org. We extracted 281 incidents reported on the BikeMaps.org global mapping platform and analyzed 21 explanatory variables representing personal, trip, route, and crash characteristics. We used a balanced random forest classifier to classify three outcomes: (i) collisions resulting in injury requiring medical treatment; (ii) collisions resulting in injury but the bicyclist did not seek medical treatment; and (iii) collisions that did not result in injury. Results indicate the ranked importance and direction of relationship for explanatory variables. By knowing conditions that are most associated with injury we can target interventions to reduce future risk. The most important reporting pattern overall was the type of object the bicyclist collided with. Increased probability of injury requiring medical treatment was associated with collisions with animals, train tracks, transient hazards, and left-turning motor vehicles. Falls, right hooks, and doorings were associated with incidents where the bicyclist was injured but did not seek medical treatment, and conflicts with pedestrians and passing motor vehicles were associated with minor collisions with no injuries. In Victoria, British Columbia, Canada, bicycling safety would be improved by additional infrastructure to support safe left turns and around train tracks. Our findings support previous research using hospital admissions data that demonstrate how non-motor vehicle crashes can lead to bicyclist injury and that route characteristics and conditions are factors in bicycling collisions. Crowdsourced data have potential to fill gaps in official data such as insurance, police, and hospital reports.