Gorio Tech Blog search

Data-to-Value: An Evaluation-First Methodology for Natural Language Projects | Summary

한국어로 읽기

19 Jan 2022 | Paper Review Model Evaluation Data Curation Supervised Learning
Contents

This article explains the key points of Data-to-Value: An Evaluation-First Methodology for Natural Language Projects.

  • 2022-01-19 (arXiv)
  • Leidner, Jochen L.
  • Coburg University of Applied Sciences, University of Sheffield
  • Paper, arXiv v1

Read this article in Korean

Summary

  • Natural language projects need a process for defining success and managing text data, annotation, ethics, and deployment. The paper proposes Data to Value (D2V), an evaluation-first process model that, in the author’s feature comparison with existing data-mining methodologies, places evaluation design earlier and explicitly includes unstructured text; it reports no measured project-performance improvement.
  • D2V commits to an evaluation protocol early, creates annotated reference data, and repeatedly evaluates system changes. Its guidance extends from project planning through deployment, monitoring, and possible model retraining.
  • The paper provides a methodological comparison and a historical usage poll, but no controlled experiment or case study testing D2V. It acknowledges that the process is less suited to structured data mining and cannot predict project duration.

1 Introduction

The author frames predictable delivery as an engineering goal alongside forecasts of runtime, memory, and output quality. D2V aims for more consistent delivery through finer-grained phases, quantitative evaluation, and phase-specific questions; the paper does not provide a theory that predicts those outcomes.

The paper reviews Knowledge Discovery in Databases (KDD), an iterative process from data selection to interpretation [5,6,4]; SEMMA, a Sample, Explore, Modify, Model, and Assess cycle [10]; and CRISP-DM, the Cross Industry Standard Process for Data Mining [3,11,1]. It also describes Marr’s Strategy Board as a business-focused approach with six guidance questions [8].

The author argues that these approaches provide insufficient specific guidance for text, supervised learning, ethical concerns, and large-scale processing, and place evaluation too late. D2V therefore specifies output evaluation and typically implements a test harness early in the project.

3 Data to Value (D2V)

The development model begins with planning, an ethics review, requirements, data acquisition, and an optional feasibility study on a static sample. Evaluation Design commits the project to an experimental design and protocol before data cleansing and annotation.

The 18-phase development sequence places Evaluation Design before preprocessing and annotation, then loops feature implementation through quantitative evaluation before project closure.

Data-to-Value Process Model, with two steps containing sub-processes.
Data-to-Value Process Model, with two steps containing sub-processes.

Gold data, annotated reference data for evaluation and, when supervised learning is used, training, is produced through guidelines, overlapping annotation, agreement measurement, and resolution of disagreements. The paper recommends that at least three annotators label the same 5–10% of a larger sample; insufficient inter-annotator agreement, a measure of consistency among annotators, prompts guideline revision and re-annotation.

The annotation workflow connects written guidelines and multiple annotators to agreement measurement, adjudication, guideline revision, and a training, development-test, and test split.

Gold Data Annotation as a D2V sub-process.
Gold Data Annotation as a D2V sub-process.

The paper gives typical splits of 60% or 80% for training and 10%–20% each for development testing and final testing; the training set is needed only for supervised learning. It proposes blind regular evaluation on the development test set and a single final evaluation on the test set; feature design and quantitative evaluation repeat until the quality target, time limit, or diminishing returns justify stopping.

The operational extension adds a second ethics review, deployment, monitoring, periodic quantitative evaluation, and retraining when warranted by the findings. The author suggests reassessing the model when incoming data has substantially changed and maintaining the cycle until end of life.

The operational extension adds an ethics review and deployment to the research project, followed by monitoring and a cycle of evaluation and model retraining. Its caption notes that project handover and closure steps may need to move or be repeated when these phases are included.

From Development to Deployment and Operations.
From Development to Deployment and Operations.

4 Discussion

The author recommends adapting the detailed process to project size and complexity. In a kdnuggets.com poll reported for 2007 and 2014, with total N = 200, CRISP-DM accounted for 42% and 43% of responses; the poll did not include text- or big-data-specific processes and does not evaluate D2V.

The reported poll puts CRISP-DM well ahead of SEMMA and KDD in both years, but contains no D2V outcome comparison.

Internet poll of methodology use in 2007 and 2014, with total N = 200.
Internet poll of methodology use in 2007 and 2014, with total N = 200.

The paper’s feature comparison lists D2V as 30 phases and 96 guidance questions, versus six phases for CRISP-DM and six guidance questions for Marr’s Strategy Board. These are descriptive design counts, not evidence that D2V delivers projects faster or more reliably.

The author’s comparison records D2V as supporting unstructured data, rule-based and learning-based approaches, and evaluation-first practice, while listing the other models as lacking explicit unstructured-data support.

Evaluative comparison of D2V and earlier process models by phases, data type, approaches, guidance questions, and evaluation timing.
Evaluative comparison of D2V and earlier process models by phases, data type, approaches, guidance questions, and evaluation timing.

5 Summary, Conclusion & Future Work

The conclusion emphasizes evaluation-first planning, annotated gold data, and iterative evaluation for text projects, especially those using supervised learning. Future work proposes collecting resource-use data by phase and modeling cost and quality in relation to time and expenditure.

Appendix

  • A D2V Question Catalog (Excerpt) presents sample questions from approximately 100, including data reliability and growth rate, licensing, ethics, multilingual requirements, and responsibility for post-deployment retraining. The full catalog and a supporting software tool are left for future work.

Brief Thoughts

The early commitment to an evaluation protocol and explicit separation of development and final testing offer concrete guidance for teams building text systems. Its practical benefit remains unverified in this paper: the comparison describes process features, and the historical poll measures adoption of other methodologies rather than project outcomes.

Related Posts