redacted-contextual-question-answering/CITATION.cff at main · isi-vista/redacted-contextual-question-answering · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
cff-version: 1.2.0
title: Redacted Contextual Question Answering
abstract: Data for a new question answering task, along with associated code
type: dataset
repository-code: https://github.com/isi-vista/redacted-contextual-question-answering
# Semantics are "AND" instead of "OR" but the `CITATION.cff` specification
# does not the support the full SPDX license grammar.
# See the "License" section of the README for more detailed information.
license:
  - Apache-2.0
  - CC-BY-4.0
  - CC-BY-SA-4.0
  - MIT
license-url: https://github.com/isi-vista/redacted-contextual-question-answering/blob/main/README.md#license
message: If you use this dataset or software, please cite the paper from `preferred-citation`.
authors:
  - given-names: Jacob
    family-names: Lichtefeld
    affiliation: USC Information Sciences Institute
  - given-names: Joe A.
    family-names: Cecil
    affiliation: USC Information Sciences Institute
  - given-names: Alex
    family-names: Hedges
    affiliation: USC Information Sciences Institute
  - given-names: Jeremy
    family-names: Abramson
    affiliation: USC Information Sciences Institute
  - given-names: Marjorie
    family-names: Freedman
    affiliation: USC Information Sciences Institute
preferred-citation:
  authors:
    - given-names: Jacob
      family-names: Lichtefeld
      affiliation: USC Information Sciences Institute
    - given-names: Joe A.
      family-names: Cecil
      affiliation: USC Information Sciences Institute
    - given-names: Alex
      family-names: Hedges
      affiliation: USC Information Sciences Institute
    - given-names: Jeremy
      family-names: Abramson
      affiliation: USC Information Sciences Institute
    - given-names: Marjorie
      family-names: Freedman
      affiliation: USC Information Sciences Institute
  title: Redacted Contextual Question Answering with Generative Large Language Models
  type: conference-paper
  collection-title: Proceedings of the First International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security
  collection-type: proceedings
  publisher:
    name: International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security
  year: 2024
  month: 6
  start: 230
  end: 237
  url: https://aclanthology.org/2024.nlpaics-1.25/
  abstract: Many contexts, such as medicine, finance, and cybersecurity, require _controlled_ release of private or internal information. Traditionally, manually redacting sensitive information for release is an arduous and costly process, and while generative Large Language Models (gLLM) show promise at document-based question answering and summarization, their ability to do so while redacting sensitive information has not been widely explored. To address this, we introduce a new task, called redacted contextual question answering (RC-QA). This explores a gLLM's ability to collaborate with a trusted user in a question-answer task as a proxy for drafting a public release informed by the redaction of potentially sensitive information, presented here in the form of _constraints_ on the answers. We introduce a sample question-answer dataset for this task using publicly available data with four sample constraints. We present evaluation results for five language models and two refined models. Our results show that most models—especially open-source models—struggle to accurately answer questions under these constraints. We hope that these preliminary results help catalyze further exploration into this topic, and to that end, we make our code and data available at <https://github.com/isi-vista/redacted-contextual-question-answering>.