Skip to content

Commit 18bc4d8

Browse files
vanzueCopilot
andauthored
feat(agent-pane): paste clipboard images into the chat with Alt+V (#211) (#354)
* feat(agent-pane): paste clipboard images into the chat with Alt+V (#211) Support pasting an image from the Windows clipboard into the WTA agent-pane chat input via Alt+V, mirroring the Copilot CLI. The image is sent to the agent as an ACP `ContentBlock::Image` in the existing `session/prompt`. - clipboard_image.rs (new): Win32 clipboard read (CF_DIBV5/CF_DIB screenshots re-encoded to PNG; CF_HDROP image files read raw with mime-from-ext). - app.rs: Alt+V key handler gated on input nav-focus + the agent advertising `promptCapabilities.image`; per-tab pending image attachments; three clear system-message outcomes (attached / unsupported / empty clipboard). - client.rs: thread `images` through PromptSubmission -> dispatch_prompt -> build_prompt_content, appended as ContentBlock::Image after the text block. Capability read from initialize's prompt_capabilities.image. No master change (cached_init_resp replay + verbatim PromptRequest forward already propagate). - locales: 4 new keys translated across all 88 locale files. - cgmanifest.json / NOTICE.md regenerated for the new `image` crate + deps. Tests: - clipboard round-trip (live OS clipboard DIB -> read_clipboard_image -> PNG). - mock-agent end-to-end: clipboard image -> dispatch_prompt -> real ACP -> agent receives the exact ContentBlock::Image (mime + base64). - ItE2E Feature.AgentImagePaste: injects Alt+V via win32-input-mode and asserts it reaches the helper's crossterm reader with the ALT modifier (the path the unit tests stub). Verified live against a deployed build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(agent-pane): address Copilot review on image paste (#211) - clipboard_image: cap clipboard payload at 256 MiB (MAX_CLIPBOARD_BYTES) so a corrupted/hostile GlobalSize can't drive an OOM allocation on Alt+V; oversized payloads are treated as non-pastable. - clipboard_image: free the HGLOBAL via GlobalFree on the GlobalLock-fail and SetClipboardData-null paths in the test helper set_clipboard_dib (ownership is only transferred to the clipboard on success). - app.rs: clear pending_images in clear_chat_history so /clear, /new, and session resets drop a queued-but-unsent attachment; doc comment updated to match. - en-US.yml: lock the "Enter" key name in system.image_pasted per the WTA localization instructions (all 88 locales already keep it verbatim). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 60371f5 commit 18bc4d8

103 files changed

Lines changed: 1696 additions & 17 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

NOTICE.md

Lines changed: 74 additions & 4 deletions
Large diffs are not rendered by default.

test/e2e/ItE2E/ItE2E.psm1

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,10 @@ $publicFns = @(
3838
# Agent / Autofix / Sessions
3939
'Open-AgentPane', 'Set-AgentPaneFocus', 'Wait-AgentReady', 'Send-AgentPrompt', 'Wait-AgentState',
4040
'Test-AgentPaneOpen', 'Stop-AgentPane', 'Restore-AgentPane', 'Get-AgentPaneSession', 'Get-AgentPaneText',
41-
'Send-AgentKey', 'Clear-AgentInput', 'Open-AgentCommandMenu', 'Get-AgentMenuSelection', 'Invoke-AgentMenuItem',
41+
'Send-AgentKey', 'Clear-AgentInput', 'Send-AgentWin32Key', 'Send-AgentAltV',
42+
'Open-AgentCommandMenu', 'Get-AgentMenuSelection', 'Invoke-AgentMenuItem',
4243
'Test-AgentPopupShown', 'Wait-AgentPermission', 'Resolve-AgentPermission', 'Assert-AgentPaneText',
44+
'Set-ClipboardImage',
4345
'Wait-Autofix', 'Wait-WtCommandFailure', 'Send-AutofixState', 'Invoke-FailingCommand', 'Get-WtSessions',
4446
'Open-SessionList', 'Close-SessionList', 'Test-SessionListShown', 'Get-SessionRows',
4547
'Get-SessionListSelection', 'Select-SessionRow', 'Resume-Session', 'Get-SessionListJson',

test/e2e/ItE2E/Public/AgentInput.ps1

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,55 @@ function Clear-AgentInput {
6161
}
6262
}
6363

64+
function Send-AgentWin32Key {
65+
<#
66+
.SYNOPSIS
67+
Inject a single MODIFIED key (e.g. Alt+V, Shift+Enter) into the agent pane via
68+
win32-input-mode raw key-event sequences — the only way to express a modifier, since
69+
wtcli's tmux tokens and bare ANSI escapes carry no modifier state.
70+
.DESCRIPTION
71+
Emits a keydown then keyup as `ESC[Vk;Sc;Uc;Kd;Cs;Rc_`, where:
72+
Vk = virtual-key code, Sc = scan code, Uc = UTF-16 unit (0 if none),
73+
Kd = 1 down / 0 up, Cs = control-key-state bitmask, Rc = repeat count.
74+
ConPTY decodes these back into INPUT_RECORD key events for the helper's console, so
75+
crossterm sees a real KeyEvent with the modifier — exactly like a physical keypress.
76+
.PARAMETER Modifiers
77+
Control-key-state bitmask: 0x02 = LEFT_ALT_PRESSED, 0x10 = SHIFT_PRESSED,
78+
0x08 = LEFT_CTRL_PRESSED (combine with -bor).
79+
#>
80+
[CmdletBinding()]
81+
param(
82+
[Parameter(Mandatory, ValueFromPipeline)]$App,
83+
[Parameter(Mandatory)][int]$Vk,
84+
[int]$Sc = 0,
85+
[int]$Uc = 0,
86+
[int]$Modifiers = 0,
87+
[string]$PaneSessionId
88+
)
89+
process {
90+
if (-not $PaneSessionId) {
91+
$PaneSessionId = (Wait-Until -TimeoutSec 20 -Because "agent pane session id" -Condition { Get-AgentPaneSession -App $App }).PaneSessionId
92+
}
93+
$e = $script:ItEsc
94+
$down = "$e[$Vk;$Sc;$Uc;1;$Modifiers;1_"
95+
$up = "$e[$Vk;$Sc;$Uc;0;$Modifiers;1_"
96+
Invoke-WtCli -App $App -Arguments @('send-keys', '--raw', '-t', $PaneSessionId, '--', "$down$up") | Out-Null
97+
Start-Sleep -Milliseconds 150
98+
$App
99+
}
100+
}
101+
102+
function Send-AgentAltV {
103+
<#
104+
.SYNOPSIS
105+
Simulate Alt+V (clipboard image paste) in the agent pane. VK_V=0x56, scan=0x2F, no
106+
character (Alt+letter is a menu accelerator, so Uc=0 — matching real hardware),
107+
LEFT_ALT_PRESSED=0x02. The agent pane session is resolved automatically.
108+
#>
109+
[CmdletBinding()] param([Parameter(Mandatory, ValueFromPipeline)]$App, [string]$PaneSessionId)
110+
process { Send-AgentWin32Key -App $App -Vk 0x56 -Sc 0x2F -Uc 0 -Modifiers 0x02 -PaneSessionId $PaneSessionId }
111+
}
112+
64113
function Open-AgentCommandMenu {
65114
<#
66115
.SYNOPSIS
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Clipboard.ps1 — OS clipboard primitives for tests.
2+
#
3+
# The agent pane's Alt+V image paste (issue #211) reads the *live* OS clipboard from the
4+
# wta-helper (a real Win32 process sharing the interactive session's clipboard). To exercise
5+
# it end-to-end a test must place a real image on the clipboard first.
6+
#
7+
# `Clipboard.SetImage` requires an STA thread, so we run it on a dedicated STA runspace
8+
# (pwsh 7 defaults to MTA and has no -STA switch). The bitmap lands as CF_BITMAP/CF_DIB,
9+
# which `read_clipboard_image` picks up via its CF_DIBV5/CF_DIB path (label "screenshot").
10+
11+
function Set-ClipboardImage {
12+
<#
13+
.SYNOPSIS
14+
Put a synthetic bitmap on the OS clipboard (CF_DIB) so the agent pane's Alt+V has a
15+
real image to capture. Returns nothing; throws if the clipboard write fails.
16+
.PARAMETER Width / -Height
17+
Size of the generated bitmap (default 24x24).
18+
#>
19+
[CmdletBinding()]
20+
param([int]$Width = 24, [int]$Height = 24)
21+
22+
$worker = {
23+
param($w, $h)
24+
Add-Type -AssemblyName System.Windows.Forms, System.Drawing
25+
$bmp = [System.Drawing.Bitmap]::new($w, $h)
26+
for ($x = 0; $x -lt $w; $x++) {
27+
for ($y = 0; $y -lt $h; $y++) {
28+
$color = [System.Drawing.Color]::FromArgb(255, ($x * 10 % 256), ($y * 10 % 256), 128)
29+
$bmp.SetPixel($x, $y, $color)
30+
}
31+
}
32+
[System.Windows.Forms.Clipboard]::SetImage($bmp)
33+
$bmp.Dispose()
34+
}
35+
36+
$rs = [runspacefactory]::CreateRunspace()
37+
$rs.ApartmentState = 'STA'
38+
$rs.ThreadOptions = 'ReuseThread'
39+
$rs.Open()
40+
try {
41+
$ps = [powershell]::Create()
42+
try {
43+
$ps.Runspace = $rs
44+
$null = $ps.AddScript($worker).AddArgument($Width).AddArgument($Height)
45+
$ps.Invoke() | Out-Null
46+
if ($ps.Streams.Error.Count) { throw "Set-ClipboardImage failed: $($ps.Streams.Error[0])" }
47+
}
48+
finally { $ps.Dispose() }
49+
}
50+
finally { $rs.Close(); $rs.Dispose() }
51+
}
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#Requires -Modules @{ ModuleName='Pester'; ModuleVersion='5.0.0' }
2+
# Feature: agent pane image paste (Alt+V) — issue #211.
3+
#
4+
# Verifies the end-to-end Alt+V flow against a deployed package:
5+
# screenshot on clipboard -> focus agent pane -> simulate Alt+V -> image captured & queued.
6+
#
7+
# The key thing this proves that the Rust-side tests cannot: that Alt+V, injected through
8+
# WT/ConPTY as a win32-input-mode key event, actually reaches the helper's crossterm reader
9+
# as a KeyEvent carrying the ALT modifier (the path the unit tests stub out with a synthetic
10+
# KeyEvent). If this assumption were wrong, Alt+V would type a literal 'v' or be dropped, and
11+
# none of the handler's three outcome messages would appear.
12+
#
13+
# Invoke-Pester test/e2e/tests/Feature.AgentImagePaste.Tests.ps1 -Tag Feature
14+
15+
BeforeDiscovery {
16+
$script:Ready = [bool](
17+
(Get-AppxPackage | Where-Object { $_.Name -like '*IntelligentTerminal*' }) -and
18+
(Get-Command copilot -ErrorAction SilentlyContinue) -and
19+
(Get-Command winapp -ErrorAction SilentlyContinue)
20+
)
21+
}
22+
23+
Describe 'Feature: agent pane image paste (Alt+V)' -Tag 'Feature' -Skip:(-not $script:Ready) {
24+
BeforeAll {
25+
Import-Module (Join-Path $PSScriptRoot '..\ItE2E\ItE2E.psd1') -Force
26+
$script:app = Start-Terminal -Package (Get-ItTestPackage) -PassFre $true -Settings @{ acpAgent = 'copilot' }
27+
Open-AgentPane -App $script:app | Out-Null
28+
Wait-AgentReady -App $script:app -TimeoutSec 60 | Out-Null
29+
}
30+
AfterAll { if ($script:app) { Stop-Terminal -App $script:app } }
31+
32+
It 'Alt+V reaches the handler and captures the clipboard image (not typed as a literal key)' {
33+
# 1) Put a real image (CF_DIB) on the shared OS clipboard.
34+
Set-ClipboardImage
35+
36+
# 2) Focus the agent pane input and dismiss any popup / stale char.
37+
Set-AgentPaneFocus -App $script:app | Out-Null
38+
Clear-AgentInput -App $script:app | Out-Null
39+
40+
# 3) Simulate Alt+V via win32-input-mode raw key events.
41+
Send-AgentAltV -App $script:app | Out-Null
42+
43+
# 4) The handler must have run. With an image on the clipboard it either ATTACHES it
44+
# (📎 title / "Image attached") or — if copilot doesn't advertise the image prompt
45+
# capability — reports it UNSUPPORTED. Either proves Alt+V arrived as a KeyEvent
46+
# with the ALT modifier. It must NOT report an empty clipboard (we set one).
47+
Assert-AgentPaneText -App $script:app -Pattern '📎|Image attached|support image input' -TimeoutSec 15
48+
49+
$after = Get-AgentPaneText -App $script:app -MaxLines 80
50+
$after | Should -Not -Match 'No image on the clipboard'
51+
}
52+
53+
It 'Queued image is sent to the agent as an ACP image content block (when the agent supports images)' {
54+
$paneText = Get-AgentPaneText -App $script:app -MaxLines 80
55+
if ($paneText -match 'support image input') {
56+
Set-ItResult -Skipped -Because 'the configured agent does not advertise the image prompt capability'
57+
return
58+
}
59+
# The image is queued (the 📎 input-box title). Submit it, then confirm an image
60+
# content block reached the agent on the ACP wire. The wire trace only lands when
61+
# debug logging is on, so treat its absence as inconclusive rather than a failure.
62+
$acpLog = Get-ItLogText -App $script:app -Name 'wta-acp-debug.log' -SinceStart
63+
if ([string]::IsNullOrWhiteSpace($acpLog)) {
64+
Set-ItResult -Skipped -Because 'wta-acp-debug.log is empty (ACP wire trace requires WTA_LOG=debug)'
65+
return
66+
}
67+
68+
Send-AgentKey -App $script:app -Key Enter | Out-Null
69+
Wait-AgentState -App $script:app -State Working -TimeoutSec 30 | Out-Null
70+
71+
$sent = Test-Until -TimeoutSec 30 -IntervalSec 1 -Condition {
72+
$log = Get-ItLogText -App $script:app -Name 'wta-acp-debug.log' -SinceStart
73+
($log -match '"type"\s*:\s*"image"') -or ($log -match 'image/png')
74+
}
75+
$sent | Should -BeTrue -Because 'the pasted image must reach the agent as an ACP ContentBlock::Image'
76+
}
77+
}

tools/wta/Cargo.lock

Lines changed: 99 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

tools/wta/Cargo.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,16 @@ windows-sys = { version = "0.61", features = [
2828
"Win32_Foundation",
2929
"Win32_Globalization",
3030
"Win32_Storage_Packaging_Appx",
31+
"Win32_System_DataExchange",
3132
"Win32_System_Environment",
33+
"Win32_System_Memory",
3234
"Win32_System_Registry",
3335
"Win32_System_Threading",
36+
"Win32_UI_Shell",
3437
] }
38+
# Image decode/encode for clipboard-paste (Alt+V). Minimal feature set: decode
39+
# the clipboard DIB (BMP) and re-encode to PNG for ACP `ContentBlock::Image`.
40+
image = { version = "0.25", default-features = false, features = ["bmp", "png"] }
3541
tracing = "0.1"
3642
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
3743
tracing-appender = "0.2"

0 commit comments

Comments
 (0)