Epic 2026.1: User facing logs in project sessions

[hubertp]

Current session logs contain extensive engineering-focused information about project execution. Logs are written to a file and are not available within the IDE while working on a workflow.

In hybrid and cloud, there is a single log file per session; in local execution, there is a single log file per day.

• Generate logs per project execution #14754

The goal of this epic is to provide logging capabilities to achieve a few different aims:

1. Debugging at different levels: Engine, Library and End User (workflow).

• Note that shared user libraries will mean that end users will have an interest in similar capabilities as the in-house devs.

2. Audit logging (aimed at IT - record access to audit)

Additionally, but separate to the main logging story, we also want to gather crash reports and telemetry information (aimed at Enso).

Types of Events:

Audit Log

• Aimed for IT.
  • Helps determine what resource is using what.
• It is understood that a user with enough programming knowledge could bypass this.
• Record read and write events. Including things like email, post to API, etc.
• Associated with a specific resource (file, URL, database).
  • For database actions, should the query.
  • For web actions, records the method, URL
• Records time, user, workflow, action, session id, and runtime (hybrid, local, cloud, async).
  • Session ID would allow connecting back to other logs.
• If a workflow uses an audit resource (something inside Enso Cloud), then all IO (local or cloud) should be audited. new
  • Audited resources are those inside the Enso cloud or referenced by a data link.
• Created only by Enso developers.
  • TBD: how to cover 3rd party library use of auditted resources.
• The current Audit panel needs more fields!
  • Want to be able to access the audit from the audit resource.
  • Audit panel should eventually be a "middle" panel.

Runtime logs (diagnostic)

• Aimed for developers.
  • GUI, Engine or library developers.
  • Including end-users building shared libraries.
• Associated with a session ID (including local executions).
  • Should be able to get sessions for local projects and get the logs from the execution. new
  • Each engine session should be its own file.
  • Hybrid and cloud should all gather logs back to the cloud store.
• Should not be gathering TRACE logs by default. Currently, we send TRACE log to the cloud
  • Allow to opt in an environment variable.
• Log panel: Work here to overhaul the panel
  • Improve the current log panel with filtering support.
  • Ability to filter based on level.
  • Rules to hide engine and standard library logs.
  • Can be downloaded from the UI as a single file.
  • Easy to get to the underlying log file for the current session.
• Libraries to increase runtime logs for easier debugging.
• Sent via SLF4J or Any.log_message.

Message Events (normally data driven)

• Associated with an executing component
• Three types:
  • Data Flow Error.
  • Warning attached to a value.
  • Message Event. new could be using the progress API log
• Aimed for users.
• Either:
  • Created by users for monitoring workflow.
  • Generated by the libraries
    • Filtered row count, Join row count, etc.
    • Warnings and errors as before.
• Associated with a component in the workflow.
• Any message event in the main method should be logged to the runtime log as well.
• Within the IDE:
  • Should have a panel which gathers message events, warnings and errors.
  • Imagined as a new tab in the bottom section (code editor).
  • When a component is rerun, its messages are flushed, and only the current ones are shown.
  • When selecting a log entry, choose the component in the editor.
  • The selection of components within the UI makes the associated messages bolder than the others. (could have similar behaviour in code view)
  • Messages can contain a link allowing the user to click and open the targeted object (open in data catalogue / explorer).
    • For example, a file name (opens explorer), url (open browser), enso link (open data catalog to it). (could be a specific warning or error type).

Future work

• Logs will be associated with result gathering when running within the scheduler.
• Users could choose to include a component's output in the results.
  • Could be a selection within the IDE.
  • Return from the workflow.
  • To be further designed.
• Possible that a result would be written as a log message telling the scheduler where to get the "result" from.
• Could be a serialised version of objects.
• Further design to be covered when result gathering is in scope.

Examples

Writing a CSV

• User makes a Table of data
• write to a location in their Enso Cloud as a CSV
  • Runtime log: DEBUG writing started.
  • Progress: every n rows record progress tick.
  • Runtime log: DEBUG writing completed.
  • Audit event: Write to file (CSV, n rows).
  • Telemetry: Wrote n in CSV format.
  • Message: Wrote n rows to in CSV format. (including a hyperlink to the file (enso://))
    • As in main, also written to Runtime log.