Harden extraction edge cases and add upstream API parity - #10
Open
SupperTomato wants to merge 7 commits into
Open
Harden extraction edge cases and add upstream API parity#10SupperTomato wants to merge 7 commits into
SupperTomato wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens rs-trafilatura’s extraction pipeline for tricky inputs (UTF-8 truncation, malformed/large HTML, nested/large tables, structured-data fallbacks) and expands feature parity with upstream trafilatura-style helpers and richer media preservation/export.
Changes:
- Add upstream-style public helpers (
bare_extraction,extract_with_metadata,extract_metadata,load_html*,baseline,html2txt) and make extraction results serializable viaserde. - Improve robustness/perf: UTF-8-safe truncation, JSON-LD “hook” scanning before parsing, baseline fallback enhancements, and Markdown pre-processing to flatten pathological tables and represent video/audio as links.
- Extend content preservation options for media (
include_videos,include_audio) and add/expand tests + benchmarks for edge cases and regressions.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/table_test.rs | Adds a regression test to guard against superlinear Markdown growth from nested tables. |
| tests/spike_markdown_validation.rs | Suppresses deprecation warnings in a spike test and avoids unused variables. |
| tests/robustness_test.rs | Adds UTF-8 truncation boundary coverage for max_extracted_len. |
| tests/official_api_test.rs | New tests validating the new “official/compat” helper APIs and JSON serialization. |
| tests/image_link_toggle_test.rs | Expands coverage for srcset/picture selection and media (video/audio) preservation in exports. |
| tests/image_license_test.rs | Adds coverage for <meta name="image"> metadata extraction. |
| tests/boilerplate_test.rs | Adds coverage ensuring <fencedframe> content is excluded as boilerplate. |
| src/result.rs | Derives Serialize/Deserialize for public result structs and introduces BaselineResult. |
| src/page_type/mod.rs | Minor warning hygiene and test import adjustments around page-type profiling. |
| src/page_type/ml.rs | Simplifies per-section feature accumulation logic. |
| src/options.rs | Adds include_videos / include_audio options and defaults/tests. |
| src/metadata/mod.rs | Adds module-level warning suppression and keeps metadata extraction wiring. |
| src/metadata/meta_tags.rs | Extends meta image extraction to include name="image". |
| src/markdown.rs | Adds HTML pre-processing for Markdown conversion (flatten large/nested tables; replace video/audio with links). |
| src/lib.rs | Exposes upstream-style public helper functions and re-exports BaselineResult. |
| src/html_processing.rs | Updates cleaner behavior to conditionally preserve image/video/audio-related tags and attributes. |
| src/extractor/tags.rs | Adds fencedframe to the cleaning tag list and updates tag-count tests. |
| src/extractor/state.rs | Tracks additional potential tags for images/video/audio (figure/picture/source/track). |
| src/extractor/mod.rs | Adds module-level warning suppression in the extractor module. |
| src/extractor/fallback.rs | Optimizes JSON-LD scanning and expands structured-text collection for baseline fallback paths. |
| src/extract.rs | Implements input/output truncation safeguards, metadata gating (only_with_metadata), richer media HTML export, and Markdown safety pre-processing. |
| src/etree.rs | Adds a targeted allowance for unused imports in re-export module. |
| README.md | Updates project status and option count to reflect new capabilities. |
| Cargo.toml | Bumps crate version and enables chrono serde support. |
| benches/benchmark.rs | Adds baseline/html2txt benchmarks and structured-data baseline scenarios. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
653
to
655
| if total > best_total_len { | ||
| best_total_len = total; | ||
| best_group = Some(texts); | ||
| } |
Comment on lines
+136
to
+155
| let result = extract_with_options( | ||
| html, | ||
| &Options { | ||
| max_extracted_len: 3, | ||
| min_extracted_len: 0, | ||
| min_output_size: 0, | ||
| ..Options::default() | ||
| }, | ||
| ) | ||
| .expect("extraction should not panic or fail"); | ||
|
|
||
| assert!(result | ||
| .content_text | ||
| .is_char_boundary(result.content_text.len())); | ||
| assert!(result.content_text.len() <= 3); | ||
| assert!(result | ||
| .warnings | ||
| .iter() | ||
| .any(|w| w.contains("Content truncated"))); | ||
| } |
| //! This module provides functions for extracting metadata from HTML documents, | ||
| //! including JSON-LD parsing, HTML meta tags, Open Graph, and other sources. | ||
|
|
||
| #![allow(unused_imports)] |
| //! pruning::prune_unwanted_sections(&tree, state.potential_tags(), &opts); | ||
| //! ``` | ||
|
|
||
| #![allow(unused_imports)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification