ed-news/tests/ at main · ebardelli.com/ed-news

Update calmatters preprocessor They turned off the rss feed, so I need to extract the articles from the page directly.

3 weeks ago

Move to ednews python module

9 months ago

Add article_exists helper and skip existing DOIs in ISSN lookup - add article_exists(conn, doi) - fetch_latest_journal_works skips existing DOIs and logs inserted/skipped counts - add tests: tests/test_issn_lookup_skip.py and tests/test_article_exists.py (including negative cases)

9 months ago

test_build.py

Implement mojibake recovery function across articles, headlines, and feeds; add XML declaration to RSS templates

3 months ago

test_build_filters_empty_items.py

Rename articles to research

7 months ago

test_build_includes_news.py

Add news section

9 months ago

test_cli_date_translation.py

Enhance ISSN lookup date handling and add tests for date normalization - Implement date normalization for CLI inputs in issn-lookup command. - Update documentation to clarify accepted date formats. - Add tests for date normalization functionality to ensure correct behavior.

9 months ago

test_cli_edworkingpapers_processor.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_cli_embed_flags.py

Add headlines embeddings

9 months ago

test_cli_embed_headlines.py

Add headlines embeddings

9 months ago

test_cli_fetch_flags.py

Enhance feed fetching with processor support - Updated the `fetch` command to support processors for feeds, allowing for flexible processing of entries. - Introduced a new `crossref_enricher_processor` to enrich feed entries with metadata from Crossref. - Added a `sciencedirect_feed_processor` to fetch and augment ScienceDirect entries with DOIs. - Modified `load_feeds` to include processor information in the feed configuration. - Updated tests to cover new processor functionalities and ensure proper integration with the fetching mechanism. - Enhanced documentation to reflect changes in project structure and usage.

9 months ago

test_cli_postprocess_selects_feed_processor.py

Use preferred postprocessor when running `postprocess`

8 months ago

test_cli_rematch_dois.py

Rework maintenance task

3 weeks ago

test_cli_rematch_only_articles.py

Fix rematch code [skip ci]

8 months ago

test_cli_rematch_reports_article_counts.py

Try to clean up mismatching items and articles

8 months ago

test_cli_remove_feed_articles.py

Fix article matching bugs

8 months ago

test_cli_remove_feed_discovery.py

Fix article matching bugs

8 months ago

test_cli_rss_preprocessor.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_combined_articles_published_formats.py

Parse mixed published formats and add regression test

6 months ago

test_crossref.py

Crossref parsing & date normalization

9 months ago

test_crossref_authors.py

Remove authors from citations and prefer creation date if available

9 months ago

test_crossref_choose_pref_nonfirst.py

Add command to rematch wrong dois by publication_id

8 months ago

test_crossref_integration.py

Fix crossref test

3 weeks ago

test_crossref_integration_publication_match.py

Match crossref query on preferred publication string

8 months ago

test_crossref_json_created.py

Remove authors from citations and prefer creation date if available

9 months ago

test_crossref_parsing_extra.py

Crossref parsing & date normalization

9 months ago

test_crossref_publication_match.py

Match crossref query on preferred publication string

8 months ago

test_crossref_wrappers.py

Fix rss preprocessor `rss.py`: call feeds.fetch_feed with explicit keyword args (and a guarded positional fallback) and always return a list Reason: the preprocessor previously passed positional args that could mismatch fetch_feed signature; that caused certain feeds (e.g. nber_ed configured with processor.pre == "rss") to be skipped in the normal fetch flow while the debug flow (which calls fetch_feed directly) succeeded Make Crossref helpers accept positional or keyword calls and be monkeypatch-friendly `crossref.py`: expose compatibility wrappers for query_crossref_doi_by_title(...) and fetch_crossref_metadata(...) that normalize positional/keyword arguments and delegate to internal implementations add a small indirection wrapper for the cached title-lookup so tests can monkeypatch the uncached implementation and still affect cached calls (preserves lru_cache while remaining test-friendly) minor typing/coercion in the wrapper to validate args Tests `test_rss_preprocessor.py`: unit tests for rss_preprocessor (normal dict result, non-dict fallback, positional-fallback path) test_crossref_wrappers.py: tests exercising positional and keyword calls to the Crossref wrappers (patches internal implementations to avoid network)

8 months ago

test_db.py

Add fix-encoding command to manage-db for mojibake recovery; implement repair_text_encoding function

3 months ago

test_db_get_article_by_title.py

Move hosting to s3

9 months ago

test_db_init.py

Add database initialization command and tests - Introduced `db-init` command to create the database schema and views. - Updated CLI to manage database initialization via the new command. - Added tests to verify schema creation and idempotency of the `db-init` command.

9 months ago

test_db_postprocessor_invocation.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_db_rematch_delete_on_unique_conflict.py

Fix rematch code [skip ci]

8 months ago

test_db_rematch_entries_none_for_postprocessor.py

Rework maintenance task

3 weeks ago

test_db_rematch_fallback_to_crossref.py

Fix rematch code [skip ci]

8 months ago

test_db_rematch_only_missing.py

Rework maintenance task

3 weeks ago

test_db_rematch_only_missing_title_update.py

Rework maintenance task

3 weeks ago

test_db_rematch_only_wrong_skips_when_empty.py

Rework maintenance task

3 weeks ago

test_db_rematch_orphan_articles.py

Fix rematch code [skip ci]

8 months ago

test_db_rematch_publication.py

Fix rematch code [skip ci]

8 months ago

test_db_rematch_publication_id_fence.py

Fix rematch code [skip ci]

8 months ago

test_db_rematch_retry_limit.py

Rework maintenance task

3 weeks ago

test_db_rematch_uses_feed_postprocessor.py

Fix rematch code [skip ci]

8 months ago

test_db_sync_articles_from_items.py

Try to clean up mismatching items and articles

8 months ago

test_db_utils.py

Split db utils to its own file

8 months ago

test_delete_rows.py

Add delete_rows script and corresponding GitHub workflow for row deletion

5 months ago

test_doi_helpers.py

Move to ednews python module

9 months ago

test_embed_compat.py

Add headlines embeddings

9 months ago

test_embeddings.py

Update page template And fix a bunch of stuff in the database

9 months ago

test_feeds.py

Add RSS feed fixture and update tests for feed parsing and ISSN handling - Created a new RSS feed fixture file `aera.rss` containing entries from the American Educational Research Journal. - Updated `test_fetch_feed_parses_entries` to utilize the new fixture for testing feed parsing, ensuring at least one entry is present. - Added a new test file `test_issn_tuple_shapes.py` to verify that feeds with extra fields are processed correctly, including inserting articles from a mocked Crossref JSON response.

9 months ago

test_feeds_crossref_augment.py

Add item deduping

8 months ago

test_feeds_dedup_by_url_hash.py

Add item deduping

8 months ago

test_feeds_doi_extraction.py

DOI extraction, dedup, and Crossref lookup caching

9 months ago

test_feeds_edge_cases.py

Add filter for empty items

9 months ago

test_feeds_save_filters.py

Implement filtered title cleanup and add related tests

9 months ago

test_fetch_postprocessor_items_shape.py

Use preferred postprocessor when fetching articles

8 months ago

test_filtered_titles_prevent_inserts.py

Fix article matching bugs

8 months ago

test_headline_embeddings.py

Add headlines embeddings

9 months ago

test_issn_enrich.py

Add ISSN lookup and Crossref enrichment functionality - Implemented `issn-lookup` command to fetch latest works for journals by ISSN. - Added sanitization for inputs in `upsert_article` to prevent SQLite binding errors. - Enhanced `fetch_latest_journal_works` to use LEFT JOIN for Crossref enrichment. - Created tests for ISSN lookup and Crossref enrichment.

9 months ago

test_issn_fetch_retries.py

Add backoff timing to crossref requests

9 months ago

test_issn_lookup_skip.py

Add article_exists helper and skip existing DOIs in ISSN lookup - add article_exists(conn, doi) - fetch_latest_journal_works skips existing DOIs and logs inserted/skipped counts - add tests: tests/test_issn_lookup_skip.py and tests/test_article_exists.py (including negative cases)

9 months ago

test_issn_tuple_shapes.py

Add RSS feed fixture and update tests for feed parsing and ISSN handling - Created a new RSS feed fixture file `aera.rss` containing entries from the American Educational Research Journal. - Updated `test_fetch_feed_parses_entries` to utilize the new fixture for testing feed parsing, ensuring at least one entry is present. - Added a new test file `test_issn_tuple_shapes.py` to verify that feeds with extra fields are processed correctly, including inserting articles from a mocked Crossref JSON response.

9 months ago

test_manage_db.py

Add functionality to remove articles with empty DOIs when a publication stub is present [skip ci]

8 months ago

test_migrate_add_items_url_hash.py

Add item deduping

8 months ago

test_migrate_resolve_collisions.py

Add item deduping

8 months ago

test_news.py

Add news section

9 months ago

test_news_db.py

Add headlines embeddings

9 months ago

test_news_fetch_and_save.py

Introduce database management module with initialization and maintenance functions - Added `ednews.db.manage_db` for database schema initialization, maintenance tasks, and publication synchronization. - Implemented functions for creating tables, logging maintenance runs, and fetching journal works from Crossref. - Added a compatibility shim in `ednews/manage_db.py` to maintain backward compatibility with existing imports. - Created a new CLI entry point in `ednews/main.py` to streamline command execution. - Updated `pyproject.toml` to reflect the new versioning scheme and added console script entry. - Added tests for database management functions, including vacuuming and cleaning up empty articles. - Refactored main CLI logic to delegate to the new `ednews.cli.run` function. - Updated various test files to ensure compatibility with the new structure and functionality.

9 months ago

test_news_ordering.py

Add news section

9 months ago

test_news_upsert_normalization.py

Add headlines embeddings

9 months ago

test_pd_feed_filter.py

Refactor local news category check and enhance test assertions for PD feed

5 months ago

test_pressdemocrat_helpers.py

Update local news category tests to require all keywords across tags

5 months ago

test_processors_calmatters.py

Update calmatters preprocessor They turned off the rss feed, so I need to extract the articles from the page directly.

3 weeks ago

test_processors_crossref.py

Enhance feed fetching with processor support - Updated the `fetch` command to support processors for feeds, allowing for flexible processing of entries. - Introduced a new `crossref_enricher_processor` to enrich feed entries with metadata from Crossref. - Added a `sciencedirect_feed_processor` to fetch and augment ScienceDirect entries with DOIs. - Modified `load_feeds` to include processor information in the feed configuration. - Updated tests to cover new processor functionalities and ensure proper integration with the fetching mechanism. - Enhanced documentation to reflect changes in project structure and usage.

9 months ago

test_processors_edpolicyinca.py

Update calmatters preprocessor They turned off the rss feed, so I need to extract the articles from the page directly.

3 weeks ago

test_processors_edworkingpapers.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_processors_edworkingpapers_authors_trim.py

Fix issues with the edworkingpapers postprocessor

8 months ago

test_processors_edworkingpapers_no_doi.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_processors_edworkingpapers_postprocessor.py

Fix issues with the edworkingpapers postprocessor

8 months ago

test_processors_edworkingpapers_published_date.py

Fix issues with the edworkingpapers postprocessor

8 months ago

test_processors_sciencedirect.py

Enhance feed fetching with processor support - Updated the `fetch` command to support processors for feeds, allowing for flexible processing of entries. - Introduced a new `crossref_enricher_processor` to enrich feed entries with metadata from Crossref. - Added a `sciencedirect_feed_processor` to fetch and augment ScienceDirect entries with DOIs. - Modified `load_feeds` to include processor information in the feed configuration. - Updated tests to cover new processor functionalities and ensure proper integration with the fetching mechanism. - Enhanced documentation to reflect changes in project structure and usage.

9 months ago

test_read_articles_limit.py

Add news section

9 months ago

test_read_articles_publications.py

Fix warning

9 months ago

test_rss_description.py

Add source titles to headlines rss

2 weeks ago

test_rss_description_empty_items.py

Rename articles to research

7 months ago

test_rss_feeds.py

Rename articles to research

7 months ago

test_rss_link_escaping.py

Escape links in rss feed There was a problem with links containing unescaped characters. This should now escape all characters in the url

4 months ago

test_rss_preprocessor.py

Fix rss preprocessor `rss.py`: call feeds.fetch_feed with explicit keyword args (and a guarded positional fallback) and always return a list Reason: the preprocessor previously passed positional args that could mismatch fetch_feed signature; that caused certain feeds (e.g. nber_ed configured with processor.pre == "rss") to be skipped in the normal fetch flow while the debug flow (which calls fetch_feed directly) succeeded Make Crossref helpers accept positional or keyword calls and be monkeypatch-friendly `crossref.py`: expose compatibility wrappers for query_crossref_doi_by_title(...) and fetch_crossref_metadata(...) that normalize positional/keyword arguments and delegate to internal implementations add a small indirection wrapper for the cached title-lookup so tests can monkeypatch the uncached implementation and still affect cached calls (preserves lru_cache while remaining test-friendly) minor typing/coercion in the wrapper to validate args Tests `test_rss_preprocessor.py`: unit tests for rss_preprocessor (normal dict result, non-dict fallback, positional-fallback path) test_crossref_wrappers.py: tests exercising positional and keyword calls to the Crossref wrappers (patches internal implementations to avoid network)

8 months ago

test_rss_title_escaping.py

Escape titles in RSS feeds and add tests for title escaping

4 months ago

test_sciencedirect_postprocessor_db.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

test_skip_crossref_when_article_exists.py

Move hosting to s3

9 months ago

test_strip_img_build.py

Add source titles to headlines page

3 weeks ago

test_sync_publications_collision.py

Add command to rematch wrong dois by publication_id

8 months ago

test_sync_publications_updates.py

Add command to rematch wrong dois by publication_id

8 months ago

test_tui.py

Add fix-encoding command to manage-db for mojibake recovery; implement repair_text_encoding function

3 months ago

test_tui_more.py

Add edworkingpapers - Implemented tests for the edworkingpapers processor to ensure proper parsing of HTML fixtures and validation of output fields. - Added tests for the edworkingpapers processor to confirm that no DOI is fabricated during preprocessing. - Created tests for the edworkingpapers postprocessor to verify fetching and upserting articles into the database. - Developed tests for the sciencedirect postprocessor to ensure articles are upserted correctly and DOIs are attached. - Introduced tests for the RSS preprocessor to validate its integration within the CLI. - Added tests for the database postprocessor invocation to ensure proper execution of pre and post-processing steps. - Removed obsolete test for enrich_crossref flags from the TUI tests.

8 months ago

Configure Feed

Configure Feed