Editing Development/Adding a source (section)

=== Inline downloader (for non-archive sources) ===

For sources that do not follow the tar.gz archive pattern (e.g., CSV files, API endpoints), write a standalone downloader class that handles its own parsing and DB insertion. This is called an "inline source".

The CADA downloader is a good example of an inline source. Key patterns:

<syntaxhighlight lang="python">
class CadaDownloader:
    """Downloads CADA CSV from data.gouv.fr and ingests into corpus.documents."""

    def __init__(self, dsn: str, data_directory: Path) -> None:
        self.dsn = dsn
        self._cache_directory = data_directory / "cada"

    def run(self, errors: IngestErrors | None = None) -> list[str]:
        """Download, parse, and insert. Returns []."""
        # 1. Check upstream for changes (compare last_modified)
        # 2. Download or use cached file
        # 3. Parse rows into corpus.documents records
        # 4. Insert in batches using insert_batch()
        # 5. Clean up orphaned records
        # 6. Update ingest state
        ...
</syntaxhighlight>

Key conventions for inline downloaders:

* Accept <code>dsn</code> (PostgreSQL connection string) and <code>data_directory</code> (root for cached files) in the constructor
* Accept an optional <code>errors: IngestErrors | None</code> parameter in <code>run()</code> for error tracking
* Cache downloaded files to disk with a <code>.tmp</code> extension during download and atomic rename on completion
* Store a sidecar metadata file (e.g., <code>.meta</code>) for change detection
* Use <code>get_ingest_state()</code> / <code>set_ingest_state()</code> from <code>duralex.ingest.state</code> to track whether the upstream data has changed
* Use <code>insert_batch()</code> from <code>duralex.ingest.database.batch_writer</code> for bulk insertion
* Return an empty list (inline sources handle their own insertion)