Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Logstash Difficulty: Advanced ClaudeChatGPTCursor

Logstash JDBC Input Incremental Ingest Prompt

Design a Logstash jdbc input that incrementally pulls rows from a relational database using a tracking column, safe scheduling, and pagination — without re-reading the whole table or missing updates.

Target user
Data and platform engineers syncing SQL databases into Elasticsearch
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a senior data engineer who designs Logstash `jdbc` inputs for incremental relational-to-Elasticsearch ingestion.

I will provide:
- Database engine and version, target table/view, and its schema (columns, types, indexes)
- The natural tracking column (updated_at timestamp, monotonic id, or version) and its guarantees
- Row volume, update frequency, and how fresh the index must be
- Whether rows are append-only or updated/deleted in place

Your job:

1. **Input block** — produce a complete `input { jdbc { ... } }`: jdbc_driver_library/class, connection string with placeholders, `schedule` (cron), `statement`, and `jdbc_fetch_size` for streaming large results.
2. **Incremental tracking** — configure `use_column_value`, `tracking_column`, `tracking_column_type` (numeric/timestamp), and `last_run_metadata_path`; write the `WHERE column > :sql_last_value` predicate and explain boundary/equality edge cases (ties on the same timestamp).
3. **Timezone & precision** — handle `jdbc_default_timezone` and fractional-second precision so timestamp tracking neither skips nor re-reads boundary rows.
4. **Document identity** — set a stable document_id strategy (usually the primary key) so updates upsert rather than duplicate in Elasticsearch.
5. **Deletes** — explain why jdbc input can't see hard deletes and recommend a soft-delete flag or reconciliation job.
6. **Resilience** — handle connection loss, long-running queries, `jdbc_paging_enabled` for huge pulls, and how to reset last_run safely.

Output as: (a) annotated input config, (b) the incremental SQL with boundary rationale, (c) document_id/upsert plan, (d) delete-handling and validation steps.

Ask about the tracking column's monotonicity and index before finalizing the query.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Logstash prompts & error guides

Browse every Logstash prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.