Logstash JDBC Input Incremental Ingest Prompt
Design a Logstash jdbc input that incrementally pulls rows from a relational database using a tracking column, safe scheduling, and pagination — without re-reading the whole table or missing updates.
- Target user
- Data and platform engineers syncing SQL databases into Elasticsearch
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior data engineer who designs Logstash `jdbc` inputs for incremental relational-to-Elasticsearch ingestion.
I will provide:
- Database engine and version, target table/view, and its schema (columns, types, indexes)
- The natural tracking column (updated_at timestamp, monotonic id, or version) and its guarantees
- Row volume, update frequency, and how fresh the index must be
- Whether rows are append-only or updated/deleted in place
Your job:
1. **Input block** — produce a complete `input { jdbc { ... } }`: jdbc_driver_library/class, connection string with placeholders, `schedule` (cron), `statement`, and `jdbc_fetch_size` for streaming large results.
2. **Incremental tracking** — configure `use_column_value`, `tracking_column`, `tracking_column_type` (numeric/timestamp), and `last_run_metadata_path`; write the `WHERE column > :sql_last_value` predicate and explain boundary/equality edge cases (ties on the same timestamp).
3. **Timezone & precision** — handle `jdbc_default_timezone` and fractional-second precision so timestamp tracking neither skips nor re-reads boundary rows.
4. **Document identity** — set a stable document_id strategy (usually the primary key) so updates upsert rather than duplicate in Elasticsearch.
5. **Deletes** — explain why jdbc input can't see hard deletes and recommend a soft-delete flag or reconciliation job.
6. **Resilience** — handle connection loss, long-running queries, `jdbc_paging_enabled` for huge pulls, and how to reset last_run safely.
Output as: (a) annotated input config, (b) the incremental SQL with boundary rationale, (c) document_id/upsert plan, (d) delete-handling and validation steps.
Ask about the tracking column's monotonicity and index before finalizing the query.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Logstash Beats Input Design Prompt
Design and harden a Logstash beats input that ingests from Filebeat/Metricbeat fleets at scale, with TLS, backpressure tuning, and clean field handling before filtering.
-
Logstash HTTP Input Design Prompt
Design a Logstash http input that safely receives webhooks and application events over HTTP(S), with authentication, TLS, request validation, and codec handling before events enter the pipeline.
-
Logstash Kafka Input Design Prompt
Design a resilient Logstash kafka input that consumes from topics with correct consumer-group semantics, partition-to-worker mapping, offset handling, and at-least-once delivery guarantees.
-
Design a Logstash-to-Elasticsearch Mapping & Index Template Strategy
Design the index templates, dynamic-mapping controls, and field hygiene that keep a Logstash elasticsearch output from triggering mapping conflicts, field-limit explosions, and mapper_parsing_exception rejections at scale.
More Logstash prompts & error guides
Browse every Logstash prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.