Upload records to a dataset

Add records to datasets for cross-source correlations. Use bulk operations for high-throughput upload.

Note: For production workloads, query your data sources in place via Connect connections instead of importing.

Bulk Upload API

Infino supports bulk upload for high-throughput data loading. Use bulk operations whenever possible for better performance compared to individual record operations.

Format Requirement

JSON uploads must use Elasticsearch bulk format (NDJSON with action/source pairs). This is the standard format used by Elasticsearch, OpenSearch, and other search engines.

Endpoints

POST /{dataset}/json

This endpoint uploads records to the specified dataset in Elasticsearch bulk format.

Dataset Specification

The dataset name is specified in the URL. Each record in the bulk payload can optionally specify additional metadata.

// URL: POST /logs-app/json
{ "index": {} }
{ "message": "App started", "level": "info" }  // Record goes to logs-app dataset

Request Format (Elasticsearch Bulk Format)

Bulk requests use the Elasticsearch bulk format - newline-delimited JSON (NDJSON) where each operation consists of:

1. Action line: Specifies the operation type (index, create, or delete)
2. Source line: Contains the actual record data (except for delete operations)

{ "index": { } }
{ "field1": "value1", "field2": "value2" }
{ "create": { } }
{ "field1": "value3", "field2": "value4" }

Elasticsearch Bulk Format Required

This endpoint follows the Elasticsearch bulk API specification. Each JSON object must be on a single line, and the request must end with a newline character. The format is compatible with Elasticsearch, OpenSearch, and other search engines.

Not supported: Custom JSON arrays or other formats. Only Elasticsearch bulk format (NDJSON with action/source pairs).

Supported Operations

Infino supports the following bulk operations:

Index Operation

Creates a record or replaces it if it already exists.

{ "index": {} }
{ "message": "Application started", "level": "info", "@timestamp": "2024-01-15T10:30:00Z" }

Create Operation

Creates a record only if it doesn't already exist. Returns an error if the record exists.

{ "create": {} }
{ "message": "User login", "user": "john", "@timestamp": "2024-01-15T10:31:00Z" }

Delete Operation

Removes a record from the dataset. No record data line is required.

{ "delete": {} }

Metadata Fields

Metadata is optional for most operations as the dataset is specified in the URL.

Record ID Generation: Infino automatically generates unique record IDs for all uploaded records. Record IDs should not be specified in bulk operations as they are always auto-generated.

Timestamp Field Processing

Infino automatically recognizes and optimizes timestamp fields for query performance:

Automatic Timestamp Recognition

• @timestamp field: Recognized as the primary timestamp field for the record
• Numeric values: Must be Unix timestamp in milliseconds for proper recognition
• Performance impact: Records with proper timestamps enable time-based query optimizations

Timestamp Processing Logic

// Record with @timestamp field (recommended)
{ 
  "message": "User login", 
  "@timestamp": 1705312200000,  // Unix timestamp in milliseconds
  "user": "alice" 
}

// Record without @timestamp - upload time is used
{
  "message": "User login",
  "user": "alice"
  // Infino automatically adds @timestamp with upload time
}

Timestamp Field Behavior

• Valid @timestamp: Uses the provided timestamp for the record's time reference
• Invalid @timestamp: Field is removed and upload time is used instead
• Missing @timestamp: Infino automatically adds @timestamp with the current upload time
• Query optimization: Proper timestamps enable efficient time-range queries and sorting

Performance Optimization

Include a valid @timestamp field with Unix timestamp in milliseconds for optimal query performance, especially for time-series data and log analysis.

Example Request

POST /{dataset}/json

{ "index": {} }
{ "message": "User login", "user": "alice", "@timestamp": "2024-01-15T10:30:00Z" }
{ "create": {} }
{ "message": "API request", "endpoint": "/users", "@timestamp": "2024-01-15T10:31:00Z" }
{ "delete": {} }

Authentication: This request must be authenticated using one of the methods described in the Authentication documentation.

Example Response

{
  "took": 15,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "logs-app",
        "_id": "auto-generated-id-1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 1,
          "successful": 1,
          "failed": 0
        },
        "status": 201
      }
    },
    {
      "create": {
        "_index": "logs-app", 
        "_id": "auto-generated-id-2",
        "_version": 1,
        "result": "created",
        "status": 201
      }
    },
    {
      "delete": {
        "_index": "logs-app",
        "_id": "auto-generated-id-3",
        "result": "deleted",
        "status": 200
      }
    }
  ]
}

Response Fields

• took: Time in milliseconds to process the request
• errors: Boolean indicating if any operations failed
• items: Array of results for each operation, in the same order as the request

Error Handling

Check the errors field in the response to determine if any operations failed:

{
  "took": 10,
  "errors": true,
  "items": [
    {
      "create": {
        "_index": "logs-app",
        "_id": "auto-generated-id",
        "status": 409,
        "error": {
          "type": "version_conflict_engine_exception",
          "reason": "Document creation failed",
          "index": "logs-app"
        }
      }
    }
  ]
}

Common error status codes:

• 400: Invalid request format or syntax
• 404: Index or document not found (for updates)
• 409: Document already exists (for create operations)

Best Practices

Performance Optimization

• Batch operations: Use bulk requests instead of individual record operations
• Optimal batch size: Use 100-1000 records per bulk request for best performance
• Dataset creation: Create datasets before bulk upload
• Content-Type header: Always use application/x-ndjson for bulk requests

Data Formatting

• NDJSON compliance: Each JSON object on a single line with trailing newline
• Timestamp fields: Include @timestamp field for time-series data
• Auto-generated IDs: Record IDs are automatically generated - do not specify _id field
• Field consistency: Use consistent field names and types across records

Error Handling

• Check errors field: Always verify the response errors boolean
• Iterate failed operations: Process individual operation errors for troubleshooting
• Retry logic: Implement exponential backoff for transient failures
• Validate before sending: Ensure NDJSON format is correct before transmission

Upsert Records via SQL

Use SQL INSERT... for upsert operations (insert or update existing records).

Endpoint: POST /sql

Example - Upsert user profile:

POST /sql
Content-Type: application/json

{
  "query": "INSERT INTO user_profiles (user_id, name, email, last_login) VALUES ('123', 'Alice Smith', 'alice@example.com', '2024-01-15T10:30:00Z') ON CONFLICT (user_id) DO UPDATE SET name = 'Alice Smith', email = 'alice@example.com', last_login = '2024-01-15T10:30:00Z'"
}

Authentication: This request must be authenticated using one of the methods described in the Authentication documentation.

Response:

{
  "rows_affected": 1
}

Use Cases:

• Update existing records or create if they don't exist
• Maintain current state (e.g., user profiles, inventory)
• Idempotent data loading with unique keys

For more details on SQL syntax, see SQL Queries.

Upload Metrics

Upload time-series metrics data to a dataset.

Endpoint: POST /{dataset}/metrics

Format Requirement

Metrics uploads must use Prometheus exposition format. This is the standard format used by Prometheus and compatible monitoring systems.

Request Format:

# HELP http_requests_total The total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1234 1609459200000
http_requests_total{method="POST",status="201"} 567 1609459200000
cpu_usage{host="server1",env="prod"} 75.5 1609459200000
memory_usage{host="server1",env="prod"} 85.2 1609459260000

Format: metric_name{label="value",...} value timestamp

Requirements:

• One metric per line
• Labels in {key="value"} format
• Numeric value
• Unix timestamp in milliseconds (optional)
• Compatible with Prometheus exposition format

Response:

{
  "acknowledged": true,
  "indexed": 4
}

Enrich Dataset

Configure automatic data enrichment for a dataset using the simplified API.

Endpoint: POST /{dataset}/enrich_dataset

Request Body:

{
  "enrich_policy": {
    "match_field": "user_id",
    "enrich_fields": ["email", "name", "department"]
  }
}

Response:

{
  "acknowledged": true
}

Use Cases:

• Automatically enrich records with additional context
• Join data from multiple sources
• Add computed fields