Skip to main content

Create & Delete Datasets

Create and manage datasets for cross-source correlations.

Use Case: Create datasets when you need to store:

  • Cross-source correlations and enriched context
  • Staging data for testing transformations
  • Data that doesn't fit existing external schemas
  • Quick prototypes and experimental datasets

Create Dataset​

Creates a new dataset with the specified name and optional configuration.

Endpoint: PUT /{dataset_name}

Authentication: This request must be authenticated using one of the methods described in the Authentication documentation.

Response:

{
  "acknowledged": true
}

Dataset Naming Conventions​

Follow these rules when creating datasets:

  • Valid characters: Letters, numbers, hyphens, dots (not as first character)
  • Case sensitive: MyDataset ≠ mydataset
  • Length limits: 1-255 characters
  • Reserved prefixes: Cannot start with underscore (_) or dot (.)
  • Special characters: Use URL encoding for spaces and special characters

Valid dataset names:

logs-2024-01-15
user-events
metrics-prod
sales-data
customer-records

Invalid dataset names:

_system-dataset   # Cannot start with underscore
.hidden-dataset   # Cannot start with dot
logs with spaces  # Use URL encoding: logs%20with%20spaces

Delete Dataset​

Permanently deletes a dataset and all its data.

Endpoint: DELETE /{dataset_name}

Authentication: This request must be authenticated using one of the methods described in the Authentication documentation.

Response:

{
  "acknowledged": true
}
Data Loss

Deleting a dataset permanently removes all data. This operation cannot be undone.

Check Dataset Existence​

Check if a dataset exists without returning data.

Endpoint: HEAD /{dataset_name}/metadata

Authentication: This request must be authenticated using one of the methods described in the Authentication documentation.

Responses:

  • 200 OK: Dataset exists
  • 404 Not Found: Dataset does not exist

Error Responses​

Common error responses for dataset operations:

400 Bad Request​

Invalid dataset name or request format:

{
  "status": "error",
  "message": "Could not create dataset: Invalid dataset name"
}

Causes:

  • Dataset name starts with underscore (_) or dot (.)
  • Dataset name contains invalid characters or patterns
  • Dataset name conflicts is a reserved system dataset name

401 Unauthorized​

Missing or invalid authentication:

{
  "status": "error", 
  "message": "Could not create dataset: Authentication failed"
}

403 Forbidden​

Valid authentication but insufficient permissions:

{
  "status": "error",
  "message": "Could not create dataset: Permission denied"
}

404 Not Found​

Dataset does not exist (for delete operations):

{
  "status": "error",
  "message": "Could not delete dataset: Dataset not found"
}

409 Conflict​

Dataset already exists (for create operations):

{
  "status": "error",
  "message": "Could not create dataset: Dataset already exists"
}

500 Internal Server Error​

Server-side processing errors:

{
  "status": "error",
  "message": "Could not create dataset: Failed to setup semantic search"
}

Common causes:

  • CoreDB communication errors
  • Resource allocation issues

Best Practices​

Dataset Creation​

  • Use descriptive, consistent naming conventions
  • Consider date-based patterns for time-series data
  • Test with small data sets first

Dataset Deletion​

  • Always backup critical data before deletion
  • Use specific dataset names, never wildcards
  • Verify dataset name spelling before deletion
  • Consider archiving data instead of deleting