Skip to main content

Data Overview

The Data module in the GeniSpace platform is divided into Data Source Management and Vector Datasets, providing data support for workflows and agents.

Access Path​

  • Console → Sidebar Data module
  • Or navigate directly to /data (defaults to Data Source Management)
  • Switch between tabs at the top: Data Source Management | Vector Datasets | Platform Data
  • URL parameters: /data?tab=datasource, /data?tab=dataset, /data?tab=platform
note

The Platform Data tab is currently disabled and cannot be opened.

Data Source Management​

Data Source Management is used to connect to external relational databases and perform CRUD operations within workflows.

Feature Structure​

Data Source Management includes three sub-views:

ViewDescription
Data SourcesSQL-based query/write configurations associated with connected databases
Data TablesManage table structures in databases (create, edit, delete)
DatabasesDatabase connection management, supporting MySQL, PostgreSQL, MariaDB, and ClickHouse

Basic Workflow​

  1. Add a database connection in the Databases view (fill in host, port, credentials, database name, etc.)
  2. Create or manage table structures in the Data Tables view
  3. Create a data source in the Data Sources view, select a database, and configure SQL statements
  4. Run functional tests to verify

Data sources can be invoked in workflow nodes and can also be converted into Data Source Tools for use by agents.

Vector Datasets​

Vector datasets are used to store and retrieve vectorized data, providing semantic search and knowledge support for agents.

Features​

1. Data Management​

  • Custom Datasets: Create and manage custom datasets, supporting multiple field types
  • Data Preview: Preview dataset contents in real time, with pagination and search support
  • Data Operations: Insert, query, update, and delete records via the data-operation APIs

2. Vectorization Support​

  • Automatic Vectorization: Support automatic text vectorization without manual processing
  • Vector Search: Support similarity search based on vectors
  • Vector Fields: Support custom vector fields with flexible dimension configuration

3. Data Operations​

  • Data Query: Support complex query conditions, including filtering, sorting, etc.
  • Data Update: Support batch updates and individual record updates
  • Data Deletion: Support conditional deletion and batch deletion
  • Data Insertion: Support batch insertion and individual record insertion

Integration with Agents​

  • Select associated datasets in the agent configuration
  • Agents can retrieve relevant knowledge from datasets via vector search
  • Support connecting multiple knowledge bases simultaneously

Usage Guide​

Accessing Vector Datasets​

  1. Click the Vector Datasets tab at the top of the Data module
  2. For first-time use, confirm that the space key has been initialized (key status is shown in the statistics card)

Creating a Dataset​

  1. Click the "Create Dataset" button on the Vector Datasets page
  2. Fill in the basic dataset information:
    • Name: Unique identifier for the dataset
    • Description: Detailed description of the dataset
    • Database Type: Select Milvus, etc.
    • Database Configuration: Configure options such as auto ID
  3. Define the dataset structure:
    • Add fields: Support multiple data types
    • Set primary key: Select the primary key field
    • Configure indexes: Create indexes for fields that need indexing
  4. Click "Create" to complete dataset creation

Vector Dataset Interface Features​

  • Statistics Cards: Total datasets, total records, total data size, key status, with expandable detailed statistics and space key management
  • Create Dataset: Create a new vector dataset
  • Search & Filter: Search by name, filter by database type
note

Data import and export for vector datasets are not currently available in the interface. Use the data-operation APIs below to insert and manage records.

Data Operations​

Data-operation endpoints are mounted under /api/datasets, and the dataset is referenced by the :datasetId path parameter:

  • POST /api/datasets/:datasetId/data/insert
  • POST /api/datasets/:datasetId/data/query
  • POST /api/datasets/:datasetId/data/update
  • POST /api/datasets/:datasetId/data/delete
  • POST /api/datasets/:datasetId/data/search
  • POST /api/datasets/:datasetId/data/full-text-search

Insert Data​

POST /api/datasets/:datasetId/data/insert
{
  "data": [
    {
      "field1": "value1",
      "field2": "value2"
    }
  ]
}

Query Data​

POST /api/datasets/:datasetId/data/query
{
  "filter": "field1 == 'value1'",
  "limit": 100,
  "offset": 0,
  "outputFields": ["field1", "field2"]
}

Update Data​

POST /api/datasets/:datasetId/data/update
{
  "filter": "field1 == 'value1'",
  "update_data": {
    "field2": "new_value"
  }
}

Delete Data​

POST /api/datasets/:datasetId/data/delete
{
  "filter": "field1 == 'value1'"
}
POST /api/datasets/:datasetId/data/search
{
  "vector_field": "vector",
  "data": [[0.1, 0.2, ..., 0.5]],
  "limit": 5,
  "filter": "category == 'product'",
  "outputFields": ["id", "title", "content"]
}

Full-Text Search​

POST /api/datasets/:datasetId/data/full-text-search
{
  "query": "search keywords",
  "limit": 5,
  "outputFields": ["id", "title", "content"]
}

Supported Data Types​

Datasets support the following field types:

  • VARCHAR: String
  • INT8 / INT16 / INT32 / INT64: Integers of various widths
  • FLOAT: Floating-point number
  • DOUBLE: Double-precision floating-point number
  • BOOL: Boolean
  • JSON: JSON value
  • ARRAY: Array
  • FLOAT_VECTOR: Float vector

Vector fields use the FLOAT_VECTOR type. The dimension defaults to 1024, and the similarity metric can be L2 (Euclidean distance), IP (inner product), or COSINE (cosine similarity).

Best Practices​

  1. Data Preprocessing

    • Clean and format data before importing
    • Ensure data conforms to field type requirements
    • Handle missing values and outliers

  2. Vectorization Configuration

    • Choose appropriate text fields for vectorization
    • Set vector dimensions based on actual needs
    • Regularly update the vectorization model

  3. Query Optimization

    • Use filter conditions wisely
    • Set appropriate page sizes
    • Query only necessary fields

  4. Performance Considerations

    • Control data volume during batch operations
    • Use indexes appropriately
    • Avoid frequent small data operations

Important Notes​

  1. Dataset names must be unique
  2. At least one vector field is required
  3. Primary key fields must be unique
  4. Vector field dimensions must be fixed
  5. Ensure the dataset exists and is accessible before performing data operations
  6. Pay attention to data type compatibility
  7. Regularly back up important data

FAQ​

  1. Q: How do I choose the right vector dimensions? A: Vector dimensions typically depend on the vectorization model used. We recommend 768 or 1536 dimensions.

  2. Q: What should I do if inserting data fails? A: Check whether the data format meets the requirements, ensure field types match, and review error logs for detailed information.

  3. Q: How can I optimize query performance? A: Use indexes wisely, optimize filter conditions, control the number of returned fields, and use pagination appropriately.

  4. Q: How should I set the similarity threshold for vector search? A: Adjust based on your specific use case and requirements. Typically, 0.7–0.8 is a good starting point.