Google Cloud Storage Data Transfer API

This API provides endpoints to schedule and manage jobs that ingest data from a tenant's external Google Cloud Storage (GCS) bucket into our system. It uses a background job processing system to handle one-time historical transfers and recurring (daily or hourly) transfers.

Base URL: https:/https://api.replen.it.replen.it/ingestion/google-cloud-storage

---

🔐 Authentication

🔒 This endpoint requires authentication

Required Header

x-replenit-auth-key: YOUR_BASE64_ACCESS_KEY

🗝️ How to Get Your Access Key

1. Log in to your Replenit admin dashboard

2. Navigate to the Data Health and Management

3. Generate a new access key or use an existing one

4. Copy the key (provided in base64 format)

---

Prerequisites: Tenant GCS Bucket Permissions 🔑

Before using any of these endpoints, the tenant whose data is being ingested must grant permissions to our service account on their GCS bucket. Without this, all transfer jobs will fail with an access denied error.

Please provide the following instructions to your tenant:

• Principal (Our Service Account): [e-mail] (Replace with given address by the customer support)

• Required Role: Storage Object Viewer (roles/storage.objectViewer)

Instructions for the Tenant:

1. In the Google Cloud Console, navigate to the source GCS bucket.

2. Go to the Permissions tab.

3. Click Grant Access.

4. In the New principals field, paste our service account email address.

5. In the Assign roles field, select Storage Object Viewer.

6. Click Save.

---

Request Body: TenantGoogleCloudStorageOptions

All POST endpoints use the following JSON object as the request body.

Field	Type	Description	Required
TenantId	GUID	The unique identifier for the tenant.	Yes
BucketName	string	The name of the tenant's GCS bucket.	Yes
HistoricalDataFolderPaths	Array of strings	A list of folder prefixes to scan for the historical transfer.	No
DataFolderPaths	Array of strings	A list of folder prefixes to scan for recurring (daily/hourly) transfers.	No
HourlyJobStartingMinuteUTC	integer	The minute of the hour (0-59) when the recurring hourly job should run.	No
DailyJobStartingHourUTC	integer	The hour of the day (0-23, UTC) when the recurring daily job should run.	No
DailyJobStartingMinuteUTC	integer	The minute of the hour (0-59) when the recurring daily job should run.	No

---

Endpoints

1. Register Historical Transfer 📂

Registers a one-time, fire-and-forget job to transfer historical data. This job uses the HistoricalDataFolderPaths from the request body.

• Endpoint: POST /register/historical-transfer

• Request Body Example:

  {
    "BucketName": "my_bucket_replenit",
    "HistoricalDataFolderPaths": [
      "customers/Historical/",
      "products/Historical/"
    ]
  }

• Success Response (200 OK):

  {
    "Success": true,
    "Message": "Historical transfer registered.",
    "Data": {
      "JobId": "12345"
    }
  }

• Error Response (500 Internal Server Error):

  {
    "Success": false,
    "Message": "Failed to register historical transfer.",
    "Data": {}
  }

---

2. Register Daily Transfer 📅

Registers or updates a recurring job that runs once per day at a specified UTC time. This job uses the DataFolderPaths from the request body. The unique ID for this job is daily-transfer/{TenantId}.

• Endpoint: POST /register/daily-transfer

• Request Body Example:

    "BucketName": "my_bucket_replenit",
    "DataFolderPaths": [
      "customers/",
      "products/"
    ],
    "DailyJobStartingHourUTC": 2,
    "DailyJobStartingMinuteUTC": 30
  }

• Success Response (200 OK):

  {
    "Success": true,
    "Message": "Daily transfer registered.",
    "Data": {
      "recurringJobId": "daily-transfer/3fa85f64-5717-4562-b3fc-2c963f66afa6"
    }
  }

---

3. Register Hourly Transfer 🕒

Registers or updates a recurring job that runs every hour at a specified minute. This job uses the DataFolderPaths from the request body. The unique ID for this job is hourly-transfer/{TenantId}.

• Endpoint: POST /register/hourly-transfer

• Request Body Example:

  {
    "BucketName": "my_bucket_replenit",
    "DataFolderPaths": [
      "transactions/"
    ],
    "HourlyJobStartingMinuteUTC": 15
  }

• Success Response (200 OK):

  {
    "Success": true,
    "Message": "Hourly transfer registered.",
    "Data": {
      "recurringJobId": "hourly-transfer/3fa85f64-5717-4562-b3fc-2c963f66afa6"
    }
  }

---

4. Unregister Recurring Transfer 🗑️

Removes a recurring (daily or hourly) job schedule. This does not affect any jobs that are already running. You must provide the recurringJobId returned from the registration endpoints.

• Endpoint: DELETE /unregister

• Query Parameter:

  • jobId (string, required): The unique identifier of the recurring job (e.g., daily-transfer/3fa85f64-5717-4562-b3fc-2c963f66afa6).

• Example URL:

  https://api.replen.it/ingestion/google-cloud-storage/unregister?jobId=daily-transfer/3fa85f64-5717-4562-b3fc-2c963f66afa6

• Success Response (200 OK):

  {
    "Success": true,
    "Message": "Transfer unregistered.",
    "Data": {}
  }

• Error Response (500 Internal Server Error):

  {
    "Success": false,
    "Message": "Failed to unregister transfer.",
    "Data": {}
  }