Batch queries (Preview)

Perform multiple data source queries using a single batch request.

Please note that this functionality is in Preview and might be changed without prior notice. Please let us know of any issues you might encounter to help us improve this solution.

Overview

Query batches are intended for situations when the number of queries is very high, and both requesting and polling each query individually become impractical.

Starting a query batch

Send one request containing multiple queries. They are processed in the order they appear in the list.

POST /enterprise/v2/query/batch
Authorization: Bearer <your api key>
Content-Type: application/json

{
    "queries": [
        {
            "ds_id": "AW",
            "ds_accounts": "123456789",
            "start_date": "2022-09-01",
            "end_date": "2022-09-03",
            "fields": "date,impressions,cost"
        },
        {
            "ds_id": "AW",
            "ds_accounts": "234567890",
            "start_date": "2022-09-01",
            "end_date": "2022-09-03",
            "fields": "date,clicks,views"
        }
    ]
}

You can use split_accounts_policy to duplicate the same query for each account ID mentioned in the query, to run them separately from each other.

Polling for status

Using the returned batch_id, you can then poll for the status of all the queries in the batch. They are returned in the same order they were created.

GET /enterprise/v2/query/batch/qrb_3feb5ff5c71e
Retrieving results

As queries finish, you can retrieve query results from the returned result URLs. Please remember to provide authentication credentials when accessing URLs.

"results_info": {
  "total_rows": 47,
  "results_url": "https://.../results?schedule_id=qrb_3feb5ff5c71e_query_1"
}
Handling errors

Any errors that might occur while running the query are reported for each query separately.

"error_info": {
  "status": 500,
  "code": "QUERY_ACCOUNT_UNAVAILABLE",
  "message": "Account is not available",
  "description": "Google Ads account \"123456789\" is no longer available."
}

Start batch run

POST https://api.supermetrics.com/enterprise/v2/query/batch
  • queries object[] | string[]
    Required. List of objects containing parameters for a get data request, or a list of query URL strings. Both Supermetrics API version 1 and version 2 URLs are accepted.
  • description string
    Optional. Custom description for the batch run.
  • split_accounts_policy string
    Optional. Whether each provided query should be split into separate queries based on account IDs. Supported values are "all" to split each account ID into its own query, and "none" to do no splitting at all. Defaults to "none". If queries refer to a team list, splitting uses account IDs inside the list.
  • split_dates_policy string
    Optional. Whether each provided query should be split into separate queries based on query date range. Supported values are "day" to split each day into its own query, and "none" to do no splitting at all. Defaults to "none".
  • timeout_minutes int
    Optional. Total number of minutes the batch is allowed to run. Minimum value of 5 minutes is enforced. Maximum allowed value is 60 minutes. Defaults to 30 minutes.
  • query_concurrency int
    Optional. Number of queries to run concurrently inside the batch. Minimum value of 1 is enforced. Maximum allowed value is 50. Defaults to 5. Please note that actual query processing concurrency still follows your license limits.
  • query_output_format string
    Optional. Output format for all query results in the batch. Defaults to json. See output formats for all supported formats.
  • add_exclude_account_errors bool
    Optional. Add settings into every query to exclude account access related errors. Defaults to false.
  • add_clean_data_types bool
    Optional. Add settings into every query to clean numeric data types and date values. Defaults to false.
Request
POST https://api.supermetrics.com/enterprise/v2/query/batch
Authorization: Bearer <your api key>
Content-Type: application/json

{
    "description": "My AW run at...",
    "queries": [
        {
            "ds_id": "AW",
            "ds_accounts": "123456789",
            "start_date": "2022-09-01",
            "end_date": "2022-09-03",
            "fields": "date,impressions,cost",
            "settings": {
                "no_headers": true
            }
        }
    ]
}
Response

HTTP 201 Created
Response from Get batch information

Errors

BATCH_CREATE_FAILED, BATCH_QUERIES_LIMIT_EXCEEDED, BATCH_QUERY_CONCURRENCY_LIMIT_EXCEEDED, BATCH_QUERY_ITEM_SPLIT_FAILED, BATCH_QUERY_ITEM_TYPE_INVALID, BATCH_QUERY_ITEM_VALUE_INVALID, BATCH_QUERY_OUTPUT_FORMAT_INVALID, BATCH_SEARCH_FAILED, BATCH_SPLIT_FAILED, BATCH_TIMEOUT_MINUTES_LIMIT_EXCEEDED, BATCH_UPDATE_FAILED


Get batch information

GET https://api.supermetrics.com/enterprise/v2/query/batch/:batch_id
  • batch_id string
    Required. Supermetrics query batch ID.
Response
PropertyTypeDescription
meta > request_idstring API request ID
dataquery_batch Query batch object
HTTP 200 OK
{
    "meta": {
        "request_id": "WnjTGeKqOXTnkZQMbZCfVMdmswv2wAzx"
    },
    "data": {
        "@type": "query_batch",
        "batch_id": "qrb_3feb5ff5c71e",
        "status_code": "RUNNING",
        "schedule_time": "2022-09-30T13:25:17+00:00",
        "start_time": "2022-09-30T13:25:20+00:00",
        "end_time": null,
        "cancel_time": null,
        "description": "My AW run at...",
        "split_accounts_policy": "none",
        "split_dates_policy": "none",
        "timeout_minutes": 60,
        "query_amount": 3,
        "query_concurrency": 50,
        "query_output_format": "json",
        "add_exclude_account_errors": false,
        "add_clean_data_types": false,
        "links": {
            "status": {
                "href": "https://.../query/batch/qrb_3feb5ff5c71e/status"
            }
        }
    }
}
Errors

BATCH_NOT_FOUND, BATCH_SEARCH_FAILED


Get batch status

GET https://api.supermetrics.com/enterprise/v2/query/batch/:batch_id/status
  • batch_id string
    Required. Supermetrics query batch ID.
  • offset int
    Optional. Pagination offset for the number of queries to return. Defaults to 0.
  • limit int
    Optional. Maximum number of queries to return. Defaults to 100, maximum of 1000 is enforced.
Response
PropertyTypeDescription
meta > request_idstring API request ID
meta > status_codestring Batch status code from query_batch
meta > progressbatch_progress Batch query progress information
meta > paginatepaginate Pagination information
databatch_query[] List of query objects in the batch
HTTP 200 OK
{
    "meta": {
        "request_id": "WnjTGeKqOXTnkZQMbZCfVMdmswv2wAzx",
        "status_code": "RUNNING",
        "progress": {
            "queued": 0,
            "running": 1,
            "success": 1,
            "failure": 1
        },
        "paginate": {
            "offset": 0,
            "limit": 100,
            "total": 3
        }
    },
    "data": [
        {
            "@type": "batch_query",
            "schedule_id": "qrb_3feb5ff5c71e_query_1_acc_1",
            "status_code": "RUNNING",
            "query_start_time": "2022-09-30T13:25:17+00:00",
            "query_end_time": null,
            "split_info": {
                "ds_accounts": [
                    "account1"
                ],
                "start_date": null,
                "end_date": null
            },
            "error_info": null,
            "result_info": null
        },
        {
            "@type": "batch_query",
            "schedule_id": "qrb_3feb5ff5c71e_query_1_acc_2",
            "status_code": "SUCCESS",
            "query_start_time": "2022-09-30T13:25:17+00:00",
            "query_end_time": "2022-09-30T13:26:45+00:00",
            "split_info": {
                "ds_accounts": [
                    "account2"
                ],
                "start_date": null,
                "end_date": null
            },
            "error_info": null,
            "result_info": {
                "total_rows": 47,
                "results_url": "https://..."
            }
        },
        {
            "@type": "batch_query",
            "schedule_id": "qrb_3feb5ff5c71e_query_2_acc_1",
            "status_code": "FAILURE",
            "query_start_time": "2022-09-30T13:26:19+00:00",
            "query_end_time": "2022-09-30T13:27:45+00:00",
            "split_info": {
                "ds_accounts": [
                    "account3"
                ],
                "start_date": null,
                "end_date": null
            },
            "error_info": {
                "status": 500,
                "code": "QUERY_ACCOUNT_UNAVAILABLE",
                "message": "Account is not available",
                "description": "Google Ads account \"account3\" is no longer available."
            },
            "result_info": null
        }
    ]
}
Errors

BATCH_NOT_FOUND, BATCH_SEARCH_FAILED, BATCH_STATUS_NOT_FOUND


Cancel batch run

Cancelling a query batch run will stop it from starting new queries and from waiting responses of currently running queries. Already collected query results will remain available. For more details, see Restrictions.

DELETE https://api.supermetrics.com/enterprise/v2/query/batch/:batch_id
  • batch_id string
    Required. Supermetrics query batch ID.
Response

HTTP 200 OK
Response from Get batch information

Errors

BATCH_CANCEL_FAILED, BATCH_NOT_FOUND, BATCH_STATUS_NOT_FOUND, BATCH_SEARCH_FAILED, BATCH_UPDATE_FAILED


Restrictions

Please contact us to raise limits or to address other restrictions.

Usage limits
  • Maximum allowed runtime for a batch run is 60 minutes.

  • Number of concurrently running batches is limited to two batches at-a-time. Any additional batch runs will be placed in a queue, and processed in the order they were received.

  • Number of individual queries in a single batch, after splitting is applied, is limited to maximum of 100 queries.

  • Each performed query is still affected by the same license limitations that would be used when running queries individually, such as query concurrency.

Splitting
Data retention
  • Query batch information is available for 90 days from the moment the batch was created.

  • Query batch status is available up to one hour after the batch run has stopped.

Object reference

query_batch
PropertyTypeDescription
batch_idstring Supermetrics query batch ID
status_codestring
Current status code of the batch run
QUEUEDBatch is queued for execution
RUNNINGBatch is running
STOPPEDBatch is stopped
schedule_timestring ISO 8601 datetime for when the batch was sent to queue
start_timestring ISO 8601 datetime for when the batch started running
end_timestring ISO 8601 datetime for when the batch stopped running
cancel_timestring ISO 8601 datetime for when the batch was cancelled
descriptionstring Custom description for the batch run
split_accounts_policystring How each query should be split based on its accounts
split_dates_policystring How each query should be split based on its date range
timeout_minutesint Total number of minutes the batch is allowed to run
query_amountint Total number of queries in the batch run after splitting them
query_concurrencyint Number of queries to run concurrently inside the batch
query_output_formatstring Output format for all query results in the batch
add_exclude_account_errorsbool Whether to add query settings which exclude account access related errors
add_clean_data_typesbool Whether to add query settings which clean numeric data types and date values
links > statusresource_url Resource URL to get batch status

batch_query
PropertyTypeDescription
schedule_idstring Schedule ID for the query request. Generated from the Supermetrics batch ID, sequential query number, and splitting logic.
status_codestring
Current status code of the query
QUEUEDQuery is waiting to be started
RUNNINGQuery is running
SUCCESSQuery succeeded
FAILUREQuery failed
query_start_timestring ISO 8601 datetime for when query started to run
query_end_timestring ISO 8601 datetime for when query stopped running
split_infobatch_query_split Additional splitting information, if any
error_infoerror Query error object, if any
result_infobatch_query_result Query result object, if any

batch_query_split
PropertyTypeDescription
ds_accountsstring[] If query was split by accounts, list of account IDs in this query
start_datestring If query was split by dates, start date for this query in YYYY-MM-DD format
end_datestring If query was split by dates, end date for this query in YYYY-MM-DD format

batch_query_result
PropertyTypeDescription
total_rowsint Number of rows in the query results
results_urlstring Full URL to get the query results with the results endpoint. URL will not contain pagination parameters or any authentication credentials.

batch_progress
PropertyTypeDescription
queuedint Number of queries waiting for to be run
runningint Number of queries currently running
successint Number of queries succeeded
failureint Number of queries failed

paginate
PropertyTypeDescription
offsetint Used pagination offset
limitint Used pagination limit
totalint Total number of items available

error
PropertyTypeDescription
statusint Numeric HTTP response status code
codestring Error code for the error. See error codes for more.
messagestring Short localized error message or the error code when localization is not available
descriptionstring Detailed localized error description, if available

resource_url
PropertyTypeDescription
hrefstring Full URL to the resource

Errors

StatusErrorDescription
500BATCH_CANCEL_FAILEDFailed to cancel query batch run due to an internal error. Please try again.
500BATCH_CREATE_FAILEDFailed to store query batch information due to an internal error. Please try again.
404BATCH_NOT_FOUNDRequested query batch ID was not found or you don't have access to it.
400BATCH_QUERIES_LIMIT_EXCEEDEDAmount of queries after splitting exceeds allowed limit. See Restrictions.
400BATCH_QUERY_CONCURRENCY_LIMIT_EXCEEDEDRequested query concurrency exceeds allowed limit. See Restrictions.
400BATCH_QUERY_ITEM_SPLIT_FAILEDFailed to apply requested splitting logic to a query.
400BATCH_QUERY_ITEM_TYPE_INVALIDOne of the query items is not an object nor a string.
400BATCH_QUERY_ITEM_VALUE_INVALIDOne of the query items was not recognized as a valid query. Please check that enough query parameters are given, and that all provided strings are valid URLs.
400BATCH_QUERY_OUTPUT_FORMAT_INVALIDRequested query output format is not valid. See output formats for all supported formats.
500BATCH_SEARCH_FAILEDFailed to find query batch information due to an internal error. Please try again.
500BATCH_SPLIT_FAILEDFailed to split queries for query batch run due to an internal error. Please try again.
404BATCH_STATUS_NOT_FOUNDStatus for requested query batch run was not found. Data for the query batch run might have expired.
400BATCH_TIMEOUT_MINUTES_LIMIT_EXCEEDEDRequested maximum duration for a batch run exceeds the allowed limit. See Restrictions.
500BATCH_UPDATE_FAILEDFailed to update query batch information due to an internal error. Please try again.
Last updated 2023-08-16 UTC.