Most APIs hand you records 100 at a time and a cursor for the next batch. fetchAllPages(fetchPage, opts) follows that cursor for you. You supply fetchPage(cursor) plus two small extractors (opts.getItems, opts.getNextCursor), and it loops until there's no next cursor, it hits maxPages, or the time budget runs out. It returns { items, pages, done, nextCursor }, so when done is false you can save nextCursor and resume on the next run.
When to use
- APIs with
nextPageToken,nextcursors,offset/pagenumbers orLink: rel="next"headers - Nightly imports that might outgrow one 6-minute execution
- Anywhere you've written the same
while (cursor)loop twice
How to use this snippet
/**
* Follow an API's next-page cursor, with a page cap and a time budget.
*
* @param {function(*): *} fetchPage called with the cursor (null first), returns the parsed page
* @param {Object} opts
* getItems(page) → array of records on this page
* getNextCursor(page) → next cursor, or null/'' when finished
* startCursor (default null) resume point from a checkpoint
* maxPages (default 50)
* timeBudgetMs (default 4.5 minutes)
* @return {{items: Array, pages: number, done: boolean, nextCursor: *}}
*/
function fetchAllPages(fetchPage, opts) {
opts = opts || {};
var maxPages = opts.maxPages || 50;
var budget = opts.timeBudgetMs || 4.5 * 60 * 1000;
var started = Date.now();
var cursor = opts.startCursor == null ? null : opts.startCursor;
var items = [], pages = 0;
while (pages < maxPages && Date.now() - started < budget) {
var page = fetchPage(cursor);
pages++;
items = items.concat(opts.getItems(page) || []);
cursor = opts.getNextCursor(page);
if (!cursor) return { items: items, pages: pages, done: true, nextCursor: null };
}
return { items: items, pages: pages, done: false, nextCursor: cursor };
}
Example
function importContacts() {
var props = PropertiesService.getScriptProperties();
var res = fetchAllPages(function (cursor) {
var url = 'https://api.example.com/v1/contacts?limit=100' +
(cursor ? '&page_token=' + encodeURIComponent(cursor) : '');
var r = UrlFetchApp.fetch(url, {
headers: { Authorization: 'Bearer ' + props.getProperty('API_TOKEN') },
muteHttpExceptions: true
});
if (r.getResponseCode() !== 200) throw new Error('HTTP ' + r.getResponseCode());
return JSON.parse(r.getContentText());
}, {
getItems: function (p) { return p.contacts; },
getNextCursor: function (p) { return p.nextPageToken; },
startCursor: props.getProperty('CONTACTS_CURSOR'),
maxPages: 40
});
writeContacts_(res.items); // your writer
if (res.done) props.deleteProperty('CONTACTS_CURSOR');
else props.setProperty('CONTACTS_CURSOR', res.nextCursor);
}
Tips:
- Need retries on 429/5xx? Call retryUrlFetch inside
fetchPageinstead ofUrlFetchApp.fetch. Link: <...>; rel="next"APIs: readr.getHeaders()(response headers are available), return{ body, link }fromfetchPage, and parse the URL ingetNextCursor.- Offset APIs: make the cursor the next offset, and return
nullwhen a page comes back short. - Write each run's items before saving the cursor, so a crash never skips records.
Tip: NitroGAS Co-Pilot can wrap an existing single-page fetch in fetchAllPages and add the cursor checkpoint. Paste the function and one sample API response.
Happy Coding!
