When you learn several data tools at once, it's easy to memorize syntax without understanding why each tool works the way it does. So I ran an experiment: take one dataset (Superstore), ask one business question, and solve it in all four platforms.

The code is on GitHub: CodingFactory

The setup

• Dataset: Superstore Sales (Orders: dates, customers, products, sales, profit)
• Platforms: MS SQL Server, Google BigQuery, Python (Pandas in Colab), Power BI (DAX measures)
• Rule: the same question and the same logic on every platform, with measures used for DAX wherever possible

Challenge 1: Highest spending customer per year

SQL and BigQuery lean on a window function. Python uses idxmax. DAX needs RANKX.

MS SQL

WITH YearlySpend AS (
    SELECT CustomerID, CustomerName, YEAR(OrderDate) AS OrderYear,
           SUM(Sales) AS TotalSpend
    FROM Sales
    GROUP BY CustomerID, CustomerName, YEAR(OrderDate)
),
Ranked AS (
    SELECT *, RANK() OVER (PARTITION BY OrderYear ORDER BY TotalSpend DESC) AS Rnk
    FROM YearlySpend
)
SELECT OrderYear, CustomerID, CustomerName, TotalSpend
FROM Ranked
WHERE Rnk = 1
ORDER BY OrderYear;

BigQuery

WITH yearly_spend AS (
  SELECT CustomerID, CustomerName,
         EXTRACT(YEAR FROM OrderDate) AS order_year,
         SUM(Sales) AS total_spend
  FROM `project.dataset.Sales`
  GROUP BY CustomerID, CustomerName, order_year
),
ranked AS (
  SELECT *, RANK() OVER (PARTITION BY order_year ORDER BY total_spend DESC) AS rnk
  FROM yearly_spend
)
SELECT order_year, CustomerID, CustomerName, total_spend
FROM ranked
WHERE rnk = 1
ORDER BY order_year;

Python

df['OrderYear'] = pd.to_datetime(df['OrderDate']).dt.year
yearly = (df.groupby(['OrderYear', 'CustomerID', 'CustomerName'])['Sales']
            .sum().reset_index(name='TotalSpend'))
top = yearly.loc[yearly.groupby('OrderYear')['TotalSpend'].idxmax()]

DAX

Total Sales = SUM(Sales[Sales])

Customer Rank in Year =
RANKX(
    ALL(Sales[CustomerName]),
    [Total Sales]
)

Put Year and Customer on a table visual and filter Customer Rank in Year = 1.

Challenge 2: Running total

Window frames in SQL, cumsum() in Pandas, and a filter that rebuilds the date range in DAX.

-- SQL Server and BigQuery (same syntax)
SELECT OrderDate, Sales,
       SUM(Sales) OVER (ORDER BY OrderDate
                        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS RunningTotal
FROM Sales;

df = df.sort_values('OrderDate')
df['RunningTotal'] = df['Sales'].cumsum()

Running Total =
CALCULATE(
    [Total Sales],
    FILTER(ALL('Date'[Date]), 'Date'[Date] <= MAX('Date'[Date]))
)

Challenge 3: Month-over-month growth %

SQL, BigQuery and Python all use the same idea: grab the previous row (LAG / shift). DAX uses time-intelligence functions instead, which only work with a proper Date table.

WITH MonthlySales AS (
    SELECT DATEFROMPARTS(YEAR(OrderDate), MONTH(OrderDate), 1) AS SalesMonth,
           SUM(Sales) AS TotalSales
    FROM Sales
    GROUP BY DATEFROMPARTS(YEAR(OrderDate), MONTH(OrderDate), 1)
)
SELECT SalesMonth, TotalSales,
       ROUND((TotalSales - LAG(TotalSales) OVER (ORDER BY SalesMonth)) * 100.0
             / NULLIF(LAG(TotalSales) OVER (ORDER BY SalesMonth), 0), 2) AS MoM_Growth_Pct
FROM MonthlySales;

m = df.groupby(df['OrderDate'].dt.to_period('M'))['Sales'].sum().reset_index()
m['MoM_Growth_Pct'] = (m['Sales'].pct_change() * 100).round(2)

Prev Month Sales = CALCULATE([Total Sales], PREVIOUSMONTH('Date'[Date]))

MoM Growth % = DIVIDE([Total Sales] - [Prev Month Sales], [Prev Month Sales]) * 100

Challenge 4: 3-month moving average (where DAX bit back)

In SQL, Pandas and BigQuery this is a one-liner:

AVG(TotalSales) OVER (ORDER BY SalesMonth ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)

monthly['Moving_Avg_3M'] = monthly['Sales'].rolling(window=3, min_periods=1).mean()

In DAX, my first attempt looked right, ran without errors, and returned the wrong numbers. The moving average was identical to the monthly total on every row. Two things caused it:

1. DATESINPERIOD returns a narrow, date-only table. It has none of your other columns.
2. The table visual already filters each row to one month, so unless you clear those filters, the 3-month window never opens.

The fix is to clear the existing date filters, build the 3-month list of months, then average a measure over them:

Moving Avg 3M =
VAR CurrentMonth = MAX('Date'[Date])
VAR Last3Dates   = DATESINPERIOD('Date'[Date], CurrentMonth, -3, MONTH)
VAR Months       = CALCULATETABLE(VALUES('Date'[MonthStart]), ALL('Date'), Last3Dates)
RETURN
    CALCULATE(AVERAGEX(Months, [Total Sales]), ALL('Date'))

Always sanity-check the output against your SQL or Python numbers. DAX won't throw an error for logic mistakes.

The rest of the series

What I learned

1. SQL and Python think in sequences. DAX thinks in filter context. Window functions and shift() move row by row through ordered data. DAX recalculates a measure for whatever filters surround it, which is why ALL(), VALUES() and CALCULATE keep showing up.

2. Columns group, measures aggregate. My cohort table collapsed into a single row because I'd built the grouping fields as measures. A measure has no rows to group by. Anything you want on the rows of a table should be a column, and the numbers you aggregate should be measures.

3. Some problems belong upstream. Market basket analysis is natural in SQL or Python and painful in DAX. Knowing which tool to reach for matters as much as knowing the syntax.

4. Simple questions converge. A profit margin by region looks almost identical on all four platforms. The differences only show up with time comparisons, row-to-row matching and context shifting.

Try it yourself

Pick one dataset and solve the same five questions in every tool you're learning. You'll understand each tool better than you would from isolated tutorials.

Full code and notes: github.com/shijithpulikkal/CodingFactory

If you found this useful, let me know which challenge you'd like me to break down next.

This post walks through how we pulled DoorDash Ads reporting data into BigQuery with Cloud Run Jobs, and how that single dataset now feeds both a Looker Studio dashboard and a custom app built in Base44. It includes the architecture, the key code, and the gotchas that cost us time, so you can skip them.

What we built

• A Python job on Cloud Run that creates, polls and downloads DoorDash Ads reports through the API
• Three BigQuery fact tables (ad group, keyword, product), plus a run log
• Daily scheduling with Cloud Scheduler and the API token in Secret Manager
• A Looker Studio dashboard and a Base44 app that both read the same tables

The DoorDash Ads reporting API in 60 seconds

DoorDash Ads exposes the same reports you see in Ads Manager through an API. Authentication is a single bearer token (Authorization: Bearer <token>) against https://openapi.doordash.com.

Reports are generated asynchronously:

1. POST /ads/api/v1/sp/reports/{recordType}/create returns a reportId
2. GET /ads/api/v1/sp/reports/download/{reportId} returns a status. Poll until it is COMPLETED
3. The completed response contains a URL to a CSV file

A few rules from the docs are worth knowing up front:

• Only use endpoints under /ads/api/v1/sp/reports/* for reporting. The other endpoints manage campaigns and modify live campaigns.
• Rate limits are roughly 10 reads per second and 1 write per second. Creating a report counts as a write.
• You can look back up to 3 years, but a single report covers at most 365 days.

Architecture

flowchart LR
  A["DoorDash Ads API<br/>/sp/reports"] -->|"create, poll, download CSV"| B["Cloud Run Jobs<br/>ADGROUP / KEYWORD / PRODUCT"]
  S["Secret Manager<br/>API token"] --> B
  C["Cloud Scheduler<br/>every morning"] --> B
  B --> D["BigQuery staging tables"]
  D -->|"delete + insert date window"| E["BigQuery fact tables<br/>sp_adgroup_daily<br/>sp_keyword_daily<br/>sp_product_daily"]
  E --> F["Looker Studio"]
  E --> G["Base44 app"]

Choosing the reports (and one naming trap)

The client's team already used three Sponsored Products reports in the Ads Manager custom report builder: Campaign, Product and Keyword. In the API they map like this:

The trap: the report called Campaign in the UI is actually ad group level. The CSV has an Ad Group Id column, and the API's CAMPAIGN record type is a different, campaign-level report. If you request CAMPAIGN, you won't get the file you expected.

Request bodies are small. This is all it takes to create a report:

def create_report(session, record_type, start, end):
    body = {
        "reportName": f"{record_type.lower()}_{start}_{end}",
        "startDate": f"{start} 00:00:00",
        "endDate": f"{end} 23:59:59",
        "campaignTypes": ["SPONSORED_PRODUCTS"],
    }
    r = session.post(
        f"{BASE_URL}/ads/api/v1/sp/reports/{record_type}/create", json=body
    )
    r.raise_for_status()
    return r.json()["reportId"]

One image, three Cloud Run Jobs

We wrote one Python container and deployed it three times as separate Cloud Run Jobs, each with a different REPORT_TYPE environment variable (ADGROUP, KEYWORD, PRODUCT). Separate jobs fail independently and can have different memory settings, which turned out to matter (see the gotchas).

Each run does the same thing for a date window:

1. Create the report, poll until it is ready, download the CSV
2. Clean and type the data, using the BigQuery table's own schema as the source of truth
3. Load it into a staging table
4. In one transaction, delete that date window from the final table and insert the staging rows

BEGIN TRANSACTION;
DELETE FROM `doordash_ads.sp_adgroup_daily`
WHERE event_date BETWEEN @s AND @e;
INSERT INTO `doordash_ads.sp_adgroup_daily` (...)
SELECT ... FROM `doordash_ads_staging.stg_sp_adgroup_daily`;
COMMIT TRANSACTION;

Replacing a whole window instead of merging row by row has two advantages. Re-running a day is always safe, and rows that disappear or change upstream don't linger in your tables.

Column handling is mostly mechanical. DoorDash headers such as New to Brand Sales (180 days) become new_to_brand_sales_180d with one small function:

def norm_col(name):
    s = name.strip().lower()
    s = s.replace("(180 days)", "180d").replace("(1 year)", "1y").replace("%", "pct")
    return re.sub(r"[^a-z0-9]+", "_", s).strip("_")

Values like $55.29 and 1.49% are converted to numbers before loading. Percentages are stored as fractions (0.0149). Dates must be ISO YYYY-MM-DD, and the job stops with a clear message if they aren't. Unknown new columns from DoorDash are dropped with a warning instead of breaking the load.

Every run also writes one row to a dd_report_runs table (window, report ID, rows loaded, status), which makes "did it work this morning?" a one-line query.

Scheduling around DoorDash's refresh cycle

DoorDash documents exactly when data settles, and it shapes the schedule:

• Marketing data for today is recomputed every 3 hours
• Yesterday is restated once a day, with reports recommended after 23:00 UTC
• All dates are UTC

So each daily run re-pulls a rolling window (the last 3 days, ending yesterday UTC) and replaces it. A day first appears with the latest numbers, then settles automatically on the following runs. We set the Cloud Scheduler time zone to America/Los_Angeles and run in the morning Pacific time, so the schedule follows daylight saving without any cron edits and the data is fresh when the client opens the dashboard.

Gotchas that cost us time

1. UTC days are not California days. Ads Manager's "By dates" option shows UTC. A UTC day runs from 4 PM to 4 PM Pacific in winter. Keep UTC dates in your tables so the dashboard matches Ads Manager, and don't relabel them as local days.

2. Orders don't add up across report levels. Campaign-level reports count an order once per checkout. Product, ad group and keyword reports count it once per promoted product in the checkout. Our product and ad group totals matched exactly on impressions, clicks, spend, sales and units, and differed only on orders. We stayed away from summing orders across levels.

3. Product IDs and GTINs. Use DD_SIC as the product key. The GTIN_14 field often holds several comma-separated values, so store it as a string and never join on it. Trim whitespace on names, because some ad group names had trailing spaces.

4. Keep the "Other Clicks" bucket. DoorDash groups interactions that can't be attributed to a product under "Other Clicks". You need those rows for product totals to reconcile with campaign totals.

5. Don't open raw CSVs in Excel. A file we re-saved in Excel came back with dd-mm-yyyy dates and thousands of blank rows. The raw API files use ISO dates. A strict date parser turns this from a silent bug into a loud error.

6. BigQuery's client library returns legacy type names. Our tables were created with INT64 and FLOAT64, but get_table().schema reports INTEGER and FLOAT. Our first production run crashed on that. If you derive types from the live schema, map the legacy names.

7. The keyword report is big. The default 512 MiB Cloud Run Job memory ran out when we backfilled two months of keywords in one report. We gave that job 4 GiB and backfilled in 14-day windows.

8. Read the 401 message. "Unauthorized Advertiser" means the API recognized the token but it isn't provisioned for your advertiser account. "Expired or invalid token" means the credential isn't an API token at all. The fix for the first one is on the client or DoorDash side, not in your code.

From tables to dashboards

The dashboards only need a handful of fields, all of them base metrics:

Ratios are never averaged. ROAS, CTR and NTB share are always calculated from summed base metrics, in a view or a calculated field:

SELECT
  event_date,
  SUM(spend)         AS spend,
  SUM(ad_attr_sales) AS revenue,
  SAFE_DIVIDE(SUM(ad_attr_sales), SUM(spend))                  AS roas,
  SUM(clicks)        AS clicks,
  SUM(ad_attr_units) AS quantity,
  SAFE_DIVIDE(SUM(new_to_brand_sales_180d), SUM(ad_attr_sales)) AS ntb_sales_pct
FROM `doordash_ads.sp_adgroup_daily`
GROUP BY event_date;

Looker Studio

The Looker Studio dashboard follows the same structure as the client's existing Instacart dashboard: KPIs at a glance, daily performance, campaign performance, a search term view built on the keyword table, and a product view built on the product table. Keeping the layout the same across networks means the client reads DoorDash and Instacart the same way.

Base44

We also built a custom app in Base44 that puts every retail media network in one place, with a tab per network and DoorDash as one of them. The DoorDash tab has date presets (last 7, 14, 30 and 90 days, last month, month to date, last 12 months, all time), KPI cards, a daily performance table and a campaign performance table. Each section shows which BigQuery table it reads from, which makes the numbers easy to audit.

Because the app and Looker Studio read the same tables and use the same ratio definitions, the numbers agree. However you connect an app to BigQuery, keep the credentials on the server side and give it a read-only service account.

Running costs and operations

Three small jobs that run for a few minutes a day cost very little. For operations, we watch two things: the dd_report_runs table for FAILED or EMPTY runs, and a Cloud Monitoring alert on failed Cloud Run job executions. The API token lives only in Secret Manager, and it is rotated by adding a new secret version, since the jobs read :latest.

What's next

• Sponsored Brands and Promotions campaigns through the same pipeline
• The Business reports (sales, competitive share, interest insights)
• Views for every dashboard section, so ratio logic lives in one place

Wrapping up

The DoorDash Ads API is friendly once you know its shape: asynchronous reports, UTC dates, one restatement a day, and a few naming quirks. Pull a rolling window, replace it atomically, calculate ratios from base metrics, and every dashboard built on top stays consistent.

If you're sorting out retail media reporting across several networks, we build these data pipelines and dashboards at Xtract Studio. Get in touch and we'll help.

I built a small pipeline that fixes this: three Cloud Run jobs pull campaign, keyword and product reports from the Instacart Ads API every morning, load them into BigQuery, and feed a Looker Studio dashboard and a Base44 app. This post walks through the architecture, the code patterns that matter, and the mistakes that cost me the most time.

The architecture

Cloud Scheduler (daily, Pacific time)
        |
Cloud Run Jobs (campaign / keyword / product loaders)
        |-- Secret Manager (shared OAuth tokens)
        |-- Instacart Ads API v3 (async reporting)
        |
BigQuery (one table per report type)
        |
Looker Studio dashboard  +  Base44 app

• Three independent Cloud Run Jobs, one per report type, so one failing report never blocks the others.
• Secret Manager stores the OAuth tokens, shared by all three jobs.
• Cloud Scheduler triggers each job once a day, staggered by 15 to 30 minutes.
• BigQuery holds one table per report, with a date column on every row so you get daily trends.

Step 1: Authentication is not client_credentials

Most ad APIs use a simple client-credentials flow. Instacart Ads API v3 uses OAuth2 authorization_code plus refresh_token:

1. In Ads Manager, open Applications and create an application with a redirect URI you control. The page doesn't need to exist. You only need to read the ?code= value off the address bar after authorizing.
2. Click Authorize. You are redirected to your URI with an authorization code, valid for 5 minutes.
3. Exchange that code once for an access token and a refresh token.

requests.post(f"{BASE}/oauth/token", json={
    "client_id": CLIENT_ID,
    "client_secret": CLIENT_SECRET,
    "redirect_uri": REDIRECT_URI,
    "code": auth_code,
    "grant_type": "authorization_code",
})

The access token lives for 2 hours. The refresh token lives for 30 days and rotates every time you use it: the old one is revoked as soon as a refresh succeeds. That one detail drives the whole design.

Store tokens in Secret Manager, not on disk

Cloud Run's filesystem is ephemeral, and a rotating refresh token that is lost or stale locks you out until you re-authorize by hand. So every run reads the latest token from Secret Manager, refreshes, and immediately writes the new token back as a new secret version.

def get_access_token():
    tokens = _load_tokens()                        # latest secret version
    tokens = refresh_access_token(tokens["refresh_token"])
    _save_tokens(tokens)                           # persist the rotated token
    return tokens["access_token"]

Because one Instacart application authorizes every report type, all three jobs share the same secret. The consequence is that two jobs must never refresh at the same moment, which is why the schedules are staggered.

Step 2: Async reporting in three calls

Reporting goes through one endpoint, POST /api/v3/charts/async/tables. You choose a source, a date range, and the columns you want:

body = {
    "source": "sponsored_product_keyword",
    "query": {
        "date_range": {"start_date": "2026-01-01", "end_date": "2026-01-31"},
        "series": {"columns": ["campaign.name", "keyword.keyword",
                               "event.date", "event.spend",
                               "event.attributed_sales_linear", "event.clicks"]},
    },
    "header_map": {"event.spend": "spend", "event.date": "date"},
}

The flow is: create the report, poll its status_url until the status is completed (it moves through pending, created, in_progress), then download the CSV from the location it returns. Two practical notes:

• event.date is a real column you must request. Leave it out and you get one aggregated row per keyword for the whole range, with no daily trend.
• header_map renames columns in the output CSV, so your BigQuery columns are clean (spend, not event.spend) without a pandas rename step.

The reports I use are sponsored_product (campaign level), sponsored_product_keyword and sponsored_product_product. The API also exposes display, video, promotions, audience segments and share of digital shelf sources.

Step 3: Make the load idempotent

Every run deletes the date range it is about to load, then inserts fresh rows. Re-running a day, retrying a failed job or backfilling history can never create duplicates.

DELETE FROM `project.dataset.table`
WHERE date BETWEEN @start_date AND @end_date

For large backfills, the script splits the range into chunks of up to 31 days automatically. One command handles both the January-to-today backfill and the daily run:

python loader.py 2026-01-01 2026-07-31     # backfill, auto-chunked
python loader.py                           # daily run

Step 4: Attributed sales keep changing, so re-pull a rolling window

This was the most important lesson. My first version pulled only "yesterday" each morning, and the numbers drifted. Attributed sales for a given day kept changing after I had already loaded it.

I could not find a documented attribution-finalization window in Instacart's API reference, so I did not guess one. Because the load is idempotent, the fix was simple: every daily run re-pulls the last 15 days instead of one.

end_date = date.today() - timedelta(days=1)
start_date = date.today() - timedelta(days=15)

Late-settling attribution now corrects itself on the next run. The trade-off is that each run does more work, so check your Cloud Run task timeout.

Step 5: Deploy as Cloud Run Jobs

Cloud Run Jobs suit this better than Services, because the work runs to completion and exits instead of waiting for HTTP requests.

gcloud builds submit --tag us-central1-docker.pkg.dev/PROJECT/instacart/keyword-loader
gcloud run jobs create instacart-keyword-loader \
  --image=us-central1-docker.pkg.dev/PROJECT/instacart/keyword-loader \
  --region=us-central1 \
  --set-env-vars=GCP_PROJECT=PROJECT,INSTACART_CLIENT_ID=...,INSTACART_CLIENT_SECRET=...
gcloud scheduler jobs create http instacart-keyword-daily \
  --schedule="0 8 * * *" --time-zone="America/Los_Angeles" \
  --uri="https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT/jobs/instacart-keyword-loader:run" \
  --http-method=POST --oauth-service-account-email=SERVICE_ACCOUNT --location=us-central1

Two tips: set the scheduler time zone to where your client works (Cloud Scheduler handles daylight saving for you), and paste long gcloud commands as a single line, because stray characters after a \ continuation broke mine more than once.

The service account needs Secret Manager access to the token secret (secretmanager.secretVersionManager) and BigQuery write access (bigquery.dataEditor).

At this volume (three short jobs a day, one secret, a few small tables) the whole thing sits inside Google Cloud's free tiers. Artifact Registry image storage is the only line worth watching, since every rebuild adds an image version.

Step 6: Looker Studio dashboards

Connect Looker Studio to the BigQuery tables with the native BigQuery connector. Two things to get right:

• Never aggregate ratios by summing them. If the table sums the raw roas column across rows, you get nonsense like a 120x ROAS. Create a calculated field instead: SUM(attributed_sales) / SUM(spend). The same goes for CTR and CPC.
• The keyword and product reports will not match. For the same day, the product report showed more spend and clicks than the keyword report. Keyword-level reporting only covers traffic tied to a keyword, while product-level reporting includes everything. Label each chart with its source so nobody compares them.

Set the report time zone to match your client's. Instacart's event.date values are in PST.

Step 7: The Base44 app with user Login & Password

Looker Studio works well for analysts, but I also wanted something that felt like a product rather than a report. So I built a custom dashboard in Base44 on top of the same BigQuery data. Base44 app reads from BigQuery using live Query direct connection

What the app includes:

• One page per retail media platform. The sidebar has a page for Instacart Ads, and the same shell holds pages for other retail media networks. This pipeline feeds the Instacart one.
• A KPI strip at the top: spend, ROAS, clicks, attributed units, revenue, new-to-brand (NTB) sales and NTB sales percentage. Revenue and quantity are friendly labels over the attributed sales and attributed units columns.
• Date range presets: last 7, 14, 30 and 90 days, last month, month to date, last 12 months, all time, plus a custom range. The whole page recalculates from the selected range.
• A daily performance table with in-cell bars, sortable columns and a grand total row.
• A campaign performance table with revenue, NTB revenue, spend, ROAS, quantity, CTR and NTB sales percentage per campaign.
• A source label on each section that names the BigQuery table behind it, so anyone looking at a number knows where it came from.

The ROAS lesson from the Looker Studio step applies here too. The grand total row calculates ROAS as total revenue divided by total spend, never as a sum of the daily ROAS values.

The two front ends read the same tables, so the numbers agree. Looker Studio stays the workbench for ad-hoc analysis, and the Base44 app is the cleaner view to hand to a client.

Mistakes that cost me time

1. Deploying the wrong script into the wrong folder. All three loaders had the same filename, and a campaign script ended up inside the product job's image. The job ran happily and wrote to the wrong table. Run grep -E '"source"|"bq_table"' on the file right before every build.
2. Schema drift with WRITE_APPEND. Adding new columns to a script breaks appends to an older table with Cannot add fields. Add the columns to the table first, or load into a new table.
3. Placeholder credentials in the job config. A job created with literal your_real_app_id fails with a 401 that looks like an auth bug.
4. Missing packages outside Docker. Running the script by hand in Cloud Shell needs pyarrow and google-cloud-secret-manager installed, on top of pandas.
5. Relative URLs. The API sometimes returns a relative path for the status and download links. A small _absolute_url() helper handles both forms.

What the API does not give you

The reporting sources expose spend, impressions, clicks, CTR, CPC, ROAS, attributed sales and units (linear and last-touch), and new-to-brand metrics. I found no order count and no conversion metric. Attributed units is the closest proxy, and a clicks-to-units rate is a derived approximation you can calculate yourself, as long as you label it that way.

Wrapping up

The result is a hands-off daily pipeline: authenticate once, then every report refreshes itself, corrects late attribution, and lands in BigQuery ready for dashboards and apps. The patterns here (token rotation in Secret Manager, idempotent range loads, rolling re-pulls) carry over to any ad platform with async reporting and delayed attribution.

Need a reporting pipeline for a retail media platform? I build these as a freelance data and analytics consultant at Xtract Studio.

Retail media reporting has a familiar problem: the numbers live inside the retailer's ad platform, and everyone ends up exporting CSVs by hand. A CPG brand I work with advertises on the Kroger Ad Platform and wanted daily campaign and product performance in dashboards, with no manual work. This post walks through how I built it: the API, the BigQuery model, the Cloud Run pipeline, the dashboards and the problems I hit on the way, including a very annoying intermittent 403.

TL;DR

• The Kroger Ad Platform (KAP) reporting API uses OAuth 2.0 client credentials against a custom authorization server, with one report endpoint per ad experience (PLA, carousel, TOA).
• A small Python Cloud Run Job, triggered daily by Cloud Scheduler, pulls two daily reports (campaign level and product level) and loads them into date-partitioned BigQuery tables.
• Loads are idempotent: each run replaces the dates it pulled, so retries and backfills are safe.
• Looker Studio and a Base44 app read from BigQuery. Rates such as CTR, CPC and ROAS are calculated from summed totals, never averaged.
• The biggest surprise was not the API itself but an intermittent CDN block on cloud-provider IP addresses.

Architecture

Cloud Scheduler (daily)
        |
        v
Cloud Run Job (Python) ---> Kroger Ad Platform reporting API
        |                    (OAuth 2.0 client credentials)
        v
BigQuery (date-partitioned tables)
        |
        +--> Looker Studio dashboards
        +--> Base44 app

Secret Manager holds the client ID and secret.
Cloud Monitoring alerts on failed runs.

1. Pick the right API

Kroger has more than one API. The public developer portal covers shopper-facing data such as products, store locations and carts. Ad reporting is a different system: the Kroger Ad Platform API, with its own base URL, authorization server and credentials. I lost time reading the wrong documentation first, so check this early.

For a reporting-only integration you receive:

• An OAuth client ID and secret for a sandbox and a production environment, delivered by secure email
• Access to a shared partner Postman workspace with a ready-made collection and environments
• Learning Center documentation for reports, metrics and dimensions

The sandbox is good for learning the shape of responses, but it may hold little real data. Expect to switch to production credentials for the real pipeline.

2. Authentication: use the custom authorization server

The API uses OAuth 2.0 client credentials with the scope offline_access e451.api.access. This cost me an afternoon: that scope does not exist on the identity provider's default authorization server. Every guessed token URL returned 400 Bad Request until I used the token URL of the custom server from the partner documentation, which contains an authorization server ID, in the form .../oauth2/<server-id>/v1/token.

If your client ID is recognised but every scope attempt fails, you are probably calling the wrong authorization server, not using wrong credentials.

curl -s -X POST "$TOKEN_URL" \
  -d grant_type=client_credentials \
  --data-urlencode "scope=offline_access e451.api.access" \
  -d client_id="$CLIENT_ID" \
  --data-urlencode client_secret="$CLIENT_SECRET"

Tokens last one hour. Client credentials has no refresh flow, so the pipeline simply requests a fresh token at the start of every run.

3. Validate in Postman first

Fork the partner collection and environment into your own workspace, put your credentials in the environment's current value (which stays local to you), select the environment, and run the "Get Access Token" request.

Two mistakes I made:

• Running requests with no environment selected, so variables like {{client_id}} were sent as literal text.
• Pasting credentials into places they should not go. Keep secrets out of screenshots and out of any AI assistant built into your API client.

4. The reporting endpoints

There is one endpoint per ad experience:

My client only runs PLA, so the pipeline uses the first one. A request body looks like this:

{
  "startDate": "2026-10-19",
  "endDate": "2026-10-19",
  "advertiserIds": [12345],
  "filters": [],
  "dimensions": ["daily_date", "campaign_id", "campaign_name"],
  "metrics": ["impressions", "clicks", "cost"],
  "pagination": {"offset": 0, "size": 10000},
  "sort": []
}

Things worth knowing:

• The default page size is 250 rows and the maximum is 10,000. Always send size and page with offset. An offset beyond the data returns 200 with an empty data array, which gives a clean stop condition for the loop.
• Filters are only needed for multi-brand campaigns. My first request failed because the sample filter used placeholder field names. The error message is useful: it lists every valid metric and dimension for the endpoint. Trust that list over the documentation page, because some names differ (for example halo_roas rather than clicked_roas).
• Pull the previous day, no earlier than 3 AM Eastern. The documentation recommends this buffer, and says conversions are attributed to the day they occur, so historical rows do not change.

5. Choosing metrics and dimensions

The dashboard needed these fields:

I made two requests: a campaign-level report (which includes goal, since it is a campaign attribute) and a product-level report (which adds entity_id and entity_name). A useful sanity check: I summed the product rows per campaign and they matched the campaign report exactly for clicks, cost, impressions and transactions. It is worth building that reconciliation into any pipeline.

Quirks to handle in code:

• goal arrives as a string such as "1" even though it is declared numeric.
• entity_id arrives as a number although the response header calls it text.
• Product names can contain invisible soft-hyphen characters. Strip them, or joins and filters fail silently.
• Ratios come back as fractions (0.058 means 5.8%).
• "Viewed" metrics use impression-or-click attribution, so they already include the clicked ones. Do not add the two together.

6. BigQuery design

Two tables, partitioned by daily_date and clustered by campaign (and product for the second):

CREATE TABLE IF NOT EXISTS `my-project.kroger_ads.kap_pla_campaign_daily` (
  daily_date DATE, campaign_id INT64, campaign_name STRING,
  impressions INT64, clicks INT64, cost FLOAT64,
  -- ...remaining metrics...
  goal FLOAT64, ingested_at TIMESTAMP
) PARTITION BY daily_date CLUSTER BY campaign_id;

Loads are idempotent: each run deletes the rows for the dates it is about to load, then inserts the new rows. Two safety rules make this robust:

1. Never delete when the API returns zero rows. An empty or blocked response should not wipe good data.
2. Re-pull a short lookback window (I use 3 days) on every run, so a missed day heals itself on the next successful run.

7. The Cloud Run Job

The job is about 200 lines of Python using requests and google-cloud-bigquery. The core steps are: get a token, fetch each report with pagination, cast the types, and replace the date range in BigQuery.

client.query(
    f"DELETE FROM `{table_id}` WHERE daily_date BETWEEN @s AND @e",
    job_config=bigquery.QueryJobConfig(query_parameters=[
        bigquery.ScalarQueryParameter("s", "DATE", start),
        bigquery.ScalarQueryParameter("e", "DATE", end),
    ]),
).result()
client.load_table_from_json(rows, table_id, job_config=load_config).result()

Deployment notes:

• Store the client ID and secret in Secret Manager and mount them with --set-secrets. Never put them in code or in the container image.
• Use a dedicated service account with BigQuery Data Editor, BigQuery Job User and Secret Accessor, and nothing more.
• Deploy with gcloud run jobs deploy ... --source ., then trigger it with Cloud Scheduler. Schedule it comfortably after the 3 AM Eastern buffer, for example 5 AM Eastern.
• Exit with a non-zero code on failure so Cloud Run marks the execution as failed.

8. When the CDN says no

After deployment the job started failing on some days with 403 Access Denied. It was not a JSON error from the API but an HTML error page from the CDN, with a reference number. The token call always worked, Postman worked, and some runs worked too.

How I approached it:

1. An HTML 403 means the request was blocked at the edge, before it reached the API.
2. Intermittent failures suggested a decision per source IP. Cloud Run assigns outbound IPs from a shared, dynamic pool.
3. I logged the UTC time, the reference number and the outbound IP for every blocked request, so there was real evidence to share with the API provider.
4. I added retries with backoff, but only for HTML 403 responses. Genuine permission errors are JSON and should fail fast.

edge_block = r.status_code == 403 and r.text.lstrip().startswith("<")

I raised a support ticket with the references and timestamps. Their team pointed to IP reputation scoring on cloud-provider traffic, explained that a blanket exception for cloud ranges was not possible, and suggested trying a different region. Update: after moving the job to a different region, a full week of scheduled runs finished with no blocks. That is a small sample, so I am keeping the retries and the logging in place. If blocks continue, the dependable fix is a static outbound IP through Cloud NAT, because a single dedicated address can be allowlisted. It adds some setup and a small monthly cost.

The lesson: a retry that "fixed it" once hid the problem for days. Make intermittent failures visible.

9. Monitoring

• A log-based alert on the job's failure message that emails me
• An alert on Cloud Scheduler errors, in case a job never starts
• A small dashboard of executions by result
• A separate freshness check, because a run that succeeds but loads zero rows is not an error to Cloud Run

10. Dashboards in Looker Studio

Connect Looker Studio to the BigQuery tables. The key rule is to store raw sums and calculate rates in Looker Studio:

CTR           = SUM(clicks) / SUM(impressions)
CPC           = SUM(cost) / SUM(clicks)
CPM           = SUM(cost) / SUM(impressions) * 1000
Clicked ROAS  = SUM(halo_revenue) / SUM(cost)
Viewed ROAS   = SUM(halo_exposed_revenue) / SUM(cost)

Averaging the stored daily ratios gives wrong totals as soon as someone filters by campaign or date range.

A layout that works well for PLA reporting :

• Overview: scorecards for spend, clicks, revenue and ROAS, plus a daily trend
• Campaigns: a table with ROAS against the campaign's ROAS goal
• Products: a leaderboard by revenue and ROAS, filterable by campaign

11. Putting the data in a Base44 app

For the client-facing view I used Base44 with Individual Login system, an AI app builder, and connected it to BigQuery through its native BigQuery connector. The app runs live queries against the same two tables the Looker Studio dashboards use, so there is no export or sync step in between: when the Cloud Run job loads a new day, the app shows it.

The Kroger page of the app has these parts:

• Layout: a sidebar with one page per retail media platform, and a KPI strip at the top of each page (cost, ROAS, clicks, transactions, units and revenue).
• Daily performance: a day-by-day table with revenue and cost bars, ROAS, transactions and units, read from the campaign table.
• Campaign performance: revenue, cost, ROAS, transactions, units, clicks, impressions and CTR per campaign, from the same table.
• Campaign and product performance: a drill-down by product, read from the product table.
• Date range: presets such as the last 7, 14 and 30 days and month to date, plus a custom range, applied as a filter on daily_date.

Because every view runs a query, two habits keep it fast and cheap: filter on the partition column daily_date in every query, and select only the columns a view needs. Partitioned, clustered tables keep the data scanned small, even when someone picks "All time".

One design principle applies however you connect: never put BigQuery credentials in front-end code. Give the app a read-only path to the data, such as a service account limited to the reporting dataset behind a backend function or a small API, and keep write access for the pipeline's service account only.

Lessons learned

• Confirm which API and which authorization server you are using before debugging anything else.
• Use the API's own validation errors as the source of truth for field names.
• Reconcile your detailed report against the summary report to catch data problems early.
• Make loads idempotent and add a lookback window, so failures do not leave permanent gaps.
• Log evidence for intermittent failures, and do not let retries hide them.
• Calculate ratios in the BI layer, from sums.

What's next

• Add carousel and TOA reports if the client starts using them
• Snapshot the pacing ratio daily, since the API only returns the latest value
• Add automated data quality checks, including the campaign-versus-product reconciliation

If you are building something similar and want to compare notes, feel free to get in touch.