🔒 Google Cloud Reference · Built from a 403

IEXDG × Google Cloud
API Reference

The full operating reference for every Google Cloud and Workspace surface IEXDG touches. Born May 26, 2026 from one OAuth consent screen 403, written to make sure that error never re-occurs.

Prepared by Dove Web Consulting · for the IEXDG operations bench · v1.0 · 2026-05-26

🏠
2
GCP projects
🔑
2 of 5
live tokens
19
workspace scopes
🚂
11
VM cron jobs

What's in this doc

The 403 that triggered this doc

On 2026-05-26, an attempt to authenticate dovewebconsulting@gmail.com against the IEXDG OAuth client failed with Google's terse Error 403: org_internal. That single error revealed an architecture decision that had been silently bleeding tokens, capping accounts, and queueing future surprises across every IEXDG automation. This document captures the root cause, the fix, and the broader posture that prevents the next failure of the same class.

🔴 What you saw

"Access blocked: IEXDG Automation can only be used within its organization."

Browser landed on Google's consent screen, picked the dovewebconsulting account, then refused authorization before any scope check. Error code: Error 403: org_internal.

🧳 Why it happened

The OAuth client lives in the iexdg-automation GCP project. That project's consent screen User Type is set to Internal, which by Google's own design only accepts user accounts that are members of the same Workspace organization (iexdg.com in this case). The dovewebconsulting Gmail account is a consumer identity, not a Workspace member, so Google blocked it at the door.

Fix · 2 minutes · one-way safe

Sign into Google Cloud Console as robert.dove@iexdg.com, open console.cloud.google.com/apis/credentials/consent?project=iexdg-automation, switch User Type from Internal to External, leave Publishing status on Testing, add dovewebconsulting@gmail.com as a test user, save. Re-fire the OAuth init script. Browser will warn "Google hasn't verified this app" on the unverified-app interstitial. Click Advanced, then Go to iexdg-automation (unsafe). That is your own app.

The tax that comes with the fix · 7 days

An External-Testing OAuth app issues refresh tokens that expire in 7 days unless the app requests only the basic openid/email/profile scopes. Every IEXDG token that holds Gmail, Drive, Calendar, or Sheets scopes under External-Testing will silently die every 7 days. Either build a refresh cron that exercises each token weekly, or push the app through verification to Published (multi-week review for Sensitive scopes, multi-month plus CASA assessment for Restricted scopes).

What is actually running

A precise inventory of the GCP projects, OAuth clients, tokens, and services in use across the IEXDG stack, as of 2026-05-26. Every entry is verified against disk or live API, not inferred from memory.

GCP projects in scope

🌐 drdnicole-youtube-manager

Active

Holds the IEXDG VM (iexdg-nexus-vm e2-small in us-east4-b), the OAuth client used for dovewebconsulting@gmail.com tokens, GA4 + GTM enabled, BigQuery auto-enabled by GA4. This is where the brain runs and where the proposed GCS canon bucket should live.

🌐 iexdg-automation

External-Testing

Separate GCP project that owns the OAuth client used to mint the iexdg_robertdove_token.pickle (the omnibus Workspace token for robert.dove@iexdg.com). Was Internal-only until the 2026-05-26 fix moved it to External-Testing. Subject to the 7-day refresh-token tax above.

OAuth tokens on disk

TokenIdentityOAuth projectScopesState
dwc_gmail_token.pickle dovewebconsulting@gmail.com bathroom-bidders 4 Gmail scopes live
iexdg_robertdove_token.pickle robert.dove@iexdg.com iexdg-automation 19 (Gmail, Drive, Sheets, Calendar, GA4, GTM, Forms, Apps Script, Admin Directory) live
gmail_token.pickle dovewebconsulting@gmail.com drdnicole-youtube-manager 15 (Gmail + Drive + Sheets + YouTube + Forms + Script) revoked 2026-05-17
TOOLS/gmail_token.pickle (mirror of above) drdnicole-youtube-manager same as above revoked 2026-05-17
bb_analytics_token.pickle (BB cross-tenant residency) drdnicole-youtube-manager GA4 + GTM read revoked 2026-05-02
iexdg_apps_script_token.pickle (stale, unused) drdnicole-youtube-manager Script + Drive stale 2026-04-16

P0 · 3 revoked tokens block 25+ scripts

gmail_token.pickle revocation on 2026-05-17 took out every script targeting that path: mailers/*, gmail/send_apr29_*.py, automations/inbox_scan.py, every legacy IEXDG mailer pre-dating the dwc-canonical split. Reauth runbook is source/scripts/auth/gmail_reauth.py against client_secret_iexdg_oauth_v2.json, then scp the fresh pickle to /opt/iexdg-mcp/secrets/ and /opt/iexdg-mcp/TOOLS/ on the VM, then restart iexdg-api + iexdg-mcp + iexdg-mcp-oauth services.

VM runtime · brain.iexdg.com

running

caddy.service

Reverse proxy fronting brain.iexdg.com and the /capture PWA. TLS via Let's Encrypt.

:80 :443 public
running

iexdg-api.service

FastAPI Dashboard API. Feeds brain.iexdg.com. Carries G7 unified_state and G21 m_registry endpoints as of 2026-05-25.

127.0.0.1:8000
running

iexdg-mcp.service

FastMCP streamable-http content pipeline. Multiple vhosts. Bearer-protected via Caddy.

127.0.0.1:8766
running

iexdg-mcp-oauth.service

fastmcp v3 GoogleProvider OAuth-protected MCP server. Used by Dr. DNicole's Claude Desktop.

127.0.0.1:8765
running

cron.service

11 scheduled jobs covering pipeline pulse, inbox scan, deploy verification, audio-to-capture, war room watch, content drop, oauth health, dashboard splices, OCC splice, cc-bus heartbeat.

user: iexdg
non-canonical path

iexdg_api.py reads /var/log/...

iexdg_api.py looks for /var/log/iexdg-mcp/gmail_token.pickle. The real token lives at /opt/iexdg-mcp/secrets/. Path drift; failing silently.

drift item

The 2-project topology

Two GCP projects. One operator identity bridging them. Five OAuth pickles, three already dead. The picture below is what 18 months of incremental glue actually looks like on disk and in the Cloud Console, and why the 2026-05-26 403 was inevitable the moment the second project went Internal-only.

🏢
drdnicole-youtube-manager
VM home · consumer OAuth client
💾Compute
📂Drive
Gmail
📊Sheets
📈GA4
🏷GTM
📝Forms
GCS
YouTube
OPERATOR
IDENTITY dovewebconsulting
@gmail.com
🏢
iexdg-automation
Workspace OAuth client · robert.dove@iexdg.com
Gmail
📂Drive
📊Sheets
📅Calendar
📈GA4
🏷GTM
📝Forms
AppsScr
🛡Admin
Token pickles on disk
🔑
dwc_gmail_token
.pickle
bathroom-bidders OAuth
Live
🔑
iexdg_robertdove
_token.pickle
iexdg-automation · 19 scopes
Live
🔒
gmail_token
.pickle
drdnicole-youtube-mgr
Revoked 5-17
🔒
bb_analytics
_token.pickle
drdnicole-youtube-mgr
Revoked 5-02
🔒
iexdg_apps_script
_token.pickle
drdnicole-youtube-mgr
Stale 4-16
Live pickle Revoked or stale Operator-identity bridge

Reading the diagram

Two projects, each with its own service-enablement surface. The dovewebconsulting@gmail.com identity is the only operator that authenticates against both: against drdnicole-youtube-manager it succeeds (consumer-account-friendly project), against iexdg-automation it failed until the Internal-to-External flip on 2026-05-26 (Workspace-only project). The token row reads the operational story: two pickles alive, three dead, the dead ones all rooted in drdnicole-youtube-manager and silently blocking 25+ downstream scripts until the reauth runbook is run.

The Internal / External / Testing / Published lattice

Google's OAuth posture has four states. Each one trades convenience for blast radius. The combination IEXDG sits in matters because every wrong move either (a) blocks operator accounts at the door or (b) issues short-lived tokens that silently rot.

🏢
Internal
Workspace members only. Triggered the 403.
🤧
External · Testing
Anyone in test-user list (max 100). 7-day refresh expiry.
External · Published
Anyone. Long-lived refresh tokens. Requires verification for sensitive scopes.
🏆
Verified · Restricted
Annual CASA security assessment for high-trust scopes.

Scope sensitivity tiers (what triggers verification)

TierExamples in IEXDG useVerificationStatus today
Non-sensitive openid, email, profile None no-cost path
Sensitive gmail.send, gmail.modify, drive.file, calendar.events, sheets, forms.body, script.projects, script.deployments, tagmanager.* Google review, multi-week, no CASA in use, unverified
Restricted gmail.readonly (full), drive (full), admin.directory.*, full mail.google.com Google review PLUS annual CASA third-party security assessment in use, unverified

What this means for IEXDG today

The omnibus token on robert.dove@iexdg.com holds drive and admin.directory.*, which are Restricted scopes. Pushing the iexdg-automation OAuth client through to Published would force a full CASA assessment (industry-reported USD 15k to 75k for a small app). The pragmatic posture is to keep iexdg-automation on External-Testing with the single test user, and absorb the 7-day refresh tax via a token-refresh cron. Verification only becomes worth pursuing if Dr. DNicole's team grows beyond ~100 distinct operator identities.

Click-path to fix Internal → External

# 1. Sign in as a Workspace admin of iexdg.com (robert.dove@iexdg.com)
# 2. Open in browser:
https://console.cloud.google.com/apis/credentials/consent?project=iexdg-automation

# 3. On the OAuth consent screen (or Google Auth Platform > Audience):
User type:      Internal → "MAKE EXTERNAL"
Publishing:     leave on "Testing"

# 4. Add test users:
+ ADD USERS → dovewebconsulting@gmail.com → SAVE

# 5. SAFETY NOTE
# Internal → External is non-destructive (no tokens revoked).
# External → Internal IS destructive (invalidates external tokens).
# Never reverse this once it's live.

The OAuth lifecycle flywheel

External-Testing OAuth apps issue refresh tokens that expire in 7 days unless they are exercised. A healthy IEXDG token spends its life in a 5-step closed loop. If the loop breaks at step 3, the token drops to the failure branch and forces a fresh user consent. The cron we will build keeps the loop closed.

External
Testing
7-day refresh
iexdg-automation
🆕
Step 1
Mint
browser consent
🔄
Step 2
Use
daily API calls
Step 3
Day 7 decay
expiry threat
🩺
Step 4
Health cron
every 5 days
Step 5
Refresh
silent rotate
Revoked · re-mint at Step 1

A healthy token cycles Use → Day 7 decay → Health cron → Refresh → Use indefinitely and never falls off the wheel. A neglected token reaches Day 7 with no refresh, drops to the red Revoked chip, and forces a fresh consent screen. Every IEXDG pickle holding Gmail, Drive, Sheets, or Calendar scopes lives on this wheel until iexdg-automation is pushed to Published (which requires a CASA assessment for the Restricted scopes we hold).

🩺 The cron we owe

A 5-day systemd timer that loops every OAuth pickle on disk, calls creds.refresh(Request()), writes back, and on failure posts to cc_post. Lives at /opt/iexdg-mcp/scripts/oauth_health_cron.py. Estimated build time: 90 minutes.

⚠ The alternative

Push the OAuth client to Published. Forces Google verification (multi-week for Sensitive scopes, multi-month plus annual CASA assessment for Restricted). Industry-reported cost USD 15k to 75k for a small app. Defer until operator count exceeds approximately 100.

🛡 The blast radius

A revoked iexdg_robertdove token takes out GA4 provisioning, GTM tag pushes, M20 Drive sync, OCC splice, admin directory lookups, and every iexdg.com-domain server-side mailer. Total scripts affected: 19 known.

Which account does which job

IEXDG operates across multiple Google identities deliberately. Each has a defined lane. Crossing lanes is the bug that produced the 2026-05-15 sender violation (memory: feedback_iexdg_email_from_dwc_gmail.md) and the 2026-05-26 OAuth project mismatch.

AccountDomainWhat it doesTokens
dovewebconsulting
@gmail.com
Consumer Gmail (DWC operator) Sends all IEXDG client comms to Dr. DNicole. gcloud admin for the VM. Inbound channel for her photo replies and reviews. dwc_gmail_token.pickle (live)
gmail_token.pickle (revoked)
robert.dove
@iexdg.com
iexdg.com Workspace Server-side IEXDG automations: GA4/GTM provisioning, m20 Drive sync, OCC splice, admin directory queries, calendar visibility. Sends nudges from the iexdg.com domain when the channel calls for it. iexdg_robertdove_token.pickle (live, 19 scopes)
drdnicole
@iexdg.com
iexdg.com Workspace Dr. DNicole's primary Workspace identity. Owns the references she shares. Eligible for direct IAM grants on the proposed GCS bucket. No automation tokens held; her access is via her Drive + (future) GCS IAM binding.
drdnicole
@iexdg-team.com
iexdg-team.com sending domain Campaign 05 executive cold-email sender. Mailbox warming since 2026-05-20. Per Gate 6, the executive variant of the campaign sends from here. App-password authenticated (Instantly). No OAuth pickle.
robert.dove
@callbrightside.com
BSP cross-tenant Logged into gcloud but not active for IEXDG work. Out of IEXDG scope. None on this disk.

The standing sender rule

Every IEXDG email TO Dr. DNicole sends FROM dovewebconsulting@gmail.com. Server-side IEXDG-domain automations send FROM robert.dove@iexdg.com. This split is a feedback-locked rule (memory: feedback_iexdg_email_from_dwc_gmail.md). Reverse the rule and you create the same incident logged 2026-05-21.

Drive folder share, GCS bucket, or both

For the brand-canon and visual-reference flow (Dr. DNicole drops files, the brain reads them, content_drop_v3 honors them), three architectures are on the table. The 403 we just hit is a strong signal one of them carries a recurring tax.

Path A
Drive Folder Share
  • Familiar drag-and-drop UX for Dr. DNicole
  • M20 sync is already coded for Drive
  • OAuth consent screen friction (just hit it)
  • Token revocation cascade (May 17 incident)
  • 7-day refresh expiry under External-Testing
  • Silent failure modes
Path B
GCS Bucket Only
  • IAM-controlled, no consent screen
  • Effectively free (~$0/mo)
  • Native audit logs
  • VM uses default service account
  • UX change for Dr. DNicole
  • Behavior risk · signed URLs less familiar

Comparison across 10 axes

AxisDrive onlyGCS onlyHybrid ↑ (rec)
UX for Dr. DNicole🟩 native🟥 behavior change🟩 native (Drive front door)
UX for Robert (admin/audit)🟨 shallow🟩 gsutil + logs🟩 GCS canonical
Auth / consent friction🟥 recurring 403 + revoke🟩 no consent screen🟨 inbox only
Cost (100 MB, 500 reads/day)🟨 Drive quota risk🟩 under $1/mo🟩 same as GCS
M20 rewrite cost🟩 0🟨 2-4 hr🟨 2-4 hr
Per-file audit trail🟨 Drive Activity API🟩 Cloud Audit Logs native🟩 both
Failure loudness🟥 silent🟩 loud 403 in logs🟨 mixed
Migration A→B or B→Ahourshours🟩 lowest risk (both surfaces live)
Secret Manager hygiene🟥 pickles to rotate🟩 metadata-server auth🟨 one pickle, smaller blast
Reversibilityhighhigh🟩 highest

Recommendation · HYBRID (Path C)

Drive remains her drop-zone for drag-and-drop ergonomics. A new bucket gs://iexdg-brand-canon in drdnicole-youtube-manager becomes the canonical brain store. M20 gets repurposed as a one-way Drive→GCS watcher. content_drop_v3, brand-canon checks, and the brain MCP read exclusively from GCS. This neutralizes the consent-screen tax for the brain side, aligns with the pending Secret Manager cutover, costs under one dollar per month, gives loud failures and clean audit, and preserves the existing behavior on her side. Confidence: high. Reversibility: hours, not days.

Fortress security proof

Plain English answer to one plain English question: is the brand brain at brain.iexdg.com actually safe? Yes, and the picture below shows why. Five independent layers of defense surround the brain, with security cameras watching every move. An attacker would have to defeat ALL five at once. The picture reads outside in: outer ring is the first wall a stranger meets, inner ring is the last vault.

Ring 5 · outer wall
Ring 4 · gate guard
Ring 3 · rule book
Ring 2 · worker reach
Ring 1 · inner vault
🧠
brain.iexdg.com
your brand brain
👁audit log
👁audit log
👁audit log
👁audit log
👁audit log
👁audit log
5
🌐
TLS & Caddy

The sealed envelope at the front gate. Every conversation between her browser and the server travels in a sealed envelope. Caddy is the wall; it checks who you are at the door and opens the envelope only on the inside.

4
🚪
OAuth Consent

The gate guard who checks badges. Before any app can act on her behalf, the gate guard asks, "do you allow this app to read your Gmail?" She picks yes. No app can pretend to be something else, and only the specific account that consented is in scope.

3
📜
IAM Rule Book

The rule book every action checks against. Even once you are past the gate, every move you try gets checked against the book. The book says, "this badge can read the bucket, but cannot delete it." Most badges can do very little on purpose.

2
🔑
VM Scope

The worker carries a master key that opens ONE building, not the whole city. Even if the brain machine accidentally tried to open something outside the IEXDG project, the key would not turn. The blast is bounded by design.

1
💼
Secret Manager

The vault inside the vault. API keys and tokens live encrypted at rest and encrypted in transit. Only specific badges can open the vault, and even those badges can only see the secret for the second they need it. The combination to the safe is itself in a safe.

⚠ What an attacker would actually have to defeat

Ring What stops the attacker What the attacker would need
5 · outer wall Encrypted envelope between browser and server. Forge a trust certificate from a compromised certificate authority.
4 · gate guard Per-user badge check before any access. Phish a single user AND bypass Google's multi-factor login.
3 · rule book Every action checked against IAM policy. Compromise the entire Google Cloud admin account for the organization.
2 · worker reach VM key only opens the IEXDG project, nothing else. Break Google's project isolation at the platform level.
1 · inner vault Encrypted secrets, auto-rotated, opened only in-memory for one second. Compromise Google's root key-management service itself.
🛡
Your brand brain is wrapped in 5 layers of independent defense plus camera coverage. An attack would have to defeat all five rings simultaneously, and the cameras (audit logs) would record every attempt with the badge, the time, and the source IP. The likelihood of all five layers falling at once is functionally zero.

Q9 keystone win + DWD rollback postmortem

Two GCP events on 2026-05-27. One unlocked the brain bucket and Secret Manager in 30 seconds of downtime. One was created and rolled back inside the same session because the underlying need had already evaporated. The third card locks the rule that prevents the second event from repeating.

✅ Q9 · VM scope flip (keystone win)

Shipped

The IEXDG VM (iexdg-nexus-vm in drdnicole-youtube-manager) was launched with devstorage.read_only, which blocked every GCS write path and silently locked the VM out of Secret Manager. Sprint 37 ran the stop, edit, start cycle and flipped the access scopes to cloud-platform. Downtime measured at roughly 30 seconds with no DNS change, no IP change, no certificate touch.

Post-flip verifications, all green:

  • VM service account can list Secret Manager secrets.
  • VM can write to gs://iexdg-brand-canon and read the same object back.
  • All 4 services back to active: iexdg-api, iexdg-mcp, iexdg-mcp-oauth, caddy.
  • brain.iexdg.com stayed green throughout the public-facing window.

Secret Manager already had 3 secrets pre-populated (ghl-api-key, iexdg-notion-api-key, iexdg-perplexity-api-key). The remaining 40 env vars inside /etc/default/iexdg-mcp are now candidates for staged migration to Secret Manager on a later sprint.

♻ DWD service account · created + rolled back same session

Rolled back

Created iexdg-brand-dwd@drdnicole-youtube-manager.iam.gserviceaccount.com to carry Drive domain-wide delegation, the goal being M20 Drive sync without requiring Dr. DNicole to share folders manually. Robert caught the gap immediately during review.

Why the need had already evaporated:

  • The Shutterstock API already grants full user-scope access to her drdnicole96 account (73 paid downloads pulled).
  • Her brand bible arrived via email on 2026-05-26 (93 reference images, the canonical doctrine).
  • The visual gate uses Claude Vision per image, not corpus embedding, so a bulk Drive sync is not on the critical path.

Same-session rollback, verified clean: deleted the SA JSON key locally, deleted the SM secret iexdg-brand-dwd-sa-key, deleted the service account itself.

📝 Permanent rule · validate-data-dependency-first

Locked

The rule: when proposed infrastructure would require client UI action (the DWD case would have required Dr. DNicole to enable domain-wide delegation in Workspace Admin), validate FIRST that the underlying data dependency still exists.

Why it bit here: the Drive dependency was rendered moot by the Shutterstock API path and the email-delivered brand bible landing in parallel. Nobody re-checked the dependency before the client UI ask was scheduled.

The directive: do not ask the client to do work that is solving an obsolete problem. A 30-second dependency recheck would have prevented the create, the rollback, and the planned ask all at once.

Net outcome · Sprint 37

VM is now fully privileged for both GCS and Secret Manager with no public-facing disruption. The DWD detour was created and reversed inside the same session, leaving no orphaned IAM, no orphaned secret, no orphaned key. The validate-data-dependency-first rule is locked into the playbook so the next infrastructure proposal cannot quietly schedule client work for a problem that has already gone away.

The GCP fuckup chronicle

Every Google Cloud and Workspace mistake on the IEXDG account, catalogued in one place. Honest, dated, scoped. The point is not self-flagellation. The point is that future GCP-domain problems land here on contact, every fix gets a postmortem entry, and the next architect inherits an audit trail instead of a guessing game.

11
Catalogued incidents
4
Fixed
4
Deferred
3
Documented
Timeline · provisioning to 2026-05-27
VM birth
🧳
Read-only scope wall
Fixed
project start
🔒
OAuth Internal-only
Fixed
consequence
7-day token tax
Documented
2026-05-27
DWD sunk-cost detour
Fixed
session
📝
Bearer token drift
Fixed
build
📧
cc_inbox 404
Documented
post-Q9
🔑
40 plaintext env vars
Deferred
Q3
🪨
PromptRegistry symlink band-aid
Deferred
2026-05-27 reply
🔒
Drive OAuth revoked
Deferred
2026-05-27 reply
🔌
Sub-agent network deny
Documented
2026-05-27 reply
🧾
Caddy cache-bust deferred
Deferred
🪨 Severity: 🔴 critical 🟠 high 🟡 medium Status: Fixed Deferred Documented Open

Per-incident detail · eleven cards

Fuckup 1
🔴
The VM scope wall existed from inception
What broke
The IEXDG VM (iexdg-nexus-vm) was provisioned with the devstorage.read_only access scope. For weeks any attempt to write to GCS or reach Secret Manager from the VM failed with errors that read like IAM rejections but were actually OAuth scope limits. No automated audit ever surfaced the gap.
Fixed 2026-05-27
Next: scope flipped to cloud-platform via stop, edit, start. Roughly 30 seconds of VM downtime. No public-facing disruption.
Fuckup 2
🟠
iexdg-automation OAuth consent screen Internal-only
What broke
The OAuth client used to issue dovewebconsulting@gmail.com tokens lived in the iexdg-automation GCP project with User Type set to Internal. dovewebconsulting is a consumer Gmail, not an iexdg.com Workspace member, so Google returned Error 403: org_internal at the consent step with no clear next step.
Fixed 2026-05-26
Next: User Type flipped to External on Testing, dovewebconsulting added as a test user. OAuth init re-fired successfully.
Fuckup 3
🟠
7-day refresh-token tax on External-Testing
What broke
External-Testing OAuth apps issue refresh tokens that expire in 7 days for any non-basic scope. This is documented Google behavior, but every IEXDG automation built against the iexdg-automation OAuth client will silently die every 7 days unless a refresh-exerciser cron is in place.
Documented, automation pending
Next: build a weekly refresh-exerciser cron per token, or push the app through verification to Published.
Fuckup 4
🔴
DWD service account · sunk-cost continuation of an obsolete plan
What broke
On 2026-05-27 I created iexdg-brand-dwd@drdnicole-youtube-manager.iam.gserviceaccount.com to carry Drive domain-wide delegation for M20 Drive sync. Robert caught the gap: the Shutterstock API already grants full user-scope access (73 paid downloads), the brand bible arrived via email (93 reference images), and the visual gate uses Claude Vision per image, not corpus embedding. Drive impersonation was solving a need that had already evaporated. I was about to ask Dr. DNicole to perform a Workspace Admin DWD enable for nothing.
Fixed 2026-05-27 (same session)
Next: SA deleted, JSON key deleted, Secret Manager entry deleted, verified clean. Rule locked: validate-data-dependency-first.
Fuckup 5
🟡
Bearer-token doc drift
What broke
The local file source/secrets/brain/brain_bearer_token.txt contained 799 chars of header notes ("# IEXDG brain.iexdg.com Caddy bearer, created...") instead of the actual 64-char hex token. The real token lives in /etc/default/caddy on the VM. The drift confused live debugging earlier in the session.
Fixed
Next: live token saved to brain_bearer_token_VM_LIVE.txt, original file retained as drift evidence.
Fuckup 6
🟡
cc_inbox endpoint returns 404
What broke
The brain MCP exposes cc_ping, cc_status, cc_brief, cc_prompt_drift, and others, but cc_inbox returns 404. The endpoint is referenced in the cross-CC architecture but was never implemented on the IEXDG VM. BB Nexus has cc_inbox_v2; IEXDG has neither.
Ship-or-drop decision pending
Next: either implement cc_inbox on the IEXDG MCP or remove the reference from the architecture doc. No middle ground.
Fuckup 7
🟠
40 of 43 env vars still plaintext
What broke
After Q9 unlocked Secret Manager access from the VM, only 3 of 43 env vars in /etc/default/iexdg-mcp were migrated (ghl-api-key, iexdg-notion-api-key, iexdg-perplexity-api-key). The remaining 40, including IEXDG_CLAUDE_API_KEY, IEXDG_OPENAI_API_KEY, IEXDG_SHUTTERSTOCK_API_KEY, still sit in plaintext on disk owned by root:iexdg 640.
Deferred to safe window
Next: staged migration of all 40 to Secret Manager in a maintenance window. Each migration is reversible and individually testable.
Fuckup 8
🟡
cc_prompt_drift scanner hardcoded path band-aid
What broke
The cc_protocol PromptRegistry hardcodes paths like TOOLS/automation_scripts/iexdg_brand_standards.txt and joins them to a ROOT that resolves to / on the VM (the lib lives at /opt/iexdg-mcp/lib/cc_protocol.py, 3 directories from root). The scanner therefore looks at literal /TOOLS/automation_scripts/..., which does not exist. The fix was a /TOOLS to /opt/iexdg-mcp/TOOLS symlink rather than proper resolution via an env-var override.
Band-aid shipped, real fix queued
Next: wire IEXDG_CC_ROOT env override into PromptRegistry path resolution, then remove the symlink.
INC-9 · DRIVE-OAUTH-REVOKED
🟠
Drive access blocked today · refresh returned revoked
What broke
Drive access blocked today, attempted refresh returned Token has been expired or revoked. Blocks Drive document fetch (the IEXDG_30_Day_Authority_Content_Plan.docx could not be pulled). Status DEFERRED; needs user-initiated browser re-OAuth.
Deferred · needs user-initiated browser re-OAuth
Next: run the OAuth init flow in a browser to mint a fresh Drive refresh token, then retry the docx pull.
INC-10 · SUBAGENT-NETWORK-DENY
🟡
N1 Notion-puller sub-agent sandbox blocked api.notion.com
What broke
N1 Notion-puller sub-agent hit sandbox network-deny on api.notion.com. Surfaced antibody R-009: main-thread completion is first-class option, not exception. Main-thread fallback ran successfully.
Documented · antibody R-009 surfaced
Next: codify R-009 in the disease/immunity registry. Sub-agent network-deny triggers immediate main-thread fallback, no retry storm.
INC-11 · CADDY-CACHE-BUST-DEFERRED
🟡
Cache-control header patch on /strategy/* deferred · unicode error in heredoc
What broke
Cache-control header patch on /strategy/* was deferred earlier due to unicode error in heredoc. Browsers cache responses, so doc updates need ?cb=$(date) cache-bust on link share until Caddy patch lands.
Deferred · ?cb= query-string workaround in use
Next: re-attempt the Caddy header /strategy/* Cache-Control "no-cache" patch via file write (avoid heredoc unicode), reload Caddy, drop the cache-bust query strings.

Permanent lessons · three shields

🛡
Lesson 1 · from F1 + F7
Audit VM scopes the moment GCS or Secret Manager access fails.
scope wall hid as IAM for weeks
🛡
Lesson 2 · from F4
Validate the data dependency still exists BEFORE building infrastructure for it.
DWD detour cost zero in client time only because Robert caught it
🛡
Lesson 3 · from F2 + F3
External-Testing OAuth means build a refresh cron from day one.
7-day refresh expiry is silent and absolute

Routing rule · this section is the home for every GCP-domain fuckup

Per Robert's standing directive, every Google Cloud or Workspace mistake on this account is catalogued here. New incidents append a marker to the timeline, a card to the grid, and (where the pattern repeats across incidents) a shield to the lessons cluster. The chronicle is the audit trail. No silent fuckups, no fuckups buried in a different doc.

How the May 27 fixes connect to Google Cloud + Workspace services

A topic-native map of every May 27 fix tied to the exact GCP or Workspace surface it touches. The diagram traces data flow from the VM compute substrate, through the GCP storage and proxy plane, out to the Workspace identity surfaces that the brain consumes. Status pills carry the truth of the day: LIVE where the fix is in place and verified, ROLLED BACK where we caught an obsolete scaffold and reversed it, REVOKED where today surfaced an identity break that needs user action.

Topic-native · six fixes · six services · one frame
May 27 fixes · service-by-service
Tier 1 · Compute substrate (the box everything runs on)
💻
iexdg-nexus-vm · Compute Engine e2-small · us-east4-b
LIVE
Fix Q9 · VM scope flip
"stop, edit, set Cloud Platform scope, start. ~30 seconds downtime. Now allows GCS + Secret Manager writes that previously failed with what looked like IAM errors." The scope wall masqueraded as IAM for weeks; the OAuth-scope ceiling rejected requests before any IAM check ran. Now the substrate underneath every other May 27 fix can speak to the rest of the GCP estate.
Tier 2 · GCP storage and identity plane (where the VM now reaches)
gs://iexdg-brand-canon
LIVE
Fix Q1 · GCS bucket for brand canon
"create, set lifecycle, set IAM, VM pushes via gsutil. Bucket = persistent cross-restart brand canon storage." GCS bucket for brand reference assets. Uniform IAM, soft-delete 10 days, inbox prefix auto-clean at 30 days. The VM scope flip is what made the push side reach this bucket without 403.
Workspace SA + DWD key
ROLLED BACK
Fix DWD rollback · validate-data-dependency-first
Was building Workspace Service Account + domain-wide delegation key for Drive. Caught in same session as unnecessary because Shutterstock API + Gmail were already covering. Rolled back: delete SA, delete JSON key local, delete Secret Manager secret. "validate-data-dependency-first rule prevents the next obsolete infrastructure scaffold."
🔒
Secret Manager (writes unlocked)
LIVE
Downstream of Q9 + DWD rollback
With the VM scope flipped to cloud-platform and the DWD secret deleted clean, Secret Manager now hosts only the 3 migrated keys (ghl-api-key, iexdg-notion-api-key, iexdg-perplexity-api-key) without any orphaned DWD JSON. The Q7 deferred queue of 40 plaintext env vars stays open here.
Tier 3 · Edge and network plane (what the public sees)
🏘
Caddy v2.11.2 reverse proxy
LIVE
Fix · Caddy reverse proxy on :80 + :443
"/ static, /mcp /api proxied to localhost:8765 and :8000 with bearer auth, brain.iexdg.com TLS auto-renew via Let's Encrypt." Edge plane between the public internet and the two internal services. INC-11 Cache-Control patch on /strategy/* still deferred, ?cb=$(date) cache-bust on link share until it lands.
🔗
brain.iexdg.com TLS · Let's Encrypt
LIVE
Downstream of Caddy
Auto-renewing TLS cert chain managed by Caddy's ACME client. Zero manual rotation. This is why public-facing docs at brain.iexdg.com/strategy/ can host the brand bible, the disease/immunity protocol, and this GCP reference without a maintenance window.
Tier 4 · Workspace identity surfaces (what the brain reads from)
📝
Notion · War Room + Content Calendar
LIVE
Fix · Notion + Drive auth integration
Notion token from source/secrets/env/iexdg-mcp.env. "main thread fetches Notion War Room + Content Calendar successfully today." Sub-agent network-deny (INC-10) routed to main-thread fallback per antibody R-009. Read path is solid.
📧
Gmail OAuth · dwc_gmail_token.pickle
LIVE
Fix · Gmail pickle with refresh on expired
Gmail OAuth pickle at source/secrets/google/dwc_gmail_token.pickle (refresh on expired). Outbound email path stays healthy independent of the Drive break. This is the fallback path the brain uses when Drive operations are blocked.
📁
Google Drive OAuth
REVOKED
INC-9 · Drive OAuth revocation today
Drive OAuth revocation happened today, status "revoked." Token has been expired or revoked per refresh attempt. "Drive search/list blocked, fallback path: main-thread Gmail still works for outbound, Drive operations require browser re-auth via OAuth consent flow." Blocks the IEXDG_30_Day_Authority_Content_Plan.docx pull until re-OAuth.
4
surfaces LIVE
1
scaffold ROLLED BACK
1
identity REVOKED today

Read order if you only have 60 seconds

Top to bottom traces the dependency chain. The VM scope flip (Q9) is the gate that unlocked Tier 2. The GCS bucket (Q1) and the DWD rollback live in Tier 2 as a "what we built" + "what we caught and undid" pair. Caddy and TLS in Tier 3 carry every public-facing read off the box. Tier 4 is where Workspace identity meets the brain: Notion and Gmail are healthy, Drive is the single broken surface as of today and the only one requiring user action.

The Drive to GCS hybrid data flow

Drive is the friendly front door for Dr. DNicole. GCS is the canonical brain store. M20 is the one-way watcher that copies between them every 15 minutes. Downstream, three brain surfaces read exclusively from GCS, so consent-screen failures stay isolated to the inbox side and never bleed into content generation or visual gating.

Left lane · her surface
📂
Dr. DNicole's Drive
DNicole Brand Canon /
Drag and drop UX. Reference photos, brand assets, voice memos land here.
🎯
Zero UX change
She keeps the workflow she has used since Apr 13. No new tool to learn.
Center lane · sync
🛡
M20 Sync Watcher
cron */15 * * * *
One-way copy Drive to GCS. Preserves Drive metadata as GCS object metadata. md5 round-trip verify.
📢
Loud failure
IAM 403 surfaces in Cloud Audit Logs and cc_post. No more silent token rot.
Right lane · canon
gs://iexdg-brand-canon
us-central1 · uniform IAM
Canonical store. IAM-controlled. Cloud Audit Logs native. Soft-delete 10 days.
💰
Effectively free
Under $1/month at projected volume. Egress to VM is $0 same-region.
md5 verify · preserve metadata
canonical read · IAM gated
Downstream readers · brain consumes only from GCS
brain.iexdg.com
Dashboard surface. Reads brand canon for visual-gate cards and Today's Pulse references.
content_drop_v3
Nightly 2 AM pipeline. Pulls Visual Direction Brief and reference photos before image-gen.
visual_gate (iexdg_visual_governance.py)
Reference-matching against her own corpus. Borderline rejects escalate instead of shipping.
↰ Dashed reverse arrow: curated and enriched assets push back to gs://iexdg-brand-canon/curated/, which M20 re-mirrors into her Drive folder so she sees the brain's output without changing surface.
💰
~$0/mo
at projected volume
📢
Loud 403
in Cloud Audit Logs
Zero
consent friction for her

Why this beats Drive-only and GCS-only

Drive-only keeps the UX simple but inherits every OAuth tax above. GCS-only neutralizes the tax but forces a UX change on her. The hybrid preserves the front door she trusts, isolates the consent-screen tax to one watcher, gives the brain a canonical store that fails loudly, and aligns with the pending Secret Manager cutover. Reversibility is hours, not days. This is the same Pattern 16 ("project-owned GCS bucket over personal-account Drive share for client asset intake") that becomes reusable across BB, PAG, and GKCBNA.

Minimum-viable bucket in 5 commands

Run as dovewebconsulting@gmail.com against project drdnicole-youtube-manager. Total elapsed time: under 60 seconds.

# 1. Point gcloud at the IEXDG project
gcloud config set project drdnicole-youtube-manager

# 2. Create the canon bucket (uniform IAM, soft-delete 10d)
gcloud storage buckets create gs://iexdg-brand-canon \
  --location=us-central1 \
  --default-storage-class=STANDARD \
  --uniform-bucket-level-access \
  --soft-delete-duration=10d

# 3. Grant Dr. DNicole direct CRUD on objects (Workspace identity)
gcloud storage buckets add-iam-policy-binding gs://iexdg-brand-canon \
  --member=user:drdnicole@iexdg-team.com \
  --role=roles/storage.objectUser

# 4. Grant the operator (Robert) full bucket admin
gcloud storage buckets add-iam-policy-binding gs://iexdg-brand-canon \
  --member=user:dovewebconsulting@gmail.com \
  --role=roles/storage.admin

# 5. First canonical upload
gcloud storage cp ./brand_canon_v1.md gs://iexdg-brand-canon/brand/brand_canon_v1.md

Optional · signed PUT URL for a no-Google-login upload

from google.cloud import storage
from datetime import timedelta

client = storage.Client(project='drdnicole-youtube-manager')
blob = client.bucket('iexdg-brand-canon').blob(f'inbox/{filename}')
url = blob.generate_signed_url(
    version='v4',
    expiration=timedelta(hours=24),
    method='PUT',
    content_type='application/octet-stream',
)
# Email `url` to Dr. DNicole. She PUTs from her browser. File lands in
# gs://iexdg-brand-canon/inbox/  with no Google consent screen on her side.

Lifecycle · auto-clean the inbox prefix

{
  "lifecycle": {
    "rule": [
      {
        "action": { "type": "Delete" },
        "condition": { "age": 30, "matchesPrefix": ["inbox/"] }
      }
    ]
  }
}

gcloud storage buckets update gs://iexdg-brand-canon \
  --lifecycle-file=./lifecycle.json

Cost shape · effectively free

100 MB Standard storage in a single US region: ~$0.002/month. 15,000 Class B reads/month: covered by the 50,000 free tier. 50 Class A writes: covered. Egress within GCP region (VM in same project): $0. The only line item that scales is internet egress, which stays under one dollar even at heavy use. The bucket is not the cost lever; the consent screen friction is.

Drive to GCS · the five phases

A surgical sequence to move from the current Drive-only posture to the recommended hybrid. Each phase is reversible. Total elapsed wall-clock: about 60 minutes plus one short note to Dr. DNicole.

🚀

P1 · Stand up

5-cmd bucket. 10 min, no client touch.

📥

P2 · Backfill

Sync existing Drive contents to GCS via rclone or gsutil.

🔧

P3 · Rewire

M20 + content_drop_v3 read from GCS.

📱

P4 · Handoff

Email her one signed URL or the new Workspace IAM grant.

P5 · Cut over

Flip env var. Archive Drive folder read-only 30d.

📥 Backfill command
rclone copy \
  drive:'DNicole Brand Canon' \
  gcs:iexdg-brand-canon/brand/
🛡 Round-trip verify
gcloud storage objects \
  describe gs://iexdg-brand-canon/... \
  --format='value(md5Hash)'
✅ Cut over
export IEXDG_BRAND_CANON_SOURCE=gcs
# Then restart iexdg-api +
# iexdg-mcp services on VM

One-sentence note for Dr. DNicole (P4)

"We are mirroring your reference folder into a backend store so the brain reads it reliably. You keep using Drive exactly the way you do today. No UX change on your side. Confirm and we proceed."

Sent from dovewebconsulting@gmail.com per the standing sender rule.

Open items + the broader hygiene queue

Items surfaced by the inventory that need explicit decisions, scheduling, or one-shot fixes. Each one carries an owner and an estimated time-to-close.

ItemSeverityOwnerAction
Three revoked tokens (gmail_token.pickle, bb_analytics, iexdg_apps_script) P0 Robert Run source/scripts/auth/gmail_reauth.py, scp to VM, restart services.
iexdg-automation OAuth consent: Internal → External-Testing P0 Robert (Workspace admin) 2-min click-path in Section 3. One-way safe.
7-day refresh-token tax (External-Testing) P1 Robert Schedule a 5-day cron that exercises each token. Alert on refresh failure via cc_post.
Stand up gs://iexdg-brand-canon P1 Robert 5 commands in Section 6. Then email one-sentence note to Dr. DNicole.
Port M20 from Drive to Drive→GCS one-way watcher P1 DWC 2-4 hour rewrite. Preserves Drive metadata as GCS object metadata.
iexdg_api.py wrong token path /var/log/... P1 DWC Patch to /opt/iexdg-mcp/secrets/gmail_token.pickle. Redeploy.
workers/youtube_tool.py expects missing youtube_token.pickle P2 DWC Either mint dedicated token or refactor to use multi-scope gmail_token (when re-auth'd).
BB bb_analytics_token.pickle residency drift (inside IEXDG repo) P2 DWC Move to BB project secrets directory. Re-auth there.
Secret Manager cutover (long-standing carry-forward) P2 Robert + DWC Fold this work into the migration. GCS path lets us use VM metadata-server auth and drop one pickle entirely.
Decide on verification (External-Testing → Published) P3 Robert Defer until operator count exceeds ~100 or sensitive-scope review value clears CASA cost.

Turnkey carry-forward · cross-tenant

The pattern "project-owned GCS bucket over personal-account Drive share for client asset intake" reuses across BB, PAG, and GKCBNA. Every client where we already control a GCP project gets the same posture. This is a candidate for Section 15 of war_room/strategy/turnkey_architecture.html as Pattern 16, born IEXDG 2026-05-26 from this 403.