The Error Message Was Lying: A KV v2 Nesting Bug Hiding Behind "Variable Is Not Defined"

TL;DR: Ansible kept reporting a Vault-backed variable as undefined. Every plausible theory about why — group_vars not loading, Ansible Vault not decrypting, Vault sealed, the secret never actually written, AppRole misconfigured — checked out clean when actually tested, one at a time. The real bug wasn't any of those. It was a generic error message masking a specific one: Vault's KV v2 API always nests secret data two levels deep, and the lookup expression only unwrapped one.

The symptom

ansible k3s-master -m debug -a "var=k3s_node_token"
k3s-master | SUCCESS => {
    "k3s_node_token": "VARIABLE IS NOT DEFINED!"
}

The variable was meant to come from a community.hashi_vault.vault_kv2_get lookup against HashiCorp Vault, authenticated via AppRole. The first thing tried was the obvious fix — the play might not have had a vault password to decrypt the Ansible-Vault-encrypted variables the lookup depended on:

ansible k3s-master -m debug -a "var=k3s_node_token" --ask-vault-pass

Identical result, password prompt and all. That non-difference turned out to be the first real clue, even though it didn't look like one yet.

First theory: the variable definition isn't loading at all

This lab had already hit a real bug earlier in the same build where a group_vars file never loaded because its filename didn't match the actual inventory group name — silent, no error, just quietly ignored. Given that precedent, and given that --ask-vault-pass produced zero visible difference, the leading theory was that Ansible wasn't even reaching the file containing the lookup expression for this host. The reasoning: if Ansible were actually attempting to decrypt something, a correct password should succeed and a wrong one should throw a distinct decryption error — neither of which happened. A clean "not defined" with no decrypt-related error at all suggested the file simply wasn't in play.

ansible-inventory --host k3s-master disproved that immediately. The raw, unrendered lookup expression for k3s_node_token was right there in the merged host vars — along with vault_addr, vault_approle_role_id, and vault_approle_secret_id, all decrypted, all matching the exact variable names the lookup referenced. The file was loading fine. Group_vars precedence wasn't the bug.

That check surfaced a genuinely useful side-finding, too: ansible-vault view group_vars/all/vault.yml decrypted successfully with no password prompt at all on the first call — meaning ansible.cfg already had vault_password_file configured, and non-interactive vault password handling (tracked separately as its own open task) had quietly already been solved. The project's own handoff notes still listed it as outstanding. Worth remembering: a checklist item marked open isn't always still open — sometimes the environment has moved past its own documentation.

Second theory: something on Vault's side

With the loading theory dead, the next candidates were on the Vault side. The project's own notes described the k3s_node_token secret migration as "in progress," not confirmed complete — meaning authentication could succeed perfectly and still return nothing if the secret was never actually written to that path. Vault getting resealed after a VM reboot (a real, already-flagged risk in this build, since auto-unseal wasn't configured yet at that point) was an equally plausible candidate — a sealed Vault fails every authenticated read, silently as far as the lookup's caller is concerned.

Rather than guess further, the fix was to reproduce the lookup's exact sequence by hand, outside of Ansible entirely, from the same control host:

1. curl .../v1/sys/health — "sealed": false. Vault was fine.
2. AppRole login via curl with the actual role_id/secret_id — succeeded, returned a valid client token with exactly the expected policies attached.
3. A KV v2 read at the actual secret path, using that token — succeeded, returned a real value.

Every layer on the Vault side checked out clean. That's a clean example of collapsing an entire branch of possibilities at once: reproducing the failing component's exact behavior manually, with the same credentials and the same target, proves or disproves a whole category of theory in one shot rather than one hypothesis at a time.

The actual bug

With loading, decryption, seal status, AppRole auth, and the secret's existence all confirmed working, the only thing left was the lookup expression itself — and the only way to see what it was actually doing wrong was to stop trusting the debug module's summary and ask Ansible for the real exception underneath it:

ansible k3s-master -m debug -a "var=k3s_node_token" -vvvv

Buried in the verbose output was the real error, which "VARIABLE IS NOT DEFINED!" had been quietly standing in for the entire time:

'dict object' has no attribute 'value'

Vault's KV v2 API always wraps secret data in a double data envelope — an outer data that's just the API response wrapper, and an inner data that holds the actual key/value pairs you wrote:

{
  "data": {
    "data": { "value": "the-actual-secret" },
    "metadata": { "created_time": "...", "version": 1 }
  }
}

The vault_kv2_get lookup returns that same shape without flattening it further. The expression in group_vars/k3s/vars.yml only unwrapped one layer — ...).data.value — when the actual value lived a level deeper, at .data.data.value. Every earlier finding fit this perfectly in hindsight: authentication worked, the secret existed, Vault was unsealed — the lookup had simply grabbed the wrong branch of an otherwise completely healthy response.

The fix

One line, in group_vars/k3s/vars.yml:

- role_id=vault_approle_role_id, secret_id=vault_approle_secret_id).data.value }}"
+ role_id=vault_approle_role_id, secret_id=vault_approle_secret_id).data.data.value }}"

Worth grepping the rest of the codebase for the same vault_kv2_get(...).data.value pattern — a bug this easy to introduce once is exactly the kind that gets copy-pasted into every other secret lookup that follows the same template.

What this demonstrates

The most interesting thing about this investigation isn't the fix — it's what it means when several independent, well-reasoned theories all check out clean in a row. That's not a sign of not having looked hard enough. It's a sign the framing of the problem is wrong. "Variable is not defined" describes a category of failure — the name doesn't exist — but the actual failure was a defined value with the wrong shape, which is a completely different problem wearing the same error message. Every theory tested against the "undefined" framing was reasonable, and every one of them was correctly ruled out, precisely because none of them were the actual bug. The turning point wasn't a smarter guess — it was refusing to keep debugging around a generic abstraction of the error and going straight to the specific exception underneath it.