# Advanced DataWeave workshop: one Store order, four accountable stages

This is an original, finite BEAT IT Store transformation workshop. It adds depth after the existing sixteen DataWeave lessons; it does not replace their readings, fixtures, questions or visuals. It is not copied from a private training book and does not claim official certification alignment.

**Evidence boundary:** the JSON fixtures were parsed and their baseline arithmetic/projection consistency checked with Python. The expected outputs and error-message fragments were manually derived from the stated rules. No DataWeave engine, Mule compilation, Transform Message preview or MUnit suite was executed while authoring this package. A correct-looking solution still needs real engine verification. The Python helper neither interprets DWL nor proves that a negative fixture fails in DataWeave.

## 1. Define the finite job before choosing functions

A synthetic order contains two warehouse batches and four lines. Its catalog arrives as an object keyed by SKU. The integration must produce a compact review report, retain unknown products, reconcile known versus unresolved value, and omit internal source data. Nothing is ordered, paid, reserved or shipped. All identifiers and contact details are fictional.

| Line | Warehouse | SKU (String) | Quantity | Catalog unitMinor | Derived amountMinor |
| --- | --- | --- | ---: | ---: | ---: |
| 1 | W1 | 0012 | 2 | 2500 | 5000 |
| 2 | W1 | B20 | 1 | 800 | 800 |
| 3 | W2 | 0012 | 1 | 2500 | 2500 |
| 4 | W2 | X99 | 1 | unknown | null |

These are synthetic prices in integer minor units at an agreed demonstration scale of 100. No exchange-rate, tax or accounting policy is implemented. Integer minor units avoid an unnecessary rounding exercise; they do not eliminate the need for a real system's currency and range rules.

The two important identities are SKU "0012" and customer "0007". Neither is a quantity. Converting them to Number would destroy information. Required text must be a nonempty String after trimming. Quantity must already be a Number, integral and at least one. A string "2" is rejected rather than automatically converted. Catalog unitMinor must already be a nonnegative integral Number; zero is a known zero price, whereas a missing catalog match is unknown.

Optional giftNote deliberately maps missing or null to an empty presentation string. That rule applies only to the optional note. It does not authorize defaulting a missing quantity or customer identity.

## 2. Inspect the files and make a prediction

The module is modules/StoreRules.dwl. It contains reusable helpers and four stage functions. The numbered DWL files call one stage each; pipeline.dwl composes all four. Their payload contracts differ: stage 1 consumes fixtures/baseline.json, stage 2 consumes stage 1 output, and so on. Sending the raw source directly to stage 3 is not a valid experiment.

Expected stage outputs are in expected/01-normalized.json through expected/04-safe.json. Additional final outputs cover all-known and empty-line conditions. cases.json names fourteen supplied inputs and their predicted success or error outcomes. It is a test plan, not a list of executed engine results.

Before opening an engine, predict: four input lines become how many output lines? Four, because no line is intentionally filtered out. What is the known value? 5000 + 800 + 2500 = 8300. Is that the complete order value? No: X99 is unresolved. The summary therefore uses fullValueMinor null and valuationComplete false, while knownValueMinor remains 8300. A null total communicates an incomplete valuation instead of presenting a partial subtotal as a final amount.

Run the limited offline check from this folder:

```bash
python3 verify_fixtures.py
```

Expected: fourteen JSON inputs parse, baseline arithmetic/projection consistency checks complete, and the tool prints its explicit non-execution limitation. It derives scalar control totals from input quantities/catalog values, checks reconciliation, and proves the JSON comparator rejects a numeric-SKU mutation. It does not translate the entire DWL algorithm into Python or call an expected file a real engine result.

## 3. Stage 1 — validate, normalize and flatten deliberately

Open 01-normalize.dwl and the normalize function. Every public raw value begins as Any. objectOrFail and arrayOrFail establish container shape. textOrFail and integerOrFail perform explicit type tests before applying operations. This is intentional: DataWeave typed parameters can accept coercible inputs. A Number annotation by itself is not a reliable policy that rejects the string "2".

The typed StoreLine helper describes the normalized output. A do block gives intermediate values meaningful names and keeps them local. The two lambdas in the nested batch/line mapping have distinct responsibilities: the outer lambda supplies warehouse context, and the inner lambda turns one source line into one normalized line. flatten removes exactly the resulting batch-array level. It does not recursively flatten arbitrary data.

Catalog values are normalized with mapObject while their original keys remain intact. pluck then turns object entries into a catalog array and makes each SKU a visible String field. This separation matters: trimming keys directly into another object can conceal a collision. Here, "0012" and " 0012 " become two visible normalized rows; stage 2 rejects the duplicate identity before enrichment.

Shipping city and country are first selected into an explicit two-field object and validated. The update operator uppercases those existing fields. Selecting before updating avoids a subtle bug: an update of a missing field without an upsert does not itself guarantee the missing field will be validated. This workshop does not copy shipping.doorCode to the normalized result.

Predict the first line: warehouse W1, SKU "0012", quantity 2. Predict the optional note: empty String. The customer identity remains "0007". Extra customer phone/internalRisk, catalog internalCost and root internalToken are not selected into the normalized shape.

Changed condition: use quantity-string.json. The expected outcome is a DataWeave failure whose message includes QUANTITY_NUMBER_REQUIRED. Compare quantity-null, quantity-zero, quantity-negative and quantity-fractional separately; they test different boundaries. Do not fix the failed fixture by adding an undocumented coercion. If a future contract permits numeric strings, that is a changed boundary policy needing new examples and review.

## 4. Stage 2 — enrich without losing evidence

Open 02-enrich.dwl. The catalog is first checked for duplicate normalized SKU identities. Otherwise, one source line could join to multiple catalog rows and inflate both line counts and totals. The duplicate guard is a data-quality decision, not a performance optimization.

leftJoin preserves every normalized order line. A normal inner join would omit X99, hiding the unresolved item. Each pair contains the left line and an optional matching right product. The map explicitly distinguishes a present right product from its absence. Unknown lines retain warehouse, SKU and quantity and receive status unknown_sku, with productName, unitMinor and amountMinor null. They are not free products.

Predict the row counts: four left lines and four enriched rows because the catalog identities are unique. W1 and W2 both contain SKU "0012"; those are distinct order lines with their own quantities. A repeated SKU in the order is not the same defect as a duplicated identity in the lookup catalog.

Changed condition: catalog-key-collision.json adds a second catalog key that trims to "0012". The expected failure includes CATALOG_DUPLICATE_SKU. Do not average the prices, pick an arbitrary first row or quietly deduplicate conflicting information. The product-data owner must resolve that conflict. A different changed condition, all-known.json, adds a legitimate X99 price of 400; the line should then be priced and the complete value should become 8700.

A successful JSON parse does not establish catalog uniqueness. Likewise, an unknown SKU is not a malformed quantity. Keeping these decisions separate lets a reviewer understand which source needs correction.

## 5. Stage 3 — reduce into a reconciled summary

Open 03-summarize.dwl. reduce carries an object accumulator initialized with seven zeros. Each row increments lineCount and totalUnits. A priced row also increments pricedLineCount, pricedUnits and knownValueMinor. An unknown row increments unresolvedLineCount and unpricedUnits instead.

Trace the knownValueMinor accumulator by hand: 0 → 5000 → 5800 → 8300 → 8300. Trace totalUnits: 0 → 2 → 3 → 4 → 5. The final unknown row changes the unit and unresolved counters while leaving the known subtotal unchanged. This is not equivalent to assigning its price zero.

Two reconciliation identities must hold: pricedLineCount + unresolvedLineCount equals lineCount, and pricedUnits + unpricedUnits equals totalUnits. The completeness flag is derived from unresolvedLineCount. Only a complete valuation gets a numeric fullValueMinor.

The initialized accumulator defines empty-input behavior. With empty-lines.json, the result has zero lines, zero units, known value zero, complete valuation true and fullValueMinor zero. That means this finite report has no unresolved lines; it does not mean an empty order is acceptable to a real checkout system. An order-creation contract could reject empty orders before this reporting transformation. Do not infer a business acceptance rule from a reporting identity.

Changed condition: imagine changing the first quantity from 2 to 3. Predict totalUnits 6 and knownValueMinor 10800; the unresolved row still prevents a complete value. Write the prediction before executing. Keep this as a separate exercise, not a replacement of baseline.json.

No large-payload performance claim is made. flatten, joins, recursive walks and accumulated results can require memory. Benchmark a realistic bounded workload in your own supported runtime before claiming throughput or streaming behavior.

## 6. Stage 4 — select a safe shape, then mask known contact fields

Open 04-project-safe.dwl. The first boundary is explicit projection: construct only approved output fields. The output excludes the full catalog, internal source fields and unselected shipping/customer attributes. A recursive masking helper then traverses objects with mapObject and arrays with map. match selects object, array or scalar behavior. Scalar values otherwise pass through unchanged.

The exact known keys email, phone and token are matched case-insensitively for masking. The final selected contact email becomes "[MASKED]". Identifiers "0007" and "0012" remain Strings. This is a teaching mask, not a complete anonymization system or a guarantee that every possible sensitive value has been discovered. A field named contactAddress, a secret embedded in free text, or an organization-specific sensitive identifier needs a separate reviewed policy. In particular, giftNote is retained text and is not scanned for private information; do not put real personal data into this exercise.

Why project first? Recursively masking a few familiar keys while copying the entire input would allow unrelated internal fields to leak. Why mask after projection? It gives one consistent traversal over the selected nested shape and demonstrates reusable recursion without changing the value calculations.

Changed condition: add an arbitrary source-only customer field and predict that it is absent from the final output because it is not allowlisted. Then deliberately change the expected output's SKU to numeric 12 in a disposable copy. A real output comparison must reject that type/value change even if totals look correct.

Review evidence includes the actual final JSON, runtime/tool version, input filename, script/module revision, and a comparison to expected/04-safe.json. A screenshot of a successful editor is insufficient to establish the actual returned shape.

## 7. Execute in an actual DataWeave environment

### Standalone Playground route

Use only these fictional fixtures in the public [DataWeave interactive environment](https://dataweave.mulesoft.com/learn/). Do not upload private business records. Because custom-module file editing differs between hosted versions, generate a self-contained script rather than pretending that a browser has your local classpath:

```bash
python3 verify_fixtures.py --write-playground /tmp/store-workshop-playground.dwl
```

1. Open the generated file. It contains the same module declarations/functions followed by the pipeline expression; generation does not execute it.
2. In the Playground script editor, paste that full script.
3. Set the payload input to application/json and paste fixtures/baseline.json.
4. Run/evaluate with the site's actual action and inspect compilation errors or output. Record its displayed DataWeave version if available.
5. Save the actual JSON output locally as actual-baseline.json; do not copy the expected file into that name.
6. Compare the saved actual output using the command below. Repeat each case from cases.json, recording actual error messages for failures.

If the hosted environment does not support the update operator or required Arrays function, use a supported current DataWeave/Mule environment. Do not delete validation merely to obtain a result. This source uses features introduced after the earliest DataWeave 2.0 release; the %dw 2.0 directive alone is not an engine-version declaration.

```bash
python3 verify_fixtures.py --actual actual-baseline.json --expected expected/04-safe.json
```

The comparator ignores JSON object-key order and formatting, requires array order and the exact field set, and distinguishes Strings, Booleans, null and numeric values. Numeric JSON representations such as 2500 and 2500.0 compare as equal numeric values. It checks only the supplied files; retain evidence that the actual file came from the engine.

### Studio route with the reusable module

1. Use an authorized compatible Mule project and record its actual runtime/Java versions. This folder is a resource workshop, not a second complete Maven project.
2. Copy modules/StoreRules.dwl into src/main/resources/modules/StoreRules.dwl.
3. Copy fixtures under src/main/resources/workshop/fixtures and expected under src/test/resources/workshop/expected.
4. Add a Transform Message component and paste pipeline.dwl as its transformation. Set its input preview to the baseline JSON. If running a flow, set payload to the parsed JSON fixture before the component.
5. Execute the preview or flow and capture actual output. The source fixture's local path is not automatically the Mule payload.
6. For staged debugging, use 01-normalize first and save its actual output; feed that output into 02-enrich, then 03 and 04. Compare each actual intermediate with its corresponding expected file.
7. Build MUnit assertions in your own project that compare actual parsed output with the expected JSON, test error-message fragments for negative cases, and prove a deliberately wrong SKU assertion fails. No executed MUnit suite is supplied or claimed here.

Do not flow-reference a missing workshop flow or assume copying resources starts a server. Imported module names follow the resource path, so modules::StoreRules requires exactly the module location above. If an import fails, inspect case-sensitive filenames and packaged resources.

## 8. Submit a finite evidence pack

Complete all fourteen cases in an actual engine. Record fixture, expected outcome, observed result, tool/version, module revision and pass/fail reasoning. Success cases need actual JSON comparisons. Error cases need actual failure evidence; a Python manifest entry is not that evidence.

Submit the baseline four-stage trace, all-known and empty-line comparisons, negative/type/null/shape/collision findings, and the changed-quantity prediction. Explain which errors should reject a future request versus which unresolved facts belong in a partial report. Do not label every transformation exception HTTP 400 without a separate API error-mapping design.

The solution is provided so you can study and challenge it. A compilation error, an unexpected join shape or a mismatching output is useful feedback to investigate, not a reason to fabricate a passing result. Preserve the original course content and store workshop findings separately.

See OBJECTIVE-MAP.md for the audited sixteen-lesson mapping and SOURCES.md for current primary platform references.