Automated regression testing for Acumatica configuration-as-code (GitOps)

TLDR

Acumatica ERP configuration usually ships after manual click-through with no automated gate that rebuilds the tenant, exercises a business path, and fails on drift or a wrong outcome. With configuration-as-code for Acumatica in git, the GitOps gate is three commands: acu apply config/, acu run scenario/, acu diff config/ — each with an exit code CI can enforce. On a tenant that already has the seed applied, the path completes on the order of one minute; from an empty tenant it is a few minutes.

Manual click-through leaves indirect flows untested

Acumatica configuration ships after click-through on familiar screens, so indirect flows fail later — and Acumatica ERP still has no CI-style gate like software teams expect.

An Acumatica configuration change usually ships after click-through testing on the screens that come to mind. The flows the change touches only indirectly are rarely re-run, so the miss shows up later: a posting class that books the wrong account, an order type that stops assigning document numbers, or inventory that does not move the way the kit specification implies.

Software teams gate every code change with automated CI. Acumatica ERP configuration usually has no equivalent gate. There is no pipeline that rebuilds the tenant, exercises a business path end to end, and fails when either the configuration or the business outcome drifts.

The configuration-as-code post established desired settings as YAML in git, acu apply to write them, and acu diff to prove the live tenant matches. This post adds the missing exercise step in that GitOps loop: transaction scenarios that must pass after the seed is applied.

Build, exercise, compare as three exit codes

With Acumatica configuration-as-code in git, the GitOps regression gate is acu apply, acu run, and acu diff — three steps, three exit codes a pipeline can enforce.

The regression gate has three parts. Each maps to one command with a meaningful exit code that a pipeline can enforce.

Step	Command	Meaning
Build	`acu apply config/`	Seed the Acumatica tenant from YAML (idempotent upserts).
Exercise	`acu run scenario/`	Run lifecycle scenarios (exit 1 on any miss).
Compare	`acu diff config/`	Prove live configuration still matches the repo (exit 2 on drift).

From the Acumatica GitOps data repo (kborovik/acumatica-gitops), with acu already pointed at a target tenant:

acu apply config/
acu run scenario/
acu diff config/

acu apply config/ and acu diff config/ expand seed children under config/ in fixed order: bootstrap/, baseline/, setup/, then master/. Bare acu apply / acu diff (no path args) prefer those same trees when they exist. Bare acu run defaults to scenario/. Any CI runner can gate on those exit codes.

Order matters for the proof. acu diff runs after acu run so a green gate means the scenarios left Acumatica configuration unchanged. A scenario that dirties the config diff is a bug by definition. (You can also run acu diff config/ right after acu apply config/ to prove the seed alone is clean before exercising transactions.)

Lifecycle scenarios: capital, buy, build, sell

Four YAML files under scenario/ fund the bank, buy components, assemble kits, and sell — once-class capital, then additive buy/build/sell legs with inventory and GL delta expects.

The seed from acu apply config/ creates LAB5 Electronics Inc. in Acumatica: vendors, customers, stock items, and kit bills of materials. The happy path is four numbered scenario files, not a single monolithic script:

File	Class	What it exercises
`scenario/10-seed-capital.yaml`	once	Owner capital JE: Checking 10100 and Owner Capital 30000 both +50k
`scenario/20-buy.yaml`	additive	Four vendors: PO → receipt (CreateBill) → AP bill → AP WIRE pay
`scenario/30-build.yaml`	additive	Kit assembly: parts → GW-EDGE ×10, GW-CELL ×5, GW-RAIL ×5
`scenario/40-sell.yaml`	additive	Three customers: SO → ship → invoice → AR WIRE pay in full

once: true on seed-capital declares a present inquire-absolute gate. When Owner Capital already shows EndingBalance >= 50000, the CLI prints skip … (once: already present) and runs neither steps nor expects — capital does not restack on a warm re-run. Buy, build, and sell are additive: each warm re-run creates new documents; delta expects still hold.

Steps are ordered operations against Acumatica ERP. The important pattern is capture: Acumatica assigns document numbers, and capture stores them in ${var} references for later steps. A receipt can then reference the purchase order Acumatica just numbered:

- id: po-shenzhen
  put: PurchaseOrder
  record:
    VendorID: SHENZHEN
    Hold: false
    Details:
    - {InventoryID: MB-CM4, OrderQty: 25, UnitCost: 69.00}
    - {InventoryID: MOD-LTE, OrderQty: 8, UnitCost: 40.00}
  capture: {OrderNbr: po_shenzhen}

- id: rcpt-shenzhen
  put: PurchaseReceipt
  record:
    Type: Receipt
    VendorID: SHENZHEN
    Hold: false
    CreateBill: true
    VendorRef: "PO-${po_shenzhen}"
    Details:
    - {POOrderType: Normal, POOrderNbr: "${po_shenzhen}", POLineNbr: 1, ReceiptQty: 25}
    - {POOrderType: Normal, POOrderNbr: "${po_shenzhen}", POLineNbr: 2, ReceiptQty: 8}
  capture: {ReceiptNbr: rcpt_shenzhen}

The buy leg continues through release, filter-get for the AP bill number, and WIRE payment until the bill and check are both Closed. A valid scenario parses under acu run --dry-run, and every ${var} resolves from an earlier capture. The live files are in scenario/; the runner contract is documented in kborovik/acumatica-cli.

Delta expectations, not absolute balances

Business-outcome assertions are deltas from pre/post inquiry snapshots so additive legs re-run clean on an Acumatica tenant that already has data, without a reset.

Each scenario file asserts two kinds of outcome.

Document status is absolute: bills, checks, kit assemblies, invoices, and payments must end in the expected status (Closed, Released, and so on). Customer ACMEMFG is one of the demo customers the seed creates; the sell leg ships, invoices, and collects WIRE in full for ACMEMFG, NORTHGRID, and AGRISENSE.

Business balances are deltas, not absolute totals. The runner snapshots selected inquiries before step 1 of that file, re-probes after the last step, and asserts the difference. From the real buy file:

expect:
- get:
    entity: Bill
    keys: [Bill, "${bill_shenzhen}"]
  fields:
    Status: Closed
- get:
    entity: Check
    keys: [Payment, "${chk_shenzhen}"]
  fields:
    Status: Closed
- inquire: InventorySummaryInquiry
  parameters: {InventoryID: MB-CM4, WarehouseID: WH01}
  delta: {QtyOnHand: 25.0}
- inquire: AccountSummaryInquiry
  parameters: {Ledger: ACTUAL, Period: "072026"}
  match: {Account: '10100'}
  delta: {EndingBalance: -4037.00}

One inventory line is easy to follow by hand: the buy leg puts 25 of part MB-CM4 on hand at WH01. The build leg then consumes 20 of that part into kit assemblies (delta: {QtyOnHand: -20.0} in 30-build.yaml). Cash 10100 moves by −4037 on the four vendor WIRE payments; PO accrual and AP clear to zero once bills are paid. The sell leg asserts kit qty down (EDGE −7, CELL −3, RAIL −2), revenue 4138, COGS 1848, and cash +4138 when all three customers pay.

Deltas are why additive scenarios re-run clean on a tenant that already holds prior runs. There is no tenant reset between runs; absolute balances would fail on the second pass. The once-class capital file is the exception: on a warm tenant it skips entirely so Owner Capital stays 50k, not 100k.

Exit 0 means every step completed and every expectation held (or a once-file was correctly skipped). Exit 1 means a step error or a missed assertion.

What the GitOps gate catches

Two failure modes, two exit codes: diff exit 2 for Acumatica configuration drift, run exit 1 for a business-outcome miss.

Two failure modes show up as different exit codes.

Configuration drift. Someone changes a field on a live Acumatica screen and never writes that value back into the configuration-as-code YAML. acu diff names the entity and field, shows source versus live, and exits 2. The pipeline fails before anyone can assume the tenant still matches the repo.

Example shape (credit terms edited in the UI, repo still net-30):

CreditTerms [NET30].DayDue00: source=30 live=45

If the live edit was accidental, acu apply writes the repo state back onto the tenant. If it was intentional, the new value is codified into YAML in a pull request (configuration-as-code post), then apply and the gate both go green.

Business-outcome miss. The configuration still matches the files, but a change breaks the path the scenarios exercise. acu run fails an expect line: wrong inventory delta, wrong GL account movement, or a document stuck open. Exit 1. Close that loop by fixing the Acumatica configuration that broke the path, or by correcting an expectation that no longer matches the intended design — then re-run the gate.

What the gate costs

On an Acumatica tenant that already has the seed, apply, run, and diff complete on the order of one minute; from an empty tenant the path is a few minutes — cheap enough for every configuration-as-code change.

A warm tenant already has the seed from a prior acu apply config/. On that tenant, apply, run, and diff complete on the order of one minute (lifecycle scenarios alone finish in under a minute on the lab instance once capital once-skips).

A virgin path starts from an empty Acumatica tenant: bootstrap, apply the whole config/ tree, run the scenarios, then diff. That path is a few minutes:

acu bootstrap
acu apply config/
acu run scenario/
acu diff config/

Optional, only if you need a new login on the instance and have host access: acu tenant create creates the tenant and publishes AcuBootstrap first (needs ACU_SSH). Everything else in the gate is REST.

Either path is cheap enough to run on every configuration-as-code change, not only before go-live.

Transactions never write configuration

Scenarios create documents only; after run, diff must still be clean, or the Acumatica config change belongs in the seed tree, not the scenario.

Scenarios create documents; they do not upsert posting classes, order types, or preferences. The GitOps gate asserts that separation: after run, diff must still be clean. If a scenario step had to change configuration to pass, that change belongs in the configuration-as-code seed directories under config/ (bootstrap/, baseline/, setup/, or master/ — see the configuration-as-code post), not in the scenario file.

Where AI enters later

This post uses hand-written Acumatica scenarios as the format contract; generating consistent linked transaction sets from plain language is a later post.

The sample scenarios were written by hand once, as the format contract and the first green path. A later direction is to generate scenarios from a plain-language request: a manufacturing client, dozens of customers, months of orders, some overdue AR. The hard part is consistency (orders link to customers, invoices to orders, payments to invoices), not inventing row counts. That generation path — consistent AR/AP cycles from plain language, not just fake rows — is a later post. This post only needs the hand-written lifecycle that already fails the gate when the configuration or the deltas are wrong.

Acumatica GitOps as three commands

An Acumatica configuration-as-code change gets build, exercise, and compare steps with pipeline-enforceable exit codes.

That is the GitOps service claim in this series: not a slide, but files and commands a pipeline can run against Acumatica ERP. The configuration-as-code substrate is the configuration-as-code post. The instance the tenant runs on is built by kborovik/acumatica-devops (DEV/TEST environments post). The CLI that runs apply, run, and diff is kborovik/acumatica-cli (PyPI: acumatica-cli).