Arine Environment Build

arine-server is the one service that did not exist when the original environment was captured, so there was no image of it in this account to start from. This step builds one and pushes it to the account's container registry.

It builds from public PyPI plus one internal GitHub repository via token, on Arm. It does not need the private PyPI mirror or the shared build host - that constraint applies to the Lambda and ECS images, not to this one.

Everything in this environment that reaches the internet leaves through a single fixed address, including the pulls from your private PyPI mirror. That mirror only answers addresses on its allowlist, so the address has to exist and be known before any build here can install anything.

This was pre-agreed, so all that was needed was to produce the address and hand it over. It is held outside the Terraform that builds the environment, deliberately, so that tearing the environment down and rebuilding it does not release it.

The end-of-August target was set before the integration services were measured, and they came to 87 more functions than the original count assumed. Going through the repositories on 17 August also raised scope questions that could have moved the number either way.

The outcome was a split rather than a slip: the core environment on 31 August, the integration services on 11 September. Those are the dates the rest of this page is built around.

Terraform keeps a file describing everything it has created, and that file is what lets it tell the difference between something that needs making, something already correct, and something that has drifted. If two people run it at once against the same file, that record can be corrupted.

So this step creates a dedicated bucket to hold it, with a lock so only one run can write at a time. It is a new bucket rather than the one the existing environment uses, so that the two cannot interfere with each other.

The encryption key for this environment's secrets, plus the six values that are generated fresh rather than copied: the Django secret key, the database password, the cache auth token, the message broker username and password, and the mail webhook secret. The existing file is sealed to the old account's key and cannot be reused, so these have to be new ones.

We had this recorded as the one irreversible step in the build. That was overstated and is worth correcting rather than leaving as a warning nobody can act on. Nothing here touches your existing setup - the key is new, in the DuploCloud account, under its own alias, and it does not overlap with the keys already in that account or with the ones your own repositories encrypt against.

The real risk is narrower than irreversible. Sealing a file is a hand edit that Terraform does not own, so Terraform cannot roll it back. The six values exist nowhere else, so losing the key means generating them again. And a KMS key cannot be deleted on the spot - AWS enforces a waiting period, which we set to the 30-day maximum so there is as much room to recover as possible.

The check that actually matters is that the sealed file decrypts. If it does not, nothing fails until much later in the build, a long way from the cause.

For your CI to push a container image into this account, it needs permission to do so. The usual way is to store an AWS key in GitHub, which means a long-lived credential sitting in a place it can leak from.

This does it the other way round. GitHub proves its identity to AWS at the moment a workflow runs, and AWS checks the request genuinely came from the expected repository and branch before allowing it. No stored key, and nothing to rotate.

The permission is deliberately narrow - push an image to the named registries, and nothing else. It is not a copy of the existing Terragrunt runner role, which is far broader than this needs.

The environment is described as a set of units - network, cluster, databases, functions, and so on - and each declares what it needs to exist before it can be built. Together those declarations form an order.

This step is the check that the order is complete and consistent: nothing circular, nothing depending on something that was never defined. It sounds procedural, and it is, but it earns its own line because until it passes, nothing can be planned at all - and the failure surfaces in whichever unit happened to be evaluated first, which is rarely where the problem actually is.

A network of its own, so that nothing in this environment shares address space or routing with anything else in the account.

It takes the range 10.32.0.0/16. The obvious choice, 10.30.0.0/16, is already used by the devops network, and overlapping ranges cannot be connected to each other later even if you want to. The range comes out of a reserved 10.32.0.0/12 block, so further environments can be stamped out beside this one without running into the same problem again.

Nodes must be arm64. The image is arm64-only, and an x86 node group fails later at Helm time in a way that presents as a registry problem.

This is not a change from what you run today. Your production cluster, arine-core-prod-use1-server-eks, is already entirely arm64 - a t4g system node group plus an m7g Karpenter pool. We are matching it.

Raised on the 19 August call as a question about Graviton generally. It is worth separating: the Lambda estate is x86_64 and stays that way, and the ECS services get pinned per service to what they run now. Only the Kubernetes nodes are arm64, and that follows the image.

Three pieces of cluster plumbing that everything later quietly assumes is already there.

An ingress class, so that a service asking to be published gets an AWS load balancer in front of it. A storage class, so that a pod asking for a disk gets a gp3 volume rather than waiting for one that never arrives. And a node pool, so that a pod which does not fit on the running machines causes another to be started instead of sitting unscheduled.

Worth listing separately because none of these fail loudly. A missing storage class does not produce an error - the pod simply stays pending, and the cause is several steps removed from the symptom.

arine-server runs on its own PostgreSQL instance and its own cache, separate from the Aurora clusters and Redis in Phase 3 — those belong to the existing estate. Both were built alongside the release. Celery worker, Celery beat and the CDC consumer start disabled: the CDC consumer needs a logical-replication database URL, and the chart installs atomically, so it would roll back the whole release.

The first end-to-end proof that the environment works: a request from outside, through the load balancer, into the application, and back.

Up to this point everything can look correct and still not serve traffic. If this returns a 400 rather than a 200, it is almost always that the application's allowed-hosts setting does not include the load balancer's hostname - which reads like a routing fault and is not one.

This phase brings up the stored-data side of the environment: two Aurora PostgreSQL clusters, Redis, two OpenSearch domains, EFS, and the shared buckets. They are grouped together because nearly everything else depends on at least one of them, and because they are the slowest things in the whole build - a database cluster takes tens of minutes, which sets the pace of the phase.

This item is the first half of the check: every service reporting itself available. The second half - that they can actually be reached from inside the cluster - is the step below, because the two fail for different reasons and one does not imply the other.

A data service can report itself perfectly healthy and still be unreachable by the thing that needs it, because reachability is a matter of network rules rather than of the service's own state.

So this is checked from a pod running inside the cluster, not from a workstation. A workstation test would pass or fail for entirely different reasons and would not tell us what we need to know.

Stamping this environment a second time is what the naming had to allow for. The two OpenSearch domains are named arine-core-dev-usw2-users and arine-core-dev-usw2-drugs, following the same convention as every other resource here. They were previously users and drugs with no prefix, which only ever works for one environment in an account.

That change was necessary and it stands. The prefix we chose to do it with does not.

On 24 August this environment collided with your dev-dr environment, which runs in the same account and region and derives names the same way, so the two produce identical names. We had picked arine-core-dev-usw2 specifically to match the convention your own arine-server Terragrunt uses, and matching it that closely is what caused the collision. The fix for the OpenSearch domains was right in principle and the prefix has to change. That is being tracked as its own question, under the resource name collision item.

The wfe_api template declares a serverless ElastiCache cache. It appeared in no inventory before the 19 August measurement, and it is a data service rather than a Lambda concern, so it belongs in this phase.

It is dated with the integration work rather than with the rest of the data services, because that is the wave it belongs to.

All sixteen table names this environment needs were already taken in the account, by the tables the earlier environment created. DynamoDB names are unique per account and region, and the application refers to these tables by their bare names in source code, so a second set beside them was never an option.

From 24 August this environment shared those tables. On 9 September you approved deleting and recreating them, on the condition that the existing records were exported first and loaded back in. That was done on 11 September.

Every table was exported, and each export was checked against a live record count before anything was removed. Nothing had written to any of the tables in the previous week, and nothing wrote to them between the export and the deletion. The tables were then recreated under the same names, now managed by this environment rather than the earlier one, and every record was loaded back in. Each table's record count matches its export exactly - 91 records across the sixteen.

Because the names are unchanged, no application change was needed.

The change-stream triggers were reconnected to the new tables. That includes provider_service's trigger on the ProviderService table, which is now switched on. The earlier environment's trigger on that table reads the new table's stream as well, so two readers share it - DynamoDB's documented limit, so a third should not be added.

Both environments are still in the same account, so anything in the earlier environment that uses these tables by name now reaches the new ones. Nothing has written to them in over a week. The new environment's arine-server role still carries no DynamoDB permissions.

Production sets a customer-managed key explicitly on its tables rather than relying on the AWS default. This step was written to match that.

While this environment shared the earlier environment's tables, they stayed on the AWS-managed DynamoDB key, because changing it would have meant re-keying tables holding your test data. Recreating the tables on 11 September removed that concern, so all sixteen were created on this environment's customer-managed key.

The key lets DynamoDB use it on behalf of any role in the account that is already allowed to use a table, which is how the AWS-managed key behaves. Nothing that could read these tables before lost access.

Point-in-time recovery on, and deletion protection where production has it.

Checked on 11 September after the tables were recreated: point-in-time recovery and deletion protection are both on for all sixteen. That includes PatientEtl and PdcDefaultThreshold, which your Terraform declared as protected but which had never actually been protected in the earlier environment.

Your Lambda functions are deployed from packages that have already been built - roughly 800 objects and 493 MB of them, already sitting in the shared bucket. Standing the estate up here is therefore a copy rather than a rebuild.

That is worth knowing because it takes the private PyPI mirror off the critical path entirely. Rebuilding the packages would need it; copying them does not.

Twenty-one resources that the functions read at start-up: four buckets, seven Kinesis streams and ten queues, counting each queue and its dead-letter partner separately. The Cognito resources and the two IAM roles are the same class of dependency and are tracked in the step below, because the roles and the identity pool reference each other and have to be built together.

We are calling this out as its own step because it changes the order of the work. The function definitions carry placeholders that are filled in from these resources - queue URLs, stream names, bucket names - so the functions cannot be stood up correctly until these exist. A function whose environment is half-filled deploys without complaint and then fails on its first call, which is the failure mode we are trying to design out of this phase.

The streams port across directly; they were already written against AWS rather than through DuploCloud. The buckets and queues take new names, because DuploCloud prefixes those two resource types and the new environment does not. That rename is safe here in a way it is not for the DynamoDB tables: every one of these is read from an environment variable rather than named in the code.

One deliberate difference from what runs today: the streams are encrypted with the environment's own key. They are not encrypted in the current environment, two of them carry patient data, and everything else we have built here is encrypted, so we did not want to reproduce that gap. Say the word if you would rather we matched the existing setup exactly.

The API's authentication. It is already defined in Terraform in your repository, so this is a re-home rather than a conversion. It is on the page because it did not appear on the first version of this plan and it is not optional - the API is not usable without it.

Also covers the two IAM roles that go with the pool: the one your portal users assume to call the API, and the one the admin-portal functions assume for sandbox work. They are here rather than in the step above because the role and the identity pool each reference the other.

Moved ahead of the functions: several of them read the pool and client identifiers from their environment at start-up, so the pool has to exist before they can be stood up correctly.

The layer inventory is now measured rather than estimated, and the earlier figure of 14 was wrong in both directions. Across all ten templates there are 16 layer declarations resolving to 15 distinct layers. Two are published and in use today - arine_api's and provider_service's own. Thirteen are not built anywhere.

Ten of those thirteen belong to the integration stacks, and they are what the work below covers. The other three belong to arine_api itself, and they turn out to matter more than a counting correction - see the last section here.

The Terraform for the ten integration layers is now written and planning cleanly: 20 resources, no changes, no deletions. Every property comes from your templates rather than from a list we keep by hand.

One detail in there is worth stating because it would have been easy to get wrong. wfe_lib is declared twice, once in wfe_api and once in wfe_worker, with the same name and the same source directory. Under CloudFormation those are two stacks and therefore two independent version series behind one name. In a single Terraform tree they have to be one layer handed to both stacks, otherwise each stack's next build would move a version the other stack's functions are pinned to. It is built as one, and the two declarations are checked against each other on every plan so they cannot quietly drift apart.

The remaining work on these ten is the build itself, and it has to happen on a host inside the allowlisted network path. Every one of them installs from pypi-prod.arine.io, which is where arine-utils, arine_schemas and arine_tools come from. That was blocked until 26 August; the NAT address this environment egresses from, 32.188.203.54, is now allowlisted and we re-tested it from inside the cluster rather than take it on trust - the index responds and serves the exact pinned version.

The build recipes are not uniform and we have read each one off its own directory rather than assuming a pattern. Two of them ship compiled curl and libpsl binaries alongside the Python packages. One stages its own build context in a preparatory step, and driving it the obvious way skips that step and fails on a missing path. One has no Dockerfile at all and needs two files copied in by its repo's deploy script first, both of which are gitignored where the build looks for them - skip that and you get a layer that is missing its own library and fails only when something touches it.

Now the part that changes an earlier answer on this page.

On 28 August we reported that five functions still fail to start after the database credential fix - missing pandas, missing dynamodb_json in two cases, missing arine_pdfs, and one case where a package is present but the wrong version. We said this was not introduced by the migration, because your environment shows the identical failure. That remains true and is now better understood, because these are two different problems rather than one.

For four of the five, the packages really are absent from the shared layer's own requirements file. dynamodb_json and arine_pdfs appear nowhere in it, and ua-parser is pinned at 0.18.0, which does not provide the function user_feedback imports. Those are contents of the layer as your source defines it, and reproducing that source faithfully reproduces the failure.

The fifth is different, and it is not a missing package. arine_api's template declares four layers, not one. workflow_engine_consumer is the only function that asks for the workflow_engine layer, and that layer's dependencies are task_engine and arine-utils - which is where its pandas comes from. That layer has never been published. There are exactly four Lambda layers in the account, and it is not among them, in either environment. Both copies of the function are attached to the general arine_api layer instead, which has no pandas in it and was never meant to serve that function.

The same applies to ten more functions: six ask for an OpenSearch layer and four are declared on Python 3.12 with a matching 3.12 layer. All ten get the general 3.10 layer. This is how your existing deployment tooling has always wired it - it attaches one layer to every function - so the new environment reproduces it exactly, and nothing here is a regression.

We are not changing it unilaterally, because publishing three more layers and rewiring eleven functions is a behaviour change in an environment whose purpose is to match yours. It is a small piece of work and we can do it whenever you want it. The question is the same one already open about these five functions and the seven that exist only on main: fix here, or reproduce as-is. One answer settles all three.

All ten integration layers are now built and uploaded. This is the first time any of them has existed - they were never in your artifact bucket or ours, so there was nothing to copy and they had to be produced from source against pypi-prod.arine.io. Each one is stamped with its own repository's commit, which matters here because these ten come from five separate repositories with five separate histories; a single version number across them would have been fiction.

The Terraform for them plans clean with its artifact check and its safety catch both switched on, which is the state we require before applying anything. Nothing is applied yet.

Two things came out of doing it that are worth passing on, because both would have bitten quietly.

The build had to happen inside the new environment's own network. We had recorded that it would run on an existing shared build machine, and that turned out to be wrong: that machine sits in a different VPC and leaves the internet from a different address than the one you allowlisted. It would have hung and looked like a Docker problem. The address you allowlisted, 32.188.203.54, belongs to the new environment, so the build ran there - on a temporary machine we created for the purpose and destroyed afterwards. Nothing was left running, and nothing touched the shared machine your ETL work uses.

The auto task assignment layer builds differently from the other nine. The other nine each specify their own build container; that one does not, because it relies on the SAM tooling to supply one. Built outside that tooling it silently picks up whatever Python the build machine happens to have - which in our case was 3.9, against a 3.10 runtime. It failed outright for us rather than producing a subtly wrong layer, and we have made the build supply the correct container so it cannot happen again. Worth knowing if anyone ever builds that layer outside your normal deploy path.

The ten layers are now deployed. That makes this the first piece of the integration work to be applied rather than only planned, and the check that your existing environment is unaffected passed twelve out of twelve both before and after - the two reports are byte for byte identical, so nothing about dev01 or the legacy account moved.

One correction to something said earlier on this page. We described the layers as the last remaining build. They were the last missing piece of Terraform, but not the last missing artifact: the 89 function packages for these eight stacks have never been built either, and the artifact store holds only the arine_api and provider_service ones. Building those is the next step, using the same approach that worked for the layers. The version stamps are already wired so each stack's functions and its layer come from the same commit.

Correction, and it matters more than the rest of this entry. Before building the function packages we checked how current our copies of your repositories were, and they were not: mail_service was 662 commits behind, fax_service 460, wfe_service 47. The layers we had just built and deployed came from those stale copies, and four of them carried dependency pins your team has since raised for security reasons - cryptography and pypdf both moved in response to published CVEs.

We have refreshed to the current head of each repository and rebuilt everything. Nine of the ten layers moved to a new version; the tenth, patient generator, is unchanged because that repository has not moved since May. The check that your existing environment is unaffected passed twelve out of twelve again, byte for byte identical to before.

Regenerating from current source changed less than we expected, which is reassuring. The queues, topics, layers and triggers are all identical. The one change is that the workflow engine's subject rule run router and worker have been merged upstream into a single function, so this environment now builds 88 rather than 89.

All 88 function packages are now built and uploaded, from the same commits as the layers, so each function and the layer it loads come from one place. Each stack's version is recorded separately rather than under one number, because these eight stacks come from five repositories that move independently.

Nothing is deployed from this yet beyond the layers themselves.

We originally planned to verify this phase by function count. That check turns out to be unsafe: comparing the current source against what is deployed today, several functions are live under names the source no longer produces — provider_service is deployed as dev01_web_api, for example, while its template now declares provider_svc_web_api. A count would match while the names differed, and anything referencing a function by name would silently point at nothing. We now compare the full set of names instead, checked against production rather than against the dev01 environment; production agrees with your source in every case, so the source names are what we build to.

187 of arine_api's functions are deployed in the new environment under those names today, each connected to its stream, queue or schedule where it has one.

Seven functions are in your source but not in the new environment. Our earlier note here said a fresh build would cover them, and that turned out to be wrong, so here is the accurate version.

The version this environment reproduces is commit 09402725, which sits on the ak-dc-deployment branch and dates from 4 February. None of the seven exist at that commit - they are all on main only. There was never a package to build at the pinned version. They are not missing from the build, they are newer than the branch being deployed.

We checked what including them would take. Five of the seven would run against the layer as it stands. The other two, get_operations_task_filtering_dynamic_options and selectable_static_inserts_router, need shared code that is not in it - one needs a module that is not there at all, the other needs a function added to an existing module later on. That shared library ships inside the layer rather than alongside each function, so those two cannot be added without republishing the layer, and that moves the layer version all 187 deployed functions point at.

Our recommendation is to leave all seven out and treat 187 as complete here, because matching what is deployed in your environment today is the whole point of this phase. If you would rather the new environment track main instead of the deployment branch, that is a fair thing to want - but it is a different goal, and worth deciding deliberately rather than arriving at by way of seven functions.

One thing to flag while we were wiring these up. Your source declares sqs_drainer's trigger as the task-maintenance dead-letter queue, reading up to 1000 messages at a time with partial-batch-failure reporting turned on. What is actually running in the dev01 environment is a trigger on the workflow-engine dead-letter queue instead, one message at a time, with no partial-failure reporting. That is a different queue, not a smaller version of the same one.

We have built the new environment to match what is running rather than what the template says, because changing which queue a drainer drains is a behaviour change and not one we want to make on your behalf. It is worth a look on your side though - if the running configuration has been wrong since it was built, the task-maintenance dead-letter queue may have messages sitting in it unprocessed today.

Seven functions, defined separately from arine_api and converted with the same module. They were the clearest case of the naming drift raised in the questions tab, and that is now settled: production runs them as provider_svc_web_api and its siblings, which is what your template declares, so we are building to the source names.

The API is defined by a single specification file rather than assembled route by route, and we feed it to API Gateway whole. That keeps it faithful to what you run today and stops 359 hand-written resources drifting away from your source.

One finding worth putting in front of you. There are two versions of this specification: the source in arine_api, and the generated one checked into the infrastructure repository. We compared both against the live API on 19 August. The generated file matches production exactly - 173 paths, 359 methods, verb for verb. The source it was generated from is 21 paths and 44 methods behind.

So regenerating the specification from today's source would silently remove 44 live methods from the API. We are treating the generated file as the thing of record and copying it rather than rebuilding it. This is the same source-versus-deployed drift we raised for function names, showing up again in the API surface.

Deployed on 24 August. The API stands up with 264 resources and 359 methods, which matches your existing environment verb for verb - we compared both sides path by path and there is no difference in either direction. All 186 Lambda integrations resolve to functions in the new environment, and none of them names a function that does not exist.

We also called it rather than just counting it. An unauthenticated POST to /pharm/login returns a 400 carrying your application's own message, "Missing credentials in parameters" - which means the request routed, API Gateway invoked the function, the function cold-started with its layer intact, and your code ran and answered. Same for /admin/login and /pharm/refresh-token.

That test also found a real problem, and it is not in the API. Your code takes the database secret name from the database username - postgres_utils.py asks Secrets Manager for a secret named after whichever user it is connecting as. We had scoped the secret names to this environment while keeping the usernames exactly as they are in yours, because Postgres grants are written against the usernames. Each choice is right on its own; together they mean 58 functions look for a secret under a name this environment does not have. 41 of those are reachable through the API. They currently fail on start-up rather than doing anything harmful, and we are deliberately leaving them that way - the alternative would have them reading the other environment's master database credentials. We will come to you with options.

The functions are deployed and the API routes to them. What has been missing is data for them to read.

The first half is solved and was ours: arine_utils derived the database secret name from the username, which meant the username and the secret name were a single value. We split them, so the username stays as your Postgres grants expect and the secret name is scoped to this environment.

The second half is now agreed. The dumps taken in early September were schema only, so the tables existed and held nothing. On 9 September we settled that a full copy of the dev databases is fine for both metadata and operations, and Gowtham is providing both by the end of that week. Once they land this can be finished and demonstrated.

The six integration stacks are described by templates written for your existing deploy tool, and we convert them into Terraform with a generator rather than by hand. It now handles the four things it needs to. We measured all four out of the templates themselves rather than working from recollection - nine templates, 91 functions - then built the generator against those measurements and tested it by deliberately breaking each one to confirm the failure is caught rather than absorbed.

Pointed at all nine templates it produces 89 functions, every one of them with a runtime, a handler and a memory size. The two it does not produce are correct: both are gated by conditions that are false outside production.

First, settings inherited from a template's Globals section rather than written on each function. This was the one most likely to do quiet damage. All 91 functions inherit their runtime and their handler from Globals, 68 inherit their memory size and 74 inherit their layer. A generator that reads only per-function properties would emit 91 functions with no runtime and no handler at all. Every one of them would deploy without complaint and then fail the first time it was called.

Second, roles built from an inline policy rather than named directly. Not one of the 91 functions names a role. Ninety carry an inline policy block and one carries neither, so every execution role in these stacks has to be synthesised, which is what your existing deploy tool does for you today.

Third, the ways those policies refer to other resources. Ref, Sub and GetAtt account for 1,670 references across the nine templates, and each one has to be resolved to a real identifier rather than copied across. Where a reference points at something outside the template - a value your deploy tool supplies at deploy time - the generator records it as an input to be wired rather than guessing at it. That list is a useful by-product: it is precisely the set of values we will need from you for each service.

And fourth, one we had not previously counted. The templates carry conditions, and 33 resources are either created or skipped depending on how they evaluate. Most are straightforward - if something is production-only, this environment does not get it. One is not: a condition named IsEnvProdStageOrDev reads like a production gate but actually includes dev, and it decides 10 resources. Reading that one the obvious way would silently leave those 10 out of the build. The generator reads it correctly and we have a test that fails if that ever regresses.

Update, 31 August. The Terraform that consumes this now exists and validates - a full plan comes to 506 resources across the 89 functions, their roles and policies, with the database settings correctly pointed at this environment's own logins rather than yours.

It is deliberately not deployed. The package index access was resolved on 26 August, so layers can now be built, but two things still stand in the way: none of these six services has ever been built into an artifact, so there is nothing to deploy yet, and the alert destinations need settling first - see the integration services item below.

Two updates since that was written.

The count is now 88 rather than 89, and 90 declared rather than 91. That is not a regression on our side: we refreshed our copies of your repositories to current head on 31 August, and the workflow engine's subject rule run router and worker have been merged upstream into a single function. Everything else the generator produces is unchanged.

That refresh also surfaced something worth knowing about the templates. Three logical names are now declared in two stacks each - the callbacks function, the logger function, and now subject rule run. Where that happens the generator has to disambiguate, and which of the pair keeps the plain name depends on the order the templates are read. We have fixed that order and written it into the code and both test suites, because a different order silently renames Terraform resources with nothing in the plan to show it.

The generator is finished. It reads all nine templates, and the four things it needed to handle - inherited settings, synthesised roles, intrinsic functions and conditions - are each covered by tests that deliberately break them and confirm the break is caught.

Every Lambda function runs as a role, and that role is what decides which queues, buckets and databases it can touch. The two halves of the estate handle this very differently.

arine_api names an explicit role on 192 of its 196 functions, so converting it is a matter of referring to roles that already exist. The six integration stacks name no roles at all - they rely on the deploy tool to invent one per function from an inline policy. Terraform does not do that, so each one has to be produced.

Of those, 118 attachments are standard AWS managed policies and copy across directly, 86 point at a shared policy declared in the stack, and the remaining 91 are bespoke documents that have to be translated. They are regular enough to do with the generator rather than by hand.

Updated 7 September. This step is no longer separate work. The roles and policies described above are built, and they are part of the function deployment rather than a step of their own - eighty-eight roles, eighty-four inline policies and two hundred and forty managed policy attachments, all produced from your templates. They arrive when the functions do, so this step will complete with that one rather than ahead of it.

The six integration stacks - 87 further functions - along with their supporting resources and APIs.

Paused on 9 September at Amit's decision: these services are not to be deployed or enabled in this environment for now. That is a deliberate pause rather than something we are stuck on, and it also sets aside the question of how mail, fax and the workflow engine's alert topics avoid reaching real recipients, which had to be answered before they could safely run.

The conversion work already done for them is kept, so this picks back up from where it stopped whenever these services are next in scope.

The queues, alarms and topics the integration functions read at start-up. 114 resources were applied on 31 August - 53 queues, 8 topics, policies, roles, buckets and schedules - and verified against the live environment.

No topic subscribes a real email address or phone number. That was checked against the running system rather than inferred from the configuration: the three topics that carry Arine staff contact details in your source have no subscriptions at all here.

This now moves with the integration services, which were paused on 9 September, so the remaining triggers wait with them.

This service is CDK rather than SAM, so the generator used for the others does not apply. It needs a decision: port it to Terraform, or run its CDK application as-is.

integrations_service ships its Lambdas as container images rather than the pre-built packages everything else uses, so it needs images built and pushed before it can run at all.

This moves with the integration services group, which was paused on 9 September, so it is not being scoped further for now.

The four services that already ran in the original environment, rebuilt unchanged. Nothing about them is being redesigned - the point is that they come back up exactly as they were.

Applied and running as of 9 September: 46 resources - the cluster, four load balancers and listeners, four target groups, service discovery and a registration for each service, security groups, log groups, and a separate role per service rather than the shared one they used before. All four services are running and all four report healthy.

Clara took the longest because its database, cache and reference files all had to be pointed at this environment rather than the old one. The last of those was a database permission that is worth passing on to your team: the application reads a view owned by one role over tables owned by another, and on PostgreSQL 14 a view runs with its owner's permissions rather than the caller's. The error names the underlying table, which makes it look like an application fault when it is not. We found the same pattern on three further views and have made our setup script correct it automatically.

These four services are not all built for the same kind of processor. pharmacist-portal is Arm; the others are not.

It is called out separately because getting it wrong does not produce a clear error. The task simply refuses to start, and the reason is several layers down.

The four services do not address one another by IP - addresses change every time a task is replaced. They look each other up by name, through a registry that tracks where each running task actually is.

Registering each service as it is created is the whole of the work. It gets its own line because nothing about it is visible until it is missing: the services start, report themselves healthy, and then fail the first time one of them tries to call another.

The batch jobs, the Step Functions state machines that sequence them, and the Kinesis streams they read from - the part of the estate that moves and processes data rather than serving requests.

It comes last because it sits on top of nearly everything else: the buckets and streams from Phase 5, the databases from Phase 3, and the queues and rules in the step below. It also has more moving parts per unit of configuration than anything else here, which is why it is proven by running the smoke tests rather than by confirming that resources exist.

Updated 7 September. The shared ETL infrastructure is built and deployed - the two Batch compute environments, the job queue, both container registries with their cleanup rules, the cache subnet group, the log group and the three roles the pipeline runs as. Eighteen resources. The check that your existing environment is unaffected passed twelve out of twelve before and after, and the two reports are identical.

Two things we did not carry across, both deliberate. The file system for the archive already exists in this environment from the data services work, so this step grants access to it rather than creating a second one. And the job role has no database table access at all yet. Your version grants it on fourteen named tables, but in this environment those are the tables we adopted from your existing environment rather than created - so granting them would point this environment's pipelines at your live data. We would rather establish what these jobs actually need than copy that across.

What remains before this step is finished is a smoke test, which needs a pipeline to run, and that is the per-customer work rather than this shared layer.

The queues, topics and rules the pipeline runs on. Applied on 9 September once the naming convention was settled: two queues, a failure topic and a log group.

The two ETL queues carry the arine-<account id>-staging prefix, because the names your configuration uses are already taken in this account by the existing environment. Everything else keeps the name it had.

This is four resources rather than the sixteen an earlier version of this page quoted. Twelve of those were arine_api and provider-service queues that Phase 5 had already delivered, and counting them here counted them twice.

Runtime configuration for the pharmacist portal - the help menu, release notes and the maintenance page.

Settled on 9 September and it turned out to need no new infrastructure at all. AppConfig application names are unique per account, so this environment could not create its own copy, and you decided the two environments share the existing one. We have wired it that way: nothing is created, the application is referenced, and the four functions that read it were checked and already have permission to.

That also closes the content question. Sharing the application means the documentation profile is already there and already serving, and the single sign-on profile - the one that might carry a secret - never has to be read at all.

One thing to carry forward: this environment now depends on an application the old environment owns, in the same way it depends on the shared password encryption secret. Neither should be removed when the old environment is cleared down.

Originally scoped as copying the production secret across verbatim, then suggested on 24 August as a new key for this environment. Neither is needed.

The secret is already in this account, in both the region it is stored in and the region the application reads it from, and it is the value that encrypted the credential records in the tables this environment will share. Leaving it exactly as it is means logins against those records keep working.

What remains is to make sure the functions that need it are allowed to read it, and to confirm nobody removes it as part of clearing down the old environment.

This is the real acceptance test, and the reason the rest of it is written as Terraform at all.

Reproducibility is the actual deliverable. If the environment only works because of something someone did by hand once and did not write down, then it is not finished - it is just currently running. The only way to know the difference is to take it all down and build it again from the code.

Everything else on this page is, in a sense, preparation for this step passing.

One thing we found on 24 August changes how this step can run. The role we operate this account with has an explicit deny on deleting RDS instances and clusters, and on deleting ElastiCache clusters. We found it the direct way: an apply that needed to replace a database failed on it, and the database was left untouched, which is the guardrail behaving exactly as intended.

The consequence is that a full teardown cannot include the two Aurora clusters or the two Redis clusters. Everything else - the network, the Kubernetes cluster, the functions, the API, the buckets, the roles - can be destroyed and rebuilt from code today.

So there are two versions of this step, and we would like your steer on which one you want. The first is a rebuild of everything except those four stateful stores. We can run that without any change on your side, and it still proves the part most likely to be hiding undocumented manual work. The second is a full rebuild including the databases, which needs that deny lifted for a scheduled window.

Our recommendation is the first one now, and the second later if you want the stronger guarantee. Rebuilding a database that holds no data proves very little, and these two hold none yet.