I thought I had a clean upgrade. The stderr was empty, the logs looked tidy, and the application launched without throwing a fit. It was a lie. The silence wasn’t proof of success; it was the sound of a critical module leg being skipped entirely because the test harness didn’t know how to fail when the thing it was supposed to check didn’t exist.
This wasn’t a simple version bump. The target was a desktop application heavily customized with a sprawling surface of plugins and user profiles. The goal was to make a minor upgrade safe, which meant reviewing breaking changes, remediating them through git, proving the live surface worked, and codifying the whole mess into a pre-upgrade preflight skill. I wanted to stop treating upgrades as gambling events. I wanted a deterministic gate.
The first failure was subtle. I had a fix for a "clean stderr" issue that looked perfect. The logs were pristine. But when I dug into the execution path, I realized the test runner was silently skipping a module leg because of a conditional check that evaluated to false under the new runtime. The fix worked, but the proof was hollow. I was validating the absence of noise, not the presence of function.
Then came the data layer. The upgrade introduced new patch rows that needed to be applied at boot. I wrote the migration script, ran it, and watched the application start. It looked fine. But the new rows were silently dropped. The database engine had switched from an UPDATE form to an INSERT form for these specific records, and my script was trying to update rows that didn’t exist yet. The application didn’t crash because it defaulted to the old state, masking the fact that the new configuration was never actually loaded.
The preset registry added another layer of fragility. The new version required the registry to exist before any preset rows could activate. If the registry wasn’t initialized in the correct order, the presets were ignored. My remediation script handled the data, but it didn’t handle the dependency graph of the initialization sequence. I had to rewrite the git history to ensure the registry creation commit landed before the preset activation commit.
Then the lockfiles broke. The application uses a dependency manager that generates lockfiles with relative link paths. These paths were relative to the checkout that generated them. When I moved the sandbox to a different directory structure for testing, the links pointed to non-existent locations. The application failed to load plugins because it couldn’t resolve the dependencies. I had to regenerate the lockfiles at the point of use, not transport the ones from the source checkout.
Finally, the application itself refused to quit via script. It had a GUI-based confirmation dialog that blocked automated shutdown. This meant I couldn’t cleanly restart the app to test the new configuration without manual intervention. The upgrade was gated on the operator being present, which defeated the purpose of a preflight skill. I had to find a way to bypass the GUI or force-kill the process, which introduced race conditions in the file system.
What settled it was abandoning the idea that static analysis or log inspection was enough. I booted the live composition against the pending artifact in a sandbox. I treated each activation diagnostic as evidence. If a plugin didn’t load, I didn’t just check the logs; I checked the file system, the process tree, and the memory state. I built a harness that forced every leg to have a discriminator. If the leg was absent, the harness failed. No more silent skips.
I repeated that harness until it was stable, then codified it as a skill. The next upgrade doesn’t start from scratch. It starts with the harness.
The lesson is hard-won: A vendor config migration is only proven when every leg has a discriminator that fails if the leg is absent. Generated artifacts, like lockfiles, extracted trees, and sandbox profiles, must be regenerated at the point of use, never transported. Transporting them assumes the environment is identical. It never is.
Before any dependency or app upgrade, boot the current composition against the NEW runtime in a sandbox. Gate on activation diagnostics. Then ship that gate as the skill. The next upgrade starts from the harness, not from hope.