Ode to compile time

It's gonna be a short story, I'm a bit drunk and quite tired again. Had an eventful week and if I want to raise one thing that would be the difficulty of quality assurance on a process heavy project.
I was working on a web service that is expected to work with hundreds of thousands of unique items when it's gathering data from 3rd parties. Considering your time and resources you usually try to follow the conventions of the existing project and make it smooth. I did the same, I added some new functionality and made it a similar way so the risk of change will be the smallest - but the I soon realized cutting corners fails when scaling up. Scaling up means running one test for half a day at least. That's definitely not an interval that you can spend with a coffee or a random sword fight.

The lucky ones were surly had some experience with developing C or similar projects. I had this fantasy about developing a game and thinking through the way of debugging the code. When you work with websites you better to get your result in couple of seconds. That means it's really not necessary to add any reporting mechanism or debugging facility. Any mistakes you can hunt down in seconds and rerun the code again in an extremely short time. What I've seen in many lower level projects (games and desktop apps and alike) was some kind of overlay debugging. You could fire up a dev console or some display containers presenting you extra information. You could interact with the system any time.

I mean it's not something you can just apply on a PHP project. Supposedly there is no running daemon and alive process ever. It's one run. So here you are some of my ideas I could imagine - helping the inspection of the underlying code:

To be able to interact with the running process - let's say it's a long process so I can interact with it - I would hook in some environmental variables - or local files, DB frequently. So there should be a context where changing the data could be reflected in PHP.

I'd add thorough logging. Syslog forever - that's for sure. But I'd log everything. Unfortunately there are no macros in PHP - but we can apply some workaround and simulate - so it's gonna be easy to turn it off for security reasons.

I'd definitely add callable hooks in order to change the data - and make all the process as fragmented as possible. It's a commonplace but the more segments you have the less likely that you break something in a middle of an uberfunction and then you have to restart it.

Data integrity checks and on-demand inside tests. Data should be able to be verified at any given point. The better the heuristics the less process heavy action is needed - the less time it requires the more occasion it can be run. I guess it's needless to say - apart from fragmented model it has to be batch-able. Taking as small footprint at any given time as possible.

I guess there are certain patterns that could be used here as well. Working with actions, using queue-s and some kind of a priority system, having redo-undo functionality, etc.

---

If you have any other idea how to make a process heavy service developer and debug friendly let me know.

Peter