A few months ago we started using bashrc to passthrough schema passwords from a bash.profile list to the various microservice cd_storage_conf.xml files. We had witness a few issues here and there with a DB schema getting locked but thought it was random.
Recently we were able to recreate this issue consistently through multiple environments. The major offender is the discovery service and the deployer service. Normally with discovery the service will stop after several failed attempts and we can manually start. Deployer just hammers the DB schema until it locks itself out.
We created a delay in the startup of the microservices so that they start 60 seconds after the server starts. That works for some of the microservices but not for Discovery or deployer. We then tried to increase the time, all the way to 6 minutes. It does not seem to matter how long the system starts the services, always one or more microservices getting locked out from bad username/password error.
One thing we noticed if the server is up, we stop the services and then start them we don't seem to run into this issue *much. If so it tends to be model or deployer.
To prove out the bashrc password pass through was the issue, we hardcoded the passwords back into the cd_storage_conf xmls and ran several reboots. That solves the issue (of course) and not db lockouts.
The issue seems to be that the microservices start too quickly and are not able to pass the password variable in the configs to the bash.profile and pull back the actual password.
I am wondering if anyone else has used a method similar to this. I spoke with SDL support and they mentioned they have clients using similar methods with AWS to pass through the passwords to the configs. I did try several ideas from SDL as well but none panned out and we ended up back to hard coding the passwords as I mentioned.
Look forward to any input,