I'm not much of a Websphere admin but managed to fix it after a while.
The problem started when a datasource was updated and all of the sudden all node agents stopped trusting the deployment manager (dmgr). Syncing the nodes failed and hence pretty much everything failed to restart / deploy. Running servers where fine though.
We got this message in the logs:
[1/17/08 13:30:52:020 GMT] 00000028 RoleBasedAuth A SECJ0305I:My best guess is that node agent configuration was corrupted in some way.
The role-based authorization check failed for admin-authz operation
SSLAdmin:temporarilyDisableCertificateAuthentication:java.lang.Long.
The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted
any of the following required roles: administrator.
How do you fix the problem then?
Fairly easy actually.
# Stop all node-agents that seem broken (that would probably be all of them!).
# Go to the node agents bin directory on the node (usually something like $WAS_HOME/profiles/
# Manually sync the node with syncNode.sh, point to the SOAP connector (default is 8879) on the DMGR server. See example
./syncNode.sh dmgrhost 8879 -username websphere -password webfear# Start the node agent and verify that the logs are happy. Kick off a cell sync from dmgr. You should see entries similar to this in the logs:
[17/01/08 16:14:59:872 GMT] 0000002f NodeSyncTask A ADMS0003I:
The configuration synchronization completed successfully.