Enterprise API client code in LMS making an API call to an enterprise endpoint on LMS was failing because the JWT in the request was failing validation due to an incorrect JWT_AUTH['JWT_ISSUERS'] list in settings.
Instructions for testing this are in the Jira ticket above.
PR that fixed it: https://github.com/edx/configuration/pull/4656
Discovery service API client code in Credentials received a JWT with invalid issuer and audience claims signed with an incorrect secret. This was due to a configuration bug in LMS settings (LMS is the authz service). Also, JWTs are cached in memcached by the API client code for the life of the token (1 day).
https://credentials.stage.edx.org/credentials/3446ff661f044df785afcc44bcab2cf8/
Visit the credentials link above to verify that this is working on stage.
PR that fixed it: https://github.com/edx/configuration/pull/4654
Also we needed to clear the invalid cached JWT by having DevOps run the following from a stage credentials machine:
python manage.py shell --command="from django.core.cache import cache; cache.delete('siteconfiguration_access_token_1')"
Initial implementation: https://github.com/edx/edx-platform/pull/18429
Unmerged revert PR (in case you need it): https://github.com/edx/edx-platform/pull/18484
curl --header "Content-Type: application/x-www-form-urlencoded" \
--request POST \
--data 'grant_type=client_credentials&client_id={Application.client_id}&client_secret={Application.client_secret}&token_type=jwt&scope=grades%3Aread' \
https://{LMS_HOST}/oauth2/access_token
Use https://www.base64decode.org/ to decode the middle third (part between the first and second periods) of the access_token received:
Example access_token:
eyJhbGciOiJIUzI1NiJ9.eyJzY29wZXMiOiBbInByb2ZpbGUiLCAib3BlbmlkIiwgImVtYWlsIiwgInBlcm1pc3Npb25zIl0sICJhZG1pbmlzdHJhdG9yIjogdHJ1ZSwgImF1ZCI6ICJJbmtvY3VqTGlreXVjc0Vkd2lXYXRkZWJyRWFja21ldkxha0R1aWZLb29zaGtha1dvdyIsICJmYW1pbHlfbmFtZSI6ICIiLCAiaXNzIjogImh0dHBzOi8vY291cnNlcy5zdGFnZS5lZHgub3JnL29hdXRoMiIsICJmaWx0ZXJzIjogW10sICJwcmVmZXJyZWRfdXNlcm5hbWUiOiAiY3JlZGVudGlhbHNfd29ya2VyIiwgIm5hbWUiOiAiIiwgInZlcnNpb24iOiAiMS4xLjAiLCAiZ2l2ZW5fbmFtZSI6ICIiLCAiZXhwIjogMTU2MTc3NTU2NywgImlhdCI6IDE1MzAyMzk1NjgsICJpc19yZXN0cmljdGVkIjogZmFsc2UsICJlbWFpbCI6ICJlY29tbWVyY2UrY3JlZGVudGlhbHNfd29ya2VyQGVkeC5vcmciLCAic3ViIjogIjU2MjhkODM3MjNlY2VjMjQ5MzdhMGRlOWFiYTYwMzBmIn0.C3HLfQQYPLgy-2S-zGCZGi5UQJ7okTNm9bBYByuVXvM