The system provides ample logging to diagnose errors, and your first responsibility is to review that information in any situation. You should also be familiar with all the logs involved, and what their function is, to understand where to look.
The core logs consist of:
- The IdP process log, usually in idp.home/logs/idp-process.log (warnings and errors will be mirrored to idp.home/logs/idp-warn.log), is the primary troubleshooting resource for most problems. It should be the first place you look.
- A servlet container log from Tomcat, Jetty, etc. This is usually NOT all that useful unless the IdP is actually refusing to initialize and start up at all, but may be important in those cases. Note that if the Status page is viewable, that likely means the IdP started properly.
- A web server log in the case that you are proxying traffic from Apache or the like. This is also probably not involved unless the problem is very low-level and involves connectivity to the system.
- Web access logs, which track requests to the web server. These will be wherever traffic is handled (servlet container, Apache, etc.) and help with understanding what clients are asking for. This can help diagnose issues with the advertised locations for IdP functionality.
- Back-end logs of authentication and attribute sources may also be necessary in some cases, but most of the time the IdP should provide enough in its own log to indicate what the problems are.
Slow startup time
There are two known reasons for experiencing unusually slow startup: using Tomcat in certain ways and a blocking PRNG. Taken in order:
If you're using Tomcat, this issue is discussed on the prep page along with some workarounds, primarily forcing it to expand the warfile, counter to the usual advice we give with Jetty.
With all containers, it's not uncommon for startup to be delayed sometimes several minutes because Java defaults to using /dev/random as a source of randomness, which is a blocking device. The most common workaround for this is to switch to /dev/urandom instead, which can be done by passing a system property to Java during startup (e.g. on Jetty via start.ini):
Traditionally this has been viewed as less safe. There are alternate views of this. The Shibboleth Project does not take an official position on this question.
Common Log Messages
Invalid XXXXService configuration
The XXXX above might be "NameIdentifierGeneration" or "RelyingPartyResolver" or "AttributeResolver", or any of the other service names. When you see this message while debugging something that's not working, you are missing the real problem and focusing on the aftermath.
The log is not just the last message you see; you have to look at all of it, particularly during startup or reloading a new configuration for a particular part of the system. The real error is that you've broken a part of the configuration in a particularly significant way, usually with a syntax error of some sort, and you need to look at the error that describes that problem to make any progress.
Sometimes that will be a fairly complex Spring message because of a problem creating Java objects, but occasionally it will be fairly obvious what the error is if the problem mentions an XML syntax error or a property name that turns out to be misspelled. Regardless, that's the real problem and you should ignore whatever comes after it until you resolve the original issue and get the configuration of the affected service at least loading again.
If you're really confused, try changing the relevant "failfast" property in the services.properties file to force the IdP to fail at startup. At least then you'll know where the real issue is.
Error: "unable to find resource 'status.vm'"
The first time the status page is accessed after starting the IdP, the following log message will be seen:
Although this is presented by the Velocity library as an
ERROR level message, this should be ignored unless you have customised the status page yourself. In fact, the default status page is implemented as a JSP and not as a Velocity template, and the message merely indicates that the fallback from Velocity to JSP is occurring for the status view the first time it is invoked. This message is filtered in the logging configuration in newer IdP versions.
Error: "unable to find resource 'login.vm'"
If the default login UI is changed from a Velocity template to a JSP, the following error will be seen at least once:
Although this is presented by the Velocity library as an
ERROR level message, this can be ignored. The message merely indicates that the fallback from Velocity to JSP is occurring. You may wish to filter it with your logging configuration.
IdP stops sending authentication statements after 2 weeks; restarting the IdP temporarily resolves the issue
IdPs that consume metadata using the
FileBackedHTTPMetadataProvider regularly download metadata from an external metadata source. The IdP has no control over the size of that metadata file, and it may have sufficiently increased in size from one download to the next that the IdP cannot process the later file. Since the IdP stores the previous file, it continues to use that until the metadata it contains expires (perhaps a couple of weeks after the initial out-of-memory error) at which point the IdP stops recognising SPs. Re-starting the IdP temporarily resolves the issue because it allows the IdP to download new metadata until the next time it runs out of memory.
Symptoms include the idp-warn.log showing entries like this:
Please see the Installation pages for your particular deployment to determine how to set an appropriate Java heap size, and what that size should be.
Unhelpful logging information from exception traces.
The default logging provided in logback.xml is for a truncated information from exceptions. Particularly when debugging a development system it may be helpful to turn up the amount of information in the log. This page has mpore details but replacing this line in
Can often provide a useful hint of where the fault has come from. For instance:
Is more helpful when debugging than