Page tree
Skip to end of metadata
Go to start of metadata

Current File(s): conf/logback.xml, conf/errors.xml, conf/audit.xml, conf/intercept/consent-intercept-config.xml

Format: Logback, Native Spring

Legacy V2 File(s): conf/logging.xml


Overview

The audit logging feature provides a detailed record of every request and response handled by the IdP to allow tracing of user activity, statistical analysis of usage, legal record keeping, etc. It is highly extensible, allowing the format to be customized, selection of fields to log on a per-profile basis, and the addition of custom audit fields using additional Java plugins without modifying core code in most cases. Unlike the diagnostic logs, the audit log is generally meant to be machine-readable.

There are initially two such logs provided, and there might be more added later if other use cases emerge. The core audit log is used for general request/response auditing, and is routed by default through a logger named "Shibboleth-Audit" to a file called idp-audit.log. A second log is used for logging decisions made by users over attribute release and terms of use acceptance, and is routed by default through a logger named "Shibboleth-Consent-Audit" to a file called idp-consent-audit.log.

It's possible through additional configuration to send audit records configured in multiple formats to different logging categories; for example you might have custom tools that want particular formats and the IdP can produce those formats for you in real time while still generating other formats at the same time. No parsing or converting needed.

General Configuration

As with the diagnostic logs, the actual logging is handled by the Logback implementation and so the portion of configuration that deals with how the logs are written and managed is in the conf/logback.xml file through the manipulation of the two <logger> elements and related content for the "Shibboleth-Audit" and "Shibboleth-Consent-Audit" categories. However, there is minimal formatting applied there, just direct output of the audit message.

The configuration of the audit format(s) to use for particular transactions is handled separately to allow customizing of the audit fields themselves, using formatting tokens that are themselves configured within the IdP and are extensible. You can think of it like a second logging abstraction on top of the underlying one to avoid reliance on Logback-specific extensions.

There are really two halves to the auditing system: field extraction and log output. While suitable defaults are provided, both halves can be changed using conf/audit.xml (or conf/intercept/consent-intercept-config.xml for the Shibboleth-Consent-Audit category). Extending field extraction typically requires additional Java code to pull new data out of the request state and make it available to the log output stage.

Field Extraction

Field extraction is the process of populating a labeled field in an AuditContext object that is built up over the life of a transaction. The context is just a map of field names to collections of values and the field names are just conventions built-in to the software to provide a standard library of useful fields to audit.

Fields are extracted at various points through the life of a transaction so that adequate detail can be exposed about the request, the processing of the request, and the response. These extraction points are associated with collections of field extraction beans that do the actual work to pull data out of the state of the transaction and store it for output.

Prior to V3.3, if you examine conf/audit.xml you'll see a number of empty map beans that are being merged into a number of default beans declared inside the system. These are empty but declared in this manner so that you can extend them if you want to by adding additional (or replacement) field extractors to inject into the process.

Starting with V3.3, the empty beans are no longer present to reduce clutter but you can simply add them as needed (an example is below).

The built-in extraction points and beans are as follows:

  1. shibboleth.PostDecodeAuditExtractors
    • Extractors focused on the input message, runs after decoding the input message
  2. shibboleth.PostLookupAuditExtractors
    • Runs after establishing the sender's identity, supplements the input side with information about the relying party
  3. shibboleth.PostAssertionAuditExtractors
    • Extractors focused on authentication, session, and attribute information, runs after those processes and after the initial creation of the outgoing assertion (if relevant to the request)
  4. shibboleth.PostResponseAuditExtractors
    • Extractors focused on the outgoing message, runs after construction of the message
  5. shibboleth.LogoutRequestAuditExtractors 3.2
    • Extractors focused on information specific to a SAML LogoutRequest message, runs after decoding of the message
  6. shibboleth.LogoutAuditExtractors
    • Extractors focused on information in a non-SAML logout
  7. shibboleth.ErrorViewAuditExtractors
    • Extracts subject, session, and attribute information, runs in the event that a local error is generated
  8. shibboleth.consent.PreConsentAuditExtractors
    • Extracts basic information about the consent step, runs before a decision is made
    • Empty map beans are declared in conf/intercept/consent-intercept-config.xml
  9. shibboleth.consent.ConsentAuditExtractors
    • Extracts information about the decision, runs after a decision is made
    • Empty map beans are declared in conf/intercept/consent-intercept-config.xml

Each entry in these maps is a key containing the string label to associate (the name of the field), and a value containing a Function<ProfileRequestContext,Object> instance. Functions can return either single objects that get converted to a string for logging, or a Collection of objects that are converted to strings and separated by commas within the logged field.

In this manner, you can add your own fields by implementing a function (or writing a script) to return any data you can find in the context tree and associating it with a custom field label by adding it to one of the maps named above.

Examples

A skeletal example follows to demonstrate how to create a map to contain a custom field extraction rule:

Adding custom fields to the audit log via audit.xml
<!-- Adds a function to extract a custom field called MYFIELD after the final response has been generated. -->
<bean id="shibboleth.PostResponseAuditExtractors" parent="shibboleth.DefaultPostResponseAuditExtractors">
	<property name="sourceMap">
		<map merge="true">
            <entry key="MYFIELD" value-ref="MyExtractionFunctionBean" /> 
		</map>
	</property>
</bean>

A real world example: if you're logging fields containing a URL, you may need to escape characters in a URL because of the delimiter you choose to use in the log format. You can do this using a script:

Encoding 'pipe' character appearing in CAS service URLs
     <bean id="shibboleth.CASValidationAuditExtractors" parent="shibboleth.DefaultCASValidationAuditExtractors" lazy-init="true">
         <property name="sourceMap">
             <map merge="true">
                 <entry>
                     <key>
                         <util:constant static-field="net.shibboleth.idp.cas.protocol.CASAuditFields.SERVICE_URL"/>
                     </key>
                     <bean parent="shibboleth.ContextFunctions.Scripted" factory-method="inlineScript">
                         <constructor-arg>
                             <value>
                             <![CDATA[
                                 var serviceLookupFunctionClass = Java.type("net.shibboleth.idp.cas.audit.impl.ServiceLookupFunction");
                                 var serviceLookupFunction = new serviceLookupFunctionClass();
                                 var service = serviceLookupFunction.apply(input);
                                 if (service != null) {
                                     var serviceNew = service.replace(/\|/g, '%7C');
                                 }
                                 else {
                                     var serviceNew = null;
                                 }
                                 serviceNew;
                             ]]>
                             </value>
                         </constructor-arg>
                     </bean>
                 </entry>
             </map>
         </property>
     </bean>

Built-In Fields

The fields that are supported out of the box are as follows (note that not every field is always populated, it depends on the timing of errors and the specific transaction being audited):

Generic Fields
FieldDescription
TTimestamp
eWebFlow Event
URLURL
URIURI
sIdP session ID
AF 3.3Authentication flow ID
SSO 3.3SSO flag
aClient address
UAUser agent string
PProfile ID
uUsername
HASHEDu 3.2Hashed username
attrAttribute(s)
SAML Fields
FieldDescription
SPService provider name
IDPIdentity provider name
pProtocol
bInbound binding
bbOutbound binding
nNameID value
fNameID format
iAssertion ID
dAssertion timestamp
IInbound message ID
DInbound message timestamp
IIInResponseTo
IIIOutbound message ID
DDOutbound message timestamp
tAuthenticationInstant
xSessionIndex
acAuthenticationContext
SStatus code
SSSub-status code
SMStatus message
pasvIsPassive
fauthForceAuthn

Audit Log Output

In the output stage, a bean named shibboleth.AuditFormattingMap is used to write any number of log records containing any fields you configure to the logging API, at which point the Logback configuration takes over and decides how to send that data to particular log files or other log sinks. The bean is a map between logging categories and formatting strings.

Formatting strings contain fields denoted with a '%' character followed by a field label, and any whitespace or punctuation terminates a field label and is included in the record directly.  Using '%%' will output a single percent character.

The consent audit log relies on two other maps, shibboleth.consent.terms-of-use.AuditFormattingMap and shibboleth.consent.attribute-release.AuditFormattingMap, to format log records for the terms-of-use and attribute-release intercept flows respectively.

An additional bean, shibboleth.AuditSuppressedProfiles, contains a list of profile identifiers that should not result in audit log records. The default list is used to exempt the "status" handler from being logged. The complete list of profile IDs can be found TBD.

Finally, when errors are handled "locally" (see ErrorHandlingConfiguration), the IdP can be told whether to output an audit log record or not when the error event occurs. This is useful to suppress logging noise when common errors occur that would lead to a lot of useless auditing. The flag controlling this is inside the shibboleth.LocalEventMap bean in conf/error.xml (the value of each map entry is the flag indicating whether to log).

Reference

Beans

Beans defined in audit.xmlerror.xml, consent-intercept-config.xml, and related system configuration follow:

Bean IDTypeFunction
shibboleth.AuditFormattingMap                                                                     Map<String,String>Map of logging categories to audit formatting strings for general audit logging
shibboleth.consent.terms-of-use.AuditFormattingMapMap<String,String>Map of logging categories to audit formatting strings for logging of Terms of Use decisions
shibboleth.consent.attribute-release.AuditFormattingMapMap<String,String>Map of logging categories to audit formatting strings for logging of Attribute Release consent decisions
shibboleth.AuditDateTimeFormat 3.3StringDateTimeFormat string to apply to DateTime fields (but does not apply to the logback-generated field at the beginning of the log line)
shibboleth.AuditDefaultTimeZone 3.3BooleanWhether to convert DateTime fields into the machine's default time zone (UTC is used otherwise)
shibboleth.AuditFieldReplacementMap 3.3Map<String,String>Map of replacement strings to substitute in when populating audit fields (simple way to shrink long constants down to size)
shibboleth.AuditSuppressedProfilesList<String>List of profiles to skip auditing
shibboleth.LocalEventMapMap<String,Boolean>Map of local error events to flags indicating whether to output an audit log record
shibboleth.SuppressedEvents 3.3List<String>List of events that should not be logged as "errors" by the process log (generally leaving them to the audit log)
shibboleth.DefaultSuppressedEvents 3.3List<String>Default list of events used in place of shibboleth.SuppressedEvents bean
shibboleth.PostDecodeAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after message decode
shibboleth.PostLookupAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after relying party lookup
shibboleth.PostAssertionAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after assertion build
shibboleth.PostResponseAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after outbound message build
shibboleth.LogoutRequestAuditExtractors 3.2Map<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after SAML LogoutRequest decode
shibboleth.LogoutAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run during a non-SAML logout
shibboleth.ErrorViewAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run when a local error is handled
shibboleth.consent.PreConsentAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run before a consent prompt
shibboleth.consent.ConsentAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>User-supplied fields to extraction functions to run after a consent prompt
shibboleth.DefaultPostDecodeAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after message decode
shibboleth.DefaultPostLookupAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after relying party lookup
shibboleth.DefaultPostAssertionAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after assertion build
shibboleth.DefaultPostResponseAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after response build
shibboleth.DefaultLogoutRequestAuditExtractors 3.2Map<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after SAML LogoutRequest decode
shibboleth.DefaultLogoutAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run during a non-SAML logout
shibboleth.DefaultErrorViewAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run when a local error is handled
shibboleth.consent.DefaultPreConsentAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run before a consent prompt
shibboleth.consent.DefaultConsentAuditExtractorsMap<String,Function<ProfileRequestContext,Object>>Default fields to extraction functions to run after a consent prompt

Properties

Properties are defined in services.properties to customize various aspects of audit logging:

PropertyTypeDefaultFunction
idp.service.logging.saml1ssoStringSSO







Suffix added to audit logging category when various profiles/flows are audited, you can use this to route different kinds of audit records to different destinations based on general function

idp.service.logging.saml1attrqueryStringAttributeQuery
idp.service.logging.saml1artifactStringArtifactResolution
idp.service.logging.saml2ssoStringSSO
idp.service.logging.saml2attrqueryStringAttributeQuery
idp.service.logging.saml2artifactStringArtifactResolution
idp.service.logging.saml2sloStringLogout
idp.service.logging.logoutStringLogout
idp.service.logging.casStringSSO
idp.service.logging.statusStringStatus
idp.service.logging.resolvertestStringResolverTest
idp.service.logging.serviceReloadStringReload

Notes

TBD

  • No labels