Upgrades MUST be done in place!
If you are upgrading from V3, PLEASE follow the directions on the Upgrading page and do NOT install V4 separately and then attempt to apply your old configuration files. You MUST upgrade "in place" by using an installation directory that contains a copy of the previously working configuration.
Failure to do so will result in a non-working system in the majority of cases because there are absolutely essential differences between the output of the installer in the two cases to prevent your configuration from being altered in unexpected ways.
Please review these release notes before upgrading your system. You should review the notes for all the versions subsequent to the one you're running prior to upgrade, including referring back to older V3 notes.
Also be aware of the following issues regarding container or Java compatibility:
These are known issues in the code that are likely to affect deployers and have yet to be fixed.
IDP-1623Getting issue details...
- This is a bug that impacts DataConnectors that rely on the new "exporting" feature and then fail to return any results. The simplest workaround for now is to insure such connectors either fail with errors, or are backed up by a Failover connector that ensures some kind of dummy result is returned.
- - IDP-1625Getting issue details... STATUS
4.0.1 (June 3, 2020)
This is a patch release to address bugs identified and to shore up some missing LDAP features that arose due to the loss of JNDI support, such as SASL authentication and referral handling. The LDAP features are an exception to our usual policy on new features in a patch release and configuration options are denoted with the 4.0.1 designation where applicable.
A bug was corrected that was causing metadata resolver crashes on invalid metadata containing empty elements. Previous releases inadvertently accepted and ignored these elements. The patch adds warnings about the invalid content but corrects the crashing bug.
Jetty has been updated for the Windows installer to version 9.4.28 and an issue corrected with the use of wildcard certificates. The full fix for this in the case of back-channel deployment involves an update to the plugin we supply for certificate validation bypass, and this plugin has been renamed to reflect that it is designed solely for use with Jetty 9.4. The Jetty94 page has been updated to reflect this. The older plugin and back-channel XML configuration remains compatible for now, but will fail in the unlikely even a wildcard or multi-SAN certificate were used.
Changes to Defaults
- The transcoding rule shipped with 4.0.0 for the "telephoneNumber" attribute had the SAML 1 and SAML 2 names swapped, so upgraded systems will carry this mistake until corrected by hand.
- A bug was introduced into the default web.xml file in the interminable attempts to fix warnings around the default method constraints. The element
<authn-constraint/>should have been
<auth-constraint/>. This may cause warnings or result in the rule being ignored.
4.0.0 (March 11, 2020)
This is the first release of the fourth-generation Identity Provider software. The key documentation links are located on the IDP4 space Home page, such as SystemRequirements, Installation, and Upgrading material. Note the new SystemRequirements as they have substantially changed with regard to Java and container versions.
This release should interoperate with all previous releases of Shibboleth and other software that supports the same standards.
As a major upgrade, the list of issues fixed and features added is numerous. Many significant changes are highlighted below, with particularly noteworthy issues highlighted in note or warning boxes.
There are API additions, changes, and removals, and any third-party extensions will need to be checked for code compatibility and recompiled for use with this version, and any custom scripts created by deployers should be carefully tested and reviewed.
Changes to Defaults
The following defaults have changed for both new installs and upgrades and should be reviewed for possible incompatibilities:
- The idp.cookie.secure property now defaults to true if not set by the deployer. In the unlikely event that HTTP compatibility is a requirement, the property would need to be adjusted. (Note that the container's session cookies are controlled by policy in web.xml and while we have also changed that default, only upgraders without their own web.xml customizations would be exposed to that change.)
- The default LDAP provider is now UnboundID. See the LDAP section below for more information.
- Some legacy code that was used to maintain the V2-style formatting of the %T audit field (the timestamp of the record) was removed and the log now outputs all date/time fields using the supplied formatter or a default. The format of the %T field may therefore change for some deployers.
- Related, the shibboleth.AuditDateTimeFormat bean is interpreted as a pattern by the Java DateTimeFormatter class, rather than the deprecated joda-time version, and there are some differences in those classes that may result in changes to your logs. If you require identical output, review is warranted.
The following defaults have changed for new installs but will not change for properly upgraded systems (generally these are set through explicitly uncommented properties or beans that have to be present to enable a behavior):
- The default XML Encryption algorithm is now AES128-GCM rather than AES128-CBC. See GCMEncryption for much more on this, and do note that newly installed systems will fail to interoperate with a large number of SAML SPs without modern algorithm support.
- HTML Local Storage is now enabled by default when the client-side session storage feature is used (which has been and remains the default option as well).
- In addition the logout-enabling idp.session.trackSPSessions and idp.session.secondaryServiceIndex properties are also enabled by default. They can be disabled by deployers that do not wish to try and support logout propagation.
- The set of profiles enabled by default no longer includes any SAML 1.1 support, or SAML 2.0 Attribute Query.
- The default trust engines used at runtime now omit PKIX support. Support for only the Metadata Interoperability Profile is now the default behavior with no attempt at a fallback.
- A few ReloadableServices that were configured by default to fail-fast have changed to non-fail-fast behavior.
- Audit output has been changed to include a much more useful default format and various improvements to how specific fields are logged to shrink repetitive and unnecessary constant values.
Removal of Deprecated Features
A number of previously deprecated features, settings, APIs, etc. have been removed. The use of these deprecated bits should have produced a standard warning either at startup or during use in the most recent releases of the software, and most of them are now non-functional. Eliminating warnings before upgrading is strongly advised.
The entire code base has been converted from the joda-time Date/Time API to the new Date/Time API introduced in Java 8. This change is widespread in the code, with most uses of long/Long types to represent timestamps and durations converted into Instant and Duration classes.
The change is mostly invisible to deployers because of the widespread support for ISO Duration notation in the configuration. There are a couple of areas of direct impact that are now deprecated and will be removed from a future version:
- Passing back a login timestamp as a joda-time object in the External login flow.
- Use of a joda-time DateTimeFormatter object, primarily (but not required) for ExpiringPasswordInterceptConfiguration.
It was also observed that V3 may have been improperly interpreting timestamps without the trailing 'Z' indicator as compliant UTC values, which was not a correct assumption. Using such values is a violation of the SAML specification in any event, and V4 does not handle such values and will reject them in most cases as expired or in the future depending on the default time zone of the system relative to UTC.
The entire code base has been converted from the Guava Predicate/Function API to the new functional API interfaces introduced in Java 8. This change is widespread in the code, but we have supplied migration aids via our numerous utility/parent beans and a support class that converts between the legacy Guava Predicate.apply() signature and the new standard Predicate.test() signature.
We expect most custom code and scripts based on Guava Predicates will work unmodified, but may produce warnings when the apply() method is called to allow code cleanup and prevent future problems.
Beware of Guava's "compose" method
Legacy use of Guava Functions may work for the most part, but there is a known issue in Spring's wiring behavior that breaks the use of Guava's "compose" helper. If you have custom bean wiring anywhere that contains a reference like this, you will likely have to update it to avoid problems.
Using that syntax will often result in the two constructor arguments to that bean getting swapped/reversed, which reverses the order of the composition, usually producing bad output at best, and a crash or even more catastrophic outcome at worst.
Unfortunately there's no easy fix for this because of the way Guava accomodates the conversion to Java 8's Function interface. As a result, the best thing you can do is simply scan for this and convert the above syntax to:
That is functionally equivalent and should avoid the re-ordering behavior we're seeing with Spring.
Profile Configuration APIs
A lot of cleanup was performed to eliminate inconsistent method names in the APIs used for configuring and overriding profile behavior in relying-party.xml using overrides. In a small number of advanced cases, people may encounter errors in their configuration due to type incompatibilities, which may require reviewing the appropriate class Javadoc. But as a rule of thumb, all the property names now follow these conventions:
- p:something takes a literal value such as a boolean, integer, long, String, Duration, etc.
- p:somethingPredicate takes a Java Predicate when the equivalent literal would be a boolean
- p:somethingLookupStrategy takes a Java Function when the equivalent literal would be other than a boolean
As an example, the V3 property
signAssertions was Predicate-valued, while the V4 version takes a literal boolean and
signAssertionsPredicate takes a Predicate. There are a number of such cases but not many with common use.
The default LDAP implementation is now the UnboundID library and JNDI is no longer officially supported. There will be no JNDI support at all in a future version and we expect to migrate to a native implementation being built within a future version of Ldaptive at that time.
This release includes configuration updates to facilitate that change, particularly in the LDAPAuthnConfiguration and to a lesser degree in the LDAPConnector. Deployers using this feature should review and update their configurations when possible to modernize them and minimize the need for future changes. In many cases, the ldap-authn-config.xml file can be replaced outright with no impact, along with an additional change noted in the LDAPAuthnConfiguration page under "Note Regarding Upgrades".
Most previously necessary
<LDAPProperty> elements can be replaced by various new (or recently added) settings that configure the equivalent behavior in a non-JNDI fashion. For example, specifying binary attributes in the LDAPConnector is now possible directly via the
<BinaryAttributes> element added in a V3.4 patch. If you encounter properties that have no obvious replacement, seek assistance.
The ldaptive JAAS module has an incompatible change in its handling of timeout settings. Specifying values in millseconds will result in a DateTime parsing exception. The values must be converted to ISO syntax (e.g., 2000 → PT2S).
In addition, various LDAP pooling properties prefixed by "idp.pool.LDAP." (such as idp.pool.LDAP.validatePeriod) previously defaulted to numeric values expressed in seconds (e.g. 300 == 5 minutes). These are now being interpreted as milliseconds, which causes overly frequent pool validation if the defaults are used with an unmodified V3 ldap-authn-config.xml file. This is dicussed further on the LDAPAuthnConfiguration page.
Password Flow Extension
The Password login flow has undergone significant internal reimplementation to support a much higher degree of flexibility. The redesign has resulted in the elimination of one existing extension mechanism that we don't believe was heavily used, described in the V3 documentation under the heading Custom Back-Ends.
A simpler extension point is now provided for implementation of such custom validators, and converting is not a complex task.
A major new addition to V4 is the AttributeRegistry service that replaces, or at least supplements, the AttributeEncoder mechanism for translating attributes. While upgraded systems should operate unchanged, deployers are advised to review this new feature and may achieve substantial simplification of their configuration by taking advantage of it. Many existing encoders may be removeable.
Note that installing from scratch and then applying a legacy configuration will in most cases result in duplicate Attributes appearing in SAML messages due to the overlap between the existing encoders and the new rules. The upgrade process is designed to prevent that by excluding the new rules from the default configuration.
In the unlikely event that you have any attribute display name or description metadata present for an attribute definition but do not have an encoder, the compatibility support is not going to attach that display metadata to the resulting attribute. This does not seem like a likely scenario, but it's a change.
The rules for attribute IDs are tightened in this release to preclude whitespace and single quotes. With advance warning this set may be tightened further if warranted. We strongly advise sticking to simple ASCII characters.
The class hierarchy around IdPAttributeValue has been simplified and the use of generics eliminated. Some scripts may be impacted by changes to method names. There is no longer a universal getValue method exposed, but getNativeValue generally returns the equivalent result save for scoped values, where a Pair is returned. This change prevents unintentional "lossy" conversions to String for non-String values.
The removal of the deprecated
<SourceAttribute> element from the TemplateAttributeDefinition syntax creates a rare scenario that can break existing behavior in a small set of cases if an InputDataConnector is supplied with the
allAttributes option set. Using this syntax requires that all inputs either have the same number of values (or no values). Instead of using the
<SourceAttribute> element in conjunction with
allAttributes, simply enumerate the attributes to be sourced from the connector.
Due to additional "up front" processing of some SAML metadata content, certain invalid SAML constructs that used to be ignored are no longer being tolerated during the initial load of a MetadataProvider, in particular the appearance of empty XML elements such as
<md:ServiceDescription/>, and all the child elements within the
Such elements are invalid according to the standard and the IdP's behavior is correct, but the initial release has a bug that results in a NullPointerException instead of ignoring the element. For the time being, the only mitigation for metadata that can't be corrected is to mark the offending MetadataProvider with failFastInitialization="false" to allow startup of the service to continue. A future patch will resolve the issue, but deployers should never rely on the IdP's tolerance of invalid SAML constructs.
Due to a Spring bug that causes problems with configuration reloading, we have deprecated the original set of HttpClient parent beans and any inheritence from those beans in custom HttpClient configurations. We have introduced new parent beans without predefined settings that were causing the problems, and adjusted the HttpClientConfiguration documentation in various ways to refer to them.
Internal components that rely on a default HttpClient configuration now rely on a system bean definition intended only for internal use, configured as before via the services.properties file.
If you have beans that are inheriting from the deprecated versions, a warning will be issued. They will be removed from V5. This extends to several properties that were used to configure the caching versions of the clients, which will be removed.
The IdP includes a new Cross-Site Request Forgery (CSRF) Protection feature that is enabled by default for new installs. The idp.csrf.enabled property is a general on/off switch if problems are encountered or upgraders wish to enable it. The feature can also be easily incorporated in custom webflow views with minimal effort.
Mostly of interest to packagers, the installer is now built with Java and has explicit extension points. See Programming the V4 Installer.
Noteworthy New Features
- The aforementioned new AttributeRegistryConfiguration and Cross-Site Request Forgery (CSRF) Protection features
- Producing encodeable attributes directly from DataConnectors with the
- More flexible PasswordAuthnConfiguration allowing back-ends to be combined JAAS-style
- SAML proxy login flow (SAMLAuthnConfiguration) and various related features
- shibboleth.authn.SAML.* objects
- c14n/SAML2ProxyTransform c14n flow
- shibboleth.authn.Password.Validators list and related validators
- Many objects connected to AttributeRegistryConfiguration