There's a grab-bag of smaller topics of general interest this month.
If you missed the IAM Online last month, the slides and a recording are available at https://www.incommon.org/iamonline/
An IdP patch was released of relatively minimal criticality unless you're in the process of or planning to move to Java 11, in which case you will likely be impacted by an LDAP bug that's not been fixed to this point, and for which we have documented a workaround. The new patch simplifies the workaround by including the necessary libraries out of the box.
We've published example configuration files in the form of a downloadable starter kit for using Jetty 9.4, the most current release, designed for use on either Java 8 or 11. Jetty 9.3 is probably going to reach its end of life in the relatively near future so anybody still using it (which included myself until early this year) is certainly advised to look at an upgrade soon. I found the process very painless.
On a further related topic, please stay tuned for some imminent documentation on our plans for supporting OpenJDK 8 and 11 as the expected transition away from Oracle starts to pick up steam. As a preview, I can say with some confidence that we are leaning towards Amazon Coretto as a solution of choice for mult-platform support, particularly on Windows, plus support for Red Hat's version. Our plan is to establish a matrix of versions to continuously test that will include both officially supported versions and important alternatives (Oracle's included). This expanded testing will necessitate a revamping of our CI infrastructure, and Internet2 has made access to AWS available so we can offload this testing from our primary project server.
On the development front now, we have a number of feature branches under active development along with ongoing investigatory and design work into various topics. A new feature for generalized encoding and decoding of attribute information is nearing completion and will at least supplement, if not replace, the use of AttributeEncoders. The new design will support more configuration flexibility as we add more protocols to the system. It should reduce the need for unrelated content in the already-complex AttributeResolver configuration. Most particularly, it will allow for more natural ways to configure inbound processing of data from SAML, OIDC, or CAS proxy authentication even when that data isn't being passed along for outbound use. A side effect should be faster processing of configuration lookup based on SAML metadata EntityAttribute extensions (so-called "tags") by allowing tags to be decoded into internally optimized hashmaps instead of repeated linear searching of XML objects in memory.
Finally, note that Red Hat 8 was announced today. A CentOS 8 release is probably weeks away at best, so don't expect much on the SP packaging front there for a while.
We have an IAM Online on April 10th providing an update on the Consortium and an overview of work completed in 2018 and the roadmap covering roughly 2019 through probably the early part of 2021. This will mainly cover our expected scope of work for IdP V4 and V5 (the former already in development), with each planned for roughly a 12 month development and release cycle, faster than our major upgrades tend to be. This is a reflection of the stability of the V3 design, allowing us to make more targeted changes to the software that may necessitate a major version change but won't make upgrades a complex process. We hope simpler upgrades are the norm now rather than an exception.
Most of the basic outline of the roadmap has been mentioned in the course of previous blog entries:
- V4 will be primarily a release focused on technical debt, moving to Java 11 and Spring 5 and removing a lot of deprecated V2 features.
- It will also include new design work to allow development of new proxying features for at least OIDC and possibly some limited SAML login flows. This will allow some degree of delegation of authentication to other IdPs, much as is supported by a lot of other SAML software today. This will be a first attempt, so is likely to have rough spots, but represents the direction of a lot of future work.
- V4 may or may not end up including an integrated version of the GEANT-funded OIDC extension for V3.4. The first official release of this extension was announced recently. The more testing and feedback they get on that release, the more likely it might be considered for inclusion, but if it slips, there will be an updated version compatible with V4 in any case.
- V5 will be focused on polishing the new work integrated into V4, allowing us to adjust APIs where need be. It will probably include more complete SAML proxying support, and will almost certainly include OIDC OP support natively if it isn't incorporated into V4.
- We also have some backlogged work on improving the installation process, so it's possible that will be slotted into V5 as well.
If you have questions or input to give, please attend the IAM Online or send something to the development list.
We have an IdP V3.4 patch coming this month (hopefully) to address some bugs and add some additional warnings to some features that might be removed from V4 if we decide they're simple enough to remediate for deployers. This release will ship with defaults supporting the UnboundID LDAP provider to work around a Java bug people are just starting to run into as they move past Java 8, and by including the necessary jar files, it will be simpler for existing deployments to switch over to it as required.
I've started actively working on new designs for handling the mapping of data between the IdP, SAML, and OIDC to begin to lay the groundwork for the proxying support. There's not much to evaluate yet but the intent is to factor out the current AttributeEncoder support in the AttributeResolver and create a more general omni-directional mapping facility that can be configured by, as much as possible, "shared" definition files for common attribute mappings in different protocols, while still allowing customization where required. It isn't necessarily a given that the older AttributeEncoder approach will be deprecated, it depends on the maintenance burden of keeping it. But it will definitely be historical at best, or a more convenient way to support less common per-SP attribute encoding rules that may be more convenient to express that way.
There will also be code released in the relatively near future that implements client support for Amazon's RedShift JDBC and ODBC database drivers that works with the ECP support in Shibboleth or other SAML-compliant IdPs. There are some challenges federating database clients, but it does functionally work and Amazon approved the release of the code that references a couple of their Java interfaces, so we will be posting it in some form, though not officially supporting it. If you're interested, stay tuned.
The latest SP advisory leads off this month's update, of course. Not super serious, but deeply frustrating to continue to have issues with uncaught exceptions. This time I did a complete survey of the Xerces code base to locate all the non-inherited exception types and added catches for all of them to the outer stack frame that's supposed to catch them. Hopefully that will hold for longer than a couple of months this time. Worth noting for those running very critical systems and that don't want to implement process monitoring, you can add a
catchAll property to the OutOfProcess element in the configuration to force a trap of all exceptions raised, though that can have destablizing effects if errors occur that leave the system in an unknown state.
Work on the next generation of the IdP continues amongst our recently expanded team, and we'll be firming up our scope for this release in time for an IAM Online webinar next month.
There's still not much in the way of clarity around open Java 11 sourcing in a few months time when Oracle drops out, but we're at least putting the code through its paces now, and I've been running in production at OSU with Red Hat's version for a month or so. It seems likely that their version may be a logical choice for a lot of people, particularly on RHEL/CentOS. I've also been running successfully with UnboundID for LDAP and it's very easy to swap that in to avoid the bug I noted last month.
We have made significant progress on the conversion of Date/Time classes to Java's new (since 8) Time libraries, and this includes revising many of our public APIs to make use of the Instant and Duration Java types to better express intent. This will create some compatibility issues for extensions, but nothing too time consuming to fix in most cases, and we don't expect to see major impact for deployers outside of a couple of features. This is an important modernization step for the code base.
Work continues on the removal of deprecated code and features, which is important to complete before we begin work on new features. Parallel work streams continue on some of the items I highlighted last month, and we expect to see the final release of the GEANT OIDC extension for V3 this month so we can start evaluating the best timeline for incorporating that work.
The next project for me is probably going to be a revamping of the internal handling of EntityAttributes in metadata, to implement more efficient caching and matching data structures. So much of the system is tag-driven that it's important to optimize this and get away from the repeated linear searches we're doing and replace them with hashed/map lookups.
The SP and legacy IdP branch have stablized so V4 development has started in earnest. Deprecated code is being actively removed from all the projects and that work should be mostly complete this month. We've completed two major development tasks: moving the projects up to Spring 5 and Java 11, and replacing the Guava Predicate/Function interfaces throughout the code base with the standard Java 8 functional interfaces.
V4 will be built with and require Java 11, but we're still evaluating exactly what versions of Java 11 to officially support or recommend. There may or may not be reliably "pure" distributions of "free for use" Java 11 after September, and we expect to end up officially testing and certifying a (very) limited number of possible implementations with the rest left to the "probably works" category that OpenJDK used to live in. All of this is subject to change, and there will probably be some additional information available during Spring or Summer. Of course, feedback from members will have to be taken into consideration.
Java 8+ includes a number of new standard features that have replaced, or will replace, various proprietary libraries. The first conversion done was the Predicate/Function mechanism that is heavily used across the code, but also by deployers in various ways to customize behavior. For the most part, existing scripts and plugins will work compatibly after the conversion, with some small issues that should lead to warnings in V4 before causing any breakage in future versions.
We will also be migrating the Date/Time classes and interfaces to Java's standard classes, and this is likely to lead to a small number of incompatibilities, but the details will be clearer once that work is done.
Other open investigations and projects underway:
- Evaluating Velocity alternatives for possible use as a replacement, principally FreeMarker since it remains officially supported by Spring. Velocity will be supported for the time being, but we have to maintain that supporting code so moving away from it is a possibility, though not an urgent need.
- Work continues to clean up an OIDC authentication module, which will lead into more significant design work on proxy features over the summer.
- Discussions about how/when/who to incorporate the V3 OIDC extension. It is not officially targeted for V4 but remains a possibility.
A revised dependency management process is being worked on to avoid the need to privately host all of the dependencies of the codebase by pinning artifacts with PGP keys and/or checksums, and extending a Maven plugin to verify our dependencies at build-time. We continue to seek the best way to keep the build process safe from the attacks that standard Maven projects leave themselves open to and that have caused huge security vulnerabilities in other non-Java platforms, demonstrating our paranoia was well founded.
Finally, a regression was identified in the Java 9+ JNDI LDAP provider that causes connection leaks and/or crashes. So far the regression hasn't appeared in Java 8 but does affect all later versions. The bug has been reported to Oracle but so far the only workaround appears to be switching to an alternate LDAP provider. The ldaptive library has been heavily used by its authors with the UnboundID LDAP client, so we are tenatively planning to move to it as a V4 default, as JNDI has been a frequent source of bugs. We also have documentation prepared describing how to do this with V3. We may, if we decided it's warranted and safe, decide to ship a V3 patch with this change, but it depends on what Oracle does with the bug in the next couple of months.
We were unusually busy in December handling a few different security issues and patches and a few work items.
The SP received a long gestating update to add support for OpenSSL 1.1.1 that turned into a security release at the last minute due to a regression introduced in V3. We haven't done significant testing yet, but these OpenSSL and curl updates theoretically add support for TLS 1.3 for the first time. I've done a bit of investigation into the possibility of HTTP/2 support and it turns out that curl doesn't really support it natively, it's only supported through the addition of another library. I haven't seriously entertained the thought of adding that to the mix; I don't think there's sufficient benefit from doing so at this point. The advantages of HTTP/2 don't really apply to the SP's use cases for outbound HTTP traffic, and technically it isn't even allowed by SAML's SOAP 1.1 binding. The TLS 1.3 support is more relevant long term.
The IdP has continued to receive patches due to a combination of security issues and regressions, most of them due to a late decision to deprecate some syntax we probably should have left alone, but c'est la vie. Another patch is coming this week to fix two more regressions identified by deployers.
I completed most of the planned Logout enhancements, and the changes are largely if not completely backward-compatible with the existing view templates, which was unexpected. Since these are V4 features, we don't have updated documentation yet, but there are essentially just some new messages and a property or two that control the sequence of events and the choices presented to the user during logout, and it's now possible to preempt the entire operation if you want to give the user that option, which was, oddly, requested by a number of people.
We haven't seriously opened up the V4 branches for larger work yet, but that's imminent, probably starting with the move to Java 11 as our development baseline. A feature branch was pushed containing an OIDC login flow and I'll probably merge a version of that pretty soon. I was greatly disappointed, though not surprised, to learn how little actual support for OIDC exists among the major social identity sources but we're not going to get into the business of supporting one-off protocols in our core code. I would anticipate us producing unsupported examples as add-ons to demonstrate the non-standard ones that are of interest.
I don't have a real update regarding the broader OIDC support, but if it's not already been finalized, I expect the add-on will be getting a 1.0 release soon.
Mostly a follow up from November's update...
We're ready to ship a small SP patch as soon as curl ships this month, which should be this week, so expect the patch late this week and I'll be circulating advance notice of that to announce today along with the early announcement that we'll be releasing a second IdP 3.4 patch update probably next week to address a CAS security issue and some miscellaneous cleanup work around warnings and deprecations.
After some discussion amongst the team and the Board, most of the consensus settled around aiming to produce a relatively scaled back V4 release by the end of 2019 rather than start a multi-year effort again. The exact scope of that release will depend on what priorities people bring to the table and on what we can manage to finish in that time frame.
I've started work on finishing as much of the undone logout enhancements as possible by the end of the year, and progress has been pretty good. SOAP-based logout is now working and I've started on the UI enhancements requested to provide more control over the sequence of events during logout. The former took more time than expected and the latter looks like it will probably take less.
We're also working on pulling in a third party extension to support OIDC authentication to the IdP as the very early steps of looking at proxying functionality.
Since last month we've released IdP V3.4.0 and the first patch for it once the initial round of bug reports came in. We have one or two remaining annoyances around deprecation warnings outstanding and the ReleaseNotes are being kept up to date to provide the most accurate information we have on what to watch out for. We can't do patches every week obviously but as these things accumulate there will be additional patches coming to get things cleaned up. Overall the release seems to be solid and I've had it in production for a couple of weeks now.
The GEANT OIDC extensions is now in beta, with an improved installation process, so please check it out and provide feedback if you're interested in that.
In parallel with the occasional maintenance commit, we've branched the source tree so the master branches are open for development on the V4 code base. All commits are now intended to be on those master branches, with only specific authorized commits backported to the 3.4 maintenance branch.
The master branches have been moved to a new parent POM based on Java 8 and Spring 5 as the first "step forward" and all but a couple of tests are passing and the IdP seems to be functional, so that's a good sign that we won't have too much remediation needed. Eventually this will be moved to Java 11, but right now the Maven toolchain isn't stable on Java 11, so we're on Java 8 for now.
We also have a buildable artifact containing a curated and well-organized Jetty 9.4 configuration that works with the IdP with minimal adjustments and will probably make that available in some capacity, with an eye towards including it in V4 to help bootstrap new installs (assuming 9.4 is a current release at that point in time).
What isn't really decided at this stage is what the schedule and scope of V4 will be, with the most likely options being a 12 month development cycle with minimal feature additions vs. a longer cycle of perhaps 24 months to accomodate more new features. Both options are possible and make some sense, and hopefully we'll be able to engage people on this question once we have some initial discussions with the Board about it. Regardless of the decision, we don't expect the upgrade process to V4 to be onerous, though we can definitely promise there won't be a supported path to upgrade from V2 directly to V4.
On the SP front, we are in the process of getting a patch out the door, but the latest curl release (7.62.0) was a big one that has a number of regressions, so right now we're probably waiting another month to include the next version rather than ship this one, but that could change depending on bugs reported and the urgency to get an update done.
As I draft this, IdP V3.4 is tagged and will be announced tomorrow, just shy of two years (!) since the last significant feature release. The last month saw a large number of commits from the whole team to finish up late-arriving features and complete internal testing for the release. Pretty much everything originally planned for this release made it in (along with a lot more) except for a significant redesign of the logout functionality, in keeping with the project's long tradition of never getting logout work done on time.
As I outlined before, the list of new features is extensive. This will be the last feature update before V4 when we make the move to Java 11 and Spring 5. This release provides a good window to freshen and optimize configurations and get rid of legacy options and cruft that will be removed from V4.
Once this release is complete and we take a breath or two, work will begin soon on initial scoping of V4. I would like to work on some opportunities for direct interaction with the Consortium Members on what people would like to see from that work, aside from the long list of issues we already have open. One obvious task for the Board will be to have the necessary discussions on the viability of incorporating the OIDC extension being developed for V3.4, which is mostly a question of long term code ownership and maintenance.
Activity on the SP bug front is still very low, but we have a couple of possible issues to get cleaned up and then we can get a new build that supports OpenSSL 1.1.1 out the door in time for the end of the year. This may be an opportune time, with another Java developer coming on board to help out on the IdP, for me to spend a bit of time looking at the viability of an NGINX SP module.
I'll be at Internet2 Tech Exchange and ACAMP next week in Orlando, so I'll see many of you there.
The last month has been spent ramping IdP development back up and preparing for a release of V3.4 in October. The number of bug fixes and features is quite large (pushing 150+), so this is a pretty big new release that is intended to bridge to V4 as the next big update. There are a couple of things that haven't made the cut, primarily redesigning the logout behavior in response to a lot of feedback. Both time constraints and the need to maintain compatibility made it appealing to delay that work until V4 when we have more freedom to change the UI in ways that might require a small bit of rework for deployers instead of guaranteeing drop-in compatibility.
In the meantime, there should be enough to keep people busy with V3.4 for a while:
- A nice HTTPConnector for web services integrations (I built it so I'm biased but I've been using it for Grouper and AWS integration projects and it's been really, really convenient to use)
- Built-in wiring and convenience functions that make it possible to drive a ton of configuration options using metadata "tags"
- Non-browser Duo AuthAPI support
- Improvements to the deployability of the DynamicHTTPMetadataProvider and LocalDynamicMetadataProvider plugins
- Improvements for embedding scriptlets in a variety of configuration scenarios
- A new "attended startup" mode that prevents on-disk access to an unlocked or trivially encrypted private key
- The ability to provision CAS services using SAML metadata for consistency and to support the new metadata-driven configuration mechanisms
- Added support for SAML proxying constructs to detect/react to proxied SPs and to express upstream proxying to SPs
- Greatly enhanced context-check interceptor that can handle multiple scenarios at the same time
- A new impersonation interceptor that supports advanced debugging or testing scenarios
- Some small features to support the drop-in addition of the under-development GEANT OIDC extension
And more, but that's plenty for now.
We'll be wrapping up work this month, testing over the next couple of weeks, and should be freezing around the end of the month with a release in time for TechExchange in Orlando.
We're also working on a more modular and minimal configuration for Jetty 9.4 that will form the basis of not only the Windows "quick install" package but a maintained/curated Jetty configuration to make deployment simpler on other platforms for people without experience with Jetty. We're not planning to fully "embed" Jetty because that's nearly impossible to maintain over time for mature use as a production web server, but it should be a good starting point for most people.
On the SP front, things have been pretty quiet, so one presumes either upgrades are going reasonably smoothly or people are stuck on software with some known security bugs, but I'll choose to assume the former. There will most likely be a patch update later this year if only because OpenSSL 1.1.1 is now out, and 1.1.0 will be EOL soon.
Both OpenSSL and Jetty are introducing the first iterations of TLS 1.3, which is different enough from TLS 1.2 that it will require some time to digest (pun intended). Hopefully if there are some adjustments needed the less frequent use of the back-channel in the SP should soften the impact.
I skipped an update in July due to the workload of getting the SP finished, which has been dominating the work for least a couple of us most of the last few months, and I'm happy to note we finally shipped 3.0 (and 3.0.1 and 3.0.2). Given the magnitude of the internal changes, I'm reasonably happy with the results so far as the bugs have been easy to fix, and the release process is now more streamlined than before due to huge improvements in the Windows build process and the elimination of Solaris as a supported platform. That means patches come more quickly. Most people I think will find the upgrade to be painless, which is good since we very quickly hit a security issue that impacts all the older versions.
The documentation is about 50% reviewed and cleaned up at this point, better than I hoped but not as good as I would like. It's at least better organized and out from under the old IdP material.
In between patches work has picked back up on the backlog of IdP issues and features. I finished up and started documenting a new feature for limiting access to keys by requiring attended startup of the service, which is a weird concept but is about the only way I can see to protect private keys in a cost effective way. That was done on OSU time donated to the project so we can make use of the feature.
I'm starting work on extending the Duo login flow to include support for non-browser clients using work donated by UMBC and it should hopefully require minimal effort by existing Duo deployers to enable.
The IdP is now leapfrogged again by the SP in terms of some of the dynamic metadata features, so we're circling back to add some of those features into the IdP now that some more deployment experience is available.
We're also finishing up work on a stable configuration set for Jetty 9.4 and are discussing how to make that available to deployers. The embedded version for Windows will be updated to that version and we would like to make deployment on other platforms easier by delivering a customizable starting point that will mostly work out of the box and then be simple to extend. Hopefully we can help people stay current by shipping base configuration sets that can be used to jump start upgrades of local container installs without actually taking on responsibility for the installs ourselves.
The Java platform situation remains very fluid but we now know from Oracle's public statements that their JDK is about to become a for-fee product with Java 11 and that free use will require use of OpenJDK. It is unknown whether there will be free options for OpenJDK support that last longer than the 6 months Oracle will be patching each version of OpenJDK, so there may not be anything like a "long term" Java version that we can rely on for future products. At minimum, this means we have to revisit our recommendations for JDK usage, and at worst it may mean adapting to a much more rapid development cycle to keep up and maintain compatibility. That in turn means more time and money devoted to testing and maintenance.
The SP V3 work has reached the beta stage, and we really could use some testing. A 3.0.1 is probably inevitable but there's always hope.Windows packages can be downloaded from our site, and RPMs are available from a test repository. Neither is guaranteed to be directly upgradable to the final release, so please act accordingly. The RPMs get rebuilt occasionally so they're not formally a beta, more like snapshots.
The new documentation space for the SP is readable now though far from complete. More short term work is needed to get it in shape for the release but it's pretty close, and should eventually be much more readable. Eventually we hope to add more examples and the sorts of explanations of things that the IdP documentation has. Most of the new features are not yet documented, but we'll aim for getting that done in the next couple of weeks. A final round of improvements has been made to the just-in-time metadata support and some of that will hopefully make their way back into IdP V3.4 now that we have some practical experience.
We will begin to get some of the dependencies released very shortly and the release itself is probably just 2-3 weeks out if we don't get any bug reports from testers. It would be nice to get a second mirror online if possible beforehand so please contact us if you can do that.
IdP work has continued off and on at a slow pace but is cranking up again and should pick back up in mid-July. A high priority will be to get the changes requested by the OIDC extension developers into the API so they can get back to work on it after their summer break, and then I think we will start to get pretty ruthless with identifying what else we actually have the time to do. I would like to get V3.4 shipped by the end of Q3 at the latest. We should have a reference configuration for Jetty 9.4 to include with that release, and the Windows installer will be updated to that version.
An item that just came to our attention is - JPAR-121Getting issue details... STATUS , and could have major implications for deployers. It's a good example of the need to keep devoting resources to simply staying aware of what's going on with the platforms.
As we previewed last week, we have a security patch (V3.3.3) for the IdP coming on Wednesday that impacts CAS deployments and includes a Spring security fix that might affect a very small percentage of deployers on Windows (realistically none, perhaps, but we're erring on the side of caution). The main purpose of the patch is the CAS issue. This is one of the first times we've provided advance notice publically of a security fix, and that seems to be common practice for widely used software these days. Obviously the full details will be available with the patch's release.
We have been finalizing a new project position statement on our plans for supporting the Java platform long term, which you can find under Product Platforms. This addresses a few questions we've gotten lately about support for current and future Java versions, and largely clarifies that we believe our focus should be on supporting so-called Long Term Support (LTS) releases, currently Java 8, and Java 11 later this year. Feedback on this policy proposal can be provided in any of our many communication channels. Uttimately we want and need to support what you need, but we think most people haven't yet really digested what's going on with Java so we're hoping to provide a clear statement to help with any compatibility questions going forward.
Development on SP V3.0 is nearing completion and we have begun a round of internal testing, after which we will provide a beta release once a pass over the documentation is done and basic notes on upgrading are drafted. Some polishing work is left to do on the enhanced Dynamic metadata features but a feature freeze is very close.
The last development update included a planned change to the default configuration filename that has since been rescinded for the final release. Upgrades, particularly via RPM, are much cleaner and more reliable if the shibboleth2.xml filename is left alone, so we're prepared to live with the fallout of leaving it.
A number of planned changes to default settings will be compiled fairly soon and circulated for feedback, particularly in the area of SAML 1.1 and attribute query support, which increasingly create confusion, but final decisions will be open to discussion.
Significant improvements in this upgrade include:
- A newly implemented IIS7+ native module to replace the creaky ISAPI module, including support for REMOTE_USER and non-header Server Variables.
- Stateless clustering, though with even more limitations on logout.
- Substantial fixes and improvements to the Dynamic metadata support, including a LocalDynamic option for consuming IdP metadata fragments from the local file system (the IdP has a similar feature).
- Elimination of Application Overrides for trivial "call the SP something different per virtual host" use cases.
- An experimental feature to define and reference Application Overrides via external files that can be added at runtime.
- Sensible logging defaults for Apache/IIS modules relying on the Event Log (Windows) or syslog (everything else), addressing a range of problems with file permissions and log rotation.
- Updated XML libraries with substantial security and maintenance improvements. Reduces surface area will hopefully prevent certain kinds of likely security bugs from impacting the project.
- An improved build process on Windows, making the delivery of library patches less labor intensive.
- Dropping Solaris support, saving the project an average of $1500-$3000 testing time per release, without factoring in support costs. It adds up.
- Many small and not-so-small bug fixes ( ) Getting issues...
More on a lot of this as the documentation gets cleaned up. Needless to say, the upgrade got bigger as work progressed, and we think this will be a worthy upgrade, with the added bonus that it will be safe and easy to apply to existing systems.
ETA at this point is probably late June depending on testing and feedback. The sooner we get this out, the sooner we can turn full attention back to getting IdP V3.4 completed by this fall.
The last few weeks have been dominated by the recent cross-industry SAML vulnerability discovered by a Duo Security researcher, a lot of research into the impact of Java 9, 10, and 11 on our dependencies and build process, and continued work on the SP 3.0 features and documentation.
The vulnerability has dominated a lot of conversation of late. It was embargoed for about a month and I've been dropping various hints about it, mostly by alluding to the previous vulnerability that we fixed in January because it's actually almost an identical bug, at least in our SP. Once the new issue came to light, the importance of XML Encryption became very evident, since it mitigated both the original and the new class of vulnerabilities, and the fallout in pure Shibboleth deployments was much less as a result.
The real fallout from this is with cloud integrations, both because they are the most likely to forgo encryption, but also because there are so many different, and often substandard, SAML implementations in use. This is the reason we're so vocal as a project in advising people not to even consider building their own code on top of OpenSAML. We don't have the resources to document it adequately, and there are very few developers with the skills necessary. I say that with full self-awareness of the fact that we were vulnerable; that reinforces the point rather than weakens it.
Anyway, a lot of the time on this has been the fallout and considering how to approach the problem of finding and reporting all these vulnerable systems to companies who simply don't have the resources to be supporting the SAML stacks they now have to fix. We'll be cleaning this up for months, if not years.
Since people seem to take blogs so seriously, I'll say a couple of things that might help with some vendor conversations: This is an SP bug, period, and there is no justification whatsoever to point fingers at the IdP or claim that the IdP can fix this. There are some obscure things an IdP can do to make the bug hard or impossible to exploit, but that's a workaround, just like XML Encryption is. It's a bug. Fix the bug. Secondly, do not think that because your SAML code doesn't use one of the libraries that was explicitly noted as being vulnerable that you're safe. That is not a list of the affected libraries, it's a list of lbraries known to be affected. Others are affected. Other libraries are not broken per se, but have been misused by developers so as to create a vulnerability (this is particularly true of the Java version of OpenSAML).
A beneficial side effect of this has been to force us to invest the time necessary to come up with a package distribution mechanism that bypasses the broken opensuse.org mirroring infrastructure. Simply put, they allow sites to mirror without valldating the mirrors actually work, and keep serving up broken links using a server-side redirector that makes bypassing the mirrors impossible. Things were bad enough this time that we were forced to sort of insist that one of our members with the necessary infrastructure step up to deal with this, and Jisc agreed to put a long term solution into place this summer. In the meantime, we have put out a call for volunteers to host the files, and we have a "final" Yum repository builder in place that will de-reference the mirrors to choose from properly, on the client side, so these repo files should keep working in the future as we add new mirrors. Please update your servers to use these fresh repository files.
On a lighter note...
The impending release of Java 10, and soon 11, has managed to break a ton of Maven plugins and we've been spending significant time ramping up full matrix testing, identifying incompatibilities, and waiting for new versions of the plugins so the build and deploy process can be tested again. The expected pace of Java releases that introduce breaking changes is going to be a major problem for a lot of projects, and we have a lot of dependencies that, unlike Spring, are less well-maintained. We're going to have to divert even more time than in the past to keeping things working, and it's likely that there will be less of a guarantee that our major releases will continue to work on the non-LTS (long term support) versions of Java. We can only do so much, and as an "enterprise" platform, we need to focus our time on making sure the LTS releases work and that we have versions released to support those Java platforms in time. That means, practically speaking, that Java 9 and 10 matter a lot less than Java 8 and Java 11, and that we have to make sure V3.4 works on Java 11 since it supersedes Java 8 this fall. Expect some kind of official policy about this when we have more time to study the issues.
On the SP front, we should be in the home-stretch in getting the SP out the door within the next couple of months at most. The build process on Windows is now fairly streamlined and scriptable, even down to our dependencies. Repeatable processes are always nice. We have a basically "final" plan for the configuration, which is that we have built support for a new V3 configuration schema that is essentially identical to the V2 schema with some deprecated features removed, but the SP will load either one. This guarantees safe upgrades but provides the right warnings to get old settings cleaned up in case we ever do a V4. We have renamed the default file to shibboleth3.xml but will fall back to the old filename, so the whole thing is pretty seamless.
I also completed work on a cookie-based session recovery feature that allows most of the important data in a session to migrate between server nodes or across restarts using a shared key. It should alleviate most of the need for back-end session store sharing, and the cookie sizes don't seem too risky in most cases.
Finally, another highlighting of the OIDC work, which has reached alpha 2 and needs testing and feedback: https://marc.info/?l=shibboleth-dev&m=152111752828021&w=2
Most of our practical progress of late has been centered on the SP and the Metadata Aggregator, the latter made possible with some of the increased funding obtained over the last several months. The significance of the latter work is that the MA will be based on a newer Java version and library set that will lay the foundation for moving the IdP to the same (or newer) versions in V4.0, saving us a lot of the up front legwork to identify compatibility issues and changes we need to make.
The IdP work continues in the background; probably the most serious question is whether to hold the V3.4 release to wait for a significant revamp of the logout support, or ship without it and push that to V4.0 or a V3.5 interim update. In most other respects we're close to having a shippable release with some useful new features, though lots of small improvements and features remain in the queue.
On the SP side, we're making progress building out a new documentation space and have started working on a way to adapt the system to handle both a new and old configuration schema. The intention is for essentially complete com patibility with V2 configuration, but to allow for a newer format that will contain mostly small improvements such as removal of deprecated settings and possibly some long-asked for improvements like compositing the configuration from multiple sources. We have also started looking seriously at building in a client-side session migration feature that would simplify clustering in a similar fashion to the IdP's support for client-side sessions.
Our recent spate of security issues with the SP continued last month with a serious XML vulnerability in an area that we had correctly guessed would be a source of ongoing concerns (without having identified the specific issue discovered, unfortunately). Our defensive measures over the last couple of years prevented more widespread impact, but Red Hat's refusal to backport various security fixes and defensive code changes to their version of the Xerces library eventually caught up with us and led to the most serious impact from this bug on RHEL 7 and CentOS 7.
One of the takeaways from this issue was the recognition that XML Encryption is really not an optional feature anymore when it comes to use of the POST binding that most SAML deployments have stuck with. Even without data confidentiality concerns, encrypting the assertion mitigates or prevents attacks against the XML content that is of most importance to applications (i.e., the user's identity and attributes), both attacks we know about and attacks we don't yet know about. This will likely be reflected in SP features to require encryption, and a more aggressive stance on moving systems to the AES-GCM encryption algorithm, which fixes serious flaws that current deployments remain vulnerable to many years after the original discovery that CBC mode was broken. Certainly it will be a V4.0 default.
Expect more discussion of this important issue over the course of this year. Also note that this is really almost exclusively an IdP to SP concern. While the IdP supports decryption in some isolated cases, mainly logout, they are not common use cases, and there is very little deployment of SP to IdP encryption out there today. We don't expect that to change.
We made progress over the holidays on a couple of Service Provider work items, reimplementing the on-demand metadata implementation to align with the IdP software and finishing up a revision of the xml-security library with OpenSSL 1.1 support. A release candidate build will be available this week and the final release will probably be this month. We have begun moving cleaned up SP documentation into a dedicated wiki space for the eventual 3.0 release. We're beginning to work through some of the issues associated with producing a new version because of the tension between removing some deprecated settings and guaranteeing that upgrades are safe.
We've largely finished implementing deprecation warnings in the IdP and will be producing a list of features scheduled for possible removal in V4.0 so that federations can begin to publicize the changes ahead of the actual release of V3.4.
The GÉANT OIDC team posted regarding the availability of an alpha release of their Shibboleth OIDC extension, and are targeting mid-March for another update. They would welcome testing and input.
We hope everybody's new year is off to a good start and are looking forward to a productive 2018. Expect the next SP and IdP updates during the first half of the year.