As we previewed last week, we have a security patch (V3.3.3) for the IdP coming on Wednesday that impacts CAS deployments and includes a Spring security fix that might affect a very small percentage of deployers on Windows (realistically none, perhaps, but we're erring on the side of caution). The main purpose of the patch is the CAS issue. This is one of the first times we've provided advance notice publically of a security fix, and that seems to be common practice for widely used software these days. Obviously the full details will be available with the patch's release.
We have been finalizing a new project position statement on our plans for supporting the Java platform long term, which you can find under Product Platforms. This addresses a few questions we've gotten lately about support for current and future Java versions, and largely clarifies that we believe our focus should be on supporting so-called Long Term Support (LTS) releases, currently Java 8, and Java 11 later this year. Feedback on this policy proposal can be provided in any of our many communication channels. Uttimately we want and need to support what you need, but we think most people haven't yet really digested what's going on with Java so we're hoping to provide a clear statement to help with any compatibility questions going forward.
Development on SP V3.0 is nearing completion and we have begun a round of internal testing, after which we will provide a beta release once a pass over the documentation is done and basic notes on upgrading are drafted. Some polishing work is left to do on the enhanced Dynamic metadata features but a feature freeze is very close.
The last development update included a planned change to the default configuration filename that has since been rescinded for the final release. Upgrades, particularly via RPM, are much cleaner and more reliable if the shibboleth2.xml filename is left alone, so we're prepared to live with the fallout of leaving it.
A number of planned changes to default settings will be compiled fairly soon and circulated for feedback, particularly in the area of SAML 1.1 and attribute query support, which increasingly create confusion, but final decisions will be open to discussion.
Significant improvements in this upgrade include:
- A newly implemented IIS7+ native module to replace the creaky ISAPI module, including support for REMOTE_USER and non-header Server Variables.
- Stateless clustering, though with even more limitations on logout.
- Substantial fixes and improvements to the Dynamic metadata support, including a LocalDynamic option for consuming IdP metadata fragments from the local file system (the IdP has a similar feature).
- Elimination of Application Overrides for trivial "call the SP something different per virtual host" use cases.
- An experimental feature to define and reference Application Overrides via external files that can be added at runtime.
- Sensible logging defaults for Apache/IIS modules relying on the Event Log (Windows) or syslog (everything else), addressing a range of problems with file permissions and log rotation.
- Updated XML libraries with substantial security and maintenance improvements. Reduces surface area will hopefully prevent certain kinds of likely security bugs from impacting the project.
- An improved build process on Windows, making the delivery of library patches less labor intensive.
- Dropping Solaris support, saving the project an average of $1500-$3000 testing time per release, without factoring in support costs. It adds up.
- Many small and not-so-small bug fixes ( ) Getting issues...
More on a lot of this as the documentation gets cleaned up. Needless to say, the upgrade got bigger as work progressed, and we think this will be a worthy upgrade, with the added bonus that it will be safe and easy to apply to existing systems.
ETA at this point is probably late June depending on testing and feedback. The sooner we get this out, the sooner we can turn full attention back to getting IdP V3.4 completed by this fall.
The last few weeks have been dominated by the recent cross-industry SAML vulnerability discovered by a Duo Security researcher, a lot of research into the impact of Java 9, 10, and 11 on our dependencies and build process, and continued work on the SP 3.0 features and documentation.
The vulnerability has dominated a lot of conversation of late. It was embargoed for about a month and I've been dropping various hints about it, mostly by alluding to the previous vulnerability that we fixed in January because it's actually almost an identical bug, at least in our SP. Once the new issue came to light, the importance of XML Encryption became very evident, since it mitigated both the original and the new class of vulnerabilities, and the fallout in pure Shibboleth deployments was much less as a result.
The real fallout from this is with cloud integrations, both because they are the most likely to forgo encryption, but also because there are so many different, and often substandard, SAML implementations in use. This is the reason we're so vocal as a project in advising people not to even consider building their own code on top of OpenSAML. We don't have the resources to document it adequately, and there are very few developers with the skills necessary. I say that with full self-awareness of the fact that we were vulnerable; that reinforces the point rather than weakens it.
Anyway, a lot of the time on this has been the fallout and considering how to approach the problem of finding and reporting all these vulnerable systems to companies who simply don't have the resources to be supporting the SAML stacks they now have to fix. We'll be cleaning this up for months, if not years.
Since people seem to take blogs so seriously, I'll say a couple of things that might help with some vendor conversations: This is an SP bug, period, and there is no justification whatsoever to point fingers at the IdP or claim that the IdP can fix this. There are some obscure things an IdP can do to make the bug hard or impossible to exploit, but that's a workaround, just like XML Encryption is. It's a bug. Fix the bug. Secondly, do not think that because your SAML code doesn't use one of the libraries that was explicitly noted as being vulnerable that you're safe. That is not a list of the affected libraries, it's a list of lbraries known to be affected. Others are affected. Other libraries are not broken per se, but have been misused by developers so as to create a vulnerability (this is particularly true of the Java version of OpenSAML).
A beneficial side effect of this has been to force us to invest the time necessary to come up with a package distribution mechanism that bypasses the broken opensuse.org mirroring infrastructure. Simply put, they allow sites to mirror without valldating the mirrors actually work, and keep serving up broken links using a server-side redirector that makes bypassing the mirrors impossible. Things were bad enough this time that we were forced to sort of insist that one of our members with the necessary infrastructure step up to deal with this, and Jisc agreed to put a long term solution into place this summer. In the meantime, we have put out a call for volunteers to host the files, and we have a "final" Yum repository builder in place that will de-reference the mirrors to choose from properly, on the client side, so these repo files should keep working in the future as we add new mirrors. Please update your servers to use these fresh repository files.
On a lighter note...
The impending release of Java 10, and soon 11, has managed to break a ton of Maven plugins and we've been spending significant time ramping up full matrix testing, identifying incompatibilities, and waiting for new versions of the plugins so the build and deploy process can be tested again. The expected pace of Java releases that introduce breaking changes is going to be a major problem for a lot of projects, and we have a lot of dependencies that, unlike Spring, are less well-maintained. We're going to have to divert even more time than in the past to keeping things working, and it's likely that there will be less of a guarantee that our major releases will continue to work on the non-LTS (long term support) versions of Java. We can only do so much, and as an "enterprise" platform, we need to focus our time on making sure the LTS releases work and that we have versions released to support those Java platforms in time. That means, practically speaking, that Java 9 and 10 matter a lot less than Java 8 and Java 11, and that we have to make sure V3.4 works on Java 11 since it supersedes Java 8 this fall. Expect some kind of official policy about this when we have more time to study the issues.
On the SP front, we should be in the home-stretch in getting the SP out the door within the next couple of months at most. The build process on Windows is now fairly streamlined and scriptable, even down to our dependencies. Repeatable processes are always nice. We have a basically "final" plan for the configuration, which is that we have built support for a new V3 configuration schema that is essentially identical to the V2 schema with some deprecated features removed, but the SP will load either one. This guarantees safe upgrades but provides the right warnings to get old settings cleaned up in case we ever do a V4. We have renamed the default file to shibboleth3.xml but will fall back to the old filename, so the whole thing is pretty seamless.
I also completed work on a cookie-based session recovery feature that allows most of the important data in a session to migrate between server nodes or across restarts using a shared key. It should alleviate most of the need for back-end session store sharing, and the cookie sizes don't seem too risky in most cases.
Finally, another highlighting of the OIDC work, which has reached alpha 2 and needs testing and feedback: https://marc.info/?l=shibboleth-dev&m=152111752828021&w=2
Most of our practical progress of late has been centered on the SP and the Metadata Aggregator, the latter made possible with some of the increased funding obtained over the last several months. The significance of the latter work is that the MA will be based on a newer Java version and library set that will lay the foundation for moving the IdP to the same (or newer) versions in V4.0, saving us a lot of the up front legwork to identify compatibility issues and changes we need to make.
The IdP work continues in the background; probably the most serious question is whether to hold the V3.4 release to wait for a significant revamp of the logout support, or ship without it and push that to V4.0 or a V3.5 interim update. In most other respects we're close to having a shippable release with some useful new features, though lots of small improvements and features remain in the queue.
On the SP side, we're making progress building out a new documentation space and have started working on a way to adapt the system to handle both a new and old configuration schema. The intention is for essentially complete com patibility with V2 configuration, but to allow for a newer format that will contain mostly small improvements such as removal of deprecated settings and possibly some long-asked for improvements like compositing the configuration from multiple sources. We have also started looking seriously at building in a client-side session migration feature that would simplify clustering in a similar fashion to the IdP's support for client-side sessions.
Our recent spate of security issues with the SP continued last month with a serious XML vulnerability in an area that we had correctly guessed would be a source of ongoing concerns (without having identified the specific issue discovered, unfortunately). Our defensive measures over the last couple of years prevented more widespread impact, but Red Hat's refusal to backport various security fixes and defensive code changes to their version of the Xerces library eventually caught up with us and led to the most serious impact from this bug on RHEL 7 and CentOS 7.
One of the takeaways from this issue was the recognition that XML Encryption is really not an optional feature anymore when it comes to use of the POST binding that most SAML deployments have stuck with. Even without data confidentiality concerns, encrypting the assertion mitigates or prevents attacks against the XML content that is of most importance to applications (i.e., the user's identity and attributes), both attacks we know about and attacks we don't yet know about. This will likely be reflected in SP features to require encryption, and a more aggressive stance on moving systems to the AES-GCM encryption algorithm, which fixes serious flaws that current deployments remain vulnerable to many years after the original discovery that CBC mode was broken. Certainly it will be a V4.0 default.
Expect more discussion of this important issue over the course of this year. Also note that this is really almost exclusively an IdP to SP concern. While the IdP supports decryption in some isolated cases, mainly logout, they are not common use cases, and there is very little deployment of SP to IdP encryption out there today. We don't expect that to change.
We made progress over the holidays on a couple of Service Provider work items, reimplementing the on-demand metadata implementation to align with the IdP software and finishing up a revision of the xml-security library with OpenSSL 1.1 support. A release candidate build will be available this week and the final release will probably be this month. We have begun moving cleaned up SP documentation into a dedicated wiki space for the eventual 3.0 release. We're beginning to work through some of the issues associated with producing a new version because of the tension between removing some deprecated settings and guaranteeing that upgrades are safe.
We've largely finished implementing deprecation warnings in the IdP and will be producing a list of features scheduled for possible removal in V4.0 so that federations can begin to publicize the changes ahead of the actual release of V3.4.
The GÉANT OIDC team posted regarding the availability of an alpha release of their Shibboleth OIDC extension, and are targeting mid-March for another update. They would welcome testing and input.
We hope everybody's new year is off to a good start and are looking forward to a productive 2018. Expect the next SP and IdP updates during the first half of the year.
The support rollout has been smooth so far with fairly light but substantive use. JIRA seems to be working well for managing this kind of support. Slack has seen little use so far. Membership continues to grow.
Work on IdP 3.4 has stalled a bit due to vacations and diverting of effort toward the SP. The redesign of the dynamic metadata support exposed a serious bug that appeared to be present all the way back to 2.0, and demonstrated a need to spend time expanding unit testing in the SP as we work on the new version.
The patch itself was simple, but we/I were unwise in including some additional bug fixes that had been applied to the master branch, and caused a regression which was fixed promptly, but reinforced our traditional conservatism in what we include in patches. It was a lesson all around in violating our norms.
Most of the work redesigning the SP metadata support is complete and ready to be tested, and focus has shifted to getting a clean, documented build process for the SP and all its dependencies while we work in parallel on a new version of the Apache XML Security library. With the opportunity for breaking changes, I decided to get more aggressive and remove a lot of deprecated code and redesign some old APIs since we'll likely not make this kind of change again. Given the pace of this work, I don't expect a 2.0 release of that library until after the new year.
As the work on the SP progresses, platform support continues to be discussed on-list and internally. It is a given that Solaris will be dropped as a supported platform and it makes sense to consider adding Debian and Ubuntu due to popularity, but this requires figuring out how to properly scope that support, given that we don't package for them. SUSE is essentially in the same situation now anyway.
Some internal testing at OSU revealed that the older Apache HTTP client library in the IdP contains some bugs that don't impact occasional use but become serious under load for use cases such as web services, so we need to accelerate an upgrade of that library in the next IdP release.
Oracle also clarified (somewhat) their plans for Java and announced that Java 9 would not be a "long term support" release, so the MDA 1.0 software will likely be based on Java 8 instead, with the IdP 4 work based on the 18.9 release in September 2018. All subject to change as usual.
The last month has been dominated by travel to TechExchange and getting the new support services rolled out for testing by the members. TechExchange was a bit less exhausting than the past couple of years, and was dominated by a lot of good conversation and discussion about OpenID Connect, including a good touch-base with the CSC team working on those features for Shibboleth. I'm hopeful there will be some initial bits to play around with by early next year.
Proxies were another dominant area of discussion, and after an offhand comment/question from a member, I went ahead and implemented some policy support for the
<Scoping> element in SAML that allows the IdP to recognize different downstream services on the other side of a proxying SP.
The support rollout has been relatively smooth, with very little use so far (the next Slack message will be the first). The best part has been the dramatic and ongoing increase in members as a result of this change.
On the more traditional development front, work has been progressing on:
- Re-designing the dynamic metadata support in the SP.
- Finishing the long-delayed SOAP client support in the IdP, though that's more of a "looking ahead to doing proxying support" thing than anything the IdP itself is desperate to include.
- Getting a Java 9-compatible build process working with Maven so we can start testing on it.
Going into the holiday season, I will be working to get an updated release of the XML Security library used by the SP, and probably doing a Xerces bug fix release out of the kindness of my heart, and we'll hopefully get a working set of examples done for Jetty 9.4. IdP V3.4.0 is still on track for early next year but we'll need to get some progress made on the remaining work there during the holidays to stay on track.
Just a brief note to post the short set of slides I presented at the BOF at Internet2 TechExchange last week in San Francisco, primarily a summary of 2017 and an outline of the 2018 project goals.
Highlights of the last month of work:
- the recently-release IdP 3.3.2 IdP patch and security advisory
- some development on IdP 3.4
- ongoing cleanup work with the SP code/build and dependencies
- getting Java 9 support into our Jenkins builds and some reconnaissance on it
- prepping for managing access to the new support mechanisms
The most recent bout with defaulting the handling of TLS verification in the LDAP code has led us to deprecate that approach once and for all going forward, and we will be requiring explicit trust configuration of LDAP connections in 4.0. We're still going to have to study the impact on metadata configuration, but that's much less sensitive to this issue in current practice. We will have to consider the impact moving forward as federations transition to query-on-demand approaches, and we fully expect some federations will decide that signatures aren't what they want to do, whatever our opinions might be. But as a project, we are committed to the idea that security needs to be explicit and not left to the behavior of code we don't control.
The other major addition to the patch was a feature addition, which is not typical, but I felt was warranted because the timeline for 3.4 is unknown and I wanted to make sure the supported version included the ability to generate computed pairwise IDs that are safe for case-insensitive applications, of which there are many. Obviously this is a lost cause for many deployers today, but if we can make life better for the new ones, that's a good thing.
A new feature for 3.4 that was completed this week is something requested by one of the newer Consortium Members, a way to impersonate users for testing purposes that comes up in a lot of deployments. I knew that it could be done both simply and (I hope) safely in a particular way so I felt that it was worth spending a few hours on an example we can include in future releases. The documentation on that will be forthcoming but it's already engendered more discussion on the development list than I've seen in months. I firmly believe testing and test accounts is something that a production IDM environment should deal with directly and not leave up to applications or the IdP, but I also know from local experience that convincing IDM teams of that is a lost cause for many sites. Hopefully this will help.
Java 9 is out, and is a huge change to the Java platform, probably the biggest ever. We're still getting our minds around it but it will be a while before we consider what we might need to do to take advantage of it. More immediately, Java 8 will also be EOL by next September, as expected.
On the support front, we're on track to start rolling out the new options next month and begin to phase in that model through the end of the year. The management of all this is not perfect or where one would like it, but we're sensitive to the amount of time we can afford to spend on managing the way we do this, and we have a lot of existing system behavior with JIRA that limits how much we want to change things. I can definitely say that if JIRA had a way for us to offload the group management through, say, an entitlement attribute, we would explore that, but right now the Support Desk module does not rely on JIRA's own group mechanism for grouping customer accounts. I hope that changes at some point.
One priority of late has been to begin moving our development platform to Java 8 (and eventually 9 once it's out) by reorganizing our Maven parent project structure and establishing a new versioning scheme that will allow us to maintain both the current Java 7 code bases and start moving some of our work forward. The initial target for this work is the Metadata Aggregator, which will be the first product released requiring Java 9. This will necessitate creating maintenance branches for some of our libraries so they can be co-developed for both platforms for the rest of the life of IdP V3.
The other major area of work has been to rescue two of the SP's essentially dead dependencies and produce new releases. Xerces V3.2.0 was released last month at a cost of about 60-70 hours of project time. In addition to some general bug fixing, this release includes a greatly valuable feature allowing us to programmatically disable DTD processing in the parser in a future SP revision, limiting our exposure to a likely unending stream of security bugs in that code. The other major benefit is that we will be able to package this version in a manner that no longer conflicts with packages from Red Hat, allowing us to base the SP packages on a maintained version of the library. Red Hat has left security issues in this library unpatched for over a year so we saw no alternative but to invest the time.
We are currently working to produce a new revision of the Santuario xml-security library, primarily to address small compiler compatibility issues and to allow us to package a more minimal build of the library that excludes large sections of unsupportable code we don't use and don't need, again in the interest of reducing our attack surface.
Once this is complete, work can begin in earnest on the SP and on moving the documentation to a new wiki space. We will also have a simpler, more repeatable build process on Windows and will be moving the code base to the latest Microsoft tools. To some extent this is a race against time, but there's always the chance we'll need to do security patches for the current version in the interim.
We have a bug fix release of the IdP coming in early October which we'll be finishing up during the rest of this month, and we are working to wrap up work on some new modular configuration examples for Jetty 9.4 that will eventually find their way into the IdP.
Our intention is to finalize our plans for the new support changes in the next few days and announce a timeline for establishing the new support mechanisms for our Consortium Members.
Next month a couple of us will be at Tech Exchange, and we should hopefully be able to provide an update on the GÉANT-funded OpenID Connect work.
As we start to change some of how we interact with the user community for our software, it's been a mutual suggestion by the Board and the development team that we try and do a better job of keeping people aware of what we're up to as a project and what our upcoming plans look like. To that end, the updates that have been traditionally provided on a monthly basis to the Board will be done publically using wiki postings. Obviously, we can't share 100% of what we're doing because of, for example, security issues we haven't disclosed yet, but for the most part I'm going to try and limit the amount of sanitization.
We welcome feedback on anything we're doing via either comments or preferably the development list.
Before I cover more recent activity in another entry, it's probably appropriate to review 2017 to date.
This has been a transitional year to some degree because there tends to be a rule of "three", whereby the third version of something tends to be the thing you wished you'd done to begin with if you had had the time. While it's technically the fourth "feature" update, IdP V3.3.0 was really the third substantial one (the first was really a bug fix that necesitated API additions, so it became a minor bump instead of just a patch).
The major goal for this release was to deliver a revised framework for handling multi-factor authentication and for orchestrating more complex login flows to address a lot of the enterprise use cases we've been struggling to support well since the original release. Since then, there's been a lot of time spent documenting and supporting that work to make sure it's working for people and to figure out the best ways of doing various things with it. I personally have probably spent up to 15-20% more time on support than usual in fact.
The rest of our time not spent on the usual day to day has been divided between:
- V3.4 planning and development
- Mapping out a plan for a new SP version
- Completing the transition of all our projects to git, which is now completed
- Consulting on Internet2's TIER packaging efforts
- Transitioning the old web site off our system to relieve the developers of the burden of supporting it
- Summer-long discussions and planning for the support changes announced last month
A few of those are worth some elaboration.
The IdP V3.4 release is really more about V4. It's a transition release intended to warn deployers about any use of deprecated features or settings so that there is ample time to adapt anything before V4 is released and removes most of those bits. There are some features planned, notably new support for metadata-driven configuration, but the main purpose is to suitably sunset the V3 branch to support a hopefully-seamless transition for most deployers to V4. That will be the first major release we've ever done whose goal is not to rewrite the software or deliver radical changes but to actually pare back the code and remove old features, change defaults, etc. We hope to deliver V3.4 no later than Q1 2018, and hopefully sooner.
The SP is long overdue for a major update, but realistically that's not likely to happen given its relative maturity and the bandwidth we have as a project. What is needed however is a documentation cleanup, a lot of dependency care and feeding, and a reassessment of our platform support (i.e., Solaris, this means you). So there will be a V3 SP but it's not likely to be a rewrite or even a significant change really (with the notable exception of a native IIS7 module which is already available to test). Unfortunately that doesn't make it a small amount of work because the bulk of it is work we have to do to remediate issues in our dependencies, many of which are no longer being effectively maintained by the Apache Software Foundation. In a perfect world, we would change them to more supported options, but that's a bit prohibitive without additional developers to pick up slack in other areas.
Those are the major highlights. This month's "regular" update won't be very exciting given that a lot of it is covered above but will be put together for tomorrow's Board meeting, so look for it shortly.