Most of our practical progress of late has been centered on the SP and the Metadata Aggregator, the latter made possible with some of the increased funding obtained over the last several months. The significance of the latter work is that the MA will be based on a newer Java version and library set that will lay the foundation for moving the IdP to the same (or newer) versions in V4.0, saving us a lot of the up front legwork to identify compatibility issues and changes we need to make.
The IdP work continues in the background; probably the most serious question is whether to hold the V3.4 release to wait for a significant revamp of the logout support, or ship without it and push that to V4.0 or a V3.5 interim update. In most other respects we're close to having a shippable release with some useful new features, though lots of small improvements and features remain in the queue.
On the SP side, we're making progress building out a new documentation space and have started working on a way to adapt the system to handle both a new and old configuration schema. The intention is for essentially complete com patibility with V2 configuration, but to allow for a newer format that will contain mostly small improvements such as removal of deprecated settings and possibly some long-asked for improvements like compositing the configuration from multiple sources. We have also started looking seriously at building in a client-side session migration feature that would simplify clustering in a similar fashion to the IdP's support for client-side sessions.
Our recent spate of security issues with the SP continued last month with a serious XML vulnerability in an area that we had correctly guessed would be a source of ongoing concerns (without having identified the specific issue discovered, unfortunately). Our defensive measures over the last couple of years prevented more widespread impact, but Red Hat's refusal to backport various security fixes and defensive code changes to their version of the Xerces library eventually caught up with us and led to the most serious impact from this bug on RHEL 7 and CentOS 7.
One of the takeaways from this issue was the recognition that XML Encryption is really not an optional feature anymore when it comes to use of the POST binding that most SAML deployments have stuck with. Even without data confidentiality concerns, encrypting the assertion mitigates or prevents attacks against the XML content that is of most importance to applications (i.e., the user's identity and attributes), both attacks we know about and attacks we don't yet know about. This will likely be reflected in SP features to require encryption, and a more aggressive stance on moving systems to the AES-GCM encryption algorithm, which fixes serious flaws that current deployments remain vulnerable to many years after the original discovery that CBC mode was broken. Certainly it will be a V4.0 default.
Expect more discussion of this important issue over the course of this year. Also note that this is really almost exclusively an IdP to SP concern. While the IdP supports decryption in some isolated cases, mainly logout, they are not common use cases, and there is very little deployment of SP to IdP encryption out there today. We don't expect that to change.
We made progress over the holidays on a couple of Service Provider work items, reimplementing the on-demand metadata implementation to align with the IdP software and finishing up a revision of the xml-security library with OpenSSL 1.1 support. A release candidate build will be available this week and the final release will probably be this month. We have begun moving cleaned up SP documentation into a dedicated wiki space for the eventual 3.0 release. We're beginning to work through some of the issues associated with producing a new version because of the tension between removing some deprecated settings and guaranteeing that upgrades are safe.
We've largely finished implementing deprecation warnings in the IdP and will be producing a list of features scheduled for possible removal in V4.0 so that federations can begin to publicize the changes ahead of the actual release of V3.4.
The GÉANT OIDC team posted regarding the availability of an alpha release of their Shibboleth OIDC extension, and are targeting mid-March for another update. They would welcome testing and input.
We hope everybody's new year is off to a good start and are looking forward to a productive 2018. Expect the next SP and IdP updates during the first half of the year.
The support rollout has been smooth so far with fairly light but substantive use. JIRA seems to be working well for managing this kind of support. Slack has seen little use so far. Membership continues to grow.
Work on IdP 3.4 has stalled a bit due to vacations and diverting of effort toward the SP. The redesign of the dynamic metadata support exposed a serious bug that appeared to be present all the way back to 2.0, and demonstrated a need to spend time expanding unit testing in the SP as we work on the new version.
The patch itself was simple, but we/I were unwise in including some additional bug fixes that had been applied to the master branch, and caused a regression which was fixed promptly, but reinforced our traditional conservatism in what we include in patches. It was a lesson all around in violating our norms.
Most of the work redesigning the SP metadata support is complete and ready to be tested, and focus has shifted to getting a clean, documented build process for the SP and all its dependencies while we work in parallel on a new version of the Apache XML Security library. With the opportunity for breaking changes, I decided to get more aggressive and remove a lot of deprecated code and redesign some old APIs since we'll likely not make this kind of change again. Given the pace of this work, I don't expect a 2.0 release of that library until after the new year.
As the work on the SP progresses, platform support continues to be discussed on-list and internally. It is a given that Solaris will be dropped as a supported platform and it makes sense to consider adding Debian and Ubuntu due to popularity, but this requires figuring out how to properly scope that support, given that we don't package for them. SUSE is essentially in the same situation now anyway.
Some internal testing at OSU revealed that the older Apache HTTP client library in the IdP contains some bugs that don't impact occasional use but become serious under load for use cases such as web services, so we need to accelerate an upgrade of that library in the next IdP release.
Oracle also clarified (somewhat) their plans for Java and announced that Java 9 would not be a "long term support" release, so the MDA 1.0 software will likely be based on Java 8 instead, with the IdP 4 work based on the 18.9 release in September 2018. All subject to change as usual.
The last month has been dominated by travel to TechExchange and getting the new support services rolled out for testing by the members. TechExchange was a bit less exhausting than the past couple of years, and was dominated by a lot of good conversation and discussion about OpenID Connect, including a good touch-base with the CSC team working on those features for Shibboleth. I'm hopeful there will be some initial bits to play around with by early next year.
Proxies were another dominant area of discussion, and after an offhand comment/question from a member, I went ahead and implemented some policy support for the
<Scoping> element in SAML that allows the IdP to recognize different downstream services on the other side of a proxying SP.
The support rollout has been relatively smooth, with very little use so far (the next Slack message will be the first). The best part has been the dramatic and ongoing increase in members as a result of this change.
On the more traditional development front, work has been progressing on:
- Re-designing the dynamic metadata support in the SP.
- Finishing the long-delayed SOAP client support in the IdP, though that's more of a "looking ahead to doing proxying support" thing than anything the IdP itself is desperate to include.
- Getting a Java 9-compatible build process working with Maven so we can start testing on it.
Going into the holiday season, I will be working to get an updated release of the XML Security library used by the SP, and probably doing a Xerces bug fix release out of the kindness of my heart, and we'll hopefully get a working set of examples done for Jetty 9.4. IdP V3.4.0 is still on track for early next year but we'll need to get some progress made on the remaining work there during the holidays to stay on track.
Just a brief note to post the short set of slides I presented at the BOF at Internet2 TechExchange last week in San Francisco, primarily a summary of 2017 and an outline of the 2018 project goals.
Highlights of the last month of work:
- the recently-release IdP 3.3.2 IdP patch and security advisory
- some development on IdP 3.4
- ongoing cleanup work with the SP code/build and dependencies
- getting Java 9 support into our Jenkins builds and some reconnaissance on it
- prepping for managing access to the new support mechanisms
The most recent bout with defaulting the handling of TLS verification in the LDAP code has led us to deprecate that approach once and for all going forward, and we will be requiring explicit trust configuration of LDAP connections in 4.0. We're still going to have to study the impact on metadata configuration, but that's much less sensitive to this issue in current practice. We will have to consider the impact moving forward as federations transition to query-on-demand approaches, and we fully expect some federations will decide that signatures aren't what they want to do, whatever our opinions might be. But as a project, we are committed to the idea that security needs to be explicit and not left to the behavior of code we don't control.
The other major addition to the patch was a feature addition, which is not typical, but I felt was warranted because the timeline for 3.4 is unknown and I wanted to make sure the supported version included the ability to generate computed pairwise IDs that are safe for case-insensitive applications, of which there are many. Obviously this is a lost cause for many deployers today, but if we can make life better for the new ones, that's a good thing.
A new feature for 3.4 that was completed this week is something requested by one of the newer Consortium Members, a way to impersonate users for testing purposes that comes up in a lot of deployments. I knew that it could be done both simply and (I hope) safely in a particular way so I felt that it was worth spending a few hours on an example we can include in future releases. The documentation on that will be forthcoming but it's already engendered more discussion on the development list than I've seen in months. I firmly believe testing and test accounts is something that a production IDM environment should deal with directly and not leave up to applications or the IdP, but I also know from local experience that convincing IDM teams of that is a lost cause for many sites. Hopefully this will help.
Java 9 is out, and is a huge change to the Java platform, probably the biggest ever. We're still getting our minds around it but it will be a while before we consider what we might need to do to take advantage of it. More immediately, Java 8 will also be EOL by next September, as expected.
On the support front, we're on track to start rolling out the new options next month and begin to phase in that model through the end of the year. The management of all this is not perfect or where one would like it, but we're sensitive to the amount of time we can afford to spend on managing the way we do this, and we have a lot of existing system behavior with JIRA that limits how much we want to change things. I can definitely say that if JIRA had a way for us to offload the group management through, say, an entitlement attribute, we would explore that, but right now the Support Desk module does not rely on JIRA's own group mechanism for grouping customer accounts. I hope that changes at some point.
One priority of late has been to begin moving our development platform to Java 8 (and eventually 9 once it's out) by reorganizing our Maven parent project structure and establishing a new versioning scheme that will allow us to maintain both the current Java 7 code bases and start moving some of our work forward. The initial target for this work is the Metadata Aggregator, which will be the first product released requiring Java 9. This will necessitate creating maintenance branches for some of our libraries so they can be co-developed for both platforms for the rest of the life of IdP V3.
The other major area of work has been to rescue two of the SP's essentially dead dependencies and produce new releases. Xerces V3.2.0 was released last month at a cost of about 60-70 hours of project time. In addition to some general bug fixing, this release includes a greatly valuable feature allowing us to programmatically disable DTD processing in the parser in a future SP revision, limiting our exposure to a likely unending stream of security bugs in that code. The other major benefit is that we will be able to package this version in a manner that no longer conflicts with packages from Red Hat, allowing us to base the SP packages on a maintained version of the library. Red Hat has left security issues in this library unpatched for over a year so we saw no alternative but to invest the time.
We are currently working to produce a new revision of the Santuario xml-security library, primarily to address small compiler compatibility issues and to allow us to package a more minimal build of the library that excludes large sections of unsupportable code we don't use and don't need, again in the interest of reducing our attack surface.
Once this is complete, work can begin in earnest on the SP and on moving the documentation to a new wiki space. We will also have a simpler, more repeatable build process on Windows and will be moving the code base to the latest Microsoft tools. To some extent this is a race against time, but there's always the chance we'll need to do security patches for the current version in the interim.
We have a bug fix release of the IdP coming in early October which we'll be finishing up during the rest of this month, and we are working to wrap up work on some new modular configuration examples for Jetty 9.4 that will eventually find their way into the IdP.
Our intention is to finalize our plans for the new support changes in the next few days and announce a timeline for establishing the new support mechanisms for our Consortium Members.
Next month a couple of us will be at Tech Exchange, and we should hopefully be able to provide an update on the GÉANT-funded OpenID Connect work.
As we start to change some of how we interact with the user community for our software, it's been a mutual suggestion by the Board and the development team that we try and do a better job of keeping people aware of what we're up to as a project and what our upcoming plans look like. To that end, the updates that have been traditionally provided on a monthly basis to the Board will be done publically using wiki postings. Obviously, we can't share 100% of what we're doing because of, for example, security issues we haven't disclosed yet, but for the most part I'm going to try and limit the amount of sanitization.
We welcome feedback on anything we're doing via either comments or preferably the development list.
Before I cover more recent activity in another entry, it's probably appropriate to review 2017 to date.
This has been a transitional year to some degree because there tends to be a rule of "three", whereby the third version of something tends to be the thing you wished you'd done to begin with if you had had the time. While it's technically the fourth "feature" update, IdP V3.3.0 was really the third substantial one (the first was really a bug fix that necesitated API additions, so it became a minor bump instead of just a patch).
The major goal for this release was to deliver a revised framework for handling multi-factor authentication and for orchestrating more complex login flows to address a lot of the enterprise use cases we've been struggling to support well since the original release. Since then, there's been a lot of time spent documenting and supporting that work to make sure it's working for people and to figure out the best ways of doing various things with it. I personally have probably spent up to 15-20% more time on support than usual in fact.
The rest of our time not spent on the usual day to day has been divided between:
- V3.4 planning and development
- Mapping out a plan for a new SP version
- Completing the transition of all our projects to git, which is now completed
- Consulting on Internet2's TIER packaging efforts
- Transitioning the old web site off our system to relieve the developers of the burden of supporting it
- Summer-long discussions and planning for the support changes announced last month
A few of those are worth some elaboration.
The IdP V3.4 release is really more about V4. It's a transition release intended to warn deployers about any use of deprecated features or settings so that there is ample time to adapt anything before V4 is released and removes most of those bits. There are some features planned, notably new support for metadata-driven configuration, but the main purpose is to suitably sunset the V3 branch to support a hopefully-seamless transition for most deployers to V4. That will be the first major release we've ever done whose goal is not to rewrite the software or deliver radical changes but to actually pare back the code and remove old features, change defaults, etc. We hope to deliver V3.4 no later than Q1 2018, and hopefully sooner.
The SP is long overdue for a major update, but realistically that's not likely to happen given its relative maturity and the bandwidth we have as a project. What is needed however is a documentation cleanup, a lot of dependency care and feeding, and a reassessment of our platform support (i.e., Solaris, this means you). So there will be a V3 SP but it's not likely to be a rewrite or even a significant change really (with the notable exception of a native IIS7 module which is already available to test). Unfortunately that doesn't make it a small amount of work because the bulk of it is work we have to do to remediate issues in our dependencies, many of which are no longer being effectively maintained by the Apache Software Foundation. In a perfect world, we would change them to more supported options, but that's a bit prohibitive without additional developers to pick up slack in other areas.
Those are the major highlights. This month's "regular" update won't be very exciting given that a lot of it is covered above but will be put together for tomorrow's Board meeting, so look for it shortly.