Since last month we've released IdP V3.4.0 and the first patch for it once the initial round of bug reports came in. We have one or two remaining annoyances around deprecation warnings outstanding and the ReleaseNotes are being kept up to date to provide the most accurate information we have on what to watch out for. We can't do patches every week obviously but as these things accumulate there will be additional patches coming to get things cleaned up. Overall the release seems to be solid and I've had it in production for a couple of weeks now.
The GEANT OIDC extensions is now in beta, with an improved installation process, so please check it out and provide feedback if you're interested in that.
In parallel with the occasional maintenance commit, we've branched the source tree so the master branches are open for development on the V4 code base. All commits are now intended to be on those master branches, with only specific authorized commits backported to the 3.4 maintenance branch.
The master branches have been moved to a new parent POM based on Java 8 and Spring 5 as the first "step forward" and all but a couple of tests are passing and the IdP seems to be functional, so that's a good sign that we won't have too much remediation needed. Eventually this will be moved to Java 11, but right now the Maven toolchain isn't stable on Java 11, so we're on Java 8 for now.
We also have a buildable artifact containing a curated and well-organized Jetty 9.4 configuration that works with the IdP with minimal adjustments and will probably make that available in some capacity, with an eye towards including it in V4 to help bootstrap new installs (assuming 9.4 is a current release at that point in time).
What isn't really decided at this stage is what the schedule and scope of V4 will be, with the most likely options being a 12 month development cycle with minimal feature additions vs. a longer cycle of perhaps 24 months to accomodate more new features. Both options are possible and make some sense, and hopefully we'll be able to engage people on this question once we have some initial discussions with the Board about it. Regardless of the decision, we don't expect the upgrade process to V4 to be onerous, though we can definitely promise there won't be a supported path to upgrade from V2 directly to V4.
On the SP front, we are in the process of getting a patch out the door, but the latest curl release (7.62.0) was a big one that has a number of regressions, so right now we're probably waiting another month to include the next version rather than ship this one, but that could change depending on bugs reported and the urgency to get an update done.
As I draft this, IdP V3.4 is tagged and will be announced tomorrow, just shy of two years (!) since the last significant feature release. The last month saw a large number of commits from the whole team to finish up late-arriving features and complete internal testing for the release. Pretty much everything originally planned for this release made it in (along with a lot more) except for a significant redesign of the logout functionality, in keeping with the project's long tradition of never getting logout work done on time.
As I outlined before, the list of new features is extensive. This will be the last feature update before V4 when we make the move to Java 11 and Spring 5. This release provides a good window to freshen and optimize configurations and get rid of legacy options and cruft that will be removed from V4.
Once this release is complete and we take a breath or two, work will begin soon on initial scoping of V4. I would like to work on some opportunities for direct interaction with the Consortium Members on what people would like to see from that work, aside from the long list of issues we already have open. One obvious task for the Board will be to have the necessary discussions on the viability of incorporating the OIDC extension being developed for V3.4, which is mostly a question of long term code ownership and maintenance.
Activity on the SP bug front is still very low, but we have a couple of possible issues to get cleaned up and then we can get a new build that supports OpenSSL 1.1.1 out the door in time for the end of the year. This may be an opportune time, with another Java developer coming on board to help out on the IdP, for me to spend a bit of time looking at the viability of an NGINX SP module.
I'll be at Internet2 Tech Exchange and ACAMP next week in Orlando, so I'll see many of you there.
The last month has been spent ramping IdP development back up and preparing for a release of V3.4 in October. The number of bug fixes and features is quite large (pushing 150+), so this is a pretty big new release that is intended to bridge to V4 as the next big update. There are a couple of things that haven't made the cut, primarily redesigning the logout behavior in response to a lot of feedback. Both time constraints and the need to maintain compatibility made it appealing to delay that work until V4 when we have more freedom to change the UI in ways that might require a small bit of rework for deployers instead of guaranteeing drop-in compatibility.
In the meantime, there should be enough to keep people busy with V3.4 for a while:
- A nice HTTPConnector for web services integrations (I built it so I'm biased but I've been using it for Grouper and AWS integration projects and it's been really, really convenient to use)
- Built-in wiring and convenience functions that make it possible to drive a ton of configuration options using metadata "tags"
- Non-browser Duo AuthAPI support
- Improvements to the deployability of the DynamicHTTPMetadataProvider and LocalDynamicMetadataProvider plugins
- Improvements for embedding scriptlets in a variety of configuration scenarios
- A new "attended startup" mode that prevents on-disk access to an unlocked or trivially encrypted private key
- The ability to provision CAS services using SAML metadata for consistency and to support the new metadata-driven configuration mechanisms
- Added support for SAML proxying constructs to detect/react to proxied SPs and to express upstream proxying to SPs
- Greatly enhanced context-check interceptor that can handle multiple scenarios at the same time
- A new impersonation interceptor that supports advanced debugging or testing scenarios
- Some small features to support the drop-in addition of the under-development GEANT OIDC extension
And more, but that's plenty for now.
We'll be wrapping up work this month, testing over the next couple of weeks, and should be freezing around the end of the month with a release in time for TechExchange in Orlando.
We're also working on a more modular and minimal configuration for Jetty 9.4 that will form the basis of not only the Windows "quick install" package but a maintained/curated Jetty configuration to make deployment simpler on other platforms for people without experience with Jetty. We're not planning to fully "embed" Jetty because that's nearly impossible to maintain over time for mature use as a production web server, but it should be a good starting point for most people.
On the SP front, things have been pretty quiet, so one presumes either upgrades are going reasonably smoothly or people are stuck on software with some known security bugs, but I'll choose to assume the former. There will most likely be a patch update later this year if only because OpenSSL 1.1.1 is now out, and 1.1.0 will be EOL soon.
Both OpenSSL and Jetty are introducing the first iterations of TLS 1.3, which is different enough from TLS 1.2 that it will require some time to digest (pun intended). Hopefully if there are some adjustments needed the less frequent use of the back-channel in the SP should soften the impact.
I skipped an update in July due to the workload of getting the SP finished, which has been dominating the work for least a couple of us most of the last few months, and I'm happy to note we finally shipped 3.0 (and 3.0.1 and 3.0.2). Given the magnitude of the internal changes, I'm reasonably happy with the results so far as the bugs have been easy to fix, and the release process is now more streamlined than before due to huge improvements in the Windows build process and the elimination of Solaris as a supported platform. That means patches come more quickly. Most people I think will find the upgrade to be painless, which is good since we very quickly hit a security issue that impacts all the older versions.
The documentation is about 50% reviewed and cleaned up at this point, better than I hoped but not as good as I would like. It's at least better organized and out from under the old IdP material.
In between patches work has picked back up on the backlog of IdP issues and features. I finished up and started documenting a new feature for limiting access to keys by requiring attended startup of the service, which is a weird concept but is about the only way I can see to protect private keys in a cost effective way. That was done on OSU time donated to the project so we can make use of the feature.
I'm starting work on extending the Duo login flow to include support for non-browser clients using work donated by UMBC and it should hopefully require minimal effort by existing Duo deployers to enable.
The IdP is now leapfrogged again by the SP in terms of some of the dynamic metadata features, so we're circling back to add some of those features into the IdP now that some more deployment experience is available.
We're also finishing up work on a stable configuration set for Jetty 9.4 and are discussing how to make that available to deployers. The embedded version for Windows will be updated to that version and we would like to make deployment on other platforms easier by delivering a customizable starting point that will mostly work out of the box and then be simple to extend. Hopefully we can help people stay current by shipping base configuration sets that can be used to jump start upgrades of local container installs without actually taking on responsibility for the installs ourselves.
The Java platform situation remains very fluid but we now know from Oracle's public statements that their JDK is about to become a for-fee product with Java 11 and that free use will require use of OpenJDK. It is unknown whether there will be free options for OpenJDK support that last longer than the 6 months Oracle will be patching each version of OpenJDK, so there may not be anything like a "long term" Java version that we can rely on for future products. At minimum, this means we have to revisit our recommendations for JDK usage, and at worst it may mean adapting to a much more rapid development cycle to keep up and maintain compatibility. That in turn means more time and money devoted to testing and maintenance.
The SP V3 work has reached the beta stage, and we really could use some testing. A 3.0.1 is probably inevitable but there's always hope.Windows packages can be downloaded from our site, and RPMs are available from a test repository. Neither is guaranteed to be directly upgradable to the final release, so please act accordingly. The RPMs get rebuilt occasionally so they're not formally a beta, more like snapshots.
The new documentation space for the SP is readable now though far from complete. More short term work is needed to get it in shape for the release but it's pretty close, and should eventually be much more readable. Eventually we hope to add more examples and the sorts of explanations of things that the IdP documentation has. Most of the new features are not yet documented, but we'll aim for getting that done in the next couple of weeks. A final round of improvements has been made to the just-in-time metadata support and some of that will hopefully make their way back into IdP V3.4 now that we have some practical experience.
We will begin to get some of the dependencies released very shortly and the release itself is probably just 2-3 weeks out if we don't get any bug reports from testers. It would be nice to get a second mirror online if possible beforehand so please contact us if you can do that.
IdP work has continued off and on at a slow pace but is cranking up again and should pick back up in mid-July. A high priority will be to get the changes requested by the OIDC extension developers into the API so they can get back to work on it after their summer break, and then I think we will start to get pretty ruthless with identifying what else we actually have the time to do. I would like to get V3.4 shipped by the end of Q3 at the latest. We should have a reference configuration for Jetty 9.4 to include with that release, and the Windows installer will be updated to that version.
An item that just came to our attention is - JPAR-121Getting issue details... STATUS , and could have major implications for deployers. It's a good example of the need to keep devoting resources to simply staying aware of what's going on with the platforms.
As we previewed last week, we have a security patch (V3.3.3) for the IdP coming on Wednesday that impacts CAS deployments and includes a Spring security fix that might affect a very small percentage of deployers on Windows (realistically none, perhaps, but we're erring on the side of caution). The main purpose of the patch is the CAS issue. This is one of the first times we've provided advance notice publically of a security fix, and that seems to be common practice for widely used software these days. Obviously the full details will be available with the patch's release.
We have been finalizing a new project position statement on our plans for supporting the Java platform long term, which you can find under Product Platforms. This addresses a few questions we've gotten lately about support for current and future Java versions, and largely clarifies that we believe our focus should be on supporting so-called Long Term Support (LTS) releases, currently Java 8, and Java 11 later this year. Feedback on this policy proposal can be provided in any of our many communication channels. Uttimately we want and need to support what you need, but we think most people haven't yet really digested what's going on with Java so we're hoping to provide a clear statement to help with any compatibility questions going forward.
Development on SP V3.0 is nearing completion and we have begun a round of internal testing, after which we will provide a beta release once a pass over the documentation is done and basic notes on upgrading are drafted. Some polishing work is left to do on the enhanced Dynamic metadata features but a feature freeze is very close.
The last development update included a planned change to the default configuration filename that has since been rescinded for the final release. Upgrades, particularly via RPM, are much cleaner and more reliable if the shibboleth2.xml filename is left alone, so we're prepared to live with the fallout of leaving it.
A number of planned changes to default settings will be compiled fairly soon and circulated for feedback, particularly in the area of SAML 1.1 and attribute query support, which increasingly create confusion, but final decisions will be open to discussion.
Significant improvements in this upgrade include:
- A newly implemented IIS7+ native module to replace the creaky ISAPI module, including support for REMOTE_USER and non-header Server Variables.
- Stateless clustering, though with even more limitations on logout.
- Substantial fixes and improvements to the Dynamic metadata support, including a LocalDynamic option for consuming IdP metadata fragments from the local file system (the IdP has a similar feature).
- Elimination of Application Overrides for trivial "call the SP something different per virtual host" use cases.
- An experimental feature to define and reference Application Overrides via external files that can be added at runtime.
- Sensible logging defaults for Apache/IIS modules relying on the Event Log (Windows) or syslog (everything else), addressing a range of problems with file permissions and log rotation.
- Updated XML libraries with substantial security and maintenance improvements. Reduces surface area will hopefully prevent certain kinds of likely security bugs from impacting the project.
- An improved build process on Windows, making the delivery of library patches less labor intensive.
- Dropping Solaris support, saving the project an average of $1500-$3000 testing time per release, without factoring in support costs. It adds up.
- Many small and not-so-small bug fixes ( ) Getting issues...
More on a lot of this as the documentation gets cleaned up. Needless to say, the upgrade got bigger as work progressed, and we think this will be a worthy upgrade, with the added bonus that it will be safe and easy to apply to existing systems.
ETA at this point is probably late June depending on testing and feedback. The sooner we get this out, the sooner we can turn full attention back to getting IdP V3.4 completed by this fall.
The last few weeks have been dominated by the recent cross-industry SAML vulnerability discovered by a Duo Security researcher, a lot of research into the impact of Java 9, 10, and 11 on our dependencies and build process, and continued work on the SP 3.0 features and documentation.
The vulnerability has dominated a lot of conversation of late. It was embargoed for about a month and I've been dropping various hints about it, mostly by alluding to the previous vulnerability that we fixed in January because it's actually almost an identical bug, at least in our SP. Once the new issue came to light, the importance of XML Encryption became very evident, since it mitigated both the original and the new class of vulnerabilities, and the fallout in pure Shibboleth deployments was much less as a result.
The real fallout from this is with cloud integrations, both because they are the most likely to forgo encryption, but also because there are so many different, and often substandard, SAML implementations in use. This is the reason we're so vocal as a project in advising people not to even consider building their own code on top of OpenSAML. We don't have the resources to document it adequately, and there are very few developers with the skills necessary. I say that with full self-awareness of the fact that we were vulnerable; that reinforces the point rather than weakens it.
Anyway, a lot of the time on this has been the fallout and considering how to approach the problem of finding and reporting all these vulnerable systems to companies who simply don't have the resources to be supporting the SAML stacks they now have to fix. We'll be cleaning this up for months, if not years.
Since people seem to take blogs so seriously, I'll say a couple of things that might help with some vendor conversations: This is an SP bug, period, and there is no justification whatsoever to point fingers at the IdP or claim that the IdP can fix this. There are some obscure things an IdP can do to make the bug hard or impossible to exploit, but that's a workaround, just like XML Encryption is. It's a bug. Fix the bug. Secondly, do not think that because your SAML code doesn't use one of the libraries that was explicitly noted as being vulnerable that you're safe. That is not a list of the affected libraries, it's a list of lbraries known to be affected. Others are affected. Other libraries are not broken per se, but have been misused by developers so as to create a vulnerability (this is particularly true of the Java version of OpenSAML).
A beneficial side effect of this has been to force us to invest the time necessary to come up with a package distribution mechanism that bypasses the broken opensuse.org mirroring infrastructure. Simply put, they allow sites to mirror without valldating the mirrors actually work, and keep serving up broken links using a server-side redirector that makes bypassing the mirrors impossible. Things were bad enough this time that we were forced to sort of insist that one of our members with the necessary infrastructure step up to deal with this, and Jisc agreed to put a long term solution into place this summer. In the meantime, we have put out a call for volunteers to host the files, and we have a "final" Yum repository builder in place that will de-reference the mirrors to choose from properly, on the client side, so these repo files should keep working in the future as we add new mirrors. Please update your servers to use these fresh repository files.
On a lighter note...
The impending release of Java 10, and soon 11, has managed to break a ton of Maven plugins and we've been spending significant time ramping up full matrix testing, identifying incompatibilities, and waiting for new versions of the plugins so the build and deploy process can be tested again. The expected pace of Java releases that introduce breaking changes is going to be a major problem for a lot of projects, and we have a lot of dependencies that, unlike Spring, are less well-maintained. We're going to have to divert even more time than in the past to keeping things working, and it's likely that there will be less of a guarantee that our major releases will continue to work on the non-LTS (long term support) versions of Java. We can only do so much, and as an "enterprise" platform, we need to focus our time on making sure the LTS releases work and that we have versions released to support those Java platforms in time. That means, practically speaking, that Java 9 and 10 matter a lot less than Java 8 and Java 11, and that we have to make sure V3.4 works on Java 11 since it supersedes Java 8 this fall. Expect some kind of official policy about this when we have more time to study the issues.
On the SP front, we should be in the home-stretch in getting the SP out the door within the next couple of months at most. The build process on Windows is now fairly streamlined and scriptable, even down to our dependencies. Repeatable processes are always nice. We have a basically "final" plan for the configuration, which is that we have built support for a new V3 configuration schema that is essentially identical to the V2 schema with some deprecated features removed, but the SP will load either one. This guarantees safe upgrades but provides the right warnings to get old settings cleaned up in case we ever do a V4. We have renamed the default file to shibboleth3.xml but will fall back to the old filename, so the whole thing is pretty seamless.
I also completed work on a cookie-based session recovery feature that allows most of the important data in a session to migrate between server nodes or across restarts using a shared key. It should alleviate most of the need for back-end session store sharing, and the cookie sizes don't seem too risky in most cases.
Finally, another highlighting of the OIDC work, which has reached alpha 2 and needs testing and feedback: https://marc.info/?l=shibboleth-dev&m=152111752828021&w=2
Most of our practical progress of late has been centered on the SP and the Metadata Aggregator, the latter made possible with some of the increased funding obtained over the last several months. The significance of the latter work is that the MA will be based on a newer Java version and library set that will lay the foundation for moving the IdP to the same (or newer) versions in V4.0, saving us a lot of the up front legwork to identify compatibility issues and changes we need to make.
The IdP work continues in the background; probably the most serious question is whether to hold the V3.4 release to wait for a significant revamp of the logout support, or ship without it and push that to V4.0 or a V3.5 interim update. In most other respects we're close to having a shippable release with some useful new features, though lots of small improvements and features remain in the queue.
On the SP side, we're making progress building out a new documentation space and have started working on a way to adapt the system to handle both a new and old configuration schema. The intention is for essentially complete com patibility with V2 configuration, but to allow for a newer format that will contain mostly small improvements such as removal of deprecated settings and possibly some long-asked for improvements like compositing the configuration from multiple sources. We have also started looking seriously at building in a client-side session migration feature that would simplify clustering in a similar fashion to the IdP's support for client-side sessions.
Our recent spate of security issues with the SP continued last month with a serious XML vulnerability in an area that we had correctly guessed would be a source of ongoing concerns (without having identified the specific issue discovered, unfortunately). Our defensive measures over the last couple of years prevented more widespread impact, but Red Hat's refusal to backport various security fixes and defensive code changes to their version of the Xerces library eventually caught up with us and led to the most serious impact from this bug on RHEL 7 and CentOS 7.
One of the takeaways from this issue was the recognition that XML Encryption is really not an optional feature anymore when it comes to use of the POST binding that most SAML deployments have stuck with. Even without data confidentiality concerns, encrypting the assertion mitigates or prevents attacks against the XML content that is of most importance to applications (i.e., the user's identity and attributes), both attacks we know about and attacks we don't yet know about. This will likely be reflected in SP features to require encryption, and a more aggressive stance on moving systems to the AES-GCM encryption algorithm, which fixes serious flaws that current deployments remain vulnerable to many years after the original discovery that CBC mode was broken. Certainly it will be a V4.0 default.
Expect more discussion of this important issue over the course of this year. Also note that this is really almost exclusively an IdP to SP concern. While the IdP supports decryption in some isolated cases, mainly logout, they are not common use cases, and there is very little deployment of SP to IdP encryption out there today. We don't expect that to change.
We made progress over the holidays on a couple of Service Provider work items, reimplementing the on-demand metadata implementation to align with the IdP software and finishing up a revision of the xml-security library with OpenSSL 1.1 support. A release candidate build will be available this week and the final release will probably be this month. We have begun moving cleaned up SP documentation into a dedicated wiki space for the eventual 3.0 release. We're beginning to work through some of the issues associated with producing a new version because of the tension between removing some deprecated settings and guaranteeing that upgrades are safe.
We've largely finished implementing deprecation warnings in the IdP and will be producing a list of features scheduled for possible removal in V4.0 so that federations can begin to publicize the changes ahead of the actual release of V3.4.
The GÉANT OIDC team posted regarding the availability of an alpha release of their Shibboleth OIDC extension, and are targeting mid-March for another update. They would welcome testing and input.
We hope everybody's new year is off to a good start and are looking forward to a productive 2018. Expect the next SP and IdP updates during the first half of the year.
The support rollout has been smooth so far with fairly light but substantive use. JIRA seems to be working well for managing this kind of support. Slack has seen little use so far. Membership continues to grow.
Work on IdP 3.4 has stalled a bit due to vacations and diverting of effort toward the SP. The redesign of the dynamic metadata support exposed a serious bug that appeared to be present all the way back to 2.0, and demonstrated a need to spend time expanding unit testing in the SP as we work on the new version.
The patch itself was simple, but we/I were unwise in including some additional bug fixes that had been applied to the master branch, and caused a regression which was fixed promptly, but reinforced our traditional conservatism in what we include in patches. It was a lesson all around in violating our norms.
Most of the work redesigning the SP metadata support is complete and ready to be tested, and focus has shifted to getting a clean, documented build process for the SP and all its dependencies while we work in parallel on a new version of the Apache XML Security library. With the opportunity for breaking changes, I decided to get more aggressive and remove a lot of deprecated code and redesign some old APIs since we'll likely not make this kind of change again. Given the pace of this work, I don't expect a 2.0 release of that library until after the new year.
As the work on the SP progresses, platform support continues to be discussed on-list and internally. It is a given that Solaris will be dropped as a supported platform and it makes sense to consider adding Debian and Ubuntu due to popularity, but this requires figuring out how to properly scope that support, given that we don't package for them. SUSE is essentially in the same situation now anyway.
Some internal testing at OSU revealed that the older Apache HTTP client library in the IdP contains some bugs that don't impact occasional use but become serious under load for use cases such as web services, so we need to accelerate an upgrade of that library in the next IdP release.
Oracle also clarified (somewhat) their plans for Java and announced that Java 9 would not be a "long term support" release, so the MDA 1.0 software will likely be based on Java 8 instead, with the IdP 4 work based on the 18.9 release in September 2018. All subject to change as usual.
The last month has been dominated by travel to TechExchange and getting the new support services rolled out for testing by the members. TechExchange was a bit less exhausting than the past couple of years, and was dominated by a lot of good conversation and discussion about OpenID Connect, including a good touch-base with the CSC team working on those features for Shibboleth. I'm hopeful there will be some initial bits to play around with by early next year.
Proxies were another dominant area of discussion, and after an offhand comment/question from a member, I went ahead and implemented some policy support for the
<Scoping> element in SAML that allows the IdP to recognize different downstream services on the other side of a proxying SP.
The support rollout has been relatively smooth, with very little use so far (the next Slack message will be the first). The best part has been the dramatic and ongoing increase in members as a result of this change.
On the more traditional development front, work has been progressing on:
- Re-designing the dynamic metadata support in the SP.
- Finishing the long-delayed SOAP client support in the IdP, though that's more of a "looking ahead to doing proxying support" thing than anything the IdP itself is desperate to include.
- Getting a Java 9-compatible build process working with Maven so we can start testing on it.
Going into the holiday season, I will be working to get an updated release of the XML Security library used by the SP, and probably doing a Xerces bug fix release out of the kindness of my heart, and we'll hopefully get a working set of examples done for Jetty 9.4. IdP V3.4.0 is still on track for early next year but we'll need to get some progress made on the remaining work there during the holidays to stay on track.
Just a brief note to post the short set of slides I presented at the BOF at Internet2 TechExchange last week in San Francisco, primarily a summary of 2017 and an outline of the 2018 project goals.
Highlights of the last month of work:
- the recently-release IdP 3.3.2 IdP patch and security advisory
- some development on IdP 3.4
- ongoing cleanup work with the SP code/build and dependencies
- getting Java 9 support into our Jenkins builds and some reconnaissance on it
- prepping for managing access to the new support mechanisms
The most recent bout with defaulting the handling of TLS verification in the LDAP code has led us to deprecate that approach once and for all going forward, and we will be requiring explicit trust configuration of LDAP connections in 4.0. We're still going to have to study the impact on metadata configuration, but that's much less sensitive to this issue in current practice. We will have to consider the impact moving forward as federations transition to query-on-demand approaches, and we fully expect some federations will decide that signatures aren't what they want to do, whatever our opinions might be. But as a project, we are committed to the idea that security needs to be explicit and not left to the behavior of code we don't control.
The other major addition to the patch was a feature addition, which is not typical, but I felt was warranted because the timeline for 3.4 is unknown and I wanted to make sure the supported version included the ability to generate computed pairwise IDs that are safe for case-insensitive applications, of which there are many. Obviously this is a lost cause for many deployers today, but if we can make life better for the new ones, that's a good thing.
A new feature for 3.4 that was completed this week is something requested by one of the newer Consortium Members, a way to impersonate users for testing purposes that comes up in a lot of deployments. I knew that it could be done both simply and (I hope) safely in a particular way so I felt that it was worth spending a few hours on an example we can include in future releases. The documentation on that will be forthcoming but it's already engendered more discussion on the development list than I've seen in months. I firmly believe testing and test accounts is something that a production IDM environment should deal with directly and not leave up to applications or the IdP, but I also know from local experience that convincing IDM teams of that is a lost cause for many sites. Hopefully this will help.
Java 9 is out, and is a huge change to the Java platform, probably the biggest ever. We're still getting our minds around it but it will be a while before we consider what we might need to do to take advantage of it. More immediately, Java 8 will also be EOL by next September, as expected.
On the support front, we're on track to start rolling out the new options next month and begin to phase in that model through the end of the year. The management of all this is not perfect or where one would like it, but we're sensitive to the amount of time we can afford to spend on managing the way we do this, and we have a lot of existing system behavior with JIRA that limits how much we want to change things. I can definitely say that if JIRA had a way for us to offload the group management through, say, an entitlement attribute, we would explore that, but right now the Support Desk module does not rely on JIRA's own group mechanism for grouping customer accounts. I hope that changes at some point.
One priority of late has been to begin moving our development platform to Java 8 (and eventually 9 once it's out) by reorganizing our Maven parent project structure and establishing a new versioning scheme that will allow us to maintain both the current Java 7 code bases and start moving some of our work forward. The initial target for this work is the Metadata Aggregator, which will be the first product released requiring Java 9. This will necessitate creating maintenance branches for some of our libraries so they can be co-developed for both platforms for the rest of the life of IdP V3.
The other major area of work has been to rescue two of the SP's essentially dead dependencies and produce new releases. Xerces V3.2.0 was released last month at a cost of about 60-70 hours of project time. In addition to some general bug fixing, this release includes a greatly valuable feature allowing us to programmatically disable DTD processing in the parser in a future SP revision, limiting our exposure to a likely unending stream of security bugs in that code. The other major benefit is that we will be able to package this version in a manner that no longer conflicts with packages from Red Hat, allowing us to base the SP packages on a maintained version of the library. Red Hat has left security issues in this library unpatched for over a year so we saw no alternative but to invest the time.
We are currently working to produce a new revision of the Santuario xml-security library, primarily to address small compiler compatibility issues and to allow us to package a more minimal build of the library that excludes large sections of unsupportable code we don't use and don't need, again in the interest of reducing our attack surface.
Once this is complete, work can begin in earnest on the SP and on moving the documentation to a new wiki space. We will also have a simpler, more repeatable build process on Windows and will be moving the code base to the latest Microsoft tools. To some extent this is a race against time, but there's always the chance we'll need to do security patches for the current version in the interim.
We have a bug fix release of the IdP coming in early October which we'll be finishing up during the rest of this month, and we are working to wrap up work on some new modular configuration examples for Jetty 9.4 that will eventually find their way into the IdP.
Our intention is to finalize our plans for the new support changes in the next few days and announce a timeline for establishing the new support mechanisms for our Consortium Members.
Next month a couple of us will be at Tech Exchange, and we should hopefully be able to provide an update on the GÉANT-funded OpenID Connect work.
As we start to change some of how we interact with the user community for our software, it's been a mutual suggestion by the Board and the development team that we try and do a better job of keeping people aware of what we're up to as a project and what our upcoming plans look like. To that end, the updates that have been traditionally provided on a monthly basis to the Board will be done publically using wiki postings. Obviously, we can't share 100% of what we're doing because of, for example, security issues we haven't disclosed yet, but for the most part I'm going to try and limit the amount of sanitization.
We welcome feedback on anything we're doing via either comments or preferably the development list.
Before I cover more recent activity in another entry, it's probably appropriate to review 2017 to date.
This has been a transitional year to some degree because there tends to be a rule of "three", whereby the third version of something tends to be the thing you wished you'd done to begin with if you had had the time. While it's technically the fourth "feature" update, IdP V3.3.0 was really the third substantial one (the first was really a bug fix that necesitated API additions, so it became a minor bump instead of just a patch).
The major goal for this release was to deliver a revised framework for handling multi-factor authentication and for orchestrating more complex login flows to address a lot of the enterprise use cases we've been struggling to support well since the original release. Since then, there's been a lot of time spent documenting and supporting that work to make sure it's working for people and to figure out the best ways of doing various things with it. I personally have probably spent up to 15-20% more time on support than usual in fact.
The rest of our time not spent on the usual day to day has been divided between:
- V3.4 planning and development
- Mapping out a plan for a new SP version
- Completing the transition of all our projects to git, which is now completed
- Consulting on Internet2's TIER packaging efforts
- Transitioning the old web site off our system to relieve the developers of the burden of supporting it
- Summer-long discussions and planning for the support changes announced last month
A few of those are worth some elaboration.
The IdP V3.4 release is really more about V4. It's a transition release intended to warn deployers about any use of deprecated features or settings so that there is ample time to adapt anything before V4 is released and removes most of those bits. There are some features planned, notably new support for metadata-driven configuration, but the main purpose is to suitably sunset the V3 branch to support a hopefully-seamless transition for most deployers to V4. That will be the first major release we've ever done whose goal is not to rewrite the software or deliver radical changes but to actually pare back the code and remove old features, change defaults, etc. We hope to deliver V3.4 no later than Q1 2018, and hopefully sooner.
The SP is long overdue for a major update, but realistically that's not likely to happen given its relative maturity and the bandwidth we have as a project. What is needed however is a documentation cleanup, a lot of dependency care and feeding, and a reassessment of our platform support (i.e., Solaris, this means you). So there will be a V3 SP but it's not likely to be a rewrite or even a significant change really (with the notable exception of a native IIS7 module which is already available to test). Unfortunately that doesn't make it a small amount of work because the bulk of it is work we have to do to remediate issues in our dependencies, many of which are no longer being effectively maintained by the Apache Software Foundation. In a perfect world, we would change them to more supported options, but that's a bit prohibitive without additional developers to pick up slack in other areas.
Those are the major highlights. This month's "regular" update won't be very exciting given that a lot of it is covered above but will be put together for tomorrow's Board meeting, so look for it shortly.