The project had a productive holiday period, with a fairly unprecedented three software releases done in one week, IdP V3.4.8, SP V3.2.0, and xmlsectool V3.0.0. We also launched a new web site.
V3.4.8 was of course the final IdPv3 release and the branch is nominally closed. We may backport some fixes if circumstances warrant for those wanting to continue maintaining their own builds, but there will be no official releases of any of the relevant projects and we will be shutting down our Jenkins builds shortly. We bid a not-so-fond adieu to supporting Java 7 (and 8 and 9 and 10). While V3.4.8 is still unsupported, we do recommend it to anybody not planning to upgrade to V4 soon simply to obtain the latest (and last) version of Spring 4 and other minor updates.
Work continued, though intermittently, on IdP V4.1. I think Q1 is still our target. We're definitely in the final phases so not much new work is happening from here out, just cleanup, testing, documentation, etc. The OIDC code has been repackaged into the net.shibboleth namespace but documentation still needs to be written. There are significant, though we hope beneficial, changes from the earlier "non-final" versions. We intend to maintain separation between plugin/extension documentation and the IdP documentation so that they can evolve independently. Not much is done yet, but it will show up in the IDPPLUGINS space.
While the SP release was completed using the existing packaging approach, we have a working prototype of a new build and packaging process using Docker that we hope to migrate to for future releases, but the side effect of that will likely be the elimination of many older and little used platforms. One side effect of this should be a more readily extensible basis for others to build their own packaging on for private use.
On the subject of the SP, we have an initial Zoom call scheduled with members and some other invited expertise to discuss the future of the SP and its sustainability challenges this week, and we will be making some additional public announcements about our plans and design ideas in the near future. We're not in a big rush on this, but hope to keep this conversation moving and maintain some momentum so that once additional time is freed up later this year we can hit the ground running.
The slides presented at CAMP a couple of weeks ago are attached.
Since the last update, a draft of the documentation on the new IdP plugin system has been put together, and we are starting to build snapshots of the current round of plugins to test installation and upgrades.
Work continues on the OIDC plugin configuration and much of the manual work to install it has been eliminated, and additional improvements to bring things in line with existing system conventions are still coming. Some "rough edges" of the original design have been identified and will be ironed out as we make changes for this "official" release. We hope most of this work will be wrapped up this month so we can get to testing the OIDC changes in earnest. Setting up an AWS-based testing environment we can use going forward will be an important part of that work.
The beta announcement of xmlsectool V3 was sent and the official release will occur this month, along with an IdP V3.4.8 patch to wrap up work on the old branch along with some final library updates to get that release on the best possible footing as it sunsets in a few weeks. This will include an update to the last release of Spring 4, which is coming this week.
Jetty 10 and 11 were released this week (the difference between them relates to a major breaking change happening in the Java world from the original javax.servlet APIs to the new Jakarta-based APIs that are replacing them. IdP V4 (and 3 of course) is based on the original APIs and will not run on Jetty 11. We expect that it will run on Jetty 10 but testing that is not a top priority for the moment and we have no configuration examples for that at present and do not plan on extending official support to it until V4.1 at the earliest. It seems likely that moving to the Jakarata APIs and Jetty 11 will be a logical step to take in IdP V5.0, but that's only practical once Spring is updated and likely would happen after Java 17 replaces Java 11 as the next LTS release. So this is all speculative and would be in 2022 at the earliest.
(A similar issue exists with Tomcat, which appears to be moving to Jakarta APIs for Tomcat 10, and so we will be left supporting only Tomcat 9 for now.)
On the SP front, the recent OpenSSL advisory was a non-issue but was a good forcing function to get V3.2 completed as planned before the end of the year. Work is mostly complete but we can take an extra week to fix an additional cosmetic bug or two and do more testing, so it should be out early next week. There have been a handful of additional deprecation warnings added to address some poor choices in configuration names, but not much else is noteworthy for this version.
An ACAMP session was held to discuss the future sustainability challenges surrounding the SP software. The notes reflect a productive, engaged discussion, and we will be contacting the people who have expressed interest in further collaboration on this topic soon.
Over the last month we have been working towards wrapping up work on IdP V4.1. There's still a lot left to do, but the end is at least visible and we are still targeting a Q1 2021 release. As I have said before, any notion of a V5.0 has been deferred more or less indefinitely until there are technical/platform/Java motivations for doing one.
The module and plugin systems are still undergoing some evolutionary changes as we start implementing them in real-world extensions and discover what other management and control points we need to manage installation and updates.
Simplifying changes to the existing configuration are mostly done at this point, and a substantial amount of new documentation as been written. The new documentation sections are generally in tabbed sections that separate V4.0 from V4.1+ (the latter really being targeted at new installs of V4.1 with some material included that explains how to adapt a pre-existing system to some of the new property-driven mechanisms). In the Reference sections, it should be evident that the changes typically involve reducing the number of Spring bean definitions and increasing the number of properties used.
I expect a certain amount of confusion over these changes but for existing deployers things should still just work as is. New installs should be simpler to configure and hopefully with some time to assess the changes people may be able to take advantage of the changes to reduce the complexity of existing deployments, more for people that might have to take them over than anything else.
Work continues on the more low-level configuration to address all of the many places that complex plugins (i.e., the OIDC OP extension) have to be able to add material to the configuration.
V3.0 of the OIDC OP extension will be made available with the V4.1 release of the IdP, and the code packages and configuration will be in their final "official" form for long term support by the project. With maybe about 30% of the work done, a lot of the OIDC configuration has already been reduced, eliminated, or moved to internal resources so that the extension can be more automatically installed and enabled without as many manual adjustments to files. Most of these approaches were simply impossible prior to V4.1. There will be substantial "mechanical" changes needed for those already using this extension but the final product should be much cleaner, and the package renaming made many changes inevitable anyway. Work is also largely complete on a SAML metadata implementation for management of OIDC RPs which will extend many of the more advanced IdP management techniques to the OIDC support.
In addition to getting the documentation in order, I did some significant testing of the latest IdP snapshot on Ohio State's V4 configuration and apart from some bug fixing, everything seemed to work compatibly. I was also able to retrofit my configuration to take advantage of properties without too much effort, but it is a little confusing to get from point A to point B just because of all the XML one is wading through. It's really a function of knowing what one has changed to make sure every change is captured in a property.
As I mentioned last month, there should be a new release of xmlsectool designed to work with Java 11 coming shortly.
Nothing to report on the SP front, but there are still plans for a (very) small bug/feature release before end of year.
Next Monday afternoon (my time) I will be providing a 2020 review and project roadmap at Internet2's virtual CAMP conference that is taking the place of TechExchange, and those slides will be posted as soon as the slides actually exist. We probably will arrange a separate presentation for Consortium Members that aren't attending CAMP.
In the last month, the APIs and design of the new IdPModule and IdPPlugin features have firmed up and we have completed converting all of the "likely" candidates for modularization in the core feature set. Much of the XML configuration across the admin, login, interceptor, and c14n flows have been reduced or eliminated in a backward-compatible way in favor of properties for most simple settings, and any remaining XML files for those features have been buried within modules and will be left out by default.
A lot of documentation updates need to be done to update things without totally disrupting existing readers, but we have a strategy using tabbed content that seems to be non-intrusive (see AuthenticationConfiguration for some examples). The changes are really just to how existing settings can be applied, not so much to function or features, so hopefully this won't be viewed as disruptive despite the volume of changes. Note that properties can be injected externally much more easily, which is hopefully of some value to those using Docker.
The TOTP and Duo Universal Prompt plugins are nearing feature completion and are on track to ship alongside this release and work will hopefully get underway this month to prepare the OIDC extension for its final "official" form as a project-delivered code base (as we have warned, there will be changes required for those who have deployed the earlier releases but we will try to minimize them). That work, documentation, and testing are the main blockers to get V4.1 completed, and a 2020 release is not impossible, though probably a bit aggressive.
At this point we expect to deliver at least one "final" 3.4 patch update to close out the legacy code branch before the end of the year and leave things as clean as we can, but we do not anticipate there being flexibility about the EOL date because Spring's plans haven't changed. We want to reinforce that trying to create a "clean" slate to upgrade from V3 to V4 is an absolute mistake. We urge people in the strongest possible terms NOT to worry about "cleaning up" anything but the things the log warns are deprecated or that are highlighted in the Release Notes. The V4 horizon is not expected to be short and there's plenty of time later to worry about cleaning up or updating settings.
On other fronts, we are working on a refresh of the xmlsectool utility to rebase it on the Java platform used by the V4 IdP as we get closer to sunsetting the older platform.
We also have some accumulated fixes to the SP and a couple of member-requested features that will probably result in a V3.2 by the end of the year. We are hoping to be able to move to a new, more automated build process for the packages with that release.
September included a Windows patch to the SP and some significant fixes for (uncommon, so far) issues with the HTTP stack in the IdP. The latter are not expected to ship unless they cause more trouble until more critical bug fixes warrant a new patch.
Over the last month's work on IdP 4.1, much of the supporting work to make the new plugin framework useful has been completed and we have been able to address most of the manual configuration challenges of adding new flows to the system. This work has led to revisiting portions of the configuration, and much of the high level authentication behavior (what was previously in conf/authn/general-authn.xml) will now be configurable with properties instead of XML.
In the process of working on the problem of populating default configuration files after install instead of all up front, a companion notion to the plugin framework patterned after Jetty modules has been developed this month called IdP modules. Along with a new command line tool to manage them, the module system is a way of bundling rules for installing, updating, and removing configuration files, views, and any other necessary files in a deployed system. The IdP need not be running (and usually shouldn't be) to alter module state.
As an example, all of the individual login flow implementations will be exposed as IdP modules, and the default configurations hidden (and generally unusable) until the module is enabled to make use of the feature, causing the files to be added to the tree. In addition, the module system understands how to manage the files it owns, and supports RPM-like semantics to preserve configurations during upgrades or on removal/disablement. So people should expect to run into .idpsave and .idpnew file extensions on occasion when defaults are changed.
We plan to extend this module design to most of the "supplemental" features in the software that are managed with dedicated configuration files today, reducing the number of files in the way out of the box.
The module system supports the plugin framework by allowing plugins to expose their own modules as needed, and taking care of much of the configuration file placement that would otherwise be left to the plugin installer. Plugins can obviously enable their modules on install and disable them on removal automatically.
We also hope that for many older deployers, reducing system clutter by simply disabling modules not being used should be easy.
As this work has taken shape, it's becoming clearer that we are targeting 4.1 as the next "big" release of the IdP rather than deferring certain work to 5.0. We have started internal discussions about exactly what work might be moved in to this version (and to the plugins that would accompany it), and will adjust the expected release date accordingly. We're already looking at no earlier than the very end of 2020 and may push the release into 2021. At this point, 5.0 probably is more like 4.0: a release done out of necessity at some future point to make changes to the Java and Spring platform, rather than a big feature release. Since we don't support older minor versions anyway, extending the useful life of 4.x is ultimately better for everybody.
We have continued to work on the new IdP plugin framework and are nearing feature completion on several of the first round of plugins we're building, which I mentioned in last month's update. The new Duo plugin in particular will require some time to polish and complete but the bulk of the new work is done.
The major remaining tasks center around completing the plugin framework itself and testing all of that, plus significant changes to some of the IdP's core configuration approaches, to address issues we've identified that make it harder for plugins to "slot themselves" into the system effectively without requiring lots of manual additions. We want to minimize "boilerplate" sorts of changes such that only changes to defaults will require any additions.
As an example, right now adding support for a new login flow via a plugin would require editing general-authn.xml to add a descriptor bean for the new flow. That's going to be eliminated so that default settings for the flow just get applied. The act of deploying the plugin will suffice to install the defaults.
This is part of a larger effort for V4.1 to refactor and improve the configuration. This is all in the vein of "we didn't have time to make it simpler" until now. In some ways, doing this as part of V4.1 is actually a better choice than V4.0 because it forces us to remain compatible as we make these changes, which in turn minimizes the pain of the V4.0 (and V4.1) upgrade process. Using major upgrades can be a crutch to make breaking changes that may be easier to implement but cause pain to others. I prefer we take the path of creating more work for us and less for everybody else.
The configuration improvements have been happening in a staged fashion:
- First, we took on the task of eliminating the system/ folder. This is a long story but suffice to say it's hard to do, requires copying code out of Spring, and won't be 100% gone until V5, but it is essentially unneeded now.
- Next we need to re-engineer some of the internals as I described above, which will allow plugins to be designed more effectively and reduce the need for new configuration of default settings.
- Then we will be retrofitting some of these changes back into the original code and in the process hopefully removing some of the extraneous configuration files from the shipping defaults. This won't help much for upgrades, but new installs will hopefully include fewer files and the system's apparent, if not actual, complexity will be reduced. This may be too-little, too-late but we'll see.
One of the features we will need to add to make this a reality is the ability to "populate" a new default configuration for a feature so that it can be generated by the deployer using a command rather than having to pre-populate all of the possible settings the way we do now. This is also part of the plan for the plugin framework, modeled on many similar systems such as Jetty's.
Lastly we've added a new team member who will be starting to get familiar with the SP build process and eventually automating the packaging we do for it, which will help reduce the amount of time new releases will take. With OpenSSL 3.0 shipping at the end of this year, we hope to be able to exercise a new process to release a compatible version early next year.
Work over the past month has included:
- Research into some complex bugs and behavior that were in the backlog.
- Progress on some of the more complex work scheduled for V4.1
- Work to cleanup and stablize the MDA API for the eventual 1.0 release of that software.
The recent Comodo CA expiration resulted in the observation that the IdP's behavior in the case of LDAP and HTTPS certificate evaluation was different (an expired trust root worked for LDAP, but failed for HTTPS). This took a significant amount of investigation to understand, but we discovered that Java actually has two different PKIX certificate validation libraries built in, and both are being used. One accepts expired roots and the other doesn't. This isn't ideal, but messing around in those layers is even less ideal so it's likely to continue behaving this way.
On the 4.1 front, work on the new plugin architecture continues and we're building several test cases for it around new features that we would like to keep out of the core. The intent is really to try and start delivering more features outside the core release process so they can be maintained with more agility, and in the process probably move to a more regular cadence for the actual software itself, to focus it more on bug fixes and library maintenance. Success will ultimately be measured by the ability to make the deployment process seamless in the sense that it shouldn't matter a great deal whether a feature is part of the core or in a plugin.
Initial plugin work includes:
- Scripting alternatives to accomodate the removal of Nashorn from Java 15+
- Implementation for the next-generation Duo WebSDK
- A TOTP login flow based on a Java port of Google's OATH library
Work on all of them is in various stages of advancement.
One thing that probably distinguishes plugins from core work is the impact on schemas. When it comes to new functionality added to the metadata resolver, attribute resolver, or filter services, which are essentially the remaining components that rely on "custom" configuration schemas, it will be much simpler to continue to include those components in the core software in order to prevent further pollution of the configuration by other XML namespaces to prevent conflicts. An example of this is a new feature to support decrypting data in the attribute resolver. The TOTP feature requires the ability to potentially decrypt token seeds, so rather than build that into the feature directly, a more generic capability was added to the core, and the plugin can leverage that feature for its specific needs.
Since the last update we completed work on the first IdPv4 patch release to address a small number of bugs and some features that were lost in the transition. We believe the bug count has been somewhat artifically low due to the slow pace of adoption but having been running it in production for several weeks I'm also pretty comfortable that there aren't a lot of big bugs lurking, at least new ones.
One issue we hit involved an internal Jetty 9.4 change with wildcard certificates that we needed to accomodate in our examples and (in rarer cases) in the back-channel module and plugin we've provided, largely unchanged, for a number of years. A new plugin was required, and the documentation was updated to reflect this.
There are definitely other pockets of stale documentation that need updating and we need people who encounter those problems to report them so we can get them fixed.
We've since opened the master branches of all the projects for new feature development on V4.1.0 and have started merging in completed new work. A new Jira state of "Pending" was created to track work committed to development branches that is complete but awaiting merge back to master. By week's end most of this work should be merged in and 4.1 superscripts will be appearing in the documentation.
There are a couple of ways to track the work on the next minor and major versions: issue filters (4.1.0, 5.0.0) as we have used in the past, and some Kanban boards for those who might prefer that kind of view. This is increasingly the most effective way to monitor our "roadmap" and the usually-stale Project Roadmap page has been updated to reflect this.
Right now we're mostly knocking off low-hanging fruit and work that's crucial to the eventual transition to V5 rather than complex new features, but we have significant MFA feature work in the plan for 4.1 and have built a PrivacyIdea testbed in AWS to support this.
Initial work in support of the new plugin design is also starting to get merged in and some simple plugins needed for future releases will be built to exercise that code.
Work on the MDA continues as well to move it closer to a V1.0 release this year.
We have completed migration of most of the project infrastructure to AWS, which will be more cost effective and simpler to manage. All services have been moved and are fully functional except for Jenkins, which is being debugged to get all the jobs working again, after which we're hoping for some improvements to the frequent networking problems we have encountered in the past. Total time spent on this is over 75 hours and will hopefully not need to be repeated at quite this scale again. Core services are running on CentOS 8 now to extend the useful life of the deployment. We should be shutting down the old system within the next week. We have not deployed our services with region or availability zone redundancy, as that would greatly increase costs and not be an apples to apples comparison (we weren't redundant before).
The SP V3.1 release was completed, with no significant issues arising to this point. The next major hurdle is tracking the changes coming in OpenSSL 3.0, which is available in alpha form. OpenSSL has about the worst API documentation I know of (as in, there isn't any), so there's a significant risk of breakage to a lot of important code in the SP and its dependencies, so the work involved remains to be seen. I would expect this is going to be the sort of change that may result in a 4.0 update because of the impact.
I'm frankly expecting we may lose all ability to support use of certificate authentication in the SP's back channel support, but that's speculation at this stage, it's just something to watch. Moving to port 443 and signed messages is the best way to inoculate against that risk, for the few deployments still in need of SOAP support. We'll get a bug created to track this work.
We have made significant progress on drafting requirements and starting preliminary work on a new plugin design for the installer to enable more factoring of the code base. This is a big deal and it's not being undertaken lightly, but we don't see how to accomplish what we need with any off-the-shelf solution that would accomodate the freedom of deployment models we have now (i.e., we could use new tools and then write off anybody who doesn't use those tools, and we just don't believe in that approach for the core software). Spring, and particularly Web Flow, also more or less preclude some of the approaches we might otherwise consider because the architecture doesn't allow for the sort of class-loader isolation that something like OSGi would require.
Note that we do not plan to move to any sort of piecemeal "patch" approach to delivering bug fixes, as that's a totally chaotic and unmanageable deployment model that undermines the careful versioning policy we adhere to. But the model will accomodate detection of compatibility problems when upgrades are done, even retroactively (the compatibility rules themselves are expected to be remotely obtained). Once the basic framework exists, we probably will find new uses for it, but the focus is on delivering more manageable installation of complex add-ons.
We will be initially designing a number of deliverables to rely on this approach:
- Non-trivial storage plugins
- New (and old) scripting alternatives
- Official OIDC support, both inbound and outbound
- Duo integration via OIDC, eventually replacing the original integration
In the future, most new work that is expected to be of more limited adoption or that requires substantive new dependencies will probably take this form. We welcome interested parties to provide feedback on the dev list. Initial work on this will hopefully start showing up in V4.1.
I would also like to call attention to the fact that yet another IdP-of-last-resort for our services, UnitedID, has ceased operating. SUNET has kindly agreed to make their open IdP (eduID) available more widely. If you have existing bugs or pages you're tracking and need to rename a Jira or Confluence account name that was formerly issued by UnitedID just drop a note to our contact address and we can rename it for you.
Project activity can best be described as a bit unfocused right now, as a somewhat indirect fallout of the pandemic, not so much due to the change in work habits but because it's self-evidently slowed any real uptake of the IdP 4.0.0 upgrade. More typically we'd be fielding a decent range of bug reports and working on a patch but with the paucity of testing so far we haven't seen much, so we're in a holding pattern on exactly what the next release needs to be. We do have at least some positive feedback that suggests the release is solid, at least for upgrades.
We'll probably be forced to open up the master branch for new work fairly soon to start work on V4.1. There's a small set of deprecation warnings that need to be added so that at least one release is done with them prior to V5, but the list of deprecated material is much shorter than the V3/4 transition was. We have no ETA for 4.1, so we do not advise waiting for that update before making the jump.
After the IdP release and with the relative breather, work is complete on the SP upgrade and posted beta packages, with testing as usual being minimal to non-existent (which is why we don't bother much with them). After some additional testing on shibboleth.net this week, the release will most likely be done next week. Apparently, there's not much immediate pressure to get any SameSite assistance out the door because Google kindly decided to exercise their total dictatorial control of the web and defer wider dissemination of their Chrome change. In any case, the ReleaseNotes for the upgrade and documentation on the changes are available.
We are also actively engaged in reviewing a lot of older issues and finding small items to close out (fixed or not) before tackling some bigger work. We have also been working to fix issues with the project javadocs caused by a range of bugs in Java, javadoc, and the Maven javadoc plugin. The javadoc search feature works again and most of the expected links between projects and dependencies are working again, with the exception of the Java EE APIs.
Most of the summer will probably be taken up by further work on advanced authentication features, designing a plugin installation and upgrade architecture, and adapting the OIDC extension into its final form. Some of this work will probably ship ahead of V5, but the bulk of it will be targeted for that release next year.
Not a long update this month, as the activity is just an extension of last month's update. Work has been hard and heavy though.
Two beta releases of the IdP were completed, with a few serious bugs found and fixed, enough that we are comfortable with an official release being cranked as I write this, with the announcement due tomorrow. V4 is the product of over 15 months of work primarily focused on technical refresh and maintaining the maximum degree of compatibility possible as we move forward 4! Java versions and a full Spring version. The OIDC extension should be finalized within a few days of the IdP release.
We believe we have achieved the goal of a true major upgrade that feels more like a patch. Just be aware that the less you notice, the more work went into making that possible. The ReleaseNotes are extensive and cover the highlights. We will be focused on feedback and bug fixing over the next couple of months so a patch or two should be expected.
On the SP front, work has continued on a 3.1 release that has accumulated a decent list of fixes and features. Work on SameSite and correlation support is complete and the remaining tasks consist of fairly minor enhancements and some documentation cleanup that should make a late March or early April release feasible.
By early summer, work will be underway on a roadmap for the IdP that will take us through V5 and full incorporation of the OIDC extension into the project. We expect that V5 in 2021 will see minimal changes for deployers except for the transition of the OIDC module from third-party to fully-supported extension.
It's been a busy month for the project primarily to get the expected beta release of the Identity Provider V4.0 done. Testing to this point has not been extensive, but the early returns from within the team have been positive and the final release is expected by early March at the latest. Upgrades should be very easy for the majority of deployers.
The lifecycle of support for V3 is essentially a foregone conclusion at this point because Spring 4 is end-of-life at the end of 2020. We don't have any compelling reason to abandon security support for V3 any sooner than that, so V3 will be supported through the remainder of this calendar year.
I undertook a mini-project to blitz a port of the OIDC extension and we are on track to produce a compatible extension along with the IdP release. After a short discussion, we agreed to forego significant breaking changes to this code for now and leave things largely as they have been for this release.
For clarity, since this is a period of transition:
- The code in GitHub is the V3-compatible extension. Issues with it can continue to be reported via GitHub if desired.
- The V4-compatible extension is largely the same but has been moved into our Git repository and will be distributed via our download site. The documentation remains in the GitHub wiki, but we are using our Jira instance for bug reporting and patch submission.
- The V5 extension when it comes will include a number of breaking changes so that the code packages can be renamed and configuration layout changed, and the documentation will be moved into our wiki. At that point we will be in a position to fully support the extension to minimize breakage in the same way we have done for the rest of the feature set.
In parallel with the beta effort, work has begun on the next SP update, V3.1. This feature upgrade is primarily focused on addressing XSRF in two areas:
- Providing a fix for the SameSite issue in a Safari-compatible way
- Adding request/response correlation along with an option to block unsolicited SSO
The SameSite work is complete and will explicitly designate a subset of SP cookies as SameSite=None, with an optional setting to add a second non-SameSite cookie to support broken browsers for hopefully a temporary period of time. It will also be optionally possible to support cross-site use of SP sessions, when XSRF is a "feature" and not a bug.
The correlation work is underway, but is being carefully implemented to maintain compatibility so it will be an optional feature for security conscious deployers to turn on.
There are a few other odds and ends to complete for the SP release, but we're moving aggressively to complete that work as soon as possible so it's quite likely March will see both IdP and SP releases.
We have a significant queue of work items for the IdP once V4 ships, primarily authentication-related, so the roadmap should be getting cleaned up and clarified once we can take stock of things.
Welcome to the third decade of the Shibboleth Project.
Various cancelled meetings, holidays, travel, and just generally getting work done have led to a lack of updates, but as of the new year we have started the process of wrapping up V4 IdP development so we can get a beta released as soon as possible. Much of the smaller items left to do are now done, and most substantive remaining issues have been pushed out to future versions, so what's left is primarily testing, finishing up work on the SAML proxy feature, and beginning to update the OIDC extension.
The installer is essentially complete at this stage, and is substantively the same as before though has been rewritten to make more extensive use of Java code in pace of ant so we can make future enhancements to it more easily by "just" writing code.
The SAML proxy feature is missing most of the necessary security code. This will hopefully start getting committed in the next week or two. It is however functionally done and documented (SAMLAuthnConfiguration) and I don't anticipate significant changes being made for this first iteration.
The biggest remaining task is prepping an upgrade of the OIDC extension for use with V4 so that it can be made available at or near the initial release. As recently announced, we have officially assumed ongoing responsibility for the code base as of last month, and have taken the first step by importing the code into our official repository. Our plan is for any required maintenance on the original V1 branch of the extension for use with IdP V3 will be carried out in the original github repository, while work on a V2 extension for use with IdP V4 has begun on the master branch in our repository.
After some consideration, our intent is to maintain most of the current configuration, XML namespaces and types, and Java packages of the original extension, with the understanding that the implementation released with IdP V5 next year would introduce a number of breaking changes to make a number of final adjustments for the long term sustainability of the code. This version is therefore a transitional release, and deployers of the OIDC extension now and with V4 should expect some additional work to upgrade to V5 later.
As part of this transition, the documentation will likely continue to be maintained in the GitHub project wiki since it will be largely the same for this release, with the updated/integrated documentation moving into our wiki for V5.
It is however, a discussion point as to whether the code will be moved into the IdP core or maintained as a separate extension for organizational purposes. If that is the decision made, we will out of necessity also take on the delivery of some kind of formalized "module" system for the automated install and update of extensions to the core software to minimize the impact of that decision. It will, in other words, not matter all that much to a deployer whether the code is separate or not, but a separation will help us design such a module feature, and potentially reduce the impact of security issues in the relatively less mature OIDC code from the rest of the code base.
Long story short, lots of work was completed over the holidays, there's still a fair bit left to do, and we're working hard to get a beta out the door as soon as possible.
Famous last words as we had to ship yet another IdP V3.4 security patch last month to address a memory leak, which required an unfortunately disruptive change to the classes supporting the External authentication flows. We don't take making breaking changes lightly, and do apologize for not fully recognizing the risk of impact to third party plugins, but it was a good chance to remind people that it's just not safe or supportable to directly copy most of our flow definitions because they include non-public classes that are subject to change. We're happy to work with people on-list to provide any missing extension points needed if they're consistent with our goals.
Work on IdP V4 has continued aggressively of late, and the installation changes are wrapping up, which should position us to ship without any major required work items left holding things up.
A new feature I just completed that's worth highlighting: it will be possible in V4 to split one's metadata configuration such that metadata filtering rules can be separated from core metadata provider configuration, allowing a separation between core federation-delivered metadata service definitions from local deployer-created filtering rules.
A few significant updates on the plans for V4:
- We're definitely not intending to ship the OIDC add-on in this release, but will work to ensure a compatible version of it is available at the same time as the release along with additional support for use of SAML metadata to manage relying parties. The final integration into the code base will be a major V5 work item.
- We are optimistic that we will be able to provide a first iteration of SAML proxy authentication support in V4, as the work has proceeded well to this point and we don't foresee any hold ups. It may not be 100% feature-complete, but will be usable certainly, so we can evaluate whatever else might be necessary.
- We continue to expect a seamless or near-seamless upgrade path to V4 from V3.
- We are hopeful of reaching a very late alpha or hopefully beta release by the TechExchange conference in December, and expect to release by early next year. We could probably make an earlier date, but the extra time will help with testing and documentation.
Some on and off work on the SP has been occurring, mostly in response to a serious crash under load caused by a bug in the newly introduced cookie session recovery feature. The fix is done, but we're probably looking at a V3.1 SP release towards the end of the year simply because of resource constraints. We would like to get a new RPM packaging process in place before doing another release, so that will take some extra time to manage. We do have packages available for CentOS 8 for the time being.
The main highlight for this month is that we're going to be issuing an IdP patch, V3.4.5, probably next week. The bug isn't a major one but is an opportunity to ship a few fixes before V4, the most significant being IDP-1491, which allows binary LDAP attributes to work again on newer Java versions where the UnboundID code has to be used. I think we will also include some optional support for addressing the SameSite cookie issue for those deploying without Apache or other software that can more easily address the problem. We have an in depth testing analysis here.
This should, once again, be the final V3 release barring any additional security issues.
I have completed work, more or less, on the revamped password validation support I described last month along with most of the basic documentation updates. We probably will be opening up the draft V4 documentation soon so people can see better what's coming, but we have more work to do on the release notes before that's going to be too helpful.
Support for a handful of new audit fields have been added, such as logging signed requests, encrypted assertions, and requested authentication context information. The default audit format for V4 is not decided at this point, I think we're open to suggestions there, but obviously it doesn't matter much for people upgrading.
An area of current study right now is to make sure all the "fail-fast" and connector validation behavior across all the various services and attribute resolver are more consistent, both with regard to defaults and to make the settings more consistent. This includes fixing the misleading errors that appear when servcies fail to start properly but are falsely attributed to the metrics support.
Most of the deprecated APIs and features have been removed from the master branches as we get closer to a release. I think the only real area of further revamp right now is probably the installer; everything else left is more around new features that we can more easily include or exclude based on the desired schedule.
We're still waiting on CentOS 8, but we're not going to spend much time on the SP again until that drops.