As discussions have continued the shape of the proposed solution has changed. Rather that rewrite the document I have added sections. It doesn't make for great readability, but it does make for easier re-review.
During the life of IdP V4 we want to put the framework in place to allow us move away from a monolithic approach where everything anybody might want is available to a slightly more modular approach. where deployers can select what services they want.
Examples might be
- Nashorn support (post JDK15)
- Rhino support
- Duo Plugin
- Database backed Storage (consent &c).
This documents explores the area.
- Easy to use as a deployer and as a developer
- Allow secure (manual) update from a server
- Flexible update detection (for instance, a module's IdP version dependency must not be statically defined when the module is installed).
- Flexible enough to do OIDC (with maybe some hand config)
- Stackable (multiple modules)
- Start development for 4.1; Full functionality ready for 5.0
Two key things have resulted from discussion so far
1) The plugin name is critical
This name must be
- Capable of being part of a file name and part of a URL without escaping.
The suggestion is to use the reverse DNS notation that the author controls hence
net.shibboleth.plugins.scripting.rhino could be used by the Shibboleth project because it owns the
*.shibboleth.net domain tree.
2) Our thoughts on trust and signing have developed.
For a product called
- A new directory
- Build-war will incorporate all of
dist/webapp-*(in a random order). This is this done after the population from
dist/webapp(as now) and before
- We might want to add a
post-edit-webappfolder to our distribution to allow us to force the jars we want.
- The plugin can thus use
postConfig.xmlto setup global beans
- The plugin can use this to define flows
- The plugin can do that to make jars available
- Build-war will incorporate all of
- A new file
credentials/plugins-trust/net.shibboleth.foo/truststore.ascwhich contains the public PGP keys that the deployer has accepted for use with this plugin. See PluginTrust
- Files may have been added across the installation (but *not* in system or dist)
- These will NOT overwrite existing files.
- Properties files auto-edited
- Files added to
List properties can be edited. For example
List merging is done using
And the installation is required (via preConfig/postConfig) to define
BEAN.idp.service.attribute.filter.resources.net.shibboleth.foowhich is the list merge of what it needs plus
This list merging is primarily aimed at adding files to existing service definitions
- Files added to
Driving the Installation
The install package is shipped as .tgz (Linux line ending) or .zip (Windows). The distributiin must be accompanied with an armoured detached PGP signatures.
The format of an unpacked package is
Example package layout
|not allowed (it is refreshed by the installation)|
Contains the bootstrap information for the plugin installer
|recommended (for licenses)|
Note: changes to this file will not be installed and so the name should change with changes to ensure that the customer site has up to date licensing documentation
wildcard copied to
The Service Interface
The installation is driven from within the (unzip'd) package itself via the service interface. So the installer just needs to unpack, add
edit-webapp/WEB_INF/lib (in the package) to the classpath and fire up service API to see what it has to do.
The Interface will have methods to return (at least) the following
|Used as per above to uniquify file and property names|
0 or more property file names to add to
Not supported in currently
|0 or more files to add to the distribution|
|0 or more files and their location. The <Path> is relative to idp-home . The user will be prompted to download these (leap of faith). The alternative for them is to download this by hand.|
|1 or more locations where a property file can be located. The properties will include those with a name derived from the plugin Id.|
A list of property files and paired with a list of properties to merge. So in the example above if would be
Not currently supported
The plugin ID and the initial keystore are included as text (so no suspect jar need be on the classpath) in a directory with is part of the package (but not installed) called "bootstrap". This contains two files
opertya file with a single property in it
keys.txtthe signing keys. This is optional. See PluginTrust
Uninstallation cannot occur because of any changes to the configuration will render the system unusable. As part of an upgrade the webapp parts of the previous distribution will be removed but nothing else.
Doing an update consists of collecting the new version, uninstalling the old version, installing the new one.
- Because the config files are unchanged, configuration is rolled forward.
- Depending on how much we care, we can do the usual ACID tricks to be able to roll forward or back from crashes during an update (mostly using file rename as the ATOMIC operation). See below for an example
- So this requires that an install (and indeed an uninstall) is idempotent.
This has many different aspects. The four most important are:
1) Jar versions.
Jars can now be sourced from four different types of place
- The IdP.
- This plugin
- Other plugins
If each supplies a jar which provides the same class, precisely which class will be used will be random.
I do not believe that it is sensible to check for this. so the best we can do is provide mitigation, probably via some orthogonal configuration. This might (for instance) be a series of regexps which would make up a black list of jars not to copy. How this is communicated is tbd.
2) IdP to Module versioning
There is a new IdP version. Will the modules work with it?
- We can be told (by the deployer) the new IdP version to check for
- We know our current version (see above)
- We can determine (see below) what the IdP version limits are (as of today, not as of when the plugin was made)
3) Module to IdP versioning
This Module has a new version. Can it be installed against the IdP?
We know the IdP version (it is written during install and upgrade to dist):
- We know the new versions which are available (see below)
- We can determine (see below) dynamically what the version limits are.
- Therefore we can propose the latest version known to work with this IdP version
Which of course doesn't help the "How do I update my IdP" story unless the plugin writers allow some spread of supported versions so allow staggered upgrade.
4) Module to Module versioning
Out of scope however we should make the following recommendations (tbd whether this is a SHOULD or a MUST
- People building plugins should eschew any dependencies which are not "part of the IdP" (where that needs to be defined - for instance what about Spring?)
- Where possible, consider shadowing dependencies.
The Update Property file
The file is is located at the URLs pointed to by
getUpdateURLs(). Multiple locations allow for redundancy. The property names are derived from the plugin id , with the specific version appended where this is relevant. All three digits of the version information should be present and dot-separated.
|Property Name||Property Value Description||Example|
|A list of versions in arbitrary order. All three digits of the version must be present|
Space separated list of directories where this version is to be found.
Multiple values allow redundancy.
The file name. the following four suffixes will be added during download
The maximum (exclusive) IdP version supported.
Trailing version and ppatch levels of zero are inferred
The minimum (inclusive) IdP version supported.
Trailing version and ppatch levels of zero are inferred.
Rules for version comparison
- Assume a patch of zero if not defined
- Version minimum is INCLUSIVE, maximum is EXCLUSIVE
- so min 4.1, max 5.0 would allow anything starting at 4.1 but not an upgrade to 5.0
- Patch versions do matter (but should not be specified if not needed)
- Iff major versions are the same then compare minor versions
- Iff major & minor versions are the same then compare patch versions
Detecting the need to update
This is all done from within the IdP in an admin flow.
- All installed plugins will be in the IdP classpath
- Then use the service interface to get all plugins
- Collect the
- Open the this as a property file and check the version/subversion/patch for increasing values against what the plugin claims
- If the number have changed we want to propose the upgrade,
In a later release we might chose to have an external program - this would build a classpath from
URL+/package.jar.tgz.sig(or zip equivalents)
- Check the signature against the keystore
- Unpack to a folder for update and do the obvious thing
- Check versioning
- Uninstall old,
- Install new
- Do as much ACID dance as we want to make this crash proof:
Example of being crash proof
- Unpackage to
- If discovered on restart this is deleted (undo)
- Rename (atomic) to
- If discovered on restart operation restarted here (redo)
- Uninstall old (from
- This is idempotent. The existence of the unpackage folder causes a post crash restart from (2).
- Install from
- This must be idempotent (so care needed with property manipulation)
- Commit changed by deleting
- Or (destructively) renaming to
- Or (destructively) renaming to
The initial install is made fraught by the whole "bootstrapping trust" thing. You have an artefact which you don't trust which is the only place you can do to find out it. For the initial bootstrapping of trust we do not want to run any code which we do not trust. The trick is therefore to
- Open the distribution as a zip or tgz file (needs commons-compress)
- Extract the plugin id and the certificate used to sign the distro
- Query the user to add the certificate to the (per plugin) trust store
- We now have "trust" established
- The distribution can be unpacked and the services interface checked.
- Check versioning
- Install in the canonical manner
Blocking IdP upgrades
Given a putative IdP version "it can be seen that" one can test whether the installed plugins are compatible. This leads us to two options
- An admin flow to "test for update". This is the easier option
- A way to probing an IdP installation for it's version and using that before the install starts to block the update. This is a harder option and I'd sooner not do it to start with
Behind all this is a feeling that there is a "turtles all the way down" solution by which the IdP also exports most of the Service Interface and this can therefore be used to drive update detection and testing and even "download and update" the IdP.
IdP Updates Insert
IdP update needs to be modfiied to preserve the
Nothing special. We should have no interest in deploying special windows installation technology. Windows user can just do what everyone else does. Anyone smart enough to want a plugin probably wants to (a) not be running Windows and (b) owning their jetty install.
My ideas here are not well formed yet, but there seem to be at least three different areas
- Testing the plugins themselves (as stand alone items)
- I think that we will discover how best to do this as we develop the plugins.
- We probably want to test not just the functionality the plugin provides but also its plug-in-ness
- Testing the IdPs interaction with the plugins
- The trick here is that we can do this testing by adding the plugin jars as test dependencies of the areas we want to excercise the IdP in
- Testing the module installer
- I this this might be best addressed by building a testing framework for the IdP Installer and extending that.
Of course there are two axes of "one off testing" and regression/unit testing, I think I'd sooner concentrate on the latter and leave the former as ad hoc testing as part of the development process.
Project Process Issues
Again, this is an area where my ideas are not well formed yet. Some observations
- We agreed that plugins live in separate git repositories. Small plugins (like nashorn or rhino) share a repository
- I Created - IDP-1595Getting issue details... STATUS to capture the work items I envisage as being required. That case, and specifically the sub tasks, might (or might not) provide a suitable location for discussions of the details.
- I have created the following "under contruction" documentation pages
- There is the usual issue of where to put the classes. Development is happening in the idp-installer project. But if we want to use classes from this in an administrative flow then they will need to be moved to "somewhere else" and it is not obvious where that somewhere else is.
These come in two flavors
The plugins need to obey relevant licensing restrictions. The plugin architecture allow the optional install time download of files which can help this
Plugin Installer licensing
Building the plugin will require some new dependencies. We need to be aware of them
- Bouncy castle (for GPG signature checking).
- This is already a dependency, so we assume that we are covered
- Commons-compress (needed to unpick tgz files)
- The is the perpetual where to put the classes question.
- This is all being developed in the idp-installer project
- But if we want to report about plugin status in an admin page or jsp or some sy