[#992] Apply the confidentiality of a backend index to the running backend

[vharseko]

Fixes #992

DefaultIndex.setConfidential only compared the setting with the one the index was opened with. Nothing moved the CryptoSuite the indexes of the attribute share to the new value, and nothing bound their codecs again, so confidentiality-enabled reached the running backend in neither direction, and the comparison never converged: every later change of that index - index-entry-limit included - untrusted it again and asked for a rebuild which could not help.

The declared admin action of the property is that the index has to be rebuilt (BackendIndexConfiguration.xml:230), not that the component has to be restarted, so this makes the implementation answer for what is documented.

What the change does

AttributeIndex.applyConfigurationChange compares the old configuration with the new one itself - setConfidential leaves the Index interface, since an index does not need to know about this setting - and when the confidentiality changed:

• puts the new setting in force on the shared CryptoSuite, through EntryContainer.setIndexConfidentiality, which keeps the cipher of the backend. Done before the write which opens the indexes this change adds, so the key hashed index that enabling confidentiality of an equality index creates binds its codec to the new setting and encrypts its records, rather than only hashing its keys. Applied outside the writes, which the storage may replay: it changes a live object and is idempotent;
• gives up the trees of the indexes which are kept: one write untrusts and deletes them, a second opens them again, which is what binds their codecs to the setting now in force. Both in a branch of their own, which takes and releases the same exclusive access the removal of an index takes; a change which leaves those trees where they are - of the entry limit alone - takes none, and shares the first of the two writes through writeUpdatedIndexes().

The trees are given up rather than left to the rebuild because their records are in the encoding of the setting being given up, and the codec of the new one does not read them back as what they are: an encrypted record read as clear text decodes to an empty set of entry IDs rather than failing, which is a search answered with no entries at all. An empty untrusted index answers "undefined" instead, and so is not used until the rebuild has run. They are deleted rather than emptied record by record, which for an index of any size would be a transaction of its own making, and in two writes rather than one because the storage engines delete and create the tree of an index as operations of their own - as removing and adding an index does. That order also decides how an interrupted change ends: the write which deletes is the one which untrusts, so what is left is an index the next open of the container creates empty and keeps degraded, never a trusted one holding nothing.

DefaultIndex.codec becomes volatile, since it is now bound again on a live instance other threads are holding.

Tests

IndexConfidentialityChangeTestCase, with a subclass per storage engine (PDBIndexConfidentialityChangeTest, JEIndexConfidentialityChangeTest) because deleting and creating the tree of an index is the engine's own operation. Five behaviours each, all failing before the change:

what it pins	before
enabling empties the index it asks to rebuild, which then answers "undefined"	the tree kept its record
what the index writes next is encrypted	the record was written in clear text
the key hashed index the change creates encrypts its records too	its records were written in clear text
disabling stops the encryption	the record was still encrypted
a later unrelated change leaves the index trusted	it was untrusted again, with the rebuild message repeated

Run green: the ten new tests together with EncryptedPDBTestCase, EncryptedJETestCase, PDBTestCase, JETestCase, ReplayedConfigChangeTest, DefaultIndexTest, EntryIDSetTest, StateTest and OnDiskMergeImporterTest - 225 tests, no failures.

Not in this change

An online rebuild-index of an index whose confidentiality was never changed is unaffected. The offline rebuild already applied the setting correctly, since it opens a container of its own; it still does.