Regression introduced by bce5268a21.
The bit size of the initialisation vector for AES GCM has been
introduced in NSS version 3.52 in the CK_GCM_PARMS struct via the
ulIvBits field.
Unfortunately, Firefox 68.8.0 and 76.0 do not set this field and thus it
gets initialised to zero, which in turn causes IV generation to fail.
I found out about this because WebRTC stopped working after updating to
NSS 3.52 and so I started bisecting.
Since there wasn't an obvious error in Firefox hinting towards NSS but
instead just the video stream ended up as a "null" stream, I didn't
suspect the NSS update to be the culprit at first. So I verified a few
times and then also started bisecting the actual commit in NSS that
caused the issue.
This turned out to be the problematic change:
https://phabricator.services.mozilla.com/D63241
> One notable change was caused by an inconsistancy between the spec and
> the released headers in PKCS#11 v2.40. CK_GCM_PARAMS had an extra
> field in the header that was not in the spec. OASIS considers the
> header file to be normative, so PKCS#11 v3.0 resolved the issue in
> favor of the header file definition.
Since the test I've used[1] was a bit flaky, I still didn't believe the
result of the bisect to be accurate, but after running the test several
times leading same results I dug through the above change line by line
to get more clues.
It fortunately didn't take that long to stumble upon the ulIvBits change
(which is actually documented in the NSS 3.52 release notes[4], but I
managed to blatantly ignore it for some reason) and started checking the
Firefox source tree for changes regarding that field.
Initialisation of that new field has been introduced[2] in preparation
for the 76 release, but subsequently got reverted[3] prior to the
release, because Firefox 76 is expected to be shipped with NSS 3.51,
which didn't have the ulIvBits field.
The patch I'm adding here is just a reintroduction of that change,
because we're using NSS 3.52. Not initialising that field will break
WebRTC and WebCrypto, which I think the former seems to gain in
popularity these days ;-)
Tested the change against the mentioned VM test[1] and also by testing
manually using Jitsi Meet and Nextcloud Talk.
[1]: https://github.com/aszlig/avonc/tree/884315838b6f0ebb32b/tests/talk
[2]: https://hg.mozilla.org/mozilla-central/rev/3ed30e6b6de1
[3]: https://hg.mozilla.org/mozilla-central/rev/665137da70ee
[4]: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.52_release_notes
Signed-off-by: aszlig <aszlig@nix.build>
aszlig