Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/skills/tests/author-verify.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,8 @@ describe.concurrent('every type file', () => {
for (const path of paths(source.params)) if (!/^https?:/.test(path)) await writeFile(join(folder, path), sample(path))
const output = join(folder, 'example.h5p')
const result = await buildPackage(source, output, { base: folder })
expect(result.notes).toEqual([])
// The demo's clip, every video's sample, has no sound, and the build says so (#218).
expect(result.notes.filter(({ message }) => !/\.mp4 has no sound$/.test(message))).toEqual([])
await checksAsSaved(output)
await playsBothWays(output)
}, 180_000)
Expand Down
16 changes: 14 additions & 2 deletions design/author.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,20 @@ text to the content check's rules, and `apps/skills` plays every example untrust
address downloaded at build time. It lands under `content/images|videos|audios|files/` with its
MIME type, an image's size read from its header (PNG, JPEG, GIF, BMP; a JPEG's segments walked
to its frame header, which a colour profile can push past 64 kB), and its copyright, on H5P's
own list of types (no SVG, no WebP). A download is given up after a minute with no byte, not
after a fixed time. A YouTube, Vimeo, Panopto or Echo360 address stays an address, as the editor
own list of types (no SVG, no WebP). A video's or a recording's container is read: an mp4 or an
m4a by its index, after `ftyp` or, as in an older file, `moov`, `mdat`, `free` or `wide`, any
other by its first bytes. One its field does
not take is refused (a video takes MP4, WebM and Ogg; a recording MP3, M4A, WAV, Ogg and WebM):
a web page at an address that ends `.mp4`, as a wiki's file page or a share link answers, built
and passed the verifier before (#218). An mp4's index is read through the normalizer's
`mp4Info`, the walk its faststart uses, in Node and on a page alike. No index to read is a
problem. Where the index lists tracks, no sound or no picture is a note, a silent video being
one an author can mean, and so is HEVC. Where it gives a length, that holds an Interactive
Video's times (`videoTimes`): a start, a bookmark or an end screen past the end is refused, an
interaction that ends past it is not, since it shows to the end. No time is named in a list
where an item was refused, since the list's indices no longer match the source's. A download
is given up after a minute with no byte, not after a fixed time. A YouTube, Vimeo, Panopto or
Echo360 address stays an address, as the editor
embeds it, and only where H5P.Video's own pattern for that host takes it, since an address none
matches goes to its HTML5 player, which cannot play a page: a Short or a live stream is written
as its video's watch address, and another form on those hosts is refused by name.
Expand Down
5 changes: 3 additions & 2 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,9 @@ with the specification is a conformance fix.
### The normalizer, `@missing-elements/h5p-normalize`

The command's options, and `normalizeArchive()`, `formatBytes()` and `archiveFingerprint()`, with
their types, and `crc32()` from `@missing-elements/h5p-normalize/crc32`, which embed and h5p-author
take through a range. The content of a package it rewrites never changes. How it lays out the zip may change in a minor, as
their types, `crc32()` from `@missing-elements/h5p-normalize/crc32`, which embed and h5p-author take
through a range, and `mp4Info()` from `@missing-elements/h5p-normalize/mp4`, which h5p-author takes
the same way. The content of a package it rewrites never changes. How it lays out the zip may change in a minor, as
long as the player streams the result at least as well.

### The verifier, `@missing-elements/h5p-verify`
Expand Down
7 changes: 5 additions & 2 deletions packages/author/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,11 @@ What `params` takes is what H5P's editor would let an author save in the same fo
- **Sub-content**: `{ "library": "H5P.MultiChoice", "params": { … }, "metadata": { "title": "…" } }`,
filled in the same way, at the pack's version.
- **Files**: `{ "path": "photo.jpg" }`, a file beside the source or an `http(s)` address, with an
optional `copyright`. Each is copied into the package with its type and, for a picture, its size;
the package is then laid out to stream, media stored and an mp4's index first. A YouTube, Vimeo,
optional `copyright`. Each is copied into the package with its type and, for a picture, its size.
A video or a recording is read from its first bytes, and one that is not what its field takes, a
web page at a file's address among them, is refused; an mp4's index says whether it has sound
and, where it gives one, its length, past which an Interactive Video's times are refused. The
package is then laid out to stream, media stored and an mp4's index first. A YouTube, Vimeo,
Panopto or Echo360 address stays where it is, if it is one H5P.Video plays: a Short's is taken
as its video's, and another address on those sites is refused. A download is given up once
nothing has arrived for a minute, however long it runs.
Expand Down
1 change: 1 addition & 0 deletions packages/author/lib/browser.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ function pageMedia(files) {
keep: (body) => new Response(body).blob(),
async open(file) {
return {
size: file.size,
read: async (offset, length) => new Uint8Array(await file.slice(offset, offset + length).arrayBuffer()),
close: async () => {}
}
Expand Down
10 changes: 8 additions & 2 deletions packages/author/lib/build.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { quizProblems, quizSource } from './quiz.mjs'
import { outlineOf } from './outline.mjs'
import { SourceError } from './source-error.mjs'
import { escapeText } from './text.mjs'
import { LAYOUTS, placesOf } from './types.mjs'
import { LAYOUTS, placesOf, videoTimes } from './types.mjs'

export { SourceError }

Expand Down Expand Up @@ -310,7 +310,7 @@ export async function composePackage(source, pack, media, said = []) {

// Media: copied, described, put in place of what was written.
for (const { field, value, path } of generator.media) {
const resolved = await media.resolve(field, value, path, report)
const resolved = await media.resolve(field, value, path, report, note)
for (const key of Object.keys(value)) delete value[key]
Object.assign(value, resolved)
}
Expand All @@ -337,6 +337,12 @@ export async function composePackage(source, pack, media, said = []) {
if (!known && !shortened(path)) problems.push({ path, message })
}
for (const added of checked.notes) if (!notes.some(({ path, message }) => path === added.path && message === added.message)) notes.push(added)
// Times past the end of a video whose length was read, which only a build, with the file, knows;
// none in a list an item of which was refused, whose indices are no longer the source's.
const timed = (/** @type {string} */ path, /** @type {string} */ message) => { if (!shortened(path)) report(path, message) }
for (const place of placesOf(main, checked.params)) {
if (place.library === 'H5P.InteractiveVideo') videoTimes(place.params, inSource(place.path), (file) => media.lengthOf(file), timed)
}
if (said.length || problems.length) throw new SourceError([...said, ...problems])
const content = checked.params

Expand Down
107 changes: 103 additions & 4 deletions packages/author/lib/media.mjs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { mp4Info } from '@missing-elements/h5p-normalize/mp4'
import { escapeText } from './text.mjs'

/**
Expand Down Expand Up @@ -29,6 +30,15 @@ const TYPES = {
/** The folder under `content/` H5P's editor keeps each kind in (h5peditor-file.class.php). */
const FOLDERS = { image: 'images', video: 'videos', audio: 'audios', file: 'files' }

/** The containers a video and a recording come in, among H5P's types, and how to name them. */
const PLAYS = {
video: { takes: ['mp4', 'webm', 'ogg'], what: 'an MP4, WebM or Ogg video' },
audio: { takes: ['mp3', 'mp4', 'wav', 'ogg', 'webm'], what: 'an MP3, M4A, WAV, Ogg or WebM recording' }
}

/** Codecs of an mp4's picture that not every browser plays, by the name its index gives them. */
const NOT_EVERYWHERE = /** @type {Record<string, string>} */ ({ hvc1: 'HEVC (H.265)', hev1: 'HEVC (H.265)' })

/** Whether a host name is one of these domains or under one. @param {string} host @param {string[]} domains */
const under = (host, domains) => domains.some((domain) => host === domain || host.endsWith(`.${domain}`))

Expand Down Expand Up @@ -100,11 +110,11 @@ const IDLE_MS = 60_000
* name(source: string): string,
* local(source: string): Promise<{ file: F } | { missing: string }>,
* keep(body: ReadableStream<Uint8Array>, extension: string): Promise<F>,
* open(file: F): Promise<{ read(offset: number, length: number): Promise<Uint8Array>, close(): Promise<void> }>
* open(file: F): Promise<{ size: number, read(offset: number, length: number): Promise<Uint8Array>, close(): Promise<void> }>
* }} MediaAccess
* `what`, what a source's path names, for problems; `name`, the file's own name it names;
* `local`, that file, or why there is none; `keep`, a download's bytes kept; `open`, a kept
* file's bytes read from an offset, fewer at its end
* file's length, and its bytes read from an offset, fewer at its end
*/

/**
Expand Down Expand Up @@ -176,6 +186,35 @@ export async function imageSize(read) {
return null
}

/**
* Whether bytes start with an MPEG audio frame's header a decoder takes: the sync bits, then no
* reserved version, layer, bit rate or sample rate.
* @param {Uint8Array} head
*/
const mpegFrame = (head) => head.length >= 4 && head[0] === 0xff && (head[1] & 0xe0) === 0xe0 && (head[1] & 0x18) !== 0x08 &&
(head[1] & 0x06) !== 0 && (head[2] & 0xf0) !== 0xf0 && (head[2] & 0x0c) !== 0x0c

/**
* What a video's or a recording's first bytes say it is: its container, a web page, or none known.
* An mp4 or an m4a starts with `ftyp`, WebM with EBML's mark, Ogg with `OggS`, WAV with RIFF's
* `WAVE`, MP3 with an ID3 tag or a frame header; a page with `<`, in UTF-8 or, after its
* byte-order mark, in UTF-16.
* @param {Uint8Array} head the file's first 16 bytes, or all of a shorter one
* @returns {'mp4' | 'webm' | 'ogg' | 'wav' | 'mp3' | 'page' | undefined}
*/
function containerOf(head) {
const ascii = (/** @type {number} */ offset, /** @type {number} */ length) => String.fromCharCode(...head.subarray(offset, offset + length))
if (ascii(4, 4) === 'ftyp') return 'mp4'
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
if (head[0] === 0x1a && head[1] === 0x45 && head[2] === 0xdf && head[3] === 0xa3) return 'webm'
if (ascii(0, 4) === 'OggS') return 'ogg'
if (ascii(0, 4) === 'RIFF' && ascii(8, 4) === 'WAVE') return 'wav'
if (ascii(0, 3) === 'ID3' || mpegFrame(head)) return 'mp3'
const text = ascii(0, head.length)
// UTF-16 read as its characters by leaving out the zero bytes, which only it has here.
const page = /^(?:\xff\xfe|\xfe\xff)/.test(text) ? text.slice(2).replaceAll('\0', '') : text.replace(/^\xef\xbb\xbf/, '')
return /^\s*</.test(page) ? 'page' : undefined
}

/** @template F */
export class MediaStore {
/**
Expand All @@ -193,18 +232,30 @@ export class MediaStore {
this.known = new Map()
/** @type {Set<string>} */
this.names = new Set()
/** @type {Map<string, number>} a video's or a recording's length in seconds, by its path in the package, where its index gave one */
this.lengths = new Map()
}

/**
* A video's or a recording's length in seconds, by its path in the package, where it was read.
* @param {string} path
*/
lengthOf(path) {
return this.lengths.get(path)
}

/**
* A written file made what H5P stores: `{ path, mime, width?, height?, copyright }`, its bytes
* staged for the package.
* staged for the package. A picture's size is read from its bytes, and a video's or a
* recording's container; an mp4's index too, for its length and what its tracks hold.
* @param {any} field the image, video, audio or file field
* @param {{ path: string, copyright?: any }} value as written
* @param {string} at where it is written, for problems
* @param {(path: string, message: string) => void} report
* @param {(path: string, message: string) => void} [note] what is worth saying of a file and no reason not to build
* @returns {Promise<Record<string, unknown>>}
*/
async resolve(field, value, at, report) {
async resolve(field, value, at, report, note = () => {}) {
const kind = /** @type {'image' | 'video' | 'audio' | 'file'} */ (field.type)
const source = value.path.trim()
const copyright = this.copyright(value.copyright, `${at}.copyright`, report)
Expand Down Expand Up @@ -245,6 +296,14 @@ export class MediaStore {
} finally {
await file.close()
}
} else if (kind === 'video' || kind === 'audio') {
const file = await this.access.open(staged.from)
try {
const length = await this.#played(file, kind, source, web, `${at}.path`, report, note)
if (length) this.lengths.set(name, length)
} finally {
await file.close()
}
}
this.files.push({ name, from: staged.from })
this.known.set(key, described)
Expand Down Expand Up @@ -283,6 +342,46 @@ export class MediaStore {
return out
}

/**
* A video's or a recording's bytes held to what its field takes: its container, from its first
* bytes, a problem where it is none of those; for an mp4, its index, a problem where there is
* none to read, and a note for a file with no sound, no picture, or a picture not every browser
* plays. A file whose index lists no track says nothing either way.
* @param {{ size: number, read(offset: number, length: number): Promise<Uint8Array> }} file
* @param {'video' | 'audio'} kind
* @param {string} source as written
* @param {boolean} web named by a web address
* @param {string} at
* @param {(path: string, message: string) => void} report
* @param {(path: string, message: string) => void} note
* @returns {Promise<number | undefined>} its length in seconds, where its index gives one
*/
async #played(file, kind, source, web, at, report, note) {
const { takes, what } = PLAYS[kind]
// An mp4 by its index, as the normalizer reads one, after `ftyp` or an older file's `moov`,
// `mdat`, `free` or `wide`; any other by its first bytes.
const info = await mp4Info((offset, length) => file.read(offset, length), file.size)
const container = info ? 'mp4' : containerOf(await file.read(0, 16))
if (container === 'page') {
report(at, `${source} is a web page, not ${kind === 'video' ? 'a video' : 'a recording'}${web ? ': give the address of the file itself' : ''}`)
return undefined
}
if (!container || !takes.includes(container)) {
report(at, `${source} is not ${what}: its first bytes are none of theirs`)
return undefined
}
if (!info) {
if (container === 'mp4') report(at, `${source} has no index (moov) a player can read: is the file whole?`)
return undefined
}
const held = new Set(info.tracks.map((track) => track.kind))
if (held.size && !held.has('audio')) note(at, `${source} has no sound`)
if (held.size && kind === 'video' && !held.has('video')) note(at, `${source} has no picture`)
const codecs = new Set(info.tracks.filter((track) => track.kind === 'video').map((track) => NOT_EVERYWHERE[track.codec]).filter(Boolean))
for (const codec of codecs) note(at, `${source} is ${codec}, which not every browser plays; H.264 plays in all of them`)
return info.duration
}

/**
* @param {string} source
* @param {Record<string, string>} types
Expand Down
5 changes: 5 additions & 0 deletions packages/author/lib/node.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,12 @@ export function nodeMedia({ base, temporary, within }) {
},
async open(file) {
const handle = await open(file, 'r')
const { size } = await handle.stat().catch(async (error) => {
await handle.close()
throw error
})
return {
size,
async read(offset, length) {
const bytes = new Uint8Array(length)
const { bytesRead } = await handle.read(bytes, 0, length, offset)
Expand Down
29 changes: 29 additions & 0 deletions packages/author/lib/types.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,35 @@ const interactiveVideo = {
}
}

/**
* An Interactive Video's times held to its video's length, where the build read one (an mp4's
* index gives it): an interaction that starts past the end never shows, a bookmark or an end
* screen past it is never reached, a start past it plays nothing. An interaction may end past it,
* and shows to the end.
* @param {any} params an Interactive Video's, as H5P stores them
* @param {string} path where they are
* @param {(file: string) => number | undefined} lengthOf a video's length in seconds, by its path in the package
* @param {(path: string, message: string) => void} report
*/
export function videoTimes(params, path, lengthOf, report) {
const files = params?.interactiveVideo?.video?.files
const length = (Array.isArray(files) ? files : []).map((file) => (typeof file?.path === 'string' ? lengthOf(file.path) : undefined)).find((each) => each !== undefined)
if (length === undefined) return
const end = `the video ends at ${Math.round(length * 100) / 100} s`
const late = (/** @type {unknown} */ time) => typeof time === 'number' && time > length
const list = (/** @type {unknown} */ value) => (Array.isArray(value) ? value : [])
const assets = params.interactiveVideo.assets
list(assets?.interactions).forEach((interaction, i) => {
if (late(interaction?.duration?.from)) report(`${path}.interactiveVideo.assets.interactions[${i}].duration`, `starts at ${interaction.duration.from} s, after ${end}`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The interaction index in this message is the generated-list index. When an earlier interaction was refused, the list is shorter, so the path can name the wrong interaction. Skip or map the index for lists the generator shortened, as build.mjs already does for other problems.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/author/lib/types.mjs, line 469:

<comment>The interaction index in this message is the generated-list index. When an earlier interaction was refused, the list is shorter, so the path can name the wrong interaction. Skip or map the index for lists the generator shortened, as `build.mjs` already does for other problems.</comment>

<file context>
@@ -447,6 +447,35 @@ const interactiveVideo = {
+  const list = (/** @type {unknown} */ value) => (Array.isArray(value) ? value : [])
+  const assets = params.interactiveVideo.assets
+  list(assets?.interactions).forEach((interaction, i) => {
+    if (late(interaction?.duration?.from)) report(`${path}.interactiveVideo.assets.interactions[${i}].duration`, `starts at ${interaction.duration.from} s, after ${end}`)
+  })
+  for (const key of ['bookmarks', 'endscreens']) {
</file context>

})
for (const key of ['bookmarks', 'endscreens']) {
list(assets?.[key]).forEach((mark, i) => {
if (late(mark?.time)) report(`${path}.interactiveVideo.assets.${key}[${i}].time`, `is ${mark.time} s, after ${end}`)
})
}
if (late(params.override?.startVideoAt)) report(`${path}.override.startVideoAt`, `is ${params.override.startVideoAt} s, after ${end}`)
}

/* ---------------------------------------------------------------- Branching Scenario */

/**
Expand Down
20 changes: 20 additions & 0 deletions packages/author/tests/browser/page.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { pack } from '@missing-elements/h5p-libraries/install'
import { describe, expect, it } from 'vitest'
import { commands } from 'vitest/browser'
import { buildPackage, openPack } from '../../lib/browser.mjs'
import { mp4 } from '../helpers/mp4'

// The page's build in Chromium, as a page runs it: the pack a bundler emits, a picture the page
// was given as a `File`, the package back as a `Blob`. Its bytes go to Node, where the verifier
Expand Down Expand Up @@ -77,4 +78,23 @@ describe('the build in a page', () => {
await libraries.close()
}
})

it('reads a video\'s index from the File it was given, as a build in Node reads it from disk (#218)', async () => {
const libraries = await openPack(pack)
try {
const clip = new File([mp4(false, { seconds: 30, tracks: [['vide', 'avc1']] })], 'clip.mp4', { type: 'video/mp4' })
const source = (from: number) => ({
...META,
type: 'H5P.InteractiveVideo',
params: { interactiveVideo: { video: { files: [{ path: 'clip.mp4' }] }, assets: { interactions: [{ duration: { from, to: from + 2 }, label: 'Look', action: { library: 'H5P.Text', params: { text: 'Look' } } }] } } }
})
const built = await buildPackage(source(5), { libraries, files: { 'clip.mp4': clip } })
expect(built.notes.filter(({ path }) => path.endsWith('.path'))).toEqual([{ path: 'params.interactiveVideo.video.files[0].path', message: 'clip.mp4 has no sound' }])
await expect(buildPackage(source(40), { libraries, files: { 'clip.mp4': clip } })).rejects.toMatchObject({
problems: [{ path: 'params.interactiveVideo.assets.interactions[0].duration', message: 'starts at 40 s, after the video ends at 30 s' }]
})
} finally {
await libraries.close()
}
})
})
Loading
Loading