DCP-o-matic 2.18.0 beta 1 released

[carl]

Download it now from here.

This is the first beta release of what will become the new stable version 2.18.0 of DCP-o-matic, which includes everything in 2.16.91 and adds some new features and fixes.

You are advised to use this version only for testing, not in production systems.


This version will convert the XML files containing your cinema, screen and DKDM recipients to SQLite3 format (leaving the old XML files behind). If you change any of these details the changes will not be visible to older DCP-o-matic versions. It is a good idea to back up your cinema and DKDM recipients files (if you use those features) before installing this version.


Full list of changes

• DCP-o-matic now takes video frame timing information from sources in a (hopefully) more reliable way, which should fix video/audio sync errors in a number of cases.
• Support has been added for a commercial (paid-for) GPU-based JPEG2000 encoder (‘Grok’) that provides dramatic encoding speed-ups with Nvidia graphics cards on Ubuntu 22.04.
• Cinema and DKDM recipient information is now stored using SQLite (rather than XML), meaning that use of shared cinema database files should be more reliable.
• There is now experimental support for playback and creation of MPEG2 DCPs.
• The user interface for making VFs has changed. You can now set them up using a separate dialogue box, which can be opened with the ‘Version File (VF)...’ option in the ‘Tools’ menu.
• A new user interface for setting up reels has been added, which allows setup of custom reel breaks.
• UTC offset for KDMs is specified when KDMs are created, not when setting up the ‘Cinema’. This is to avoid difficulties when cinemas are in places where timezones change through the year (e.g. where there is daylight saving time).
• There is now support for open captions and closed subtitles.
• A separate DCP verification tool is included.
• Add option to copy content settings from another project.
• Template features have been improved, and some default settings removed in favour of using a template.
• A menu option was added to un-map all audio channels for a piece of content.
• It is now possible to save projects with relative (rather than absolute) content paths.
• DCP-o-matic can make text or HTML reports during verification runs.
• DCP-o-matic is now linked against openjpeg 2.5.2. This provides some worthwhile speed-ups compared to the previous version.
• DCP-o-matic is now linked against FFmpeg 7.0.1.
• The audio language settings has been moved to the DCP audio tab.
• On loading a project DCP-o-matic will show the DCP panel if that was what was open when the project was last edited.
• Some XML attribute names were reformatted to ‘camel-case’.
• Fix some errors related to font IDs.
• Fix rendering of subtitles where there are colour changes within a line.
• Add ProRes LT export option.
• Add basic HTTP interface to control the player.

[Carsten]

>A separate DCP verification tool is included.

Is this different from the verification option in DCP-o-matic player?

[carl]

Not really, it uses the same code to check the DCP - it's just a new "front-end" I suppose.

[Carsten]

Wouldn't it have to be a separate download then (for Mac)? Because I don't find it in the Mac OS download options.


I'm currently verifying a few DCPs I backed up - I notice the player 2.17.20 verification now seems to check hashes first - is that konsistent/intentional? I always wanted to have an option to hash-check-only DCPs I created myself (as in DCP-o-matic player prior to 2.16), or commercial DCPs that I assume to have proper code-stream properties and be formally correct (the code stream checks take very long).

If it hash-checks first now, I may just cancel verification after the hash checks?

edit: No, seems that DCP-o-matic player still mixes hash checking and picture frame size checking.

Maybe we could have two options for verification - 'Integrity' (hash checks and completeness), and 'Verification' or ''Validation'?


Or, since the verification code has to read through the files for both hash-checks and codestream compliance - couldn't it perform both checks during one read?

- Carsten

[carl]

I missed it off the website for mac/Appimage - I just added it.

Did we already file the verification vs. validation somewhere in the bug tracker? It makes sense to me.

Perhaps the hash checks and code-stream stuff could be merged, indeed.

[Carsten]

Did we already file the verification vs. validation somewhere in the bug tracker? It makes sense to me.

Perhaps the hash checks and code-stream stuff could be merged, indeed.

Hmm, yes, I filed it a while back...


There it is: https://dcpomatic.com/mantis/view.php?id=2656

I think hashing in DCP-o-matic main after conversion is multithreaded/per reel, but in player, it seems to be single-thread? On machines with internal SSDs like the latest Macs with internal flash, performing the checks multithreaded should be very very fast?

[carl]

You're right, yes, the verifier hashing is single-threaded but the DCP creation hashing is multi-threaded.

[IoannisSyrogiannis]

I was balancing between a comment here and a new thread, with a link here.
I am in the third hour (2H40m) of verifying an 165 gigabytes long DCP.
Most times that I check the status. I see "Checking picture frame sizes" (unfortunately, I can't tell which j2c.mxf file it is, in the row, does it go from reel 1 to 2, etc. up to... reel n?).
I tried, yesterday, to use:

Code: Select all

dcpomatic2_verify_cli.exe C:\ROOT\TO\DCP1\FOLDER -o C:\ROOT\TO\LOG1\FOLDER\DCP1VerificationInfo.html && dcpomatic2_verify_cli.exe C:\ROOT\TO\DCP2\FOLDER -o C:\ROOT\TO\LOG2\FOLDER\DCP2VerificationInfo.html

for a ~200 gigabytes DCP, that took a lot of hours as well.
My goal there (with the CLI commands and log export) was to be able to verify more than one DCPs in the row, without needing to be there to choose and press 'Verify'.
The reason that I chose log export, was in order to have the results, in case of an accidental reboot or power outage. It happens every a few weeks.

I wonder if the log export is the reason that the verification is dragging for so(ooo) long. I don't remember it like that. (?)

Besides that, while there is an option to skip hash check (--no-asset-hash-check), there is no similar option to skip picture frame size check, that is useful often enough. Especially if you want to run a fast check on the integrity of a hard drive full of material you stored some time ago, or you received in a questionable fashion (like in a disc using an... exotic file system), or a file-sharing service that is not cut out for such packages.

I don't know if that is relevant, but the DCP-o-matic version I am using is 2.18.11 and the DCPs I verified were Interop.

[Carsten]

A while ago I filed a Mantis feature suggestion towards the verification offering different levels - e.g. only hash check vs. full check. Very often I need to check my own DCPs created with DCP-o-matic, and all I need is a hash check. Also, the hash check in DCP-o-matic after creating the DCP is multithreaded (when having multiple reels in a DCP), while the hash check in player or verifier is single threaded. On a fast internal (Mac) or M.2 SSD, the verification would become a lot faster when performed multithreaded. The trouble is, it will take a lot longer multithreaded when performed on external/USB or spinning disks. So one would probably also want the multithreading being optional. We are using inexpensive machines now (Ryzen 7) where single threaded hashing/checking takes longer than the encoding.

[IoannisSyrogiannis]

All that makes sense, Carsten, and I knew about the Mantis suggestion when writing here, where you have given the link to it.
On top of that, though, I wonder if there are different bottlenecks as well, behind the disk reading speed and CPU(s) used.
On my Resource Monitor (Windows 10), I see the CPU not going higher than 50% for the verifier, the disk in question having an active time close to 0%, the memory being committed being 331.272 KBs, (working set almost 300.000 KBs, total RAM in use, almost 20.430MB) and that blessed j2c.mxf file that I was hoping to be the last giving its place to another, a few gigabytes bigger.
At this point, I will find my way to bed and leave the work to automation, but four and a half hours (01:00 UTC) of verification seems quite excessive.

That's why I am now wondering if there are other "players" that make a difference. (Including the log creation parameter.)

After writing all the above and even more, I started a small experiment.
I opened in parallel the player and started verifying another movie, from an external drive. It seems that the Player is doing much better in terms of speed. That is, at least at 2.18.11. It also uses about the same CPU, but more memory:

DCPoMPlayerVsVerifier.PNG

Again, I would like to have the choice to not check picture frame sizes and do it from the CLI as well. Yet, the more-than-five-hours full verification is suspicious.

Edit, please read: When chosen the same DCP that is taking long to check with DCP-o-matic Player, I see that it's taking longer as well (even though the memory used is around twice as big and the CPU used slightly . So, I suspect that it has something to do with my choices when making the .j2c files. (The DCP was made out of kakadu-manipulated images.) I might know more when the verification finishes.