info about metadata.xml

[IoannisSyrogiannis]

I was recently practicing with encryption.
I usually decrypt and scarcely encrypt. Yet, when I made a test with a small project, I was greeted with a warning that, if I am making an encrypted DCP, it would be best to keep the metadata.xml file of the working files, to save myself troubles in the future, if issues appear with DKDMs etc.
I made the encrypted DCP (video and audio, no subtitles) and then I checked the metadata.xml file.
I found that the key was there for the video file, but not for the audio. Or, at least, I haven't found any other <key> tag in the file and that key didn't work with the audio.mxf.
Is that the case, or is there something I miss in the whole scheme?
Could someone elaborate on how the metadata.xml works (in general) and how does it help in case one looses access to the (D)KDMs?

Spoiler: I tried with the latest version of DCP-o-matic, and the <key> turned for both video and audio. I was probably missing something in the copy/paste procedure. to run `asdcp-unwrap -k <key> pcm_file.mxf`

I do understand what is the significance of "VideoEncoding", or "AudioFrameRate", or "Resolution" tags, but I would like to know what is there that I could take advantage after I create a DCP and might not be self evident. For instance, the "Digest" tag is the SHA1 digest without being base64 encoded? What else?

[Carsten]

The idea of keeping the encryption keys in metadata.xml is that you can always reopen the project and create new DKDMs for the project - even if your encryption certs have changed or expired. Of course, using the proper tools, the cleartext keys will also allow you to decrypt the assets directly if necessary. I don't know if DCP-o-matic is now creating different keys for audio and video. For a while at least, a single key was used for all assets.

[carl]

For instance, the "Digest" tag is the SHA1 digest without being base64 encoded?

The <Digest> in the metadata is a mix of the first million and last million bytes of the file, and the length. It's an attempt to make a "good enough" hash to recognise when a file has changed, without needing to read the whole file.

[IoannisSyrogiannis]

For instance, the "Digest" tag is the SHA1 digest without being base64 encoded?

The <Digest> in the metadata is a mix of the first million and last million bytes of the file, and the length. It's an attempt to make a "good enough" hash to recognise when a file has changed, without needing to read the whole file.

That is a nice idea, but how may I cross-check that digest with the file? Is there a command I can run or a DCP-o-matic function I can utilize to that end?

The idea of keeping the encryption keys in metadata.xml is that you can always reopen the project and create new DKDMs for the project - even if your encryption certs have changed or expired. Of course, using the proper tools, the cleartext keys will also allow you to decrypt the assets directly if necessary. I don't know if DCP-o-matic is now creating different keys for audio and video. For a while at least, a single key was used for all assets.

My limited experience agrees with what you describe, Carsten, in regards to the single key.
Yet, what I get from what you say is that one needs to keep the entirety (?) of the working files, in order to reopen the original project, if necessary?
That would include, I suppose the original content. Either a (number) of video, audio, subtitle file(s), or another DCP (encrypted or not).
So, would the trick with metadata.xml (or/and any other working file) be able to help re-create a project without keeping all initial ingredients?
If yes, what would be the way?
Because, otherwise, keeping the whole project (minus the created DCP) and the original essences as a back-up would be less flexible and it could as well mean -in the case where the original material is an encrypted DCP- that expiration or change of encryption certificates would get one in front of the same problems.

It goes without saying, and we are on the same page on that matter, that the whole idea of re-opening the same project, or managing to re-create it, would be the alternative to decrypt the files one by one with the key and make a new project from the decrypted images, audio, subtitles and so on. An alternative that is always there.

[Carsten]

The digest is used to check if a file that has been added previously to a project has changed. As they are external files, it is possible for them to change before the DCP is created. E.g. if you added a graphics file, and later on you notice you need to recreate/edit it, DCP-o-matic will notice the change and a pop-up will come up with a notification.

[carl]

If I remember right my only intention with the message about keeping metadata.xml was to provide a simple way for a user to backup the decryption key as a "last resort" to decrypt a DCP they made with DCP-o-matic. It's simpler/more reliable than saying "keep a DKDM, which let's hope you made correctly, and the private key to decrypt the KDM, and let's hope you keep the right one". Especially in the early days of the encryption support this felt like a good idea.

As you say, keeping a project intact for a long time is more complicated, and though I do try to make it possible to load old projects into newer DCP-o-matics I don't have much automated testing to "prove" that it works. My assumption is that the amount of effort that goes into making a DCP-o-matic project is relatively low, and the serious complexity of making DCP-o-matic projects from 10 years ago load and produce a bit-for-bit identical output is IMHO not worth it.

This is in contrast to something like a video editor, or DAW, where I think it would make a lot more sense.

[IoannisSyrogiannis]

I understand, Carl, and it makes perfect sense.
It's a happy coincidence, what happened to me today, because it gave me a story to tell about the working files of DCP-o-matic and encryption.

These days, I am checking a big number of DCPs, made in the last 12 years, give or take. Some made by obsolete DCP authoring systems, others with OpenDCP, DaVinci Resolve, Premiere Pro and a good number by DCP-o-matic. Some of them are "flawless", some were "flawless" when made, but miss some info according to the current standards, their CPLs may or may not be signed, may or may not have hash info for assets, etc., etc.

As it happens to most of us, once in a while, I find a DCP in its DCP-o-matic project folder. In most cases, that means that I remove from that folder the DCP itself and discard the rest. Being the incurable "yes-but-what-could-go-wrong" guy, I don't throw away anything before I make sure it won't turn to be necessary. So, this time, today, paid for what otherwise seems a waste of time. The DCP created was encrypted. And while it wasn't meant to be delivered encrypted without a (D)KDM, "guess what"?

That, both gave me the chance to discuss with my colleagues the rare examples where that "garbage" may turn to be useful and to reflect on the matter of the encryption certificates of the DCP-o-matic installation of my station, that are to become obsolete, come 2029. (If I didn't know me, I would thought that I am getting older.)

So, after making a DKDM for that stray DCP, and before I decrypt that DCP, in order for it to find its place in history, I wonder:
What is the absolutely necessary to keep, in order to be able to re-make the DKDM?
(Admittedly, one should consider backwards compatibility on DCP-o-matic, but let's say that we get lucky.)
Having the DCP that was created somewhere, not having the original video files (or DCP) what files and folders are necessary for DCP-o-matic to recognize the project and allow DKDM creation?
The DCP?
metadata.xml?
assets.xml?
log?

As things stand, what would I need to back-up and what wouldn't I?
I can always experiment, but that doesn't always lead to the correct conclusions.