How does DOM operate when modifying an existing Project?

[Guy Burns]

First question: is there a proper name for what I'm calling a Project (I borrowed the name from Premiere)?

Now for some questions about how DCP-o-matic operates when modifying an existing Project.

Test 1
I have a test Project that contains six files:

• five ProRes 422 HQ files
• and a previously encoded DCP file from DoM, containing 90% of the total video material.

When I made a new DCP, the encode ran through very quickly because most of the encoding had already been done on the 90% material.

Test 2
Then I made another change: I added a 5-second video clip in the middle. I assumed that the encode would really fly this time because 99.5% of the encode had already been done. I also assumed that, during encoding, each individual clip would be stored as a separate encoded file, and then assembled together at the end. But no, the encode was slow.

So I went looking for all these video files in my Project folder, and found three large ones. Two were identical in size, but different in name. One of those files is in the "video" folder, the other in the folder destined for ingestion by the cinema. And there is another large file in the "video" folder, roughly the same size. I assume this one was the previously encoded DCP.

Ques 1
When making the DCP, I asked for the previous file to be overwritten. Yet I still have 5.6GB of the previous DCP in the "video" folder. Does it serve some purpose to keep the old version?

Ques 2
Why are there two copies of the new video file? What purpose is served by keeping the extra copy in the "video" folder? Any problems in deleting it?

Ques 3
If I make a minor change to a Project, such as a 5-second addition in the middle, does DoM make use of any previously-encoded video files in the "video" folder? Meaning: does DoM say to itself: "Hey, I encoded most of those clips last week, and they ended up inside a video file I made called 'XXX'. This bloke hasn't made any changes to the clips, so clever me -- I'll just extract the relevant frames and reuse them."

Ques 4
If I make a minor change to an already-encoded Project, is it possible to reduce encode time, given that most of the encoding has already been done?

[Carsten]

For Test2 - I don't know what happened, but maybe the 'content tracking' of DCP-o-matic can't deal with that. Have to replicate that myself.

In general, DCP-o-matic keeps the 'working video' in the video folder. Once the conversion is done, it copies/links the necessary files into the final DCP folder. This may not work for all combinations of new content added somewhere in the middle. Carl may comment.

You can make minor changes, but wether it triggers a full re-encode or not depends on the kind of changes you do. Everything that needs to be encoded in the image signal will trigger a new encoding. e.g. if you change the color conversion, cropping, scaling, etc., the whole piece of contente needs to be re-encoded. If you trim off something at the end or beginning, or add audio, this will not trigger a new encoding.

- Carsten

[Guy Burns]

After playing around a bit more, I've now got several large video files in the "Video" folder of the same Project. I'm going to delete them. That shouldn't cause any problems, should it?

[Carsten]

It won't do any real harm to your project, the worst that can happen is that you need to convert a project again from scratch if the last conversion hasn't finished yet.

Now, with many files in that folder, usually each relates to a separate encoding run/different DCP version, as far as I can say.

There is something special about the file(s) in the video folder, at least if you have your project folder located on common modern filesystems:

Assume that you create a single DCP for the first time. During conversion, DCP-o-matic writes the actual video MXF file to the video-folder. Once the conversion has completed, the complete video-mxf is not copied, but LINKED into the final DCP directory. That means, the file remains in the video folder, the DCP folder only contains a link to the original file. Modern operating systems hide this mechanism from the user. From looking at the directories, you would assume that the two video files consume twice the amount of space than what's needed for the DCP, but as a matter of fact, only the file in the video folder actually exists at this size.
No, if you delete the file in the video folder, it won't be gone in the DCP folder as well - your OS recognizes that you are about to delete a file that is referenced from another location. When you delete it in the original location (=video), it is actually not deleted, but the link from the DCP folder is actually converted into a directory entry for the original file, and the original reference from video folder deleted. Now the mxf exists as an original file in the DCP directory, and no longer in the video folder.

This only works if your project resides on modern file systems that support file linking. The native filesystems of Windows, OS X and Linux do support this. It won't work e.g. on FAT32 media. If you create a DCP on file systems that do not support file linking, a real copy is made after the video conversion has finished. This actually consumes twice as much space.

Ah - I know Carl had written this down in fewer words before:

https://dcpomatic.com/faq?id=11

- Carsten