When generating a DCP, DOM creates a directory/folder above the DCP folder itself in the file system hierarchy where it creates a number of files and sub folders (on the same folder level as the DCP itself). These include:
analysis [folder]
info [folder]
video [folder]
COVER SHEET.txt
ffprobe.log
log
metadata.xml
I think that 'metadata.xml' is important, and 'COVER SHEET.txt' has useful information.
The 'video' folder seems to have an mxf version of the content. On my Windows machine, this occupies actual significant hard drive space - I say this as I seem to recall being told that this is just a symlink or similar in the past. When I duplicate the directory structure, this large file is physically copied as is the same-sized mxf video file forming part of the actual DCP folder. This almost doubles the size the data uses on the hard drive.
Can I delete the 'video' sub-folder? And of the other files and folders in the top level directory above the actual DCP folder itself, which are necessary to keep?
Many thanks.
'video' folder - can it be deleted?
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: 'video' folder - can it be deleted?
https://dcpomatic.com/manual/html/ch14.html
Actually, usually the video folder does NOT hold an actual physical copy of the video content. After a straight conversion, it is just linked to the same file in the DCP folder. So, while it looks as if you consume twice the space, there is only ONE file occupying space. The folder size is misleading.
NOW - you can delete the file in video safely, as, the moment when you delete it, the previously linked file in the DCP folder becomes the actual file.
Now - if you created multiple versions within one project, played with video settings, etc. - then yes, there may be some garbage from previous conversions in the video folder, that you might want to delete. In general, it is safe to delete everything from the video folder. The worst thing that might happen is that you may need to run through another compression run when changing something in a project you are still working on.
E.g. if you created a DCP, then went back to change audio levels. If you deleted files in the video folder already, then the video part has to be transcoded once more. If you had left the video file there, it could have been reused and only the audio changes would have been applied.
Nothing bad with cleaning up all these files once you are sure you really have completed the DCP.
Actually, usually the video folder does NOT hold an actual physical copy of the video content. After a straight conversion, it is just linked to the same file in the DCP folder. So, while it looks as if you consume twice the space, there is only ONE file occupying space. The folder size is misleading.
NOW - you can delete the file in video safely, as, the moment when you delete it, the previously linked file in the DCP folder becomes the actual file.
Now - if you created multiple versions within one project, played with video settings, etc. - then yes, there may be some garbage from previous conversions in the video folder, that you might want to delete. In general, it is safe to delete everything from the video folder. The worst thing that might happen is that you may need to run through another compression run when changing something in a project you are still working on.
E.g. if you created a DCP, then went back to change audio levels. If you deleted files in the video folder already, then the video part has to be transcoded once more. If you had left the video file there, it could have been reused and only the audio changes would have been applied.
Nothing bad with cleaning up all these files once you are sure you really have completed the DCP.
-
rtX
- Posts: 83
- Joined: Tue Apr 22, 2014 6:08 pm
Re: 'video' folder - can it be deleted?
@Carsten
Many thanks for the response. In Windows, the video seems to be stored twice - once in the 'video' folder and once in the DCP itself. When I delete the 'video' folder, after emptying the recycle bin, I have whatever the size of the video folder plus contents returned to me as available disk space. If I copy the whole file hierarchy with the 'video' folder in it to another drive, it physically copies the data, and that amount of disk space on the destination drive is used.
That's why I am keen to lose the 'video' folder as soon as I can.
Many thanks for the response. In Windows, the video seems to be stored twice - once in the 'video' folder and once in the DCP itself. When I delete the 'video' folder, after emptying the recycle bin, I have whatever the size of the video folder plus contents returned to me as available disk space. If I copy the whole file hierarchy with the 'video' folder in it to another drive, it physically copies the data, and that amount of disk space on the destination drive is used.
That's why I am keen to lose the 'video' folder as soon as I can.
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: 'video' folder - can it be deleted?
That is misleading. Windows can do file linking as well. The system folder size calculations do not take this into account. Now, some file systems do not support file linking. But NTFS does support it. The only reason to delete the video folder (yes, you may safely delete the whole folder as well) is if you created multiple versions within the same project and then have multiple versions of video files, of which you don't reuse previous ones (e.g. with bad scaling, color correction, filters, etc.). These previous video files are typically not used in the final DCP and do indeed consume unnecessary space.
When you start a DCP conversion, first the transcoded video is created in the video folder. After the transcoding is finished, DCP-o-matic creates the necessary final DCP files in the DCP folder. Some of these files there are actual self contained files (small). The video MXF, though, is linked from the video folder. The operating system makes sure, that, if you delete either of these two files, that a single autonomous version will remain.
Again, that is for file systems/operating systems that do support file linking.
On other systems/file systems, DCP-o-matic does indeed create two self contained files. That means, after transcoding, DCP-o-matic copies the full video MXF from the video folder to the DCP folder instead of linking it from the video folder. Linking goes within a snap of a second, the full copy can take very long.
When you start a DCP conversion, first the transcoded video is created in the video folder. After the transcoding is finished, DCP-o-matic creates the necessary final DCP files in the DCP folder. Some of these files there are actual self contained files (small). The video MXF, though, is linked from the video folder. The operating system makes sure, that, if you delete either of these two files, that a single autonomous version will remain.
Again, that is for file systems/operating systems that do support file linking.
On other systems/file systems, DCP-o-matic does indeed create two self contained files. That means, after transcoding, DCP-o-matic copies the full video MXF from the video folder to the DCP folder instead of linking it from the video folder. Linking goes within a snap of a second, the full copy can take very long.
-
rtX
- Posts: 83
- Joined: Tue Apr 22, 2014 6:08 pm
Re: 'video' folder - can it be deleted?
@Carsten
Many thanks for taking the time to respond. My experience of what actually happens is somewhat at odds with the way you describe that it should work.
Windows may support file linking under NTFS, but it is making two copies of the data in my case (for whatever reason). On a recent copy from one drive to another I watched it physically transfer 173GB - it didn't suddenly jump from 83GB to nothing else to copy (all done) - it physically copied the files across. And after the copy the hard drive had 173GB less space than it had before I did the copy.
You will have this problem, anyway, if you aren't using NTFS. Some Windows file systems don't support hard links, and other files system drivers (for example, HFS+ drivers under Windows) may not, either - as it happens, though, I am using NTFS.
The reason I am looking to save this hard drive space is that I have all my delivery materials on a single large capacity disk (properly backed up elsewhere). When I copy the DCP across I am right on the edge of the drive being full, and I'm unable to copy certain other files on to the drive. If I delete the 'video' folder, I can copy those files on to the drive. This shows that the drive space has been allocated - and can be retrieved. It also suggest that from within DoM the hard link has been lost (or was never activated) and two physical copies of the data exist.
Windows is physically copying the data across and preventing access on the drive to the space occupied. I don't understand why, but maybe it happens if a copy of the data is made away from the original material? Please don't take my word for it, but I'm saying that this is what happens and has always happened, in my experience.
Many thanks for taking the time to respond. My experience of what actually happens is somewhat at odds with the way you describe that it should work.
Windows may support file linking under NTFS, but it is making two copies of the data in my case (for whatever reason). On a recent copy from one drive to another I watched it physically transfer 173GB - it didn't suddenly jump from 83GB to nothing else to copy (all done) - it physically copied the files across. And after the copy the hard drive had 173GB less space than it had before I did the copy.
You will have this problem, anyway, if you aren't using NTFS. Some Windows file systems don't support hard links, and other files system drivers (for example, HFS+ drivers under Windows) may not, either - as it happens, though, I am using NTFS.
The reason I am looking to save this hard drive space is that I have all my delivery materials on a single large capacity disk (properly backed up elsewhere). When I copy the DCP across I am right on the edge of the drive being full, and I'm unable to copy certain other files on to the drive. If I delete the 'video' folder, I can copy those files on to the drive. This shows that the drive space has been allocated - and can be retrieved. It also suggest that from within DoM the hard link has been lost (or was never activated) and two physical copies of the data exist.
Windows is physically copying the data across and preventing access on the drive to the space occupied. I don't understand why, but maybe it happens if a copy of the data is made away from the original material? Please don't take my word for it, but I'm saying that this is what happens and has always happened, in my experience.
-
funkytwig
- Posts: 72
- Joined: Mon Jul 28, 2014 9:37 am
Re: 'video' folder - can it be deleted?
Bwwn reading through this and thought it is worth adding that it is 'safe' to delete everything apart from the main project DCP (as long as you keep the source file) as you can just rerun the job. Yes, it will take a long time but it is safe in that you can recreate the DCP easily. The other approach in to delete everything apart from the DCP Package Folder and main project XML as this means if you need the DCP again you have it and if you need to do another version you can also do this (again will have to redo the job from scratch.