Just delivered a DCP to a festival and I got back an error log from them. I have no idea how this would have failed or what I did wrong.
The dcp was created in dcp o matic from a 4k scope 5.1 file. After creation I used DCP transfer to validate the dcp and move it to a linux drive created by the software. Everything checked out.
I have attached the error log they sent, can someone please tell me what went wrong? Need to re deliver ASAP. Thanks!
URGENT: Hash mismatch
-
Bonusfeatures75
- Posts: 5
- Joined: Thu Oct 31, 2019 9:31 pm
URGENT: Hash mismatch
You do not have the required permissions to view the files attached to this post.
-
carl
- Site Admin
- Posts: 2548
- Joined: Thu Nov 14, 2013 2:53 pm
Re: URGENT: Hash mismatch
Hi there,
The video file in your DCP has been corrupted at some point. The two most likely reasons that come to mind are:
The video file in your DCP has been corrupted at some point. The two most likely reasons that come to mind are:
- Some problem with the drive you used. Does DCP Transfer validate the DCP before or after the copy, or both?
- It looks like maybe you copied the whole DCP-o-matic project onto the drive, rather than just the DCP. Is that right? If so, that may have caused problems (I'm guessing here...). Perhaps you could try again with just the DCP itself (see here.
-
Bonusfeatures75
- Posts: 5
- Joined: Thu Oct 31, 2019 9:31 pm
Re: URGENT: Hash mismatch
I created the DCP via dcp o matic right onto an external drive. Then via that external drive I validated and copied the dcp folder with dcp transfer (not the dcp o matic folders) onto the drive.
Is there a way to make dcp o matic validate the dcp after its been created before I do any transfers? I'm at a loss here.
Is there a way to make dcp o matic validate the dcp after its been created before I do any transfers? I'm at a loss here.
-
carl
- Site Admin
- Posts: 2548
- Joined: Thu Nov 14, 2013 2:53 pm
Re: URGENT: Hash mismatch
If you validated the DCP with DCP Transfer I would hope it would spot any problems at that stage.
You can load the DCP you made into the DCP-o-matic player, then select "Verify", but I would expect DCP Transfer to have done the same checks. If they passed it would suggest that the DCP was OK when it left you. It's quite hard to be sure, though.
Where are you located, roughly?
You can load the DCP you made into the DCP-o-matic player, then select "Verify", but I would expect DCP Transfer to have done the same checks. If they passed it would suggest that the DCP was OK when it left you. It's quite hard to be sure, though.
Where are you located, roughly?
-
Bonusfeatures75
- Posts: 5
- Joined: Thu Oct 31, 2019 9:31 pm
Re: URGENT: Hash mismatch
I'm located in NY.
I ran another validation on the same file via DCP player and it validated properly.
I just got off the phone with DCP transfer and they told me their latest version has a few issues with the version of paragon included. They advised me to roll back to a previous version, which I did.
I am now going to go through this process again... we will see.
I ran another validation on the same file via DCP player and it validated properly.
I just got off the phone with DCP transfer and they told me their latest version has a few issues with the version of paragon included. They advised me to roll back to a previous version, which I did.
I am now going to go through this process again... we will see.
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: URGENT: Hash mismatch
As Carl pointed out - it is usually best to verify the DCP during/after it has been copied to an external/transport drive. Typical errors that a verification looks for are data corruption/copy errors (which can have various technical reasons), and that is what the hash check is able to detect.
Unfortunately, there is a systematic flaw in the way DCP-o-matic creates DCP files and hash values - DCP-o-matic writes the files first, then, finally after creation performs the hash computation. If for some reason, the writing of the files creates corrupt files, the hash computation afterwards will still create a valid hash for an invalid file. As a hash is targeted at file integrity, but not structural integrity. An invalid/broken MXF file can still have a valid hash, if the hash has been computed on the broken file.
That said - in the log you supplied, it is evident that the hash doesn't match the file - so, as Carl wrote, something must have happened to the video file/MXF between YOUR and THEIR verification.
- Carsten
Unfortunately, there is a systematic flaw in the way DCP-o-matic creates DCP files and hash values - DCP-o-matic writes the files first, then, finally after creation performs the hash computation. If for some reason, the writing of the files creates corrupt files, the hash computation afterwards will still create a valid hash for an invalid file. As a hash is targeted at file integrity, but not structural integrity. An invalid/broken MXF file can still have a valid hash, if the hash has been computed on the broken file.
That said - in the log you supplied, it is evident that the hash doesn't match the file - so, as Carl wrote, something must have happened to the video file/MXF between YOUR and THEIR verification.
- Carsten
-
Bonusfeatures75
- Posts: 5
- Joined: Thu Oct 31, 2019 9:31 pm
Re: URGENT: Hash mismatch
I have done this also. DCP transfer validates both before and after the transfer and it checked out.
The people at the festival are saying something about how according to the validation report, at least the audio file doesnt match, and they are saying it could be something with dcp o matic?
Is there anything in that validation report pointing to an audio file issue?
The people at the festival are saying something about how according to the validation report, at least the audio file doesnt match, and they are saying it could be something with dcp o matic?
Is there anything in that validation report pointing to an audio file issue?
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: URGENT: Hash mismatch
No, the log is pointing at a hash mismatch with the video file.
CPL:
960e2119-ae73-4f63-beff-0704be8c42de: text/xml;asdcpKind=CPL: cpl_960e2119-ae73-4f63-beff-0704be8c42de.xml
960e2119-ae73-4f63-beff-0704be8c42de: Checking hash value: OK (00:00:00)
Video:
7c5292dc-0ca6-4271-b991-9c4f2218760b: application/x-smpte-mxf;asdcpKind=Picture: j2c_7c5292dc-0ca6-4271-b991-9c4f2218760b.mxf
7c5292dc-0ca6-4271-b991-9c4f2218760b: Checking hash value: Mismatch (00:10:40)
Audio:
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: application/x-smpte-mxf;asdcpKind=Sound: pcm_0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82.mxf
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: Checking hash value: OK (00:00:22)
So, since they are pointing at an audio file, it seems they are using a verification software, but can't judge the results.
What if you make them try to ingest that DCP onto one of their servers? Servers perform checks themselves.
Maybe their verification process has issues. Maybe their ext2/3 reader software has issues (if the verification software runs in windows or OS X). Can you ask them for the name of their verification software? The log file doesn't give me a clue.
Hash computing and checking is not specifically difficult or variable in the DCI world - if you verified successfully with DCP Transfer, there are only some very few issues that could cause this. One possible reason could be an issue with the storage device/disc, or the interface/reading of the files during checking.
Where are you located, and where is that festival? As you mention DCP Transfer, you are obviously on a Mac. Which OS X version do you use? Did you contact the DCP Transfer people already?
Don't know what these are:
---
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition type: Undetermined
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 aspect ratio (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 picture resolution (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition incomplete
---
But as they are picture related, it may be that they follow the hash failure.
- Carsten
CPL:
960e2119-ae73-4f63-beff-0704be8c42de: text/xml;asdcpKind=CPL: cpl_960e2119-ae73-4f63-beff-0704be8c42de.xml
960e2119-ae73-4f63-beff-0704be8c42de: Checking hash value: OK (00:00:00)
Video:
7c5292dc-0ca6-4271-b991-9c4f2218760b: application/x-smpte-mxf;asdcpKind=Picture: j2c_7c5292dc-0ca6-4271-b991-9c4f2218760b.mxf
7c5292dc-0ca6-4271-b991-9c4f2218760b: Checking hash value: Mismatch (00:10:40)
Audio:
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: application/x-smpte-mxf;asdcpKind=Sound: pcm_0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82.mxf
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: Checking hash value: OK (00:00:22)
So, since they are pointing at an audio file, it seems they are using a verification software, but can't judge the results.
What if you make them try to ingest that DCP onto one of their servers? Servers perform checks themselves.
Maybe their verification process has issues. Maybe their ext2/3 reader software has issues (if the verification software runs in windows or OS X). Can you ask them for the name of their verification software? The log file doesn't give me a clue.
Hash computing and checking is not specifically difficult or variable in the DCI world - if you verified successfully with DCP Transfer, there are only some very few issues that could cause this. One possible reason could be an issue with the storage device/disc, or the interface/reading of the files during checking.
Where are you located, and where is that festival? As you mention DCP Transfer, you are obviously on a Mac. Which OS X version do you use? Did you contact the DCP Transfer people already?
Don't know what these are:
---
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition type: Undetermined
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 aspect ratio (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 picture resolution (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition incomplete
---
But as they are picture related, it may be that they follow the hash failure.
- Carsten
-
carl
- Site Admin
- Posts: 2548
- Joined: Thu Nov 14, 2013 2:53 pm
Re: URGENT: Hash mismatch
It's from dcp_inspect, so I'd be confident that it's right.
I think this is just some stuff it wanted to get from the picture MXF but it failed. It sounds like the corruption was fairly serious.Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition type: Undetermined
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 aspect ratio (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 picture resolution (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition incomplete
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: URGENT: Hash mismatch
That probably means the verification and ext2/3 reading is native Linux.
Hmm, weird. I could only think of a disc related issue, maybe some issue with a USB/SATA bridge as well. We can not even be sure wether the DCP there is in fact corrupt at all - maybe it is 'just' a read error during the verification. That's why I suggest trying an ingest on other hardware (DCI server).
Do we know wether they verified the transport disc, or did they copy first to their storage and then verified their copy? The path given in the log doesn't give me a clue.
- Carsten
Hmm, weird. I could only think of a disc related issue, maybe some issue with a USB/SATA bridge as well. We can not even be sure wether the DCP there is in fact corrupt at all - maybe it is 'just' a read error during the verification. That's why I suggest trying an ingest on other hardware (DCI server).
Do we know wether they verified the transport disc, or did they copy first to their storage and then verified their copy? The path given in the log doesn't give me a clue.
- Carsten