Special rewrapping of OV/VF assets

[Carsten]

Occasionally, I have the issue that a folder on the DCI server contains all the assets for a VF to be played, but not the original OV package anymore.
That means, the server had ingested VF and OV before, then deleted the OV package and left just the assets needed to play the VF.

Now, wen I export this package from the server, I have a fully functional VF package, plus the OV assets referenced by it - but no metadata to properly ingest these assets into a server again. So, the exported VF is useless, as I have no OV to ingest. These are typically encrypted and signed packages.

So, what I would need is a way to create the necessary metadata, assetmap, PKL, to ingest either the full package (VF AND OV assets), or, just the necessary assets from the OV.

Are there ways to do this with DOM? As far as I can see, there are no ways to create just transport/ingest packages without a CPL? Also, the 'orphan' assets would need to stay as they are, as otherwise, the VF will not work. So, the rewrapping would simply need to passthrough the asset files, and create a new assetmap and PKL.

[StephW999]

Hello Carsten !
From which server are you exporting your DCP?
It's important to remember that servers don't read DCPs, but CPLs.
A DCP is only made for ingest processing.
If you are using a Doremi or Dolby ims server, the ingest procedure will recreate another DCP in its assets folder. This is called: CPL to PKL
Therefore, exporting CPL to a hard drive will always be a complete package.
If resources are identical in several CPLs, then there are hard links established (on the server) for those files.

Stéphane.

[Carsten]

This is on a Sony. After a VF is ingested, the necessary assets from the OV are hardlinked into the VF folder. I can delete the OV, and the VF stays playable, because all referenced assets are hardlinked into the VF folder. However, when I export this folder through FTP, I only get the VF metadata, not the OV metadata, so, the OV assets referenced in that VF are orphan. There should be a way to create a package that only ingests these orphan OV assets so that the Sony can relate them to the VF. I have seen packages before that do not contain a CPL - partial packages only used to ingest additional assets.

[StephW999]

I see, indeed this requires creating software that will check what assets the CPL needs.
This software needs to associate the resources IDs with the paths and filenames, to create the PKL and ASSETMAP xml files (and replaces the others)
We can copy the hash values from the CPL to avoid recalculating the digest
Not difficult, but it takes time.

[IoannisSyrogiannis]

We should mention here, for future reference, that exporting to a CRU doesn't have that effect.
I haven't exported via FTP (sms on the data port, right?) from a SRX-R51X since 7 years, completing these exact days. The UI of the server is monitoring/indicating the procedure.
Yet, I remember that on the folder of a DCP/CPL, no mater if OV or VF (same as on every server I came upon, except the Sony LMT-300) there are all necessary files. Meaning, the package list and the asset map files are including all parts listed on the CPL, instead of the parts only on the original VF.

Carsten, what you write, that the OV assets referenced in that VF are orphans seems like you consider the package as a supplementary one, while it isn't. The OV metadata (and all assets that are not there anymore, if any that were not common between OV and VF) are deleted, and so is the CPL of that OV. Isn't that also true in your case?
Without a CPL, I can't see how there could be a way to create an ingest-able package. Wouldn't it make sense to export and then ingest the VF, in that case, if we are talking about VF-1 and VF-2 and both have the same references to the OV?
Otherwise, if we are talking about exporting and ingesting the same VF, what would be the use of that package, if the exporting package is not supplemental, but a complete DCP?

So, I don't get what is the goal here. Is it something I miss, forget, or something you consider a given and don't mention?

[Carsten]

This a very special aspect of the Sony, some would consider it a bug. It doesn't matter wether you export to CRU or via FTP. The Sony hardlinks all necessary OV assets into the VF folder on it's DCP RAID storage. When you export the package, the Sony will copy the full original VF package to the destination folder, plus all OV assets that the VF uses. If you export only the VF-CPL, then only the original VF is copied to the destination. In both cases, a valid VF is created, but in the first case, the OV assets are included as files, but not in the assetmap or PKL, so, you can not reingest these assets.

I think what could work is that if you create a dummy package/CPL, that transports the original MXF file(s), untouched, to the Sony as a regular ingest. The Sony should then associate this with the VF. It wouldn't matter that the Sony is not able to play this dummy CPL.

I'm wondering wether I could create such a package by manually adding the original asset to a package, so it ingests with other assets. Currently, it wouldn't work with DCP-o-matic, because an original MXF file will never be just passed through the conversion, but will always be reformatted into a new MXF, thus the the relation to the VF is lost.

[StephW999]

Hello Carsten, Did you see my PM?

[carl]

I've worked a bit on a very long-standing request to be able to re-use existing MXFs without re-writing them. It sounds like this might make what you want possible.

[IoannisSyrogiannis]

O.K.
So, if I get it right, no matter if you export the package or the CPL (the "subfolder", when you see the title, on the user interface). You get a full DCP in terms of essences (that might be a better term from „assets“, since the PKL is an asset, with its place in the assetmap), but doesn't get an „updated“ assetmap file, nor a PKL file, contrary to the re-created ones you get from Alchemy, Doremi, Dolby, etc. I suppose that the Assetmap file and PKL file are the originals. (I like how that sounds, but not the result.)

To be honest, I didn't bump into that. I suppose that the few times I exported DCPs from Sonys, it would have been OVs.

Just out of curiosity, is the username with which you FTP-in „sms“ and the connection the „Data“ one, or other? (Of course, one can SFTP as well...)
I am asking because there are other FTP users that may connect, like „s2suser“ for ingest, „ftpuser“ for another purpose, etc.

I suppose that one could ingest that package to a Dolby (DSS) server, and then export from there, but that is as much trouble (more or less) as fixing the trouble manually. I assume one could manually add the extra info on the PKL and ASSETMAP and change the UUIDs of them, in order to not create conflicts.

I wonder if the whole job could work as a feature of the combiner...

I have a vaguely similar problem with the FTP exports of a TMS that include two assetmap files, so DCP-o-matic may not import them for editing, versioning, etc. I then have to ingest to a screen server and get the importable DCP for DCP-o-matic. Which is kind of lame.

[Carsten]

@Stéphane: I have noticed there is a private message from you, but I am not allowed to read it, since it is marked as 'on hold'. It may be that Carl needs to review this first. Probably some forum security measure.