Problems importing a large J2C sequence

[gi-jones]

Hi all
I'm trying to make a DCP of a full feature film, but DOM becomes unresponsive when I try to import the file sequence.

The sequence consiste of roughly 130000 J2C files. They are DCI compliant in 4k and made from Resolve. I have tested the full workflow with the first minute, and that works fine. But when I point DOM to the folder containing the full file set, it spends roughly 2 hours indexing (or whatnot). When that is done, the program simply crashes. One time I managed to save the project, but every time I opened it afterwards it crashed again.

Is this not a reasonable use case? Or am I doing something stupid?

[carl]

This should work - it's a shame it took 2 hours. Were the files stored on a local drive or somewhere else?

[gi-jones]

Should it happen significantly faster than 2 hours?

The files are on an external USB-C SSD disk. ExFAT

The thing is - this morning I tried using ASDCP (command line) to wrap an MXF, which worked fine. So I'm pretty sure the disk is ok.

[gi-jones]

So I tried moving the files from the external drive to my internal SSD, and suddenly the ingestion only took a couple of minutes.

I guess the problem must be with using the external drive somehow. It's a very fast one though - Samsung T7. The ExFat formatting might be a factor I guess...

[carl]

Did it crash when you used the internal SSD?

It's odd that it took so long with the external. There shouldn't be too much going on with the files; just finding them all, basically. I'll see if I can see any inefficiency in that code.

[Carsten]

Maybe - just for the fun of it, you could try to format the same external disk to NTFS (or whatever the native OS filesystem is), and just try the sequence import again. Maybe something weird happens because of ExFAT.

[IoannisSyrogiannis]

Working with Windows at work, I get a lot of ExFAT drives.
Someone had the great idea to use a couple for a film festival.
Copying a couple of terabytes took almost a day each time.
Bottom line: my bet is on ExFAT, not external vs internal drive.

[gi-jones]

Did it crash when you used the internal SSD?

Nope, everything worked fine from there.

I agree that ExFat is probably the issue. It's not possible for me to reformat the drive right now unfortunately.

[worldofcrap]

You don’t mention if you’re on a Mac, windows or Linux machine. If it’s MacOS, the support for ExFAT file system is DOG slow, particularly with many small files. Windows is slightly better, but it’s still nowhere near as fast as NTFS.