View Bug Details

IDProjectCategoryView StatusLast Update
0001902DCP-o-maticBugspublic2021-12-25 23:21
Reportercarl Assigned Tocarl  
PriorityimmediateSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Target Version2.16.0 
Summary0001902: DSS200 crashes with too low a J2K bitrate
Description

e.g. with black frames. Not sure what we can do about this, other than check the J2K size and add noise until it gets high enough?!

TagsNo tags attached.
Branch
Estimated weeks required
Estimated work requiredUndecided

Relationships

related to 0001979 acknowledgedcarl More subtle fix for DSS200 crashes on low J2K bitrate 

Activities

carl

2021-02-01 00:26

administrator   ~0004106

Possibly fixed by acfbd1c01020ea95d9e7eb3d63ddb14b9407732a

Carsten

2021-02-03 02:16

manager   ~0004107

Wow! Really?

carl

2021-02-03 07:00

administrator   ~0004108

Last edited: 2021-02-03 07:01

Apparently so, according to their tech support. A user is making black DCPs and they're crashing a DSS200. Just trying to get a test version out to the user, but the Apple notorization stuff is playing up, as per usual.

Carsten

2021-02-03 11:24

manager   ~0004109

Last edited: 2021-02-03 11:44

I assume they also use a very low bitrate in addition? Don't know wether that makes a difference, though.

Weird...unfortunately, currently I can't test on a DSS200.

I use long 'real' blacks very often in combination with timed text. Maybe, instead of adding noise, one should recommend to them to use their own 'black noise' source?

How is 'black' actually created without a source content, e.g. when I place audio in an otherwise empty project? Do you actually create a J2K from a 0data-image of container dimension? Would be interesting to find if there is a difference between that method an having a 'real' 0-padded source file. Maybe something happens there.

carl

2021-02-03 13:56

administrator   ~0004110

I assume they also use a very low bitrate in addition? Don't know wether that makes a difference, though.

Not sure, though pure black frames come out very small, no matter what your bitrate setting is.

Maybe, instead of adding noise, one should recommend to them to use their own 'black noise' source?

We could do, though I think lots of people would miss/ignore that advice and then potentially have crashes... The patch only adds noise when the size of an encoded frame is below some threshold, so it shouldn't affect things 99% of the time. And it's really a very small amount of noise.

How is 'black' actually created without a source content, e.g. when I place audio in an otherwise empty project?

A black image is created and encoded into JPEG2000. Then on the next frame it spots that it's the same as the last and re-uses the same JPEG2000 data. Apparently the manufacturer in question told them that low bitrates are a problem for the DSS-200.

Carsten

2021-02-03 15:31

manager   ~0004111

Last edited: 2021-02-03 19:19

In general, I wouldn't care to add something to assure compatibility. I feel a little bit concerned being urged to 'add something that isn't there' to every DCP. It would be nice if we could do more testing around this. e.g. would black be the same as 'near' black, does it have to be noise, etc. You remember that J2k which was caused by essentially two more or less irrelevant formal bytes. Wondering wether something like that could fix this issue as well. Do we know wether it could already be one single frame that causes issues, or does it have to be a sequence of specific length?

I think we did talk about keeping j2k streams within our bandwidth constraints (when we talked about e.g. 4k or High frame rate). If the noise is really only added when the resulting j2k is too small that would probably be acceptable. However, couldn't that lead to increased encoding time, repeatedly checking and re-encoding? I do know that compressing perfect black is very fast.

Funfact: The old Dolby DCP encoding station (SCC2000), now legacy equipment, was known for it's code-stream effectiveness, it was able to create very low bandwidth J2K while maintaining high visual quality. Wondering wether it's encoder added noise to black...

Are we sure that these low bandwidth J2K are actually compliant? Do they pass dcp_inspect, clairmeta? Maybe there is a bug in OpenJPEG? Should we contact them about it?

carl

2021-02-03 16:06

administrator   ~0004112

In general, I wouldn't care to add something to assure compatibility

I'm a little surprised by that! If somebody makes a DCP with a little black in it, plays it back on a DSS-200 and it crashes, DCP-o-matic will be blamed 100% of the time; and that's where all the FUD starts. It's unfortunate, but true, and it feels to me like we should do things to stop these crashes if they don't negatively affect anything else.

I feel a little bit concerned being urgend to 'add something that isn't there' to every DCP

Sure, but this is not every DCP - only ones with very low-bitrate frames.

It would be nice if we could do more testing around this

Indeed, I have asked the reporter to check this build, which will be a useful first data point.

would black be the same as 'near' black

If Dolby's explanation for the crash is correct, then yes.

It's true that their suggestion was to use an encoder where you could specify a minimum bitrate, but at first glance this isn't trivial with libopenjpeg so adding noise could perhaps be the next best way to test things. I'll ask the openjpeg folks if there's an easy way to ensure a minimum bitrate.

If the noise is really only added when the resulting j2k is too small that would probably be acceptable

That's exactly what is happening, yes.

However, couldn't that lead to increased encoding time, repeatedly checking and re-encoding?

Yes, slightly; but if we have 100 frames of the same black we'll only need to do this for the first frame.

Wondering wether it's encoder added noise to black...

Difficult to tell unless we have a black DCP encoded with it. Even then it might not be obvious.

Are we sure that these low bandwidth J2K are actually compliant?

No, good idea, we could check. Though I don't think dcp_inspect/clairmeta do much codestream-level checking of JPEG2000 data.

Carsten

2021-02-03 19:21

manager   ~0004113

Last edited: 2021-02-03 19:26

In general, I wouldn't care to add something to assure compatibility

I'm a little surprised by that!

I guess I put this the exact opposite way I meant it. Of course we need to do everything possible to assure DCP-o-matic DCPs do not crash any servers. Even if it is clearly the manufacturers fault. or, at least in this case where it is highly unlikely that the manufacturer will ever fix it.

It's a bit strange that this Dolby weakness comes up so late in time. Shouldn't we have seen such issues earlier? Can you supply some of the information you mentioned to me by mail?

carl

2021-02-03 19:25

administrator   ~0004114

Last edited: 2021-02-03 19:26

I guess I put this the exact opposite way I meant it.

Ah, I see now :)

It's a bit strange that this Dolby weakness comes up so late in time. Shouldn't we have seen such issues earlier? Can you supply some of the information you mentioned to me by mail?

It is indeed very strange... I'll follow up by mail.

Carsten

2021-02-03 21:38

manager   ~0004116

Last edited: 2021-02-03 21:38

Weird. I remember there is also that Doremi weakness that crashes them well below the 250MBit spec limit. We should probably implement some general bitrate watchguard. But how do we deal with this in the GUI? Keeping bitrates high enough for Dolbys is comparably easy, check for it, and apply fix. Wondering wether we should issue a warning about data rates of more than 220MBit/s or so as well?

carl

2021-02-03 21:41

administrator   ~0004117

There's a hint about 245MBit/s, and the verifier warns about individual frames above some limit as well. Maybe the 245MBit/s warning should go down, or maybe just limit the GUI to 220 unless the "let me use whatever bitrate I like" checkbox is ticked.

Carsten

2021-02-03 21:46

manager   ~0004118

Last edited: 2021-02-03 21:49

Yes. I think I suggested that 245MBit/s myself a long time ago once I heard about that Doremi weakness first time. But it was just a rough guess to keep the data rate from peaking over 250MBit/s. You may remember that I created an actual documentary DCP in late 2019 that crashed already at 230MBit/s on Doremis. I needed to solve it quickly and simply recreated it at 200MBit/s.
I also checked the actual data rate and they were precisely within specs. I guess we need a warning somewhere around 210-220MBit/s. I may do some testing around that, but it is probably wise to lower that warning threshold already now. 220 is a nice enough number... Limiting in the GUI is a good idea, but still leave that warning in, and set it to 220, I'd say. Some people are unnecessarily concerned about data rates always being to low for their precious content.

Carsten

2021-02-04 14:04

manager   ~0004120

Last edited: 2021-02-04 14:06

Speaking of blacks, blacks, dark blacks and real blacks, this is one of my favourites...

https://www.youtube.com/watch?v=wx8-mysJG2s

Maybe it's enough to add some blue for Dolbys...

carl

2021-02-04 14:14

administrator   ~0004121

You let Dougal make a DCP?!
https://www.youtube.com/watch?v=xJhbP4q32vs

Carsten

2021-02-04 14:27

manager   ~0004122

Of course!

NOOOOOOOOOOOOOOT11!!!

carl

2021-02-06 20:55

administrator   ~0004124

User reports that the noise hack with 65536 bytes minimum frame size fixes the DSS 200 crashes.

Carsten

2021-02-07 12:55

manager   ~0004125

Now, what a pretty number...

carl

2021-04-22 13:34

administrator   ~0004271

Reporter also says 16KB setting in DoM works; I'll probably hard-code this for 2.16.0 and perhaps we can investigate more later.

carl

2021-04-22 20:05

administrator   ~0004278

a2dbe09d787ad10881ac52761772d6d50d2e29e9

Bug History

Date Modified Username Field Change
2021-01-31 23:14 carl New Bug
2021-01-31 23:15 carl Estimated work required => Undecided
2021-02-01 00:14 carl Status new => acknowledged
2021-02-01 00:26 carl Note Added: 0004106
2021-02-01 00:26 carl Tag Attached: ci
2021-02-01 00:26 carl Tag Attached: ci-running
2021-02-01 00:26 carl Tag Detached: ci-running
2021-02-03 02:16 Carsten Note Added: 0004107
2021-02-03 07:00 carl Note Added: 0004108
2021-02-03 07:01 carl Note Edited: 0004108
2021-02-03 07:01 carl Note Edited: 0004108
2021-02-03 11:24 Carsten Note Added: 0004109
2021-02-03 11:44 Carsten Note Edited: 0004109
2021-02-03 13:56 carl Note Added: 0004110
2021-02-03 15:31 Carsten Note Added: 0004111
2021-02-03 15:39 Carsten Note Edited: 0004111
2021-02-03 15:41 Carsten Note Edited: 0004111
2021-02-03 16:06 carl Note Added: 0004112
2021-02-03 16:06 carl Priority normal => immediate
2021-02-03 19:19 Carsten Note Edited: 0004111
2021-02-03 19:21 Carsten Note Added: 0004113
2021-02-03 19:22 Carsten Note Edited: 0004113
2021-02-03 19:25 Carsten Note Edited: 0004113
2021-02-03 19:25 carl Note Added: 0004114
2021-02-03 19:25 carl Note Edited: 0004114
2021-02-03 19:26 Carsten Note Edited: 0004113
2021-02-03 19:26 carl Note Edited: 0004114
2021-02-03 21:38 Carsten Note Added: 0004116
2021-02-03 21:38 Carsten Note Edited: 0004116
2021-02-03 21:41 carl Note Added: 0004117
2021-02-03 21:46 Carsten Note Added: 0004118
2021-02-03 21:47 Carsten Note Edited: 0004118
2021-02-03 21:48 Carsten Note Edited: 0004118
2021-02-03 21:49 Carsten Note Edited: 0004118
2021-02-03 23:10 carl Tag Detached: ci
2021-02-04 14:04 Carsten Note Added: 0004120
2021-02-04 14:05 Carsten Note Edited: 0004120
2021-02-04 14:05 Carsten Note Edited: 0004120
2021-02-04 14:06 Carsten Note Edited: 0004120
2021-02-04 14:14 carl Note Added: 0004121
2021-02-04 14:27 Carsten Note Added: 0004122
2021-02-06 20:55 carl Note Added: 0004124
2021-02-07 12:55 Carsten Note Added: 0004125
2021-04-22 13:01 carl Tag Attached: alpha-1-blocker
2021-04-22 13:34 carl Note Added: 0004271
2021-04-22 14:54 carl Branch => 1902-min-frame-size
2021-04-22 20:05 carl Assigned To => carl
2021-04-22 20:05 carl Status acknowledged => resolved
2021-04-22 20:05 carl Resolution open => fixed
2021-04-22 20:05 carl Note Added: 0004278
2021-04-22 20:09 carl Relationship added related to 0001979
2021-04-22 20:09 carl Tag Detached: alpha-1-blocker
2021-10-08 00:05 carl Status resolved => closed
2021-12-25 23:21 carl Branch 1902-min-frame-size =>