How to read an OpenEXR file quickly

Over the past few years I've spent quite a bit of effort figuring out how to read OpenEXR files quickly, particularly files containing many channels. The short answer is that for maximum speed, all the channels in an EXR should be loaded using a single read operation. ProEXR for Photoshop has been doing this for a while; now with the channel cache, After Effects can do it too.

Continue reading for more than any sane person would ever want to know about the channel cache in AE.

OpenEXR files can hold any number of channels. A "standard" EXR file just contains the channels "R", "G", "B", and maybe "A". These are the channels that are read by AE when you drop an EXR into a comp. If there are more channels, you access them using the EXtractoR and IDentifier plug-ins.

You might think that loading the standard four RGBA channels would take the same amount of time for a 4-channel file as it would for a 50-channel file. But this is not the case! Due to OpenEXR's file structure, sorting through all those extra channels actually takes up quite a bit of time. (Note to self: don't dump superfluous channels I'm not using into EXR files because they will slow things down.)

Or if you've already rendered a 50-channel file, you might think that reading only 4 channels from it would be much faster than reading all 50. But it's actually not that much faster. Apparently sorting through channels is a real bottleneck. (Tiled EXR images require even more sorting, and therefore slow things down even more compared to a regular scanline EXR.)

Nuke is well-positioned to take advantage of this situation. With its multi-channel architecture it gets all the channels it needs out of the EXR with one read. With AE, each EXtractoR instance launches a separate read.

Let's think about this. Many channels means that every 4-channel read is slower. Many channels also means you probably have a lot of EXtractoRs. (Slow reads) x (many reads) = very slow.

Channel Cache to the rescue. When this is activated, all the channels are read into a memory cache during the first read. Subsequent EXtractoR calls get their channels from the cache for a big speedup. The cost of doing this, of course, is RAM. For example, I have a 1920x1080 EXR with 47 channels and it takes up 198 MB of memory in the cache for each frame. But in these days of big RAM, that's a small price to pay for the 5x speedup it gives.

In the input options dialog you can also set the global "Cache Size", which is the number of EXRs that will be in the cache at a time. It defaults to 3. The fourth file cached would cause the cache accessed the longest time ago to get dumped. You probably want this number to be at least as big as the number of multi-channel files loaded on a given frame. Unfortunately this setting is stored with your computer, not with the project.

Cached files stay in memory until they haven't been accessed for 30 seconds (can be changed by editing the AE preferences file). You can disable all caching by setting the number of caches to 0. Caches are also purged when you run the AE purge commands under Edit > Purge.

And that's all I have to say about that.