Support Ukraine

Yesterday I started digging around in the Blender[a] codebase. The purpose of that digging was to see how I could implement a better video sequence editor; possibly by using the Sequence Rendering architecture - ported to C.

The codebase is as one would expect a 15-year-old codebase to look: A bit of an undocumented mess, but does the job and does it well. One can describe Blender as a "struct visualizer" and one wouldn't be off by much. The whole idea behind Blender is to have a database in which we store scene data, materials and so on. The application we call "Blender" then provides multiple views of the database, which becomes not just a repository, but the entire application state. When saving a file, the database is simply written out, and when opening a file, the application state is re-read. This makes it very easy to preserve application state. Since all data is in the database at all times, saving and loading becomes trivial.

In order to get this to work Blender has something called "Structure DNA[b]", which is a data serialization protocol. Instead of writing the database as a big binary blob, Blender includes information about the data structures being written. So, for example, instead of writing a point with coordinates x=5 and y=13 as "0513" and keeping the logic for splitting the "0513" blob into the right fields in the program, the information that this is a "Coordinate" struct with fields "x" and "y" is included in the file. This means that the layout of the struct can change, but Blender can still parse and load the data.

Well, almost...

If the structures have changed, Blender might be able to parse the file, but not be able to construct the correct in-memory structures. For example, if we rename "x" to "a" in the coordinate example above, the SDNA system has no way of knowing that the field "x" in the file corresponds to the field "a" in the struct.

This problem is amplified when the changes are architectural. Since the SDNA representation is the program state, it is not an interchange format, but a format that largely inherits the internal workings of Blender. It is unavoidable that some of the internal workings end up in the format. After all, both the internal representation and the file format try to describe the same problem domain; albeit for different purposes.

When writing an algorithm it is common to have one representation to describe the domain, and one to actually run the algorithm. For example, when doing image processing, you can use a TIFF file to describe the image to be processed, but when doing the actual processing, you may want to split the image into tiles, or convert it to the frequency domain, or do any other kind of transformation.

With Blender, this is brutally difficult. The internal working format is the domain description format, even if one is much better off keeping the data in a different shape while working.

As far as consequences for me, this boils down to a choice: How do I do major changes to the VSE, if they are required? Do I...

1. ...write a "VSE 2" and create all-new structures?

2. ...create some kind of "compatibility loader" or data import filter that converts the old data to the new, as far as possible? That is, we read old and new formats, but only write new format.

3. ...convert the old structs to the new in-memory? That is, we read and write the old format, maybe with some back-compatible changes, but use a new format internally?

I put the question to the Blender developers: S-DNA and Changes[c]

The answers were enlightening:

All the options you proposed are possible.

1) This was done with our animations system, 2.4x IPO's to FCurves, in most cases this is not trivial even for experienced developers, this is handled in readfile.c's do_versions(), but Id say only do this if all else fails.

2) possible too though I cant think of any examples off hand were blender does this, save and load is used for undo so if this become common place it could make blenders undo too slow.

3) This is what's done for editbones & editmesh, depends on what you intend to do so can't say if this is a good option for the VSE.

— Campbell Barton[d]

1. Write a "VSE 2" and create all-new structures?

Try to reuse the existing structures maybe modifying them. For some big changes some DNA structs get deprecated and new one appears like for the 2.5 animation system rewrite.

2. Create some kind of "compatibility loader" or data import filter that converts the old data to the new, as far as possible? That is, we read old and new formats, but only write new format.

3. Convert the old structs to the new in-memory? That is, we read and write the old format, maybe with some back-compatible changes, but use a new format internally?

There is a huge doversion() function in readfile.c that converts the old file's DNA to new one.

So to resume:

- Even if you deprecate some DNA struct, do not erase them from the source doversion() will need them

- Internally Blender will only knows of the new system/structure except in the doversion().

In a more general point of view it would be best to make some design plan and share them with the module maintainer before starting coding.

— Xavier Thomas[e]

1. Write a "VSE 2" and create all-new structures?

this will break compatibility with older versions of blender. Should only be done as a last resort and if you *really* know, what you are doing.

2. Create some kind of "compatibility loader" or data import filter that converts the old data to the new, as far as possible? That is, we read old and new formats, but only write new format.

that is *always* necessary, otherwise, you can't open old files or make sure, that on load of an old file, your new structure elements are initialized properly. This is done in doversions() in readfile.c .

3. Convert the old structs to the new in-memory? That is, we read and write the old format, maybe with some back-compatible changes, but use a new format internally?

nope. After each editing operation, DNA data has to be in sync, since DNA load/save is also used on undo operations(!).

I'd also suggest, that you first try to make sure, that you *have* to change something, and why. Since, you guessed it, you will most likely make some people unhappy, that want to open their new .blend file with an old version and see things broken all over the place.

— Peter Schlaile[f]