The Ardour Canvas

The main area of the editor in Ardour is normally referred (by developers) to as "the canvas". It displays tracks and busses as arrangements of data of various kinds along a horizontal timeline. The term "canvas" comes from several GUI toolkits, which use this word to describe a GUI element on which arbitrary objects can be placed, stacked on top of each other and moved around. Another phrase for the same thing is a "scene graph".

Historical Background

Ardour2 and versions of Ardour3 up to and including 3.5.x used a canvas called "GnomeCanvas" which actually had nothing really to do with the GNOME project. It had all the features one would really want from a canvas/scene graph but all drawing was done using the CPU and the appearance of the canvas was first drawn into an RGBA buffer in memory before being rendered to some surface/element known to the native window system. The "libart_lgpl" library was used for drawing, along with the option to use direct pixel manipulations.

The performance of the canvas was not bad on Linux, but worse on OS X where the primitive used to render the RGBA buffer to the window was not very efficient. The main problem for Ardour with the GnomeCanvas was that it could not utilize graphics hardware to do drawing under any circumstances, which meant that drawing many common "modern" object renderings (notably, gradients) were not available as basic operations and relied on the CPU, even if a GPU was available that could the job faster and better.

This made it hard to really move forward with GnomeCanvas as the basis for any future changes to Ardour's appearance. We did not want to write our own primitives for gradient rendering, or for rendering rectangles with rounded corners, amongst other things. If a GPU was available, we wanted the canvas to use it when rendering. In addition, nobody was maintaining GnomeCanvas anymore. In fact that final few releases of GnomeCanvas were all based on patches that came from the Ardour project. The decision was clear: we should switch to a new canvas implementation.

The GTK project has steadfastly refused to identify any other canvas implementation as "blessed", meaning that there have been several efforts to improve on GnomeCanvas. In fact, the GNOME project used to have a useful page on this (https://wiki.gnome.org/ProjectRidley/CanvasOverview) that has now been deleted. Of the candidate canvases listed there, the only clearly appropriate for Ardour was "goocanvas". It was written in C++, retained most of the GnomeCanvas semantics, added a few useful items, and used Cairo for drawing/rendering. It seemed close to ideal.

However, a more detailed inspection of GooCanvas didn't leave us overwhelmed by the feeling that we had found the right solution for our purposes. So in about the middle of 2012, Carl Hetherington set about implementing a brand new canvas. It was almost entirely disconnected from GTK itself, used Cairo for all rendering, and was carefully designed to tackle some of the scaling issues that we were aware of (it is entirely possible to have thousands or even tens of thousands of items in the editor canvas). Carl's work is the basis of the canvas that was introduced in Ardour 3.6 although several basic aspects of its design and implementation have been changed since his first version.

Requirements

For a canvas-type GUI widget/object to be useful for Ardour, it has to have the following properties:

  • No limit to the number of items present on the canvas
  • A coordinate space that extends along both X and Y axes at least as far as Ardour's 64 bit timeline
  • Full use of a powerful drawing API (Cairo is the reference drawing API, and given its likely adoption as part of a future C language specification, a very desirable API to have available)
  • Ability to utilize GPU rendering operations when appropriate.
  • Common operations must scale to conditions involving on the order of 10k items. This means that they must avoid recursion wherever possible
  • Ability to independently scroll different sections of the canvas. For example, Ardour's rulers scroll left and right but not move in response to vertical scroll requests. In contrast, the main track display scrolls along both axes. At some time in the future, when track headers are a part of the canvas, the headers will scroll only up and down, not left and right.
  • Ability to place items on the canvas that span independently scrolling regions (e.g. the playhead, which covers both the rulers/timebars and the tracks, even though those two areas scroll independently)

There are also several features that we do not require or use:

  • Scaling and rotational transforms
  • 3D
It may seem puzzling that the various geometric transforms are not of interest. This is is because Ardour's use of the canvas is really nothing like the presentation of an image combined with user-driven operations to zoom or rotate the image. Ardour draws very specific images in which single pixels have semantic content inherent in their existence and placement. When we "zoom", we are not zooming text, or the outlines around boxes, but the way waveform data is presented. And even the waveform drawing itself is not just rescaled, but redrawn potentially with different (more precise data). As for 3D, perhaps we will one day find a use for this aspect of visual data presentation, but so far, we have no use for it.

The Implementation

Concepts

Canvas

A Canvas is an object on which zero or more Items can be drawn and displayed to the user. The Canvas can also receive events notifying it of interactions with the user (such as a button press, a motion event, a key press, etc). There is no limit to the number or type of Items that can appear on a Canvas, and the full extent of a Canvas spans the maximal size of a double precision floating point value. All coordinates are double precision floating point.

The base class for the Canvas has no relationship with GTK at all. But it also is an abstract class that requires a few methods to be implemented in order to be instantiable. The GtkCanvas is a concrete class that causes the Canvas to be rendered in a GtkEventBox, and to receive events via the GtkEventBox.

The Canvas implementation also provides a utility class, GtkCanvasViewport, which packages a GtkCanvas and adds two GtkAdjustments which can be used to scroll the contents of the canvas. If scrolling is not required, then it is possible to use GtkCanvas directly.

Item

An Item is a discrete entity that can be placed on the canvas. Items can be drawn, hidden, moved in either direction, and raised or lowered in the stacking order on the Canvas (sometimes called Z order or Z-axis stacking). Item implementations are completely responsible for drawing the Item's appearance under all conditions. Items can receive events notifying it of interactions with the user. All Items inherit from a simple base class (ArdourCanvas::Item). The key method of an item implementation (the one that differentiates it from any other type of item) is Item::render(), a virtual method called to draw all or part of the Item as necessary. Each Item has a pointer to a parent, and to the top level canvas. Each Item has a position specified in the coordinates of its parent. If this position is (0,0), then the Item appears at the origin of its parent. If the position is (10,10) then the Item appears 10 pixels right and 10 pixels below the origin of its parent. The Item's origin is always (0,0) in its own coordinate space.

Note that the actual drawn area of an Item may have little to do with the position of the Item. The position may just act as the origin for the coordinate system of the Item, with rendered pixels being arbitrary distances away from this origin.

The rectangular area that the Item will draw to is given by its bounding box, returned as a boost::optional<Rect>. It is an optional type because the bounding box may be undefined if (for example) a Group has no children or a Rectangle has zero area. The bounding box of a Group is the union of the bounding boxes of all of its children.

Item children

All Items may have zero or more child Items added to them, created a recursive nesting that defines the structure of a particular Canvas. In general, Items that draw particular forms (e.g. rectangles or waveviews) will not generally ever have children added to them, because their drawing code ignores the existence of children entirely. However, a Container is a derived type of Item which does pay attention to the presence of children when it draws itself, specifically by calling Item::render_children().

Container

A Container is a special kind of item that draws nothing by itself, but does draw its children. It allows the easy relative placement of a set of Items as well as a way to move a set of Items as a single entity. When the Container is moved, all of its children will draw to a new position on the Canvas.

Note that the position of a Container's children is defined by each child's position() method. A Container does not arrange its children in any automatic way, so it is the responsibility of other code to place Items correctly within the Container.

In the future there may be derived classes based on Container which do set their children's position (e.g a "Box" type which arranges them along an axis, or a "Table" type which packs its children into a tabular arrangement).

Root Group

A Canvas has a single top level Container, called "root", as its only Item. It has one special function: to track its own bounding box size and notify the canvas of changes that may affect the display of the canvas using the underlying windowing system.

ScrollGroup

A ScrollGroup is a special kind of Container which responds to instructions from the Canvas to "scroll" - that is, to alter what is visible within the window in which the Canvas is viewed. A ScrollGroup can respond differently to horizontal and vertical scroll commands (it can even not respond to them at all, if necessary). A ScrollGroup that responds to both scrolls will move its children within the window whenever any scroll command is received.

If you want items on the Canvas to move in response to scrolling, you need to add them to a ScrollGroup with the appropriate sensitivity to scroll commands. Items without a ScrollGroup in their ancestors will not move when the Canvas is scrolled.

ScrollGroups should occur immediately below the "root" group on a Canvas, and should not overlap. Their placement is achieved by setting their position. Nothing will prevent overlaps, but the result is ambiguity in which ScrollGroup covers a given area. They may be moved on demand just like any other Item.

Coordinate Systems

In a given Canvas, there are 3 coordinate systems in use, all with coordinates specified as double precision floating point. A single axis value is stored in a type called Coord, and a pair of axes values used to define a point in a coordinate space is stored in a type called Duple. These names were chosen partly to avoid with the primitives found in various native Window system APIs.

All 3 coordinate systems use the same convention as Cairo and X Window, with the origin at the upper left, increasing x-axis values indicate rightward movement along the axis, and increasing y-axis values indicate downward movement along the axis.

Window Coordinate Space

(0,0) refers to the pixel at the upper left of the window in which the Canvas is displayed. Coordinates can be negative but never for drawing purposes. Coordinates can be of any positive size, but for drawing purposes should never exceed the height or width of the window in which the Canvas displayed. Thus if the window is W pixels wide and H pixels high, then (W-1, H-1) refers to the pixel at the lower right corner of the window. All coordinates should be non-fractional because they should refer to actual pixel positions in the displayed window.

Events are delivered to the Canvas in Window coordinate space.

Canvas Coordinate Space

A vast coordinate space. (0,0) is the center, and it extends by approximately 1e307 in both directions along each axis.

The relationship to Window coordinate space depends on the scrolling commands that have been issued to the canvas and the arrangement of ScrollGroups within the canvas. Canvas::window_to_canvas() and Canvas::canvas_to_window() can be used to convert between the two spaces. If the canvas has not been scrolled, then (0,0) in Window Coordinate space corresponds to (0,0) in Canvas Coordinate space.

Note that if the Canvas contains more than 1 ScrollGroup, there is no single mapping between a Window Coordinate space axis value and a Canvas Coordinate Space axis value. The transform depends on precisely which ScrollGroup covers an (x,y) coordinate, which in turns defines the offsets between Window and Canvas Coordinate spaces.

Events are delivered to canvas items in Canvas Coordinate Space.

Item Coordinate Space
A translated version of canvas coordinate space. (0,0) refers to the origin of the item (even though it may never draw anything there). (10,10) is 10 pixels right of and below the origin. Item::item_to_window(), Item::window_to_item(), Item::item_to_canvas() and Item::canvas_to_item() exist to convert between Item Coordinate space and the other two.

Available Item Types

Many of the items listed below are derived from one or both of ArdourCanvas::Fill and ArdourCanvas::Outline. These base classes provide a few methods to control how the item is drawn, specifically how/whether it is filled and how/whether an outline is drawn. Common methods include Fill::set_fill_color, Fill::set_gradient, Outline::set_outline_width and Outline::set_outline_color.

Because the canvas uses Cairo for drawing, and because single-pixel width lines are a common requirement, it may be important to read and carefully understand this Cairo FAQ answer. The short version is that you can't optimize the design of a drawing API for both trivially obvious semantics when filling and when drawing lines. Cairo's API was optimized to make fill semantics trivially obvious and line drawing slightly more complex. Perhaps you would have done it differently.

In general, the Canvas items that draw lines or outlines default to rendering single-pixel wide lines, and do so using the 0.5 coordinate offset "trick" mentioned in the Cairo FAQ cited above.

Rectangle
Draws a rectangle. Each edge can be specified as outlined or not. Note: the edges of a rectangle are on its boundaries, not outside them. The fill and outline colors can be specified independently.
Arc
Draws some part of a circle, with a specified angle. The arc can be outlined and filled, with two separately specified colors.
Circle
Derived from Arc, but always draws the full 360 or 2*PI radians.
Line
Draws a straight line between two points. Width of the line can be controlled using ::set_outline_width().
PolyLine
Draws a series of straight line segments between any number of points. Width of the line can be controlled using ::set_outline_width().
Curve
Draws a smooth curve through an arbitrary series of data points. Internal implementation uses Catmull-Rom splines, with the option to use either Centripetal, Chordal or Uniform curves (Centripetal is generally the best for drawing smooth curves). This curve differs from those offered by Cairo, which are Bezier splines and are not guaranteed to avoids "knots" when interpolating between points, which is critical when representing various kinds of data in Ardour.
Polygon
Draws a series of straight line segments between any number of points, but always closes the path to link the first and last points. Separate outline and fill colors may be specified.
Arrow
Draws a straight line between two points, and offers the option to draw an arrow of various sizes and shapes at each end.
LineSet
Maintains information about a set of all-horizontal or all-vertical lines. Each line has its own startpoint and width, but they all have the same origin along one axis and the same extent. We use this in Ardour for the grid/measure lines, and also for the ruled background of MIDI tracks. It is significantly more efficient than using the equivalent number of Line items. Note that the drawn lines never receive Event notifications of any kind, neither does the LineSet as a whole.
Text
Draws a short piece of text. The text will be drawn using Pango, and can have its color, font and alignment characteristics fully specified (no outline option is available, however). The text to be drawn can also be limited by a pixel width, so that the item never displays "too much" text regardless of what text it is asked to display.
Image
Draws an image. The image is given to the Item as a raw data buffer with a specified width, height, stride and data format.
Pixbuf
Draws an image. The image is given to the Item a Gdk::Pixbuf.
WaveView
A specialized item for Ardour which draws a representation of an audio signal.
XFadeCurve
A specialized item for Ardour which draws a representation of the pair of curves that fade material in and out near the near the ends of a region.
Ruler
A specialized item which draws a ruler with a series of 3 levels of tick marks each with optional annotation. Methods to determine where the ticks should be placed and what annotations (if any) should be drawn near them are handed to the Ruler constructor, and called whenever the visible part of the ruler changes. A Ruler does not know about or care about the nature of the time units it is displaying.

The Coordinate Space problem

In Carl's original implementation, the Canvas Coordinate Space and the Window Coordinate Space were unified. We took advantage of Cairo's use of double for its own coordinate space, assuming that to accomplish scrolling we could simply use cairo_translate() to move objects as needed when rendering.

Unfortunately, Cairo has a deep and longstanding bug (feature?) which means that although it uses doubles for coordinates, the actual size of the coordinate space it can manage correctly is limited to roughly 32767 pixels x 32767 pixels. Any attempt to use cairo_translate() to move outside this area, or any use of coordinates outside of this area causes drawing glitches and outright errors. Although the Cairo developers are aware of this bug, no fix for it appears in sight, and so we were forced to work around it. The Ardour canvas is required to span a much larger coordinate space than Cairo can handle.

The adopted solution is not as clean as Carl's original equivalence relationship, but is still not too hard to comprehend. Items maintain their own coordinates using the full range of a double precision float. However, rendering is done by converting all coordinates used for drawing into window coordinate space. If the item's bounding box doesn't intersect with the area visible in the Window, then the item will not be asked to Render itself. If it is asked then the transformed coordinates passed to various Cairo calls. Since the limits on window dimensions more or less match Cairo's internal limitations, these coordinates are guaranteed to be in range for Cairo's API.

Rendering Model

The concrete base class (e.g. GtkCanvas) receives "expose" or "draw" notifications from the underlying window system, telling it that part or all of the window in which the canvas is displayed needs to be redrawn. These notifications arise from two sources:

  1. Window-system level changes involving the visibility of the canvas' window (e.g. inital display, becoming visible after another window is moved away from the canvas' window, etc)
  2. Redraw requests queued by the canvas itself in response to changes in its Items. For example, when an Item is added, moved or resized, some or all of the part of the Canvas visible in the window may need to be redrawn. Of course, the Item could be completely off-screen, in which cases such changes will not cause a redraw request.
Both sources for expose/draw notifications result in the same GdkEvent being sent to the GtkCanvas, which includes a specification of the area of the window that needs to be redrawn (this area is specified in Window Coordinate space). The GtkCanvas obtains the Cairo context being used for drawing, repackages the area as a Canvas-native type, and passes both into the Canvas::render() method.

The rest of the rendering implementation has no relationship to Gtk or the native window system, and consists of zero or more calls to the Cairo API to alter the appearance of a Cairo surface (assumed to be the window in which the Canvas is displayed). The Canvas calls ::render() on its root group, and this finds all children that are fully or partially visible within the redraw area and then (recursively) calls their ::render() method. Note that every item is passed the same original area to be redrawn as was given the Canvas itself. Each item converts its bounding box into Window Coordinate space, and then decides precisely what and how it will draw. Some items (e.g. the Image item) may simply render using a pre-drawn, memory-cached Cairo surface. Others will draw directly using some programmatic logic and common Cairo drawing operations (for example cairo_rectangle(), cairo_line_to(), cairo_arc(), cairo_stroke(), cairo_fill()).

Event Propagation

The concrete canvas object (e.g. GtkCanvas) receives events from the native window system (e.g. X Window on Linux, Quartz on OS X) via the GTK toolkit. The following event types are explicitly handled by the canvas:

  • draw/expose (described above)
  • button press and release
  • motion
  • enter and leave
Note that at present, key press and release events are not handled by the Canvas, but left for a parent widget to handle (which suits Ardour's general interaction model).

Current Item

By default, button press and motion events will be delivered to a designated item termed the "current item". If a drag is in process, there is an alternate item, the "grab item" to which events will be delivered. Maintaining the correct current item for event delivery is one of the central tasks of the canvas implementation.

This is implemented in Canvas::pick_current_item (Duple const& point, int state). The design of the canvas allows for the use of "smart" methods of determining which items cover a given point, but at this time (June 2014) we use a naive, O(N) linear search through each group. This is done via the LookupTable object table associated with each Item. The code contains an attempted "smart" but unfinished and untested implementation (OptimizingLookupTable). For now, DumbLookupTable is used to find items covering a point, and that just iterates over a list of children and calls Item::covers() on each one.

Enter/Leave

For every enter/leave event and for every motion event, as well as whenever there are any changes to an existing Item or when an Item is added or removed from the canvas, the canvas takes the current pointer position and determines what item covering that position should be considered the "current item". Recall that there may be many items covering the pointer position - the canvas will use the upper-most item as "current item".

Whenever the "current item" changes, a series of events must be sent to various items in the canvas to notify them of the change so that they may, if necessary, modify their appearance. The existing current item is sent a "leave" event, with the "detail" field of the event set to indicate whether the new current item is a child, parent or unrelated item. Then the parent chain for the old current item is traversed, with each of them being sent a leave event (again with the "detail" field set appropriately).

After the leave events have been sent, the parent chain for the new current item are sent an enter event (with the "detail" field correctly set), and finally the new current item is sent a enter event also.

Once this set of events has been delivered, all Items have been appropriately notified of a change in the current item, and "current item" is finally reset. Subsequent button and motion events will be delivered to that item (unless a grab is in effect).

Note that enter/leave events do not cascade to parents as described in the next section. They are explicitly delivered by the canvas, without regard for whether or not a particular item returns true or false to indicate that it handled the event.

Parent/Child chain

Each canvas item has a member: sigc::signal<bool,GdkEvent*> Event; to which arbitrary functors can be attached as handlers. Each handler returns a boolean value to indicate whether or not it handled the signal/event or not. This signal will be "emitted" as part of the event delivery mechanism for that event.

If an event is delivered to an Item, zero or more handlers for the event will be invoked. If any of them return true, the event is considered to have been handled, and no further processing will be done relating directly to the event. Note that as in GTK, the connection order of the handlers can have important implications. If there are two functors/handlers connected to an item's Event signal, and the first one always returns true, then the second one will never be invoked. sigc::signal::connect() has arguments to control the ordering of handlers as they are connected.

If there are no handlers, or none of the handlers returns true, then the event is considered unhandled, and is redelivered to the item's parent. This cascade-to-parent continues until the event is delivered to the root group. If it is still unhandled at that point, we notify GTK that the canvas has not handled the event, allowing it to continue with its own propagation strategy (e.g. to the top-level window in which the canvas occurs).

We use sigc::signal rather than PBD::Signal here because the signal is connected to and delivered ("emitted") in a GUI context that is always serialized and always single threaded. PBD::Signal (unlike sigc::signal) is thread-safe, but we don't need those semantics for GUI-related signals.

How Scrolling Works

Canvas::scroll_to (Duple const& point) is invoked to tell the canvas to redisplay its contents so that where appropriate, they are offset by point.x and point.y along each axis.

However, as noted above, the Ardour canvas contains various conceptual (and actual) groups of items which need to respond differently to scroll commands. Rulers scroll left/right only; track headers scroll up/down only; main track displays scroll left/right and up/down.

ScrollGroups are the currently implemented solution for this problem. When a ScrollGroup is created, its constructor requires a specifier of which axes (X, Y, X&Y or none) it will scroll. When Canvas::scroll_to is called, the canvas iterates over all children of its "root" group. Each ScrollGroup that it finds at this level has its ScrollGroup::scroll_to method invoked. This alters the _scroll_offset member of the ScrollGroup.

As described above, when items are rendered they first convert from their own coordinate space to the window space. Every item has a pointer to its "scroll parent" (a Scrollgroup; the ScrollGroup may actually be several "generations" from the Item, but every item has only 1 scroll parent). The implementation of Item::item_to_window uses the ScrollGroup parent's scroll_offset to determine what (if any) offset in effect due to scrolling, and incorporates this into the returned value.

As a concrete example: an Item has a scroll parent which responds to both vertical and horizontal scrolling. The canvas has been scrolled by (-100, 100). The Item sits at (40,40) in Canvas Coordinate space. It calls item_to_window on a point (x,y) within its coordinate space. It first converts to canvas coordinate space, generating (x+40,y+40). Then it applies the scroll offset of its scroll parent (-100,100) to give (x+40-100,y+40+100). The item coordinate (x,y) is thus (x-60,y+140) in window coordinates.

A similar item with a scroll parent that responds only to horizontal scrolling would do the same calculation and end up with (x-60,y+40). The two items would thus appear at different positions along the y axis, as appropriate given their scroll parents' sensitivity to scrolling.

Items that are have no scroll parent will never scroll. Although it is possible to also use a scroll group with no scroll sensitivity, scroll groups may not overlap, so this no-scroll-parent => no-scroll-behaviour feature has some potential use for global canvas items that should remain in the window at all times but that span different sections of the canvas.

Note that because items have direct pointers to their scroll parent, this design adds a very lightweight O(1) operation to compute the effect of scroll on the item_to_window/window_to_item transforms. We do not have to traverse parent/child chains, which would make the computation O(N) where N represents the depth of a given item from the canvas root.