I describe below a categorified version of the coend construction, "2-coend" for short. It takes as input a collection of 1-categories $\{W_{xy}\}$ which afford left and right representations of a (weak) 2-category $A$. The output is a 1-category $C$ which can be thought of as the self-gluing or self-tensor product of $\{W_{xy}\}$ via the left and right $A$ actions.
Questions: Does this construction appear in the literature anywhere (other than here)? Does the special case where $A$ is replaced by a tensor category (2-category with one object) appear in the literature anywhere? How about the special case where $\{W_{xy}\}$ is the product of a left $A$ representation and a right $A$ representation? (So a categorification of tensor product rather than a categorification of the more general coend.)
The Drinfeld double is a special case of the 2-coend, and dually the Drinfeld center is a special case of the 2-end. (See final remark below.)
The definition I give below seems natural and inevitable to me, so I'm a little surprised I have not (yet) come across it elsewhere. If this MO question does not turn up any citations, I will tentatively assume that the definition is new.
Final remark before the details: I'm aware of definitions for general $n$ which should, in theory, specialize to this $n=2$ case. I'm mainly interested in definitions which are specific to $n=2$ are are close is spirit to this one.
Let $A$ be a weak 2-category. ("Weak" meaning that composition of 1-morphisms is not strictly associative.) I have in mind the case where the 2-morphisms of $A$ form vector spaces, but I don't think that matters for what follows.
Define a left representation of $A$ to be a (weak) 2-functor from $A$ to the 2-category of (1-categories, functors, natural transformations). For each object $x$ of $A$ we have a 1-category $Y_x$. For each 1-morphism $e:x\to y$ of $A$ we have a (notationally overloaded) functor $e: Y_y\to Y_x$. For each 2-morphism $h:e\to f$ of $A$ we have natural transformation (also denoted $h$) between the functors assigned to $e$ and $f$. For each pair of composable 1-morphisms $e$ and $f$ there are also an invertible natural transformation $c_{ef}$ which relates the functor for $ef$ to the composition of the $e$ functor and the $f$ functor. All of this data is required to satisfy some standard relations.
Let $A^{op}$ denote the 2-category where the order of composition of 1-morphisms (but not 2-morphisms) is reversed. (Reverse horizontally but not vertically.)
Let $\{W_{xy}\}$ be a collection of 1-categories with the structure of a left $A^{op}\times A$ representation. For each object $(x,y)$ of $A^{op}\times A$ we have a 1-category $W_{xy}$. For each pair of 1-morphisms $e:x\to u$ and $f:v\to y$ we have a functor $e\times f$ from $W_{xy}$ to $W_{uv}$. And so on.
Now let's define the 2-coend of $\{W_{xy}\}$ as above. The data for the 2-coend consist of (1) a 1-category $C$; (2) for each object $x$ of $A$ a functor
$$
gl_x : W_{xx} \to C ;
$$
and (3) for each 1-morphism $e:x\to y$ of $A$ an invertible natural transformation $s_e$ between the functors $(e\times 1)\bullet gl_y$ and $(1\times e)\bullet gl_x$, as shown in the following diagram.
(My convention in the above sort of 2-dimensional commutative diagram is to enclose the natural transformations labeling the 2-cells in boxes.)
The above data is required to satisfy conditions 1 and 2 below, and also to be universal in the sense described below.
Condition 1: For each 2-morphism $h:e\to f$ of $A$, the following 2-sphere-shaped diagram commutes.
(The natural transformation $s_f$ labels the "2-cell at infinity". Saying that a diagram like this commutes means that two different natural transformations, built out of two subsets of $s_e$, $s_f$, $1\times h$, $h\times 1$ and some identity natural transformations, are equal.
Condition 2: For each composable pair of 1-morphisms $e:x\to y$ and $f:y\to z$ of $A$, the following 2-sphere-shaped diagram commutes.
The data $(C, \{gl_x\}, \{s_e\})$ is required to be universal in the following sense. For any $(C', \{gl'_x\}, \{s'_e\})$ satisfying conditions 1 and 2 above, there exists a functor $\theta: C\to C'$ and, for all objects $x$ of $A$, a natural transformation $\eta_x: \theta\circ gl_x\to gl'_x$, such that the following diagram commutes for all 1-morphisms $e$ of $A$.
Some further remarks.
-
If we reverse all the arrows above, we get a definition of the 2-end of $\{W_{xy}\}$.
-
The above abstract definitions in terms of a universal property can be converted into more concrete definitions in terms of generators and relations. In this version the 2-coend $C$ has (by definition) objects the union over all $x$ of the objects of $W_{xx}$. The morphisms of $C$ are the union of all morphisms of $\{W_{xx}\}$, plus addition invertible morphisms $s_{eu}: e\cdot u \to u\cdot e$, for all $e:x\to y$ and all objects $u$ of $W_{xy}$. These extra morphisms must satisfy relations corresponding to conditions 1 and 2. An object of the 2-end is a tuple $(r_x)$, indexed by objects $x$ of A, where each $r_x$ is an object of $W_{xx}$. In addition, each such tuple is equiped with a collection of invertible morphisms $t_{er}: e\cdot r_y\to r_x\cdot e$ of $W_{xy}$. A morphism between two such enhanced tuples is a tuple of morphisms which is compatible with the $\{t_{er}\}$ according to conditions 1 and 2.
-
In the special case where $A$ is a tensor category and $W$ is the regular 2-category bimodule ${}_AA_A$, then the 2-end is the Drinfeld center of A and the 2-coend is what I would call the Drinfeld double of $A$.
Best Answer
I don't think I have seen anything like this published before, but I have written up a similar definition here (see here too).
One thing in your definition I would take issue with is that your 2-coend's universal property needs more to deserve the '2-' in its name: what I would expect (and what my definition, which otherwise seems to be equivalent to yours, says) is a bicategorical universal property that says