Vtyjkyuk

We have

$$f_X,Y(x,y) =
begincases
1 & 0 < x < 1, x < y < x+1; \
0 & textotherwise
endcases
$$

and we're looking for $operatornameVarY$.

I got two different answers with two different methods and I'm not sure which one is correct.

Method 1:

$$operatorname E(Y) = int_0^1 int_x^x+1yf_X,Y(x,y),dy,dx = int_0^1 left(x+frac12right) , dx = 1$$

$$operatorname E(Y^2) = int_0^1 int_x^x+1y^2f_X,Y(x,y) , dy , dx = int_0^1 left(x^2 + x + frac13right) , dx = frac76$$

So $operatornameVar(Y) = frac16$

Method 2:

Marginal distribution $f_Y(y) =1$

$$operatorname E(Y) = int_x^x+1yf_Y(y),dy = x+frac12$$

$$operatorname E(Y^2) = int_x^x+1y^2f_Y(y) , dy = x^2 + x +frac13$$

So $operatornameVar(Y) = left(x^2 + x +frac13right) - left(x+frac12 right)^2 = frac112$

One of these solutions is probably wrong, but I can't really identify the error.

The first method is correct.

The second method uses the wrong marginal distribution.   It appears to be a joint

Notice that the joint distribution is uniform over a rhombus. Plot this and notice the lengths of the horizontal cross section for various $y$ values from $0$ to $2$.

It should be as follows.

$$beginsplitf_Y(y) &=int_0^1 mathbf 1_xleq yleq x+1mathrm d x\&= int_max(0,y-1)^min(1,y)mathbf 1_0leq yleq 2mathrm d x \ & = ymathbf 1_0leq ylt 1+(2-y)mathbf 1_1leq yleq 2\&=begincasesy &:& 0leq ylt 1\2-y&:&1leq yleq 2\0&:&textelseendcases endsplit$$

As a reality check, a marginal distribution's support should not include other variables, and the interval must contain all realisable values for the variable.

$$beginsplit1 &=int_0^1 ymathrm d y+int_1^2 (2-y)mathrm d yendsplit$$

Then the expectations shall be in agreement with the first method. $$beginsplitmathsf E(Y) &=int_0^1 y^2mathrm d y+int_1^2y(2-y)mathrm d y \ &= 1\mathsf E(Y^2) &=int_0^1 y^3mathrm d y+int_1^2y^2(2-y)mathrm d y \ &= dfrac 76endsplit$$

Your first method is correct, but your second message can be salvaged. What you calculate in the second method are the conditional expectation and the conditional variance, conditional on the value of $X$:

begineqnarray*
mathsf E[Ymid X=x]&=&int_x^x+1y,f_X,Y(x,y),mathrm dy=x+frac12;,\
mathsf E[Y^2mid X=x]&=&int_x^x+1y^2,f_X,Y(x,y),mathrm dy=x^2+x+frac13;,\
mathsfVar(Ymid X=x)&=&mathsf E[Y^2mid X=x]-mathsf E[Ymid X=x]^2=left(x^2+x+frac13right)-left(x+frac12right)^2=frac112;.
endeqnarray*

You can get the unconditional variance of $Y$ from these results by applying the law of total variance:

$$
mathsfVar(Y)=mathsf E[mathsfVar(Ymid X)]+mathsfVar(mathsf E[Ymid X])=frac112+mathsfVarleft(X+frac12right)=frac112+mathsfVar(X)=frac112+frac112=frac16;.
$$