Preheat oven to 450˚F.

Remove giblets from chicken and reserve for another use, if you like. Rinse chicken inside and out; pat dry with paper towels. Season the inside with salt and pepper. Slice one of the lemons in half and place in the cavity, squeezing the lemon to release some juice. Use the side of a heavy knife to smoosh the rosemary sprig, and add it to the cavity. Rub the outside of the chicken with some olive oil, and season with salt and pepper.

Lightly grease a skillet or shallow roasting dish with olive oil. Set the chicken breast-up in the pan and roast for 20 minutes. 

Place the potatoes in a mixing bowl and toss with olive oil, rosemary leaves, and salt and pepper. Slice the remaining lemon into thick slices and drizzle them with olive oil.

Reduce heat to 375˚F. Arrange the potatoes around the chicken. Return to the oven and roast for an additional 20 minutes. Remove the chicken from the oven and arrange the sliced lemon, onion and capers around it. Continue to roast for another 25 minutes, or until a thermometer inserted into the thickest part of the thigh (away from bone) reads 165F, and the juices run clear.

Transfer the chicken to a cutting board and let sit for 5 minutes before carving. Arrange the chicken and roasted vegetables, lemon and capers on a serving dish. Serve warm.