Preheat the oven to 400°F (200°C).
In a mixing bowl, whisk 2 tablespoons olive oil with the Dijon mustard, 3 chopped garlic cloves, 1 chopped shallot, chopped rosemary, 1 teaspoon smoked paprika, and a pinch each of kosher salt and black pepper.
Add the chicken to the bowl and toss to coat evenly in the mustard mixture.
Heat 1 tablespoon olive oil in a large oven-safe skillet over medium-high heat. Sear the chicken until golden, about 3–5 minutes per side.
During the last 2 minutes of searing, add the 3 tablespoons salted butter and the lemon slices to the skillet to baste the chicken, then remove the chicken and lemon slices from the skillet and set aside, reserving any pan juices.
To the same skillet, add the orzo, the remaining 1 chopped shallot, and 1 clove minced garlic. Cook, stirring, until the orzo is lightly toasted, about 2 minutes.
Pour in the chicken broth, add the chopped kale, and add 2 tablespoons lemon juice. Season with salt and pepper, then bring the mixture to a boil over high heat.
Return the seared chicken and lemon slices to the skillet, nestling them into the orzo. Transfer the skillet to the preheated oven and bake uncovered for about 15 minutes, or until the chicken is cooked through.
While the chicken bakes, make the feta sauce: combine 4–6 ounces feta, 1/4 cup plain Greek yogurt, 1 grated garlic clove, juice from 1 lemon, 1/4 teaspoon smoked paprika, and a pinch of crushed red pepper flakes in a blender. Blend until smooth, thinning with additional lemon juice if needed.
Remove the skillet from the oven and let rest a few minutes. Serve the chicken and orzo topped with the creamy feta sauce and sprinkle with chopped fresh dill.