This setting controls the resolution of the textures applied to game elements. Higher texture quality will make objects look better close up, but will require more VRAM. Texture quality typically does not impact framerate, as long as you have enough VRAM.
Anti-aliasing smooths out jagged graphics. The need for Anti-aliasing decreases with resolution.
Here are some of the many kinds of Anti-aliasing you could see in a game:
Ambient occlusion is a technique used for improving the quality of lighting in virtual environments.
Reduces the bluring of textures that are viewed at an angle.
Usually you have a choice of the amount of filtering you want to apply: None, 2x, 4x, 8x, and 16x.
This option has a minimal performance impact in games.
Usually, increasing the shadows quality increases the shadow map size. This will make shadows look less blurry and more accurate. If you are having FPS problems, shadows should be one of the first things you should try turning down.
Dynamic Reflections allow you to see objects mirrored in pools of water and other shiny surfaces. Can be performance intensive.