Quick Tip: Mouse picking

Mouse picking can be found in nearly every 3D game now. In many cases, it’s crucial to let the user interact with the world by directly indicating some objects. One of the possible approaches is to convert pointer position to some vector/ray inside the world and do hit-testing. I use GLM library what can do lots of math for mouse picking.

Model-View-Perspective Matrices

When we want to render an object, we have to convert its position to OpenGL coordinates, where visible objects are in range [-1, 1]x[-1, 1]x[-1, 1]. You can think of them as some functions. You pass vertices and get the result.

(Part of) OpenGL coordinate flow

Now, we want to go the other way. We have a 2D vector in perspective coordinates and want to get world coordinates. How to do it?

Linear algebra comes in handy here. We know that world-to-camera and camera-to-perspective transformations can be depicted as vector-matrix multiplication. Also, our matrices are invertible, so we just have to calculate inversions of these functions and we’re done.

Window coordinates

We usually know where the pointer is. For example, we have a window with inner size 800×600. If the mouse is in the middle, it is at point 400×300 (approximately). Now, OpenGL uses screen coordinates [-1, 1]x[-1, 1] not matter what size or proportion is. How to convert between these values?

glm::vec2 viewportSize = glm::vec2(800, 600);

glm::vec2 getMousePositionForRaycasting(glm::vec2 mousePosition) {
  return (mousePosition / viewportSize * 2.f) - glm::vec2(1, 1);

For example:

getMousePositionForRaycasting(glm::ivec2(400, 300)) == glm::vec2(0, 0)
getMousePositionForRaycasting(glm::ivec2(600, 300)) == glm::vec2(0.5, 0)
getMousePositionForRaycasting(glm::ivec2(400, 150)) == glm::vec2(0, -0.5)
getMousePositionForRaycasting(glm::ivec2(400, 450)) == glm::vec2(0, 0.5)

You should check how your library returns values, in some cases Y axis is inverted.

Getting ray

We have projection and view matrices calculated earlier to show objects of the world.

// Calculated earlier
glm::mat4 projectionMatrix;
glm::mat4 viewMatrix;

glm::vec3 getRay(glm::vec2 point) {
  auto ray = glm::vec4(point.x, point.y, -1.f, 1.0f);

  // 1. Projection to view
  ray = glm::inverse(projectionMatrix) * ray;
  ray = glm::vec4(ray.x, -ray.y, -1.f, 0.0);

  // 2. View to world
  ray = glm::inverse(viewMatrix) * ray;
  ray.w = 0.f;
  ray = glm::normalize(ray);

  return glm::vec3(ray);

1. Get 3D point

We just take our entry point. To make it 3D we add z coordinate (-1 mean pointing forwards). We make w coordinate 1.

2. Projection to view

We take projection matrix inverse and multiply. We want out ray to still point forwards in Z axis since it should not be changed by perspective.

3. View to the world

Now we take view matrix inverse and multiply. Now ray is in world coordinates. It’s nice to return a normalized vector (i.e. of length equal 1). To do this, we first set W to 0 (to not count this dimension) and normalize.

Using ray

Now you can use the ray in any way. For example, let’s see how to calculate where ray hits the ground at level y = 0. We need to take our ray and scale it so that its so big (vertically) as camera above ground.

const glm::vec3 cameraPosition;

glm::vec3 hitGround(glm::vec2 mousePosition) {
  const glm::vec3 ray = getRay(mousePosition);
  const glm::vec3 hit = cameraPos - (cameraPosition.y / ray.y) * ray;

  return hit;

We assume here that your camera rotation does not allow to point above the horizon.

Further reading: