Since page requests are processed by Drupal as usual, Drupal's authentication and session handling stays fully working. Thus, when requests provide an authentication cookie, authentication just works.
The project template is configured to use a shared cookie domain, e.g. drupal.example.com and nuxt.example.com both use the cookie on .example.com. That way, the cookie set during Drupal login is available to the frontend.
The Nuxt Drupal CE Connector module makes sure to forward a received cookie to the backend, such that authenticated requests just work.
Client-side requests that directly go to the Drupal API need CORS configuration, such that requests with a cookie are allowed by the backend. Lupus Decoupled Drupal comes with a CORS submodule which makes sure CORS is allowed for the configured frontend. That way, CORS requests with cookies just work.