I've managed to get something working, but it's not as good as I hoped and there may be a better way.
Add OSM
First I added the OpenLayers plugin and selected Plugins -> OpenLayers Plugin -> Add OpenStreetMap layer.
Import points
Next I created a CSV of my points like this:
LATITUDE,LONGITUDE,NAME
20.201389,14.965278,Name1
20.187278,14.97020,Name2
Then I imported it following these instructions, but with a comma delimiter, then zoomed and panned as needed.
Labels
Then in the layers panel I selected my points layer, I selected Layer -> Labelling, ticked "Label this layer with" and selected "NAME".
Print
Next I followed these instructions, starting from "click on the ‘New Print Composer’ button", adding just the things I needed and changing to portrait. I then exported as image and saved as PNG.
Results
This gave me a correctly sized (2480x3507) PNG. However, I found that often it would export an image with some white tiles. More often it would export with my points in the wrong places on the map. After retrying and restarting many times, I finally got an image with a complete OSM background and the points in the correct places. I still need to test a printout, but I think the OSM background is much more pixelated when compared with the smooth markers and labels, so perhaps there's a better way of doing it.
Well re-vectorization doesn't seem to be the key to a good solution here.
As @Andre points out, the OSM Openlayers plugin just adds a raster layer. You can pick an OSM map (as cyclemap) that is postprocessed to include height contour lines, but in general height data (DEM) aren't part of the (current) OSM data model.
So the contour lines are derived from the SRTM height dataset and transformed and local imported as it can be rendered with the default OSM renderers as Mapnik: https://wiki.openstreetmap.org/wiki/Contour
To make use of the contour lines with QGIS, you might want to download them seperately as shapefile and add them as vector layer on top of your OSM raster layer.
Best Answer
This way it works:
unselect your vector data, to show only the Openstreetmap background
File -> Save as Image
Load the image as raster, set EPSG:3857 as CRS
Delete the Openstreetmap background layer
Raster -> Projection -> Reproject
, choosing EPSG:3857 as source CRS and the CRS of the vector data as target CRS, saving as Geotiff under a different name, adding that to the canvasRemove the image loaded in step 3
Raster -> Extraction -> Clipper
checkMask layer
with the vector data, and add the result to the canvasRemove the unclipped raster layer from step 5
EDIT
If you want the clipping for all zoom levels, see my answer here:
How do I clip OSM basemap with a polygon?