It is a LONG time since I used ArcScene and Exported to VRML. However, I used to do something similar a lot. The VRML exported from ArcScene is verbose and not always well-formed so I didn't like it and therefore do not recommend that you try to export your whole model in one go. It also didn't used to properly instantiate repeated objects - making the VRML rather inefficient and difficult to edit later. However, it would export the terrain reasonably efficiently.
My approach was to export just the terrain and any extruded polygons (building footprints etc). All objects represented by points I would create as a VRML object in a modelling package and then inline the object to the VRML file and add instantiate it at the point locations.
You need to adjust your coordinates unless you opt to use real-world coordinates. I used to use Excel to quickly generate pseudo VRML as I could create rows with my coordinates and then auto-fill the VRML code around them. To get the Z value I used extract values to points.
As a general note, VRML is VERY old and was superseded many years ago by X3D. I don't know if ArcScene ever did support X3D in the end (see this thread - where you will note that the OP there also suffered a similar crashing issue to yours). Even so, I would recommend that you export just the basic terrain to VRML, convert to X3D and go from there. X3D gives you much better animation, physics, editors and viewers etc
ArcScene is really nice for very quick demos but limited for anything beyond that. A much better alternative option is to use a 'proper' 3D modelling package like Blender. There are plugins that will take georeferenced rasters, DEMs and shapefiles. You can import all your layers into Blender, use the point-data shapefiles as particle instances to instantiate your point objects (or use a raster as a particle generator to control size and density of the instantiated objects). The great advantage of this approach is you have vastly more control over animate, lighting, fly-throughs, texturing etc.
Currently extruded polygons from Pro are not supported in the web scene viewer. The scene viewer allows you to extrude polygons on feature layers but only by a set number not an attribute value. If you want extruded buildings from an attribute value then you will have to publish them to your Portal and create a scene service.
Here is a tutorial covering that publishing workflow.
http://pro.arcgis.com/en/pro-app/help/sharing/overview/author-and-share-a-local-scene.htm
Best Answer
I would guess that your layer is not a 3D layer. You can check your layer attribute table. If the Shape field just says "Point" for each record you just have a 2D layer. You can use the 3D Analyst>Convert>Features to 3D tool to make your layer 3D. Once you have done that, open the table and look at the Shape field. It should now say "Point ZM", and the "Layer features have Z values" option should be active now.