Special Aircraft Service

the SAS Hangar => The Lounge => Stainless' new Flight Simulator => Topic started by: Stainless on April 29, 2018, 12:56:18 AM

Title: Terrain
Post by: Stainless on April 29, 2018, 12:56:18 AM
After a LOT of experimentation, I have got fed up with messing about and just made a decision.

The system will use displacement mapping under a quad tree mesh.

So I need to generate a lot of texture maps.

Height map

The height map is 24 bits per pixel with each channel having the following mapping.

RedLow byte of height
GreenHigh byte of height
BlueSign byte

This allows a height range of 32767 to -32767 metres. Should be enough  :D

Normal map

24 bits per pixel encoding of vertex normal

Type map

32 bits per pixel with the following mapping

RedTexture 0 percentage
GreenTexture 1 percentage
BlueTexture 2 percentage
AlphaTexture 3 percentage

Each terrain region is assigned 4 surface textures based on altitude and slope. Think sand - grass - rock - snow. These are mixed based on the type textures pixel.

All this is auto generated from SRTM data.

At the end of this you have a "bare earth" terrain patch.

This is imported into an editor so each area can be detailed by hand, but will work as is without detailing.

If a detail map is available, the type map is only used for physics. Detail maps are stored by time range. So we can have different terrain details for each time period.

Here are some examples, I still have to split them down to terrain patches.



(https://s17.postimg.cc/awp8xq4wf/heightmap.png) (https://postimg.cc/image/dqseb672j/)

(https://s17.postimg.cc/94wa2u0z3/normal.png) (https://postimg.cc/image/m91ufit0r/)

(https://s17.postimg.cc/5y1qj78tb/type.png) (https://postimg.cc/image/70bx1qrmj/)
Title: Re: Terrain
Post by: Pursuivant on May 01, 2018, 12:30:23 PM
Question: Does water level work independently of terrain height?

That is, does your terrain mapping allow for water features at different elevations on the map or subterranean terrain features?

Both are problems with current IL2 maps.

Having water level values independent from terrain elevation values could also allow easy generation of areas subjected to flooding, drought, or artifical controls on water level (e.g., canal locks or reservoirs), as well as creating areas of shallow water which might interfere with ship movement or water landings by aircraft. It might even be possible to program rise or fall of water levels to simulate things like tides, currents, or drainage.

For WW2 maps, being able to easily map water levels independent of terrain would be incredibly useful for generating realistic maps of areas like Scotland, Norway, or the Normandy Coast (where tidal variations can be quite high) or China or Holland (where flooding was used to deny terrain to the enemy).
Title: Re: Terrain
Post by: Stainless on May 02, 2018, 02:33:55 AM
Water is encoded in the type data.

At the moment I am struggling to find a way to automatically generate this data, but if all four bytes of the terrain type pixel are 0, then it's water.

This can be at any elevation.

So rivers can flow down hill, terrain can be dry and below MSL, etc.
Title: Re: Terrain
Post by: Pursuivant on May 04, 2018, 06:41:02 AM
Water is encoded in the type data.

Excellent! Thank you!

At the moment I am struggling to find a way to automatically generate this data, but if all four bytes of the terrain type pixel are 0, then it's water.

Port data over from an albedo map?

e.g.,

https://modis.gsfc.nasa.gov/data/

https://modis-images.gsfc.nasa.gov/ALBEDO/index.html

There is a specific dataset of MODIS map data which is used to mask water areas.

https://modis.gsfc.nasa.gov/data/dataprod/mod44w.php

Alternately, use more a more general albedo map data set and set albedo of water (~0.06) = 0,0,0,0 on your STRM data map.

Not only would the latter option automatically give you water features, it would also give you rough ecosystem data allowing autogeneration of things like forests, ice fields, grasslands, or deserts. If someone wanted to get really fancy, it could also be used as a base for modeling things like dust storms, forest fires, or haze.
Title: Re: Terrain
Post by: Stainless on May 06, 2018, 05:00:07 AM
I have now solved one major problem, when I generate tiles I also have to generate vertex normals.

This turned out to be very complex, most techniques commonly used end up generating nasty linear features in the normal map. At first I thought it was a bug in my code, but then I tested using xnormal and found they have exactly the same problem. So they just use a blur filter to smooth things out.

I now generate a surface normal for every triangle in the height map, accumulate that on a per pixel basis.  Normalise it. Then blur the generated normal map.

Here is an example patch

Height

(https://s17.postimg.cc/4v0ygtcqn/T0000.png) (https://postimg.cc/image/6mtxbpw3f/)

Normal map

(https://s17.postimg.cc/efkl3p9sf/T0000_N.png) (https://postimg.cc/image/9gx2p65zf/)

Each one represents 1 minute of latitude / longitude. So from each HGT file you end up with 60*60 or 3,600 bitmaps.

This gives a resolution of about 2m per pixel at the equator, and about 1.7m per pixel at 45N

When I drop a couple of the generated files into my test code, you get a display like this.

(https://s17.postimg.cc/70v9auy27/Capture.png) (https://postimg.cc/image/5ltom4wyz/)

This is two generated patches displayed side by side.

Title: Re: Terrain
Post by: asheshouse on May 06, 2018, 09:26:31 AM
Given that we can obtain sea bed height data, would it be possible or practical to have tidal effects. In particular i am thinking of the changing visual scene in shallow tidal waters.
Title: Re: Terrain
Post by: SAS~Bombsaway on May 06, 2018, 10:22:35 AM
Since I love making shallows textures, you have my attention with this.
Title: Re: Terrain
Post by: Pursuivant on May 07, 2018, 09:36:12 AM
Since I love making shallows textures, you have my attention with this.

If water depth is easy to calculate, it might be possible to automatically generate shallow water textures by "shading" underwater objects and terrain based on light diffraction with depth and water turbidity.

That would have other benefits, such as allowing objects submerged in shallow water to remain partially visible (at least from the air). That would allow for realistic sub-hunting operations and would allow wrecks of sunken ships to remain visible on campaign maps.

Albedo mapping could also be used to automatically map things like snowy, muddy or burned out areas allowing for quick creation of seasonal maps or maps where defoliants or fires have altered the landscape.
Title: Re: Terrain
Post by: Stainless on May 08, 2018, 12:16:14 AM
The main problem is that SRTM data is from orbital scanning. So it "sees" water as terrain and reports the height of it.

I need to find a way of getting data for underwater features as well.

IF anyone has any ideas for that, please yell loudly.. ;) :D
Title: Re: Terrain
Post by: Pursuivant on May 08, 2018, 04:42:15 AM
The main problem is that SRTM data is from orbital scanning. So it "sees" water as terrain and reports the height of it.

I need to find a way of getting data for underwater features as well.

IF anyone has any ideas for that, please yell loudly.. ;) :D

STRM data can be used to estimate bathymetric data, albeit not with great precision for deep oceans. Apparently, estimation of depth ranges from +/-10m for shallow waters to +/-100 m for deep ocean features. That's one of the reasons why seafloor maps are more accurate for coastal areas, but practically useless for deep ocean areas (e.g., during the search for MH370).

It's been a while since I've worked with STRM data, but the data sets on this site might be what you're looking for since they integrate standard STRM data plus bathymetric survey data:

http://topex.ucsd.edu/WWW_html/srtm30_plus.html
Title: Re: Terrain
Post by: Stainless on June 10, 2018, 04:51:33 AM
Well some progress this week.

The tool generates a palette table for each 1 degree section of SRTM data. This defines the textures used to display the terrain and the water state of the patch.

So far I have 0 == sea, 1 == standard (sand,grass,rock,snow) 128 == deep ocean 129 = standard + sea

So we have support for 127 different palettes. Should be enough.

I have basic streaming in, but still needs work.

This video shows a lot of bugs I have to fix, and a lot of features missing, but it does have a day night cycle and I can recognise the terrain as being were I live.

For me it is quite depressing, So much work to do...... :(

https://youtu.be/OmlJQqDSnYo (https://youtu.be/OmlJQqDSnYo)

Title: Re: Terrain
Post by: Pursuivant on June 10, 2018, 10:17:49 PM
Was my link to bathymetric STRM data helpful?
Title: Re: Terrain
Post by: Stainless on June 11, 2018, 02:03:57 AM
I am looking at it.

The data is a lot lower resolution than the terrain data, but I think I can use it for the pallete map.

At least to identify water more accurately
Title: Re: Terrain
Post by: Stainless on June 12, 2018, 04:42:53 AM
So looking at the data in detail, the bathometric SRTM file covers 40 degrees of longitude and 50 degrees of latitude, giving me a 120 by 120 array of data per 1 degree cell.

This means I have a 2 by 2 array per 1 minute texture tile.

I think this means I can accurately detect open water and coastline.
Title: Re: Terrain
Post by: Pursuivant on June 12, 2018, 01:05:25 PM
Glad that the data sort of helped, even if it wasn't a perfect fix.

I've done a bit of further research on bathymetric data sets and data resolution levels are all over the place, with 1-2 meter resolutions for a few coastal areas to up to 30 arc seconds for more distant areas.

An alternate source of bathymetric data which might be helpful is:

https://www.bodc.ac.uk/data/hosted_data_systems/gebco_gridded_bathymetry_data/

I'll keep looking for better data sets, particularly for coastal areas, but I don't think that there will be any one data set which has everything. Detailed bathymetric maps for coastal areas might need to be a follow-up project for someone else to do, as long as you make it easy to port more detailed maps into the simulation.
Title: Re: Terrain
Post by: Stainless on October 04, 2018, 03:29:12 AM
Current state of play of the terrain system.


I have written a tool to generate the height maps from SRTM data.

SRTM data is available in 1 degree arcs, so a single file covers 1 degree of latitude and 1 degree of longitude.

I load this data and convert the values within to signed shorts (16 bits of height) which give us a height range of 65536 M. ( roughly +- 32768 ).

Then I split this down to 1 second of arc. So a single SRTM data file gives us 60 by 60 height maps. Each height map is 1025 by 1025 and stored as a binary file not a bitmap. (A 1025 by 1025 bitmap when loaded would actually occupy 2048 by 2048 pixels in video ram on some graphics cards, obviously not a good thing ).



This turned out to be very complex. I tried many different techniques and all of them produced visible artifacts. In the end I bought  a commercial tool and reverse engineered it. Now the normals are good enough. I basically generate a very simple vertex based normal per pixel and then apply a couple of layers of blurs on the top to get something that works.

The technique I originally used should have worked, it is still a puzzle to me why it did not. I generated face normals for all faces in the height map, and added this to a per-pixel accumulator. At the end I normalised these accumulators to create the normal.

Which produced horizontal lines across the terrain. Don't know why.

The generated normal map is stored as a 1024 by 1024 pixel bitmap and sampled in the shader.


Each 1 second of arc region can have 4 textures assigned to it. So I generate a type map where each byte of the pixel represents the blend factor for each of those textures.

On paper this should work well, but coming up with a good way of generating these type maps is proving difficult.

I am currently using 4 curves that overlap and calculating a set of values based on height, but that doesn't handle cliffs well. So I am going to swap to a system based on normals.

My plan is to have two regions. One below 6000 metres and one above.

Below 6000 I will blend between texture 0 and texture 1 based on height, then overwrite any pixels with a normal nearer horizontal with texture 2.

Above 6000 I will blend between 2 and 3 based on height then overwrite any pixels with a normal nearer horizontal with texture 2.


I have a streaming system in place which loads terrain patches from disk when needed. Based on a visibility check and a temporal filter.

Each 1025 by 1025 height map is scaled based on the latitude and longitude. Then the width and height are extended by 10 metres so they overlap with the next height map.

From this I generate a CLOD terrain patch. So only the visible parts are rendered and all have LODS applied.

In the shader I blend the pixel by the type map to get a base colour, then apply lighting.

At the moment the shader also clips the polygons against the ground plane not drawing anything below zero. I did this because the water shader draws a refracted version of the terrain for the region below the water line.

I am going to have to remove this as some parts of the world are dry and below zero altitude, sigh.




The water line is too blocky. I need to change the type map so anything under water has all the blend values set to zero. Then I can use this in the water shader to generate foam or waves to hide this artifact.



SRTM data is taken from satellite data which sees the sea. (Groan). So open water has a height depending on the sea state at the time the data was gathered. I cannot find any GIS data accurate enough to automatically solve this problem.

 
Sometimes the SRTM data has holes in it. I cannot come up with a good way of automatically fixing these issues. Everything I have tried looks awful.



So guys please join this thread if you have any ideas or questions or sources of GIS data on anything that can help.

At the moment I cannot see any alternative to manually editing all the maps to fix the issues. Something I was desperately trying to avoid.

Ideally I would find a way of getting satellite imagery for each height map and using that as a detail texture, but there are all sorts of problems with that. Time period. Copyright. Alignment. The list goes on.

Any help gratefully accepted.


Title: Re: Terrain
Post by: ildifa on October 04, 2018, 10:43:24 PM
Quote
    Water line : Location

SRTM data is taken from satellite data which sees the sea. (Groan). So open water has a height depending on the sea state at the time the data was gathered. I cannot find any GIS data accurate enough to automatically solve this problem.

Would the GIS data found in this website [https://www.naturalearthdata.com/downloads/10m-physical-vectors/ (https://www.naturalearthdata.com/downloads/10m-physical-vectors/)] be useful for you?
I used the vector files (this in particular: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_land.zip (https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_land.zip)) in my work, to produce a sea/land raster mask. You would lose the position of almost all lakes, which are not present in the dataset, but I guess it's better than nothing...
But maybe for the lakes you can keep the SRTM data, as I imagine the waves are much smaller than in the open sea.

Hope to be of help, and good luck in all your wonderful work!
Title: Re: Terrain
Post by: Stainless on October 05, 2018, 02:48:38 AM
I have looked at this data many times. I don't know, maybe it's just me being a perfectionist, but I cannot get on with it.

The basic problem is accuracy. I have height data at 1M , but the best they can offer me is 10M.

And it's just not accurate enough for me. I know the problem is me not the data  :-[

For example.

This is the coastline data drawn over the height data, the two don't match.


(https://i.postimg.cc/8kWjRppp/10m_overlay.png) (https://postimg.cc/xqffYSNh)

So I tried the bathymetry data

(https://i.postimg.cc/R0fjPDNH/bath.png) (https://postimg.cc/kVJ1GwNJ)

Which as you can see is just generated from the coastline data

So I tried the land data

(https://i.postimg.cc/TYpN74s3/land.png) (https://postimg.cc/rK29zQcB)

And it's just the reverse of the water data.

Sigh.

Maybe I can come up with some way of using the data, but at the moment it feels like that is a whole new project.

Title: Re: Terrain
Post by: Stainless on October 05, 2018, 03:12:42 AM
That started the little grey cells.

I wondered if I was thinking about it in the wrong way. Instead of trying to get more accurate data, maybe I could use the data I have to generate the data I want.

So I grabbed a height map in paint.net and ran an outline filter on it twice. Once with a small radius and once with a larger radius.

I then combined these to create a new image


(https://i.postimg.cc/KYYB5RQk/new_test.png) (https://postimg.cc/DStSvf9n)


That's looking a lot closer
Title: Re: Terrain
Post by: ildifa on October 06, 2018, 05:49:42 AM
Wow, that's a great idea!
I guess now you could use a "shallow sea" texture for the yellow outline, and preserve the position of beaches and cliffs. It also solves the problem of the non-zero height of the open sea, AND it's not blocky... Perfect!
Title: Re: Terrain
Post by: Stainless on October 11, 2018, 04:19:35 AM
Okay a lot of changes.

Originally I stored the height maps as 16 bit signed integers, this just didn't give enough accuracy.

So I now calculate the height range and scale the 16 bit value to fit in that range.

The height is now

Code: [Select]
float height = (sample * heightscale) + minimumheight;
This gives each height map much more resolution.


I have also realised I am texturing the terrain all wrong. Instead of using the height to texture a pixel, we are going to need to be able to apply a hand edited texture, or a satellite image.

So I have changed the code to blend between a supplied texture and a rock texture based on slope. So cliffs will be textured properly.

This gives me the problem that we need textures for ever height map.

So I have written an app to test a system to auto-generate the textures for all height maps. A default texture that can be edited, replaced, or what ever.

It starts by loading the heights and storing them in a floating point texture so all the rest of the processes can be done in shaders.


(https://i.postimg.cc/MTpW38TZ/Desktop_Screenshot_2018.10.11_-_08.12.10.82.png) (https://postimg.cc/94s5zKZs)

Then I generate the type map based on those heights.

(https://i.postimg.cc/qvc4k76M/Desktop_Screenshot_2018.10.11_-_08.11.19.37.png) (https://postimg.cc/68QDcthD)

Then I pass it through a couple of blurs.

(https://i.postimg.cc/MpzKfRSj/Desktop_Screenshot_2018.10.11_-_10.35.00.23.png) (https://postimg.cc/tYmHLs7X)

And then I use supplied textures to generate a default texture map.

(https://i.postimg.cc/15B1Tjkf/Desktop_Screenshot_2018.10.11_-_11.04.48.59.png) (https://postimg.cc/hfXYQpPB)

The next stage is to generate the surf map.


Title: Re: Terrain
Post by: Stainless on October 12, 2018, 04:14:05 AM
Doesn't look too bad as a starting point.


(https://i.postimg.cc/cLFf8Nw3/Desktop-Screenshot-2018-10-12-11-11-58-02.png) (https://postimg.cc/JsBs9vT7)


Desperately needs a better water shader and the surf line.


Title: Re: Terrain
Post by: Stainless on October 12, 2018, 09:12:23 AM
Well I can generate sea shore textures.


(https://i.postimg.cc/brZtwv9m/Desktop-Screenshot-2018-10-12-15-29-24-23.png) (https://postimg.cc/p5tT0R7j)

They seem to be in the right place.


(https://i.postimg.cc/8C3FnGSW/Desktop-Screenshot-2018-10-12-16-10-16-68.png) (https://postimg.cc/8FRPjqsP)

So now I just need to figure out how to render the foam.


Title: Re: Terrain
Post by: SAS~Ghost129er on October 13, 2018, 03:15:46 AM
Perhaps like IL2? An alpha channel animated texture that's a border frame of land textured like water risen slightly above the shore and water..?

Sorry struggling to explain what I mean...
Title: Re: Terrain
Post by: Stainless on October 13, 2018, 04:39:57 AM
There are two parts of the rendering, terrain and sea.

Both need to be changed to use the coastline data, I am going to add a time based delta to each vertex based on the coast line data. So the water goes up and down.

But I am trying to come up with a better way of doing it.

Maybe a separate mesh......
Title: Re: Terrain
Post by: Stainless on October 16, 2018, 05:54:02 AM
First pass, not nearly correct but interesting.


(https://i.postimg.cc/nrH97TZS/Desktop-Screenshot-2018-10-16-12-52-07-00.png) (https://postimg.cc/gXT0bVx3)

The waves animate towards the coast