Planet Modding/Real World Data: Difference between revisions

From Medieval Engineers Wiki
Jump to navigation Jump to search
(Created page with "Lots of real world data exists for the real planet Earth, including worldwide infrastructure data, elevation data, land cover (plant) data], and satellite imagery. This data is typically distributed as very large images (sometimes thousands of images) that contain georeferencing data that ties those images to points on the earth. It is possible to convert from these real world locations into Medieval Engineers' planet data files, allowing for using real world data for m...")
 
(material maps only have 3 bands)
 
(2 intermediate revisions by 2 users not shown)
Line 4: Line 4:
Medieval Engineers' planet surfaces are defined by six separate faces, each one using a specially distorted [https://proj.org/en/9.4/operations/projections/gnom.html Gnomonic] projection such that pixels are as close to a constant area as possible. This special distortion can be represented in GDAL using a [https://gdal.org/development/rfc/rfc4_geolocate.html geolocation array], and is written in code as the {{#tag:syntaxhighlight|Sandbox.Engine.Voxels.MyCubemapHelpers.UniformToProjection|inline=1}} method.
Medieval Engineers' planet surfaces are defined by six separate faces, each one using a specially distorted [https://proj.org/en/9.4/operations/projections/gnom.html Gnomonic] projection such that pixels are as close to a constant area as possible. This special distortion can be represented in GDAL using a [https://gdal.org/development/rfc/rfc4_geolocate.html geolocation array], and is written in code as the {{#tag:syntaxhighlight|Sandbox.Engine.Voxels.MyCubemapHelpers.UniformToProjection|inline=1}} method.


The arguments to [https://gdal.org/programs/gdalwarp.html gdalwarp] required to convert to/from this method vary for each face that is being converted, referencing geolocation array at [[:File:Me-geolocation-16k.png|me-geolocation-16k.png]]. These options are correct for converting from real world data to ME data. For converting ME data files to real world coordinates replace SRC with DST.
=== ME to Real World ===
The arguments to [https://gdal.org/programs/gdalwarp.html gdalwarp] required to convert to/from this method vary for each face that is being converted, and reference the geolocation array at [[:File:Me-geolocation-16k.png|me-geolocation-16k.png]]. These options are correct for converting from ME data to real world data.
{{#tag:syntaxhighlight|
{{#tag:syntaxhighlight|
REM Forward
REM Forward
-to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Left
REM Left
-to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=-90 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=-90 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Back
REM Back
-to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=-180 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=-180 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Right
REM Right
-to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=90 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=90 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Up
REM Up
-to "DST_SRS=+proj=gnom +lat_0=90 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=90 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Down
REM Down
-to "DST_SRS=+proj=gnom +lat_0=-90 +lon_0=180 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png"
-to "SRC_SRS=+proj=gnom +lat_0=-90 +lon_0=180 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
|lang=bat}}
|lang=bat}}
=== Real World to ME ===
For converting real world data to ME data conceptually the above options must just be inverted (SRC becomes DST) however GDAL does not support setting output georeferencing using transformer options. Instead the output file must be manually created, and then updated by using gdalwarp. Create template files front.vrt, left.vrt, right.vrt, back.vrt, down.vrt, and up.vrt using the following.
{{#tag:syntaxhighlight|
<<nowiki>!--</nowiki> Set these to the power of two of the output file resolution -->
<VRTDataset rasterXSize="8192" rasterYSize="8192">
  <<nowiki>!--</nowiki> band layout for elevation data -->
  <VRTRasterBand dataType="UInt16" band="1" />
 
  <<nowiki>!--</nowiki> band layout for material maps -->
  <VRTRasterBand dataType="Byte" band="1" />
  <VRTRasterBand dataType="Byte" band="2" />
  <VRTRasterBand dataType="Byte" band="3" />
  <Metadata domain="GEOLOCATION">
    <<nowiki>!--</nowiki> Set the SRS here to the corresponding SRC_SRS for the face mentioned previously -->
    <MDI key="SRS">+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000</MDI>
    <<nowiki>!--</nowiki> The path to the me-geolocation-16k.png file -->
    <MDI key="X_DATASET">me-geolocation-16k.png</MDI>
    <MDI key="Y_DATASET">me-geolocation-16k.png</MDI>
    <MDI key="X_BAND">1</MDI>
    <MDI key="Y_BAND">2</MDI>
    <MDI key="PIXEL_OFFSET">0</MDI>
    <MDI key="LINE_OFFSET">0</MDI>
    <<nowiki>!--</nowiki> Output file resolution divided by 1024 (the geolocation file resolution) -->
    <MDI key="PIXEL_STEP">8</MDI>
    <MDI key="LINE_STEP">8</MDI>
  </Metadata>
</VRTDataset>
|lang=xml}}
Convert these template files into output files using {{#tag:syntaxhighlight|gdal_translate front.vrt front.tif -co COPY_SRC_MDD=YES|inline=1}}, edit the output file using any geospatial tool, then convert the intermediate output into a game compatible file using {{#tag:syntaxhighlight|gdal_translate front.tif front.png|inline=1}}.


== Examples ==
== Examples ==
Line 24: Line 58:


=== Real World Elevation ===
=== Real World Elevation ===
This example will use the [https://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/bedrock/grid_registered/georeferenced_tiff/ ETOPO1] dataset to provide global elevation data.
This example will use the [https://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/bedrock/grid_registered/georeferenced_tiff/ ETOPO1] dataset to provide global elevation data.  Manually create the [[#Real World to ME|template files]] for elevation data.
{{#tag:syntaxhighlight|
{{#tag:syntaxhighlight|
REM Scale elevation values so the elevations are all positive, convert to float, and set the SRS to long/lat
REM Scale elevation values so the elevations are all positive, convert to float, and set the SRS to long/lat
Line 32: Line 66:
REM Disable metadata files
REM Disable metadata files
set GDAL_PAM_ENABLED=NO
set GDAL_PAM_ENABLED=NO
REM Create front.png
 
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite front.png
for %%x in (front, left, back, right, up, down) do (
REM Create left.png
echo Create %%x
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=-90 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite left.png
del %%x.tif
REM Create back.png
gdal_translate %%x.vrt %%x.tif -co COPY_SRC_MDD=YES
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=-180 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite back.png
gdalwarp -dstnodata None -r average -wt Float32 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES ETOPO1_Unsigned.vrt %%x.tif
REM Create right.png
gdal_translate %%x.tif %%x.png
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=0 +lon_0=90 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite right.png
)
REM Create up.png
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=90 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite up.png
REM Create down.png
gdalwarp -dstnodata None -r average -nomd -wt Float32 -ot UInt16 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES -te 0 0 32000 32000 -ts 8192 8192 -to "DST_SRS=+proj=gnom +lat_0=-90 +lon_0=180 +R=16000 +x_0=16000 +y_0=16000" -to "DST_GEOLOC_ARRAY=me-geolocation-16k.png" ETOPO1_Unsigned.vrt -overwrite down.png
|lang=bat}}
|lang=bat}}
[[Category:Planet]]

Latest revision as of 22:42, 23 July 2024

Lots of real world data exists for the real planet Earth, including worldwide infrastructure data, elevation data, land cover (plant) data], and satellite imagery. This data is typically distributed as very large images (sometimes thousands of images) that contain georeferencing data that ties those images to points on the earth. It is possible to convert from these real world locations into Medieval Engineers' planet data files, allowing for using real world data for modded planets without distortion.

Georeferencing

Medieval Engineers' planet surfaces are defined by six separate faces, each one using a specially distorted Gnomonic projection such that pixels are as close to a constant area as possible. This special distortion can be represented in GDAL using a geolocation array, and is written in code as the Sandbox.Engine.Voxels.MyCubemapHelpers.UniformToProjection method.

ME to Real World

The arguments to gdalwarp required to convert to/from this method vary for each face that is being converted, and reference the geolocation array at me-geolocation-16k.png. These options are correct for converting from ME data to real world data.

REM Forward
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Left
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=-90 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Back
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=-180 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Right
-to "SRC_SRS=+proj=gnom +lat_0=0 +lon_0=90 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Up
-to "SRC_SRS=+proj=gnom +lat_0=90 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"
REM Down
-to "SRC_SRS=+proj=gnom +lat_0=-90 +lon_0=180 +R=16000 +x_0=16000 +y_0=16000" -to "SRC_GEOLOC_ARRAY=me-geolocation-16k.png"

Real World to ME

For converting real world data to ME data conceptually the above options must just be inverted (SRC becomes DST) however GDAL does not support setting output georeferencing using transformer options. Instead the output file must be manually created, and then updated by using gdalwarp. Create template files front.vrt, left.vrt, right.vrt, back.vrt, down.vrt, and up.vrt using the following.

<!-- Set these to the power of two of the output file resolution -->
<VRTDataset rasterXSize="8192" rasterYSize="8192">

  <!-- band layout for elevation data -->
  <VRTRasterBand dataType="UInt16" band="1" />
  
  <!-- band layout for material maps -->
  <VRTRasterBand dataType="Byte" band="1" />
  <VRTRasterBand dataType="Byte" band="2" />
  <VRTRasterBand dataType="Byte" band="3" />

  <Metadata domain="GEOLOCATION">
    <!-- Set the SRS here to the corresponding SRC_SRS for the face mentioned previously -->
    <MDI key="SRS">+proj=gnom +lat_0=0 +lon_0=0 +R=16000 +x_0=16000 +y_0=16000</MDI>
    <!-- The path to the me-geolocation-16k.png file -->
    <MDI key="X_DATASET">me-geolocation-16k.png</MDI>
    <MDI key="Y_DATASET">me-geolocation-16k.png</MDI>
    <MDI key="X_BAND">1</MDI>
    <MDI key="Y_BAND">2</MDI>
    <MDI key="PIXEL_OFFSET">0</MDI>
    <MDI key="LINE_OFFSET">0</MDI>
    <!-- Output file resolution divided by 1024 (the geolocation file resolution) -->
    <MDI key="PIXEL_STEP">8</MDI>
    <MDI key="LINE_STEP">8</MDI>
  </Metadata>
</VRTDataset>

Convert these template files into output files using gdal_translate front.vrt front.tif -co COPY_SRC_MDD=YES, edit the output file using any geospatial tool, then convert the intermediate output into a game compatible file using gdal_translate front.tif front.png.

Examples

These examples assume you've downloaded and installed GDAL and the geolocation file.

Real World Elevation

This example will use the ETOPO1 dataset to provide global elevation data. Manually create the template files for elevation data.

REM Scale elevation values so the elevations are all positive, convert to float, and set the SRS to long/lat
gdal_translate ETOPO1_Bed_g_geotiff.tif -scale -12000 12000 0 65535 -ot Float32 -a_srs "+proj=longlat" ETOPO1_Unsigned.vrt
REM Ignore translation from Earth to 16km sphere
set PROJ_IGNORE_CELESTIAL_BODY=YES
REM Disable metadata files
set GDAL_PAM_ENABLED=NO

for %%x in (front, left, back, right, up, down) do (
	echo Create %%x
	del %%x.tif
	gdal_translate %%x.vrt %%x.tif -co COPY_SRC_MDD=YES
	gdalwarp -dstnodata None -r average -wt Float32 -multi -wo SOURCE_EXTRA=100 -wo SAMPLE_GRID=YES ETOPO1_Unsigned.vrt %%x.tif
	gdal_translate %%x.tif %%x.png
)