Fastest way to convert big raster to polyline using R or Python?Automatic point plotting in ArcGIS from...
How to not let the Identify spell spoil everything?
Lightning Data Table inline edit
Does the US government have any planning in place to ensure there's no shortages of food, fuel, steel and other commodities?
Plausible reason to leave the Solar System?
Which RAF squadrons and aircraft types took part in the bombing of Berlin on the 25th of August 1940?
Charging phone battery with a lower voltage, coming from a bike charger?
What to do with threats of blacklisting?
Potential client has a problematic employee I can't work with
Crack the bank account's password!
Will rerolling initiative each round stop meta-gaming about initiative?
If angels and devils are the same species, why would their mortal offspring appear physically different?
What is a good reason for every spaceship to carry a weapon on board?
Converting very wide logos to square formats
Cat is tipping over bed-side lamps during the night
Not a Long-Winded Riddle
Time-efficient matrix elements grouping and summing
Need help with a circuit diagram where the motor does not seem to have any connection to ground. Error with diagram? Or am i missing something?
Why avoid shared user accounts?
The No-Straight Maze
Can my friend and I spend the summer in Canada (6 weeks) at 16 years old without an adult?
Why did Luke use his left hand to shoot?
Taking headphones when quitting job
How can I prevent an oracle who can see into the past from knowing everything that has happened?
Subsurf on a crown. How can I smooth some edges and keep others sharp?
Fastest way to convert big raster to polyline using R or Python?
Automatic point plotting in ArcGIS from thresholded imageHow to get X Y coordinates and cell value of each pixel in a raster using Python?How to get raster corner coordinates using Python GDAL bindings?Convert a classification image to aggregated polygon in shapefile using gdal/pythonHow to find the distance between start of line and start of raster with arcpy/python?How to improve performance when using a large amount of data (big tables) in ArcGIS?Extract raster values to multipoint shapefile and write to new field using gdal/pythonHow to speed up Select by location?Programmatic Zonal statistics to obtain cell value at 2% and 98% cumulative frequency IN GDAL OR PYTHONvectorize raster image containing linesAligning many rasters using PyQGIS or Python?
I have a big raster file (129600 by 64800 pixel) with global water bodies (1 bit values 0 and 1) and try to extract ocean and inland water shorelines.
I've tried with ArcGIS and QGIS to convert from raster to polyline, but it takes ages.
Does anybody know a better/faster way (Python or R) or a better tool for this task?
Update
- R: rasterToContour might be fast and precise but if you have a very large dataset like mine (8,398,080,000 pixels) you need either a very big amount of RAM (more than 16GB) or you force R to do more processing on the hard drive and it will also take ages.
- Python/GDAL: gdal_poligonize creates polygons instead of polylines
Update 2
- R rasterToContour: rasterToContour does not deliver the wanted results. Compared to ArcGIS (raster to polygon followed by feature to line) it does not extract the exact pixel outline, as shown in the examples below.
rasterToContour result
ArcGIS result
UPDATE 3
Python/GDAL: I've run gdal_polygonize from command line against ArcGIS on a test dataset and the results were extremely clear:
- gdal: 49 seconds
- ArcGIS: 1.84 seconds
python r performance
add a comment |
I have a big raster file (129600 by 64800 pixel) with global water bodies (1 bit values 0 and 1) and try to extract ocean and inland water shorelines.
I've tried with ArcGIS and QGIS to convert from raster to polyline, but it takes ages.
Does anybody know a better/faster way (Python or R) or a better tool for this task?
Update
- R: rasterToContour might be fast and precise but if you have a very large dataset like mine (8,398,080,000 pixels) you need either a very big amount of RAM (more than 16GB) or you force R to do more processing on the hard drive and it will also take ages.
- Python/GDAL: gdal_poligonize creates polygons instead of polylines
Update 2
- R rasterToContour: rasterToContour does not deliver the wanted results. Compared to ArcGIS (raster to polygon followed by feature to line) it does not extract the exact pixel outline, as shown in the examples below.
rasterToContour result
ArcGIS result
UPDATE 3
Python/GDAL: I've run gdal_polygonize from command line against ArcGIS on a test dataset and the results were extremely clear:
- gdal: 49 seconds
- ArcGIS: 1.84 seconds
python r performance
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35
add a comment |
I have a big raster file (129600 by 64800 pixel) with global water bodies (1 bit values 0 and 1) and try to extract ocean and inland water shorelines.
I've tried with ArcGIS and QGIS to convert from raster to polyline, but it takes ages.
Does anybody know a better/faster way (Python or R) or a better tool for this task?
Update
- R: rasterToContour might be fast and precise but if you have a very large dataset like mine (8,398,080,000 pixels) you need either a very big amount of RAM (more than 16GB) or you force R to do more processing on the hard drive and it will also take ages.
- Python/GDAL: gdal_poligonize creates polygons instead of polylines
Update 2
- R rasterToContour: rasterToContour does not deliver the wanted results. Compared to ArcGIS (raster to polygon followed by feature to line) it does not extract the exact pixel outline, as shown in the examples below.
rasterToContour result
ArcGIS result
UPDATE 3
Python/GDAL: I've run gdal_polygonize from command line against ArcGIS on a test dataset and the results were extremely clear:
- gdal: 49 seconds
- ArcGIS: 1.84 seconds
python r performance
I have a big raster file (129600 by 64800 pixel) with global water bodies (1 bit values 0 and 1) and try to extract ocean and inland water shorelines.
I've tried with ArcGIS and QGIS to convert from raster to polyline, but it takes ages.
Does anybody know a better/faster way (Python or R) or a better tool for this task?
Update
- R: rasterToContour might be fast and precise but if you have a very large dataset like mine (8,398,080,000 pixels) you need either a very big amount of RAM (more than 16GB) or you force R to do more processing on the hard drive and it will also take ages.
- Python/GDAL: gdal_poligonize creates polygons instead of polylines
Update 2
- R rasterToContour: rasterToContour does not deliver the wanted results. Compared to ArcGIS (raster to polygon followed by feature to line) it does not extract the exact pixel outline, as shown in the examples below.
rasterToContour result
ArcGIS result
UPDATE 3
Python/GDAL: I've run gdal_polygonize from command line against ArcGIS on a test dataset and the results were extremely clear:
- gdal: 49 seconds
- ArcGIS: 1.84 seconds
python r performance
python r performance
edited Oct 17 '15 at 10:50
Generic Wevers
asked Oct 16 '15 at 8:56
Generic WeversGeneric Wevers
620413
620413
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35
add a comment |
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35
add a comment |
4 Answers
4
active
oldest
votes
I'm working with R and used rasterToPolygons
from the raster
package in the past, but now I prefer gdal_polygonizeR
by John Baumgartner. It bases on gdal_polygonize.py
and is much faster.
John Baumgartner published the code and gave an example for usage in his blog.
If you are familiar with python you could use gdal_polygonize.py
directly of course.
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
add a comment |
Try rasterToContour
from the raster package.
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- rasterToContour(r)
class(x)
> [1] "SpatialLinesDataFrame"
> attr(,"package")
> [1] "sp"
plot(r)
plot(x, add=TRUE)
You may then easily write the files to a local folder, e.g. as 'ESRI Shapefile' (.shp), using the below code. Have a look at ogrDrivers
from rgdal to find out which drivers your system is compatible with.
library(rgdal)
writeOGR(x, dsn = getwd(), layer = "coastlines", driver = "ESRI Shapefile")
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
Another hint: try to set 'maxpixels' inrasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.
– fdetsch
Oct 16 '15 at 10:46
1
If you're not willing toresample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then performrasterToContour
on each tile separately in an iterative manner and, finally,merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function calledsplitRaster
to create multiple sub-rasters from one huge raster.
– fdetsch
Oct 16 '15 at 11:42
|
show 3 more comments
While I'm a big fan of GDAL, the polygonize tool was way too slow for my applications as well.
A fast alternative is gdal_trace_outline
from Dans GDAL scripts which also has more options regarding tolerance, donuts, etc.
Like gdal_polygonize
this also produces polygons which you'd need to convert afterwards with ogr2ogr -nlt MULTILINESTRING
.
Downside to that is you need to compile it yourself, unless you are on a Linux or Mac OsX System.
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
add a comment |
For posterity, I've been having success with the stars::
package in R
for doing this type of operation quickly.
library(raster)
library(stars)
library(sf)
library(magrittr)
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- st_as_stars(r) %>%
st_as_sf() %>% # this is the raster to polygons part
st_cast("MULTILINESTRING") # cast the polygons to polylines
plot(x)
plot(r)
plot(x, add = TRUE)
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "79"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f166753%2ffastest-way-to-convert-big-raster-to-polyline-using-r-or-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
I'm working with R and used rasterToPolygons
from the raster
package in the past, but now I prefer gdal_polygonizeR
by John Baumgartner. It bases on gdal_polygonize.py
and is much faster.
John Baumgartner published the code and gave an example for usage in his blog.
If you are familiar with python you could use gdal_polygonize.py
directly of course.
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
add a comment |
I'm working with R and used rasterToPolygons
from the raster
package in the past, but now I prefer gdal_polygonizeR
by John Baumgartner. It bases on gdal_polygonize.py
and is much faster.
John Baumgartner published the code and gave an example for usage in his blog.
If you are familiar with python you could use gdal_polygonize.py
directly of course.
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
add a comment |
I'm working with R and used rasterToPolygons
from the raster
package in the past, but now I prefer gdal_polygonizeR
by John Baumgartner. It bases on gdal_polygonize.py
and is much faster.
John Baumgartner published the code and gave an example for usage in his blog.
If you are familiar with python you could use gdal_polygonize.py
directly of course.
I'm working with R and used rasterToPolygons
from the raster
package in the past, but now I prefer gdal_polygonizeR
by John Baumgartner. It bases on gdal_polygonize.py
and is much faster.
John Baumgartner published the code and gave an example for usage in his blog.
If you are familiar with python you could use gdal_polygonize.py
directly of course.
answered Oct 16 '15 at 9:46
IrisIris
724613
724613
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
add a comment |
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
1
1
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I'll give it I try. Last time I've used gdal_polygonize.py ArcGIS was still faster.
– Generic Wevers
Oct 16 '15 at 10:12
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
I didn't expect that ArcGis can be faster that gdal. @Generic Militzer
– Iris
Oct 16 '15 at 10:26
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
Ah wait, this will create polygons but I need polylines.
– Generic Wevers
Oct 16 '15 at 10:28
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
If you put your data into a File Geodatabase it is pretty fast. But still not fast enough. That's why I'm searching for alternatives.
– Generic Wevers
Oct 16 '15 at 10:30
2
2
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
It's not necessarily a problem that you get polygons, you can always convert them to polylines (although, with that many it could of course take a while as well).
– Martin
Oct 16 '15 at 11:40
add a comment |
Try rasterToContour
from the raster package.
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- rasterToContour(r)
class(x)
> [1] "SpatialLinesDataFrame"
> attr(,"package")
> [1] "sp"
plot(r)
plot(x, add=TRUE)
You may then easily write the files to a local folder, e.g. as 'ESRI Shapefile' (.shp), using the below code. Have a look at ogrDrivers
from rgdal to find out which drivers your system is compatible with.
library(rgdal)
writeOGR(x, dsn = getwd(), layer = "coastlines", driver = "ESRI Shapefile")
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
Another hint: try to set 'maxpixels' inrasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.
– fdetsch
Oct 16 '15 at 10:46
1
If you're not willing toresample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then performrasterToContour
on each tile separately in an iterative manner and, finally,merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function calledsplitRaster
to create multiple sub-rasters from one huge raster.
– fdetsch
Oct 16 '15 at 11:42
|
show 3 more comments
Try rasterToContour
from the raster package.
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- rasterToContour(r)
class(x)
> [1] "SpatialLinesDataFrame"
> attr(,"package")
> [1] "sp"
plot(r)
plot(x, add=TRUE)
You may then easily write the files to a local folder, e.g. as 'ESRI Shapefile' (.shp), using the below code. Have a look at ogrDrivers
from rgdal to find out which drivers your system is compatible with.
library(rgdal)
writeOGR(x, dsn = getwd(), layer = "coastlines", driver = "ESRI Shapefile")
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
Another hint: try to set 'maxpixels' inrasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.
– fdetsch
Oct 16 '15 at 10:46
1
If you're not willing toresample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then performrasterToContour
on each tile separately in an iterative manner and, finally,merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function calledsplitRaster
to create multiple sub-rasters from one huge raster.
– fdetsch
Oct 16 '15 at 11:42
|
show 3 more comments
Try rasterToContour
from the raster package.
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- rasterToContour(r)
class(x)
> [1] "SpatialLinesDataFrame"
> attr(,"package")
> [1] "sp"
plot(r)
plot(x, add=TRUE)
You may then easily write the files to a local folder, e.g. as 'ESRI Shapefile' (.shp), using the below code. Have a look at ogrDrivers
from rgdal to find out which drivers your system is compatible with.
library(rgdal)
writeOGR(x, dsn = getwd(), layer = "coastlines", driver = "ESRI Shapefile")
Try rasterToContour
from the raster package.
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- rasterToContour(r)
class(x)
> [1] "SpatialLinesDataFrame"
> attr(,"package")
> [1] "sp"
plot(r)
plot(x, add=TRUE)
You may then easily write the files to a local folder, e.g. as 'ESRI Shapefile' (.shp), using the below code. Have a look at ogrDrivers
from rgdal to find out which drivers your system is compatible with.
library(rgdal)
writeOGR(x, dsn = getwd(), layer = "coastlines", driver = "ESRI Shapefile")
edited Oct 16 '15 at 10:24
answered Oct 16 '15 at 9:04
fdetschfdetsch
4,05521940
4,05521940
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
Another hint: try to set 'maxpixels' inrasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.
– fdetsch
Oct 16 '15 at 10:46
1
If you're not willing toresample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then performrasterToContour
on each tile separately in an iterative manner and, finally,merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function calledsplitRaster
to create multiple sub-rasters from one huge raster.
– fdetsch
Oct 16 '15 at 11:42
|
show 3 more comments
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
Another hint: try to set 'maxpixels' inrasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.
– fdetsch
Oct 16 '15 at 10:46
1
If you're not willing toresample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then performrasterToContour
on each tile separately in an iterative manner and, finally,merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function calledsplitRaster
to create multiple sub-rasters from one huge raster.
– fdetsch
Oct 16 '15 at 11:42
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
I'll try and keep fingers crossed it will not kill my RAM. Even though I have 16GB, which hopefully is enough, R sometimes is not so efficient with big raster files. But let's see.
– Generic Wevers
Oct 16 '15 at 9:06
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Conversion worked somehow, but I wasn't able to check in detail. As I am usually more into raster data processing, can you tell me how I can transfer the SpatialLineDataFrame into a shapefile or something comparable. I've googled and still struggling, as I don't know the layer name (OGRwrite).
– Generic Wevers
Oct 16 '15 at 10:10
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
Haha, I definitely see your point. See above update.
– fdetsch
Oct 16 '15 at 10:25
2
2
Another hint: try to set 'maxpixels' in
rasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.– fdetsch
Oct 16 '15 at 10:46
Another hint: try to set 'maxpixels' in
rasterToContour
to some higher value, e.g. 1e+9. You'll end up with more details then. The default setting creates quite generalized contour lines.– fdetsch
Oct 16 '15 at 10:46
1
1
If you're not willing to
resample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then perform rasterToContour
on each tile separately in an iterative manner and, finally, merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function called splitRaster
to create multiple sub-rasters from one huge raster.– fdetsch
Oct 16 '15 at 11:42
If you're not willing to
resample
your data to a coarser spatial resolution, the only solution I can imagine then would be to split your data into multiple tiles (e.g. 16 sub-rasters), then perform rasterToContour
on each tile separately in an iterative manner and, finally, merge
the resulting shapefiles into one huge shapefile. In case you are interested, our working group's package Rsenal offers a function called splitRaster
to create multiple sub-rasters from one huge raster.– fdetsch
Oct 16 '15 at 11:42
|
show 3 more comments
While I'm a big fan of GDAL, the polygonize tool was way too slow for my applications as well.
A fast alternative is gdal_trace_outline
from Dans GDAL scripts which also has more options regarding tolerance, donuts, etc.
Like gdal_polygonize
this also produces polygons which you'd need to convert afterwards with ogr2ogr -nlt MULTILINESTRING
.
Downside to that is you need to compile it yourself, unless you are on a Linux or Mac OsX System.
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
add a comment |
While I'm a big fan of GDAL, the polygonize tool was way too slow for my applications as well.
A fast alternative is gdal_trace_outline
from Dans GDAL scripts which also has more options regarding tolerance, donuts, etc.
Like gdal_polygonize
this also produces polygons which you'd need to convert afterwards with ogr2ogr -nlt MULTILINESTRING
.
Downside to that is you need to compile it yourself, unless you are on a Linux or Mac OsX System.
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
add a comment |
While I'm a big fan of GDAL, the polygonize tool was way too slow for my applications as well.
A fast alternative is gdal_trace_outline
from Dans GDAL scripts which also has more options regarding tolerance, donuts, etc.
Like gdal_polygonize
this also produces polygons which you'd need to convert afterwards with ogr2ogr -nlt MULTILINESTRING
.
Downside to that is you need to compile it yourself, unless you are on a Linux or Mac OsX System.
While I'm a big fan of GDAL, the polygonize tool was way too slow for my applications as well.
A fast alternative is gdal_trace_outline
from Dans GDAL scripts which also has more options regarding tolerance, donuts, etc.
Like gdal_polygonize
this also produces polygons which you'd need to convert afterwards with ogr2ogr -nlt MULTILINESTRING
.
Downside to that is you need to compile it yourself, unless you are on a Linux or Mac OsX System.
answered Oct 17 '15 at 11:34
KerstenKersten
7,27332446
7,27332446
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
add a comment |
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
Unfortunately it failed with the error message: "Segmentation fault (core dumped)". I am guessing my file is too big or more precise it will produce too many small polygons.
– Generic Wevers
Oct 19 '15 at 11:35
add a comment |
For posterity, I've been having success with the stars::
package in R
for doing this type of operation quickly.
library(raster)
library(stars)
library(sf)
library(magrittr)
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- st_as_stars(r) %>%
st_as_sf() %>% # this is the raster to polygons part
st_cast("MULTILINESTRING") # cast the polygons to polylines
plot(x)
plot(r)
plot(x, add = TRUE)
add a comment |
For posterity, I've been having success with the stars::
package in R
for doing this type of operation quickly.
library(raster)
library(stars)
library(sf)
library(magrittr)
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- st_as_stars(r) %>%
st_as_sf() %>% # this is the raster to polygons part
st_cast("MULTILINESTRING") # cast the polygons to polylines
plot(x)
plot(r)
plot(x, add = TRUE)
add a comment |
For posterity, I've been having success with the stars::
package in R
for doing this type of operation quickly.
library(raster)
library(stars)
library(sf)
library(magrittr)
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- st_as_stars(r) %>%
st_as_sf() %>% # this is the raster to polygons part
st_cast("MULTILINESTRING") # cast the polygons to polylines
plot(x)
plot(r)
plot(x, add = TRUE)
For posterity, I've been having success with the stars::
package in R
for doing this type of operation quickly.
library(raster)
library(stars)
library(sf)
library(magrittr)
f <- system.file("external/test.grd", package="raster")
r <- raster(f)
r[r[] < 750] <- 0
r[r[] >= 750] <- 1
x <- st_as_stars(r) %>%
st_as_sf() %>% # this is the raster to polygons part
st_cast("MULTILINESTRING") # cast the polygons to polylines
plot(x)
plot(r)
plot(x, add = TRUE)
answered 6 mins ago
mikoontzmikoontz
1662
1662
add a comment |
add a comment |
Thanks for contributing an answer to Geographic Information Systems Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f166753%2ffastest-way-to-convert-big-raster-to-polyline-using-r-or-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did that, see Update 3.
– Generic Wevers
Oct 16 '15 at 13:06
Can you provide that test dataset, so we can see if proposed alternatives are faster and/or produce the required results?
– Kersten
Oct 17 '15 at 11:35