Vjylku

Map material from china not allowed to leave the country

What is the best way to deal with NPC-NPC combat?

What's the difference between using dependency injection with a container and using a service locator?

Why isn't everyone flabbergasted about Bran's "gift"?

Is this homebrew racial feat, Stonehide, balanced?

"Rubric" as meaning "signature" or "personal mark" -- is this accepted usage?

How to keep bees out of canned beverages?

c++ diamond problem - How to call base method only once

What do you call the part of a novel that is not dialog?

My bank got bought out, am I now going to have to start filing tax returns in a different state?

Check if a string is entirely made of the same substring

I preordered a game on my Xbox while on the home screen of my friend's account. Which of us owns the game?

Do I need to protect SFP ports and optics from dust/contaminants? If so, how?

What is the term for a person whose job is to place products on shelves in stores?

Arriving in Atlanta after US Preclearance in Dublin. Will I go through TSA security in Atlanta to transfer to a connecting flight?

Israeli soda type drink

What to do with someone that cheated their way through university and a PhD program?

Multiple fireplaces in an apartment building?

Protagonist's race is hidden - should I reveal it?

Passing args from the bash script to the function in the script

Can I criticise the more senior developers around me for not writing clean code?

A strange hotel

Expansion//Explosion and Siren Stormtamer

Would reducing the reference voltage of an ADC have any effect on accuracy?

Read files with Python GDAL using VSIGS

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

1

This question is somewhat similar to this one: How to efficiently access files with GDAL from an S3 bucket using VSIS3?, except I am trying to access bucket files from Google Cloud Storage, with Python. I am using GDAL 2.3.1, so I should be able to use the VSIGS virtual driver.

According the the example I came across, it looks like this simple piece of code should work:

from osgeo import gdal



ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work

ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

But I keep getting a "file not found" error, so it looks like GDAL does not understand that I'm trying to open a GCS file. What am I missing?

Notes:

• 
  gcloud is properly installed and configured on my computer (the
  command gsutil ls gs://earthengine-public/ works properly).


• I'm working inside a Python virtual environment, so that might be the
  issue there.

python gdal google-cloud

share|improve this question

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can find Python examples from GDAL autotests github.com/OSGeo/gdal/blob/master/autotest/gcore/vsigs.py.
  
  – user30184
  Aug 5 '18 at 19:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @user30184 I tried these examples but it didn't work for me. I really think it has something to do with the virtual environment. Anyway, I found a workaround using rasterio instead of GDAL, so my question is obsolete now.
  
  – ArMoraer
  Aug 7 '18 at 12:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you share a lead on or example of what you found on vsigs with rasterio? Looks like it is an active issue and area of discussion.
  
  – metasequoia
  Aug 10 '18 at 1:20
  

  
  
  
  

add a comment | 

1

1

This question is somewhat similar to this one: How to efficiently access files with GDAL from an S3 bucket using VSIS3?, except I am trying to access bucket files from Google Cloud Storage, with Python. I am using GDAL 2.3.1, so I should be able to use the VSIGS virtual driver.

According the the example I came across, it looks like this simple piece of code should work:

from osgeo import gdal



ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work

ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

But I keep getting a "file not found" error, so it looks like GDAL does not understand that I'm trying to open a GCS file. What am I missing?

Notes:

• 
  gcloud is properly installed and configured on my computer (the
  command gsutil ls gs://earthengine-public/ works properly).


• I'm working inside a Python virtual environment, so that might be the
  issue there.

python gdal google-cloud

share|improve this question

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can find Python examples from GDAL autotests github.com/OSGeo/gdal/blob/master/autotest/gcore/vsigs.py.
  
  – user30184
  Aug 5 '18 at 19:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @user30184 I tried these examples but it didn't work for me. I really think it has something to do with the virtual environment. Anyway, I found a workaround using rasterio instead of GDAL, so my question is obsolete now.
  
  – ArMoraer
  Aug 7 '18 at 12:00
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you share a lead on or example of what you found on vsigs with rasterio? Looks like it is an active issue and area of discussion.
  
  – metasequoia
  Aug 10 '18 at 1:20
  

  
  
  
  

add a comment | 

This question is somewhat similar to this one: How to efficiently access files with GDAL from an S3 bucket using VSIS3?, except I am trying to access bucket files from Google Cloud Storage, with Python. I am using GDAL 2.3.1, so I should be able to use the VSIGS virtual driver.

According the the example I came across, it looks like this simple piece of code should work:

from osgeo import gdal



ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work

ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

But I keep getting a "file not found" error, so it looks like GDAL does not understand that I'm trying to open a GCS file. What am I missing?

Notes:

• 
  gcloud is properly installed and configured on my computer (the
  command gsutil ls gs://earthengine-public/ works properly).


• I'm working inside a Python virtual environment, so that might be the
  issue there.

python gdal google-cloud

share|improve this question

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

from osgeo import gdal



ds = gdal.Open('/vsigs/my_bucket/image.tif') # doesn't work

ds = gdal.Open('gs://my_bucket/image.tif') # doesn't work either

share|improve this question

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

share|improve this question

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

asked Aug 5 '18 at 19:24

ArMoraerArMoraer

4,70121640

2 Answers
2

active

oldest

votes

1

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio



gs_access_key= "..."

gs_secret_access_key = "..."

url = 'bucket/image.tif'



with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):

    with rasterio.open('/vsigs/{}'.format(url)) as src:

        print(src.width, src.height)

share|improve this answer

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks, that helped me out when looking for a solution to read tifs in google cloud storage from compute nodes (without data duplication).
  
  – metasequoia
  Dec 4 '18 at 0:17
  

  
  
  
  

add a comment | 

0

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio

import os



#can also set these as a normal env var outside of python

os.environ['GS_SECRET_ACCESS_KEY'] = ''

os.environ['GS_ACCESS_KEY_ID'] = ''



#url should look like gs://...

with rasterio.open(url) as src:

    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

share

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "79"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f291791%2fread-files-with-python-gdal-using-vsigs%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio



gs_access_key= "..."

gs_secret_access_key = "..."

url = 'bucket/image.tif'



with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):

    with rasterio.open('/vsigs/{}'.format(url)) as src:

        print(src.width, src.height)

share|improve this answer

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks, that helped me out when looking for a solution to read tifs in google cloud storage from compute nodes (without data duplication).
  
  – metasequoia
  Dec 4 '18 at 0:17
  

  
  
  
  

add a comment | 

1

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio



gs_access_key= "..."

gs_secret_access_key = "..."

url = 'bucket/image.tif'



with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):

    with rasterio.open('/vsigs/{}'.format(url)) as src:

        print(src.width, src.height)

share|improve this answer

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks, that helped me out when looking for a solution to read tifs in google cloud storage from compute nodes (without data duplication).
  
  – metasequoia
  Dec 4 '18 at 0:17
  

  
  
  
  

add a comment | 

Following @metasequoia's suggestion, I'm posting the workaround I found for this issue. This doesn't actually answer the question since it relies on rasterio instead of GDAL, but it might help other people.

This minimal example worked for me (with rasterio 1.0.1):

import rasterio



gs_access_key= "..."

gs_secret_access_key = "..."

url = 'bucket/image.tif'



with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):

    with rasterio.open('/vsigs/{}'.format(url)) as src:

        print(src.width, src.height)

share|improve this answer

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

import rasterio



gs_access_key= "..."

gs_secret_access_key = "..."

url = 'bucket/image.tif'



with rasterio.Env(GS_SECRET_ACCESS_KEY=gs_secret_access_key, GS_ACCESS_KEY_ID=gs_access_key):

    with rasterio.open('/vsigs/{}'.format(url)) as src:

        print(src.width, src.height)

share|improve this answer

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

answered Aug 10 '18 at 6:47

ArMoraerArMoraer

4,70121640

0

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio

import os



#can also set these as a normal env var outside of python

os.environ['GS_SECRET_ACCESS_KEY'] = ''

os.environ['GS_ACCESS_KEY_ID'] = ''



#url should look like gs://...

with rasterio.open(url) as src:

    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

share

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

0

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio

import os



#can also set these as a normal env var outside of python

os.environ['GS_SECRET_ACCESS_KEY'] = ''

os.environ['GS_ACCESS_KEY_ID'] = ''



#url should look like gs://...

with rasterio.open(url) as src:

    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

share

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

Rasterio added support for gcs urls in version 1.0.15. If you're working with a recent version you can now use the following:

import rasterio

import os



#can also set these as a normal env var outside of python

os.environ['GS_SECRET_ACCESS_KEY'] = ''

os.environ['GS_ACCESS_KEY_ID'] = ''



#url should look like gs://...

with rasterio.open(url) as src:

    print(src.width, src.height)

This is convenient, as you no longer need to wrap rasterio requests in the env.

You can also authenticate through other gcs methods (e.g. set GOOGLE_APPLICATION_CREDENTIALS variable to service account credentials json), however, these methods currently don't work unless you've installed rasterio from a source distribution (pip install rasterio --no-binary rasterio) due to gdal version incompatibilities.

share

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

import rasterio

import os



#can also set these as a normal env var outside of python

os.environ['GS_SECRET_ACCESS_KEY'] = ''

os.environ['GS_ACCESS_KEY_ID'] = ''



#url should look like gs://...

with rasterio.open(url) as src:

    print(src.width, src.height)

share

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 9 mins ago

mmidzikmmidzik

11

New contributor

mmidzik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 9 mins ago

mmidzikmmidzik

11