R MODIS MOD13Q1 - How to handle data quality issues? The Next CEO of Stack OverflowMODIS...

If I blow insulation everywhere in my attic except the door trap, will heat escape through it?

Grabbing quick drinks

What is the point of a new vote on May's deal when the indicative votes suggest she will not win?

Whats the best way to handle refactoring a big file?

Can the Reverse Gravity spell affect the Meteor Swarm spell?

Can a single photon have an energy density?

% symbol leads to superlong (forever?) compilations

How do scammers retract money, while you can’t?

What can we do to stop prior company from asking us questions?

Describing a person. What needs to be mentioned?

Would this house-rule that treats advantage as a +1 to the roll instead (and disadvantage as -1) and allows them to stack be balanced?

If the heap is initialized for security, then why is the stack uninitialized?

How to make a variable always equal to the result of some calculations?

Why doesn't a table tennis ball float on the surface? How do we calculate buoyancy here?

Where to find order of arguments for default functions

forest, changing `s sep` such that it is at each second end node larger?

Why is there a PLL in CPU?

How do spells that require an ability check vs. the caster's spell save DC work?

Science fiction (dystopian) short story set after WWIII

How to use tikz in fbox?

Horror movie/show or scene where a horse creature opens its mouth really wide and devours a man in a stables

How can I open an app using Terminal?

How to safely derail a train during transit?

What does "Its cash flow is deeply negative" mean?



R MODIS MOD13Q1 - How to handle data quality issues?



The Next CEO of Stack OverflowMODIS MOD13Q1 ndvi calculationVI Quality dataset in MODIS imageHow can I parse modis MOD13Q1 quality layers in R?Handling MOD13Q1 NDVI Product Quality Assessment (QA) flags?Clipping raster brick object in RHow to remove cloudy pixel from MODIS NDVI (MOD13Q1)Applying MODIS Pixel Reliability and Vegetation Index (VI) Quality mask for MOD13Q1 using ArcGIS Desktop?Make linear regression in multiple raster layersStack RasterDataset taking into account the dates between imagesHow to define 3 average month values in a NDVI Modis time serie












0















I'm running into some issues with 250m 16-day MOD13Q1 NDVI data using the MODIS package in R.



Looking at the following raster plot, there are certain days where the NDVI does not match the true surface characteristics. Specifically ndvi.2017.11.01 and ndvi.2018.01.17. Even ndvi.2018.03.06 and ndvi.2018.02.18 are questionable. Is it possible to programmatically exclude these layers? Is there something to gain by using the following sds layer: MODIS_Grid_16DAY_250m_500m_VI:250m 16 days VI Quality?



enter image description here



Below is the code to produce the above plot:



library(raster)
library(MODIS)
library(parallel)
library(pbapply)
library(magrittr)
library(dplyr)
library(stringr)

# Final Study Area Boundary
study_area <- readRDS(gzcon(url('http://web.pdx.edu/~porlando/study_area.RDS')))
# Study CRS
epsg_26910 <- "+proj=utm +zone=10 +ellps=GRS80 +datum=NAD83 +units=m +no_defs "

# Download Data
runGdal(product = "MOD13Q1", collection = "006"
,tileH = 9, tileV = 4
,begin = str_replace_all("2017-09-01", "-", ".")
,end = str_replace_all("2018-09-30", "-", ".")
,overwrite = TRUE)


# Process data
ndvi_path <- "./MODIS/MOD13Q1.006" # 250 m

ndvi_files <- list.files(path = ndvi_path, pattern = "\.hdf$"
,all.files = FALSE, full.names = TRUE
,recursive = TRUE
,ignore.case = FALSE)


processNDVI <- function(file_path, study_area = study_area, proj = epsg_26910) {

cat("n")

# extract date from file path
date <- stringr::str_extract(file_path, '[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]')
date_clean <- stringr::str_replace_all(date, "\.", "-")

# coerce 16 day average to daily composite?
dates <- seq.Date(from = as.Date(date_clean)
#,to = as.Date(date_clean) + lubridate::days(15)
,to = as.Date(date_clean)
,by = "1 day")

sds <- get_subdatasets(file_path)

ndvi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days NDVI", sds)] %>% readGDAL %>% raster
evi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days EVI", sds)] %>% readGDAL %>% raster
# VI Quality may be useful?
vi_quality <- sds[grepl("16DAY_250m_500m_VI:250m 16 days VI Quality", sds)] %>% readGDAL %>% raster

# combine NDVI and EVI into a single stack
r <- stack(ndvi, evi)
names(r) <- c(paste0("ndvi.", date), paste0("evi.", date))

r <- raster::projectRaster(from = r, res = 250, method = 'ngb', crs = proj)
study_area <- spTransform(study_area, CRSobj = proj)

if(identical(crs(study_area), crs(r))) {

m <- mask(r, study_area)
cr <- crop(m, study_area)

pblapply(dates, function(x) {
pblapply(1:nlayers(cr), function(y) {
layer_name <- names(cr[[y]])
var <- gsub("\..*$", "", layer_name)
date <- gsub("-", "\.", x)
writeRaster(cr[[y]]
,filename = paste0("./output/", var, "/", var, ".", date, ".tif")
,format = "GTiff"
,overwrite = TRUE)
})
})
}

pblapply(ndvi_files, function(x) {
processNDVI(file_path = x
,study_area = study_area
)
}
,cl = detectCores()-1
)

ndvi <- stack(list.files(path = "./output/ndvi/", pattern = ".tif$", full.names = T))
plot(ndvi)









share|improve this question





























    0















    I'm running into some issues with 250m 16-day MOD13Q1 NDVI data using the MODIS package in R.



    Looking at the following raster plot, there are certain days where the NDVI does not match the true surface characteristics. Specifically ndvi.2017.11.01 and ndvi.2018.01.17. Even ndvi.2018.03.06 and ndvi.2018.02.18 are questionable. Is it possible to programmatically exclude these layers? Is there something to gain by using the following sds layer: MODIS_Grid_16DAY_250m_500m_VI:250m 16 days VI Quality?



    enter image description here



    Below is the code to produce the above plot:



    library(raster)
    library(MODIS)
    library(parallel)
    library(pbapply)
    library(magrittr)
    library(dplyr)
    library(stringr)

    # Final Study Area Boundary
    study_area <- readRDS(gzcon(url('http://web.pdx.edu/~porlando/study_area.RDS')))
    # Study CRS
    epsg_26910 <- "+proj=utm +zone=10 +ellps=GRS80 +datum=NAD83 +units=m +no_defs "

    # Download Data
    runGdal(product = "MOD13Q1", collection = "006"
    ,tileH = 9, tileV = 4
    ,begin = str_replace_all("2017-09-01", "-", ".")
    ,end = str_replace_all("2018-09-30", "-", ".")
    ,overwrite = TRUE)


    # Process data
    ndvi_path <- "./MODIS/MOD13Q1.006" # 250 m

    ndvi_files <- list.files(path = ndvi_path, pattern = "\.hdf$"
    ,all.files = FALSE, full.names = TRUE
    ,recursive = TRUE
    ,ignore.case = FALSE)


    processNDVI <- function(file_path, study_area = study_area, proj = epsg_26910) {

    cat("n")

    # extract date from file path
    date <- stringr::str_extract(file_path, '[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]')
    date_clean <- stringr::str_replace_all(date, "\.", "-")

    # coerce 16 day average to daily composite?
    dates <- seq.Date(from = as.Date(date_clean)
    #,to = as.Date(date_clean) + lubridate::days(15)
    ,to = as.Date(date_clean)
    ,by = "1 day")

    sds <- get_subdatasets(file_path)

    ndvi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days NDVI", sds)] %>% readGDAL %>% raster
    evi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days EVI", sds)] %>% readGDAL %>% raster
    # VI Quality may be useful?
    vi_quality <- sds[grepl("16DAY_250m_500m_VI:250m 16 days VI Quality", sds)] %>% readGDAL %>% raster

    # combine NDVI and EVI into a single stack
    r <- stack(ndvi, evi)
    names(r) <- c(paste0("ndvi.", date), paste0("evi.", date))

    r <- raster::projectRaster(from = r, res = 250, method = 'ngb', crs = proj)
    study_area <- spTransform(study_area, CRSobj = proj)

    if(identical(crs(study_area), crs(r))) {

    m <- mask(r, study_area)
    cr <- crop(m, study_area)

    pblapply(dates, function(x) {
    pblapply(1:nlayers(cr), function(y) {
    layer_name <- names(cr[[y]])
    var <- gsub("\..*$", "", layer_name)
    date <- gsub("-", "\.", x)
    writeRaster(cr[[y]]
    ,filename = paste0("./output/", var, "/", var, ".", date, ".tif")
    ,format = "GTiff"
    ,overwrite = TRUE)
    })
    })
    }

    pblapply(ndvi_files, function(x) {
    processNDVI(file_path = x
    ,study_area = study_area
    )
    }
    ,cl = detectCores()-1
    )

    ndvi <- stack(list.files(path = "./output/ndvi/", pattern = ".tif$", full.names = T))
    plot(ndvi)









    share|improve this question



























      0












      0








      0








      I'm running into some issues with 250m 16-day MOD13Q1 NDVI data using the MODIS package in R.



      Looking at the following raster plot, there are certain days where the NDVI does not match the true surface characteristics. Specifically ndvi.2017.11.01 and ndvi.2018.01.17. Even ndvi.2018.03.06 and ndvi.2018.02.18 are questionable. Is it possible to programmatically exclude these layers? Is there something to gain by using the following sds layer: MODIS_Grid_16DAY_250m_500m_VI:250m 16 days VI Quality?



      enter image description here



      Below is the code to produce the above plot:



      library(raster)
      library(MODIS)
      library(parallel)
      library(pbapply)
      library(magrittr)
      library(dplyr)
      library(stringr)

      # Final Study Area Boundary
      study_area <- readRDS(gzcon(url('http://web.pdx.edu/~porlando/study_area.RDS')))
      # Study CRS
      epsg_26910 <- "+proj=utm +zone=10 +ellps=GRS80 +datum=NAD83 +units=m +no_defs "

      # Download Data
      runGdal(product = "MOD13Q1", collection = "006"
      ,tileH = 9, tileV = 4
      ,begin = str_replace_all("2017-09-01", "-", ".")
      ,end = str_replace_all("2018-09-30", "-", ".")
      ,overwrite = TRUE)


      # Process data
      ndvi_path <- "./MODIS/MOD13Q1.006" # 250 m

      ndvi_files <- list.files(path = ndvi_path, pattern = "\.hdf$"
      ,all.files = FALSE, full.names = TRUE
      ,recursive = TRUE
      ,ignore.case = FALSE)


      processNDVI <- function(file_path, study_area = study_area, proj = epsg_26910) {

      cat("n")

      # extract date from file path
      date <- stringr::str_extract(file_path, '[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]')
      date_clean <- stringr::str_replace_all(date, "\.", "-")

      # coerce 16 day average to daily composite?
      dates <- seq.Date(from = as.Date(date_clean)
      #,to = as.Date(date_clean) + lubridate::days(15)
      ,to = as.Date(date_clean)
      ,by = "1 day")

      sds <- get_subdatasets(file_path)

      ndvi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days NDVI", sds)] %>% readGDAL %>% raster
      evi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days EVI", sds)] %>% readGDAL %>% raster
      # VI Quality may be useful?
      vi_quality <- sds[grepl("16DAY_250m_500m_VI:250m 16 days VI Quality", sds)] %>% readGDAL %>% raster

      # combine NDVI and EVI into a single stack
      r <- stack(ndvi, evi)
      names(r) <- c(paste0("ndvi.", date), paste0("evi.", date))

      r <- raster::projectRaster(from = r, res = 250, method = 'ngb', crs = proj)
      study_area <- spTransform(study_area, CRSobj = proj)

      if(identical(crs(study_area), crs(r))) {

      m <- mask(r, study_area)
      cr <- crop(m, study_area)

      pblapply(dates, function(x) {
      pblapply(1:nlayers(cr), function(y) {
      layer_name <- names(cr[[y]])
      var <- gsub("\..*$", "", layer_name)
      date <- gsub("-", "\.", x)
      writeRaster(cr[[y]]
      ,filename = paste0("./output/", var, "/", var, ".", date, ".tif")
      ,format = "GTiff"
      ,overwrite = TRUE)
      })
      })
      }

      pblapply(ndvi_files, function(x) {
      processNDVI(file_path = x
      ,study_area = study_area
      )
      }
      ,cl = detectCores()-1
      )

      ndvi <- stack(list.files(path = "./output/ndvi/", pattern = ".tif$", full.names = T))
      plot(ndvi)









      share|improve this question
















      I'm running into some issues with 250m 16-day MOD13Q1 NDVI data using the MODIS package in R.



      Looking at the following raster plot, there are certain days where the NDVI does not match the true surface characteristics. Specifically ndvi.2017.11.01 and ndvi.2018.01.17. Even ndvi.2018.03.06 and ndvi.2018.02.18 are questionable. Is it possible to programmatically exclude these layers? Is there something to gain by using the following sds layer: MODIS_Grid_16DAY_250m_500m_VI:250m 16 days VI Quality?



      enter image description here



      Below is the code to produce the above plot:



      library(raster)
      library(MODIS)
      library(parallel)
      library(pbapply)
      library(magrittr)
      library(dplyr)
      library(stringr)

      # Final Study Area Boundary
      study_area <- readRDS(gzcon(url('http://web.pdx.edu/~porlando/study_area.RDS')))
      # Study CRS
      epsg_26910 <- "+proj=utm +zone=10 +ellps=GRS80 +datum=NAD83 +units=m +no_defs "

      # Download Data
      runGdal(product = "MOD13Q1", collection = "006"
      ,tileH = 9, tileV = 4
      ,begin = str_replace_all("2017-09-01", "-", ".")
      ,end = str_replace_all("2018-09-30", "-", ".")
      ,overwrite = TRUE)


      # Process data
      ndvi_path <- "./MODIS/MOD13Q1.006" # 250 m

      ndvi_files <- list.files(path = ndvi_path, pattern = "\.hdf$"
      ,all.files = FALSE, full.names = TRUE
      ,recursive = TRUE
      ,ignore.case = FALSE)


      processNDVI <- function(file_path, study_area = study_area, proj = epsg_26910) {

      cat("n")

      # extract date from file path
      date <- stringr::str_extract(file_path, '[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]')
      date_clean <- stringr::str_replace_all(date, "\.", "-")

      # coerce 16 day average to daily composite?
      dates <- seq.Date(from = as.Date(date_clean)
      #,to = as.Date(date_clean) + lubridate::days(15)
      ,to = as.Date(date_clean)
      ,by = "1 day")

      sds <- get_subdatasets(file_path)

      ndvi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days NDVI", sds)] %>% readGDAL %>% raster
      evi <- sds[grepl("16DAY_250m_500m_VI:250m 16 days EVI", sds)] %>% readGDAL %>% raster
      # VI Quality may be useful?
      vi_quality <- sds[grepl("16DAY_250m_500m_VI:250m 16 days VI Quality", sds)] %>% readGDAL %>% raster

      # combine NDVI and EVI into a single stack
      r <- stack(ndvi, evi)
      names(r) <- c(paste0("ndvi.", date), paste0("evi.", date))

      r <- raster::projectRaster(from = r, res = 250, method = 'ngb', crs = proj)
      study_area <- spTransform(study_area, CRSobj = proj)

      if(identical(crs(study_area), crs(r))) {

      m <- mask(r, study_area)
      cr <- crop(m, study_area)

      pblapply(dates, function(x) {
      pblapply(1:nlayers(cr), function(y) {
      layer_name <- names(cr[[y]])
      var <- gsub("\..*$", "", layer_name)
      date <- gsub("-", "\.", x)
      writeRaster(cr[[y]]
      ,filename = paste0("./output/", var, "/", var, ".", date, ".tif")
      ,format = "GTiff"
      ,overwrite = TRUE)
      })
      })
      }

      pblapply(ndvi_files, function(x) {
      processNDVI(file_path = x
      ,study_area = study_area
      )
      }
      ,cl = detectCores()-1
      )

      ndvi <- stack(list.files(path = "./output/ndvi/", pattern = ".tif$", full.names = T))
      plot(ndvi)






      r modis ndvi






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 5 mins ago







      spacedSparking

















      asked 15 mins ago









      spacedSparkingspacedSparking

      1658




      1658






















          0






          active

          oldest

          votes












          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "79"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f317090%2fr-modis-mod13q1-how-to-handle-data-quality-issues%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Geographic Information Systems Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f317090%2fr-modis-mod13q1-how-to-handle-data-quality-issues%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Щит и меч (фильм) Содержание Названия серий | Сюжет |...

          is 'sed' thread safeWhat should someone know about using Python scripts in the shell?Nexenta bash script uses...

          Meter-Bus Содержание Параметры шины | Стандартизация |...