Cannot get driver_drake() with new regions even with correct IEA file

[robbieorvis]

Hello, I followed the instructions to add a new region online (actually, I moved the UK out of the EU-15 and into its own region. I have a copy of the IEA file (which is included in the NCA submission online, FYI).

However, when I try to run driver_drake() to update data files, I keep running into an error on misalignment with the prebuilt version:

Error in verify_identical_prebuilt(L101.en_bal_EJ_R_Si_Fi_Yh_full, L101.en_bal_EJ_ctry_Si_Fi_Yh_full, :
(converted from warning) L101.en_bal_EJ_R_Si_Fi_Yh_full is not the same as its prebuilt version

The notes in the documentation say that when the model is correctly building from the IEA dataset, these warnings should not appear. Please advise.

[pkyle]

I'd recommend just re-generating the package data
source("data-raw/generate_package_data.R")
which will re-build the PREBUILT_DATA.rda, and then run it again. If that still doesn't work, I'd use driver() instead of driver_drake()

[robbieorvis]

I ran that in RStudio but it didn’t appear to update that file, unfortunately. I’ll try driver. On Apr 4, 2023, at 5:49 PM, pkyle ***@***.***> wrote:  I'd recommend just re-generating the package data source("data-raw/generate_package_data.R") which will re-build the PREBUILT_DATA.rda, and then run it again. If that still doesn't work, I'd use driver() instead of driver_drake() — Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SJK4EYOLH7HQN52P6TW7SJQFANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[robbieorvis]

I misspoke earlier. I tried running the generate_package_data.R but it returned a similar error when it started the process of extracting metadata from inputs: [1] "module_energy_LA101.en_bal_IEA" Error in verify_identical_prebuilt(L101.en_bal_EJ_R_Si_Fi_Yh_full, L101.en_bal_EJ_ctry_Si_Fi_Yh_full, : (converted from warning) L101.en_bal_EJ_R_Si_Fi_Yh_full is not the same as its prebuilt version From: pkyle ***@***.***> Sent: Tuesday, April 4, 2023 5:50 PM To: JGCRI/gcamdata ***@***.***> Cc: Robbie Orvis ***@***.***>; Author ***@***.***> Subject: Re: [JGCRI/gcamdata] Cannot get driver_drake() with new regions even with correct IEA file (Issue #1236) I'd recommend just re-generating the package data source("data-raw/generate_package_data.R") which will re-build the PREBUILT_DATA.rda, and then run it again. If that still doesn't work, I'd use driver() instead of driver_drake() - Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SJK4EYOLH7HQN52P6TW7SJQFANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[robbieorvis]

And, running driver() produces the same exact error when it gets to that module. From: pkyle ***@***.***> Sent: Tuesday, April 4, 2023 5:50 PM To: JGCRI/gcamdata ***@***.***> Cc: Robbie Orvis ***@***.***>; Author ***@***.***> Subject: Re: [JGCRI/gcamdata] Cannot get driver_drake() with new regions even with correct IEA file (Issue #1236) I'd recommend just re-generating the package data source("data-raw/generate_package_data.R") which will re-build the PREBUILT_DATA.rda, and then run it again. If that still doesn't work, I'd use driver() instead of driver_drake() - Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SJK4EYOLH7HQN52P6TW7SJQFANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[pkyle]

Oh that makes sense--this is a problem in your settings in R, where warnings are converted to errors. This is a perfectly normal warning, expected in this case, letting you know that the pre-built data is different from the data you're generating from the source data. If you just run:
options(warn = 1)
In your R session, that should make this work fine, and I think it will remain as a global default option when you close R

[robbieorvis]

Ah okay. That does appear to have stopped it from exiting. This probably also means just running driver_drake() with the updated files would work too instead of needing to re-run the generate_data_package script. From: pkyle ***@***.***> Sent: Wednesday, April 5, 2023 11:03 AM To: JGCRI/gcamdata ***@***.***> Cc: Robbie Orvis ***@***.***>; Author ***@***.***> Subject: Re: [JGCRI/gcamdata] Cannot get driver_drake() with new regions even with correct IEA file (Issue #1236) Oh that makes sense--this is a problem in your settings in R, where warnings are converted to errors. This is a perfectly normal warning, expected in this case, letting you know that the pre-built data is different from the data you're generating from the source data. If you just run: options(warn = 1) In your R session, that should make this work fine, and I think it will remain as a global default option when you close R - Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SKAPGLLPSW5YUJZNCDW7WCTVANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[robbieorvis]

Okay, that resolved the previous issue, but now it looks there is some type of data error: ✖ fail module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU Error: target module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU failed. diagnose(module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU)$error$message: left_join_no_match: NA values in new data columns diagnose(module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU)$error$calls: gcamdata:::module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU("MAKE", c(L108.ag_Feed_Mt_R_C_Y, L110.For_ALL_bm3_R_Y, L120.LC_bm2_R_LT_Yh_GLU, L121.CarbonContent_kgm2_R_LT_GLU, L121.Yield_kgm2_R_Past_GLU, L101.Pop_thous_R_Yh, L120.LC_soil_veg_carbon_GLU)) L123.ag_Prod_Mt_R_Past_Y_GLU %>% rename(Land_Type = GCAM_commodity) %>% left_join_error_no_match(L121.Yield_kgm2_R_Past_GLU, by = c("GCAM_region_ID", "Land_Type", "GLU")) %>% mutate(MgdPast = value/pasture_yield) %>% select(-value, -pasture_yield) %>% left_join_error_no_match(L123.LC_bm2_R_Past_Y_GLU, by = c("GCAM_region_ID", "Land_Type", "GLU", "year")) %>% mutate(frac = MgdPast/value) %>% replace_na(list(frac = 0)) %>% mutate(frac = replace(frac, frac > ag I am wondering if this has to do with the file L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust.csv in \inst\extdata\aglu\LDS, which looks like it has GCAM regions mapped. Thoughts? From: pkyle ***@***.***> Sent: Wednesday, April 5, 2023 11:03 AM To: JGCRI/gcamdata ***@***.***> Cc: Robbie Orvis ***@***.***>; Author ***@***.***> Subject: Re: [JGCRI/gcamdata] Cannot get driver_drake() with new regions even with correct IEA file (Issue #1236) Oh that makes sense--this is a problem in your settings in R, where warnings are converted to errors. This is a perfectly normal warning, expected in this case, letting you know that the pre-built data is different from the data you're generating from the source data. If you just run: options(warn = 1) In your R session, that should make this work fine, and I think it will remain as a global default option when you close R — Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SKAPGLLPSW5YUJZNCDW7WCTVANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[pkyle]

@kanishkan91 or @realxinzhao do you recall what the workaround is for re-building this forest file, with modified regions? I thought there might have been a constant that can be set that would rebuild it but I don't see that in there.

[robbieorvis]

Thanks in advance @pkyle @kanishkan91 and @realxinzhao. Hoping to get this sorted out so we can get the regions adjusted.

[kanishkan91]

@pkyle that fix never made it to core. The CMP that fixes this more holistically is still under review and yet to be merged. I can add a fix here if you think useful.

[pkyle]

Oh OK I see what happened. Robbie, the version of the core model that you're working from has temporarily disabled the capability to modify country-to-region mappings because of this forest file. The permanent fix is still under internal review so can't be shared. There is a temporary workaround that would allow one to re-generate that forest file after modifying country-to-region mappings. Kanishka, if there's a single commit (or two) that you can share with Robbie, you can just make a patch and post it here. Here are instructions for how to create and apply patches:

[robbieorvis]

Thanks, Page. That would be very helpful. From: pkyle ***@***.***> Sent: Thursday, April 6, 2023 3:19 PM To: JGCRI/gcamdata ***@***.***> Cc: Robbie Orvis ***@***.***>; Author ***@***.***> Subject: Re: [JGCRI/gcamdata] Cannot get driver_drake() with new regions even with correct IEA file (Issue #1236) Oh OK I see what happened. Robbie, the version of the core model that you're working from has temporarily disabled the capability to modify country-to-region mappings because of this forest file. The permanent fix is still under internal review so can't be shared. There is a temporary workaround that would allow one to re-generate that forest file after modifying country-to-region mappings. Kanishka, if there's a single commit (or two) that you can share with Robbie, you can just make a patch and post it here. Here are instructions for how to create and apply patches<https://devconnected.com/how-to-create-and-apply-git-patch-files/>: - Reply to this email directly, view it on GitHub<#1236 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK5N6SLVUKH6SUTF52F7BZTW74JJDANCNFSM6AAAAAAWTHCCFU>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[kanishkan91]

@pkyle Ok. I'l go through and make a patch.

[robbieorvis]

Thanks @kanishkan91

[realxinzhao]

A quick fix would be removing the following code in module_aglu_LB120.LC_GIS_R_LTgis_Yh_GLU
Lines 279-314

    # scale forest to avoid negative unmanaged forest area which caused issue for yield in Pakistan and African regions
    # L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust, pulled from L123.LC_bm2_R_MgdFor_Yh_GLU before managed forest scaling, was used here.
    L120.LC_bm2_R_LT_Yh_GLU %>%
      left_join(L120.LC_bm2_R_LT_Yh_GLU %>%
                  spread(Land_Type, value, fill = 0) %>%
                  left_join(L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust %>% select(-Land_Type),
			by = c("GCAM_region_ID", "GLU", "year")) %>%
                  mutate(nonForScaler =
                           if_else((Forest - MgdFor) < 0 & Forest > 0,
                                   1 + (Forest - MgdFor)/(Grassland + Shrubland + Pasture), 1),
                         ForScaler = if_else((Forest - MgdFor) < 0 & Forest > 0,  MgdFor/Forest ,1)) %>%
                  select(GCAM_region_ID, GLU, year, nonForScaler, ForScaler),
                by = c("GCAM_region_ID", "GLU", "year") ) %>%
      mutate(value = if_else(Land_Type %in% c("Grassland", "Shrubland" , "Pasture"),
                             value * nonForScaler,
                             if_else(Land_Type == "Forest", value * ForScaler, value) )) %>%
      select(-nonForScaler, -ForScaler) ->
      L120.LC_bm2_R_LT_Yh_GLU

However, forest yield in a few regions (e.g., Afirca_eastern, Pakistan, etc.) would be wrong without the adjustment (that affected the breakout). This might be okay if the study focus is not on land or forestry.
There could be a more complicated method to regenerate aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust.csv to fix forest yield. However, this will be fixed by @kanishkan91 in a better way in the next release.