| Title: | Data from Japan Meteorological Agency |
|---|---|
| Description: | Includes climate data from Japan Meteorological Agency ('JMA') <https://www.jma.go.jp/jma/indexe.html>. Can download climate data from 'JMA'. |
| Authors: | Toshikazu Matsumura [aut, cre] |
| Maintainer: | Toshikazu Matsumura <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.5.3.9000 |
| Built: | 2026-09-26 06:51:30 UTC |
| Source: | https://github.com/matutosi/clidatajp |
Wrapper function to convert into numeric without warnings
as_numeric_without_warnings(x)as_numeric_without_warnings(x)
x |
A string. |
A numeric or NA.
Helper function for download_climate().
clean_station(station)clean_station(station)
station |
A String of station information. |
A tibble including station information.
data(station_links) station_links %>% head(1) %>% `$`("station") %>% stringi::stri_unescape_unicode() %>% clean_station()data(station_links) station_links %>% head(1) %>% `$`("station") %>% stringi::stri_unescape_unicode() %>% clean_station()
Climate data downloaded from Japan Meteorological Agency web pages. URLs of each station are listed in data(station_links). https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/
climate_jp japan_climateclimate_jp japan_climate
A data frame with 3768 (157 stations * 12 months * 2 periods) rows and 14 variable:
Station no
Month
Period of observations
Mean temperature
Mean precipitation
Mean snowfall
Mean insolation
Station name. To avoid duplication, including country name after station name. Can split by "_". Escaped by stringi::stri_escape_unicode().
Country name. Escaped by stringi::stri_escape_unicode().
Latitude. (degree)
North or South.
Longitude. (degree)
West or East.
Altitude (m)
An object of class tbl_df (inherits from tbl, data.frame) with 3768 rows and 14 columns.
library(magrittr) library(stringi) library(dplyr) data(japan_climate) japan_climate %>% dplyr::mutate_all(stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(japan_climate) japan_climate %>% dplyr::mutate_all(stringi::stri_unescape_unicode)
Monthly normals downloaded from Japan Meteorological Agency web pages for all stations in data(station_jp_full). Values are raw strings of the web pages, because they include marks of 'JMA' ("@", ")", "]" and "#" show quality of the value, "///" and "—" show no value). Use clean_detail_value() to remove the marks, and to_numeric_when_all() or as_numeric_without_warnings() to convert. download_detail() does both for newly downloaded data. Stations of "AMeDAS" have no value in 11 of the columns, because only 10 items are observed. https://www.data.jma.go.jp/obd/stats/etrn/index.php
climate_jp_fullclimate_jp_full
A data frame with 21697 rows (1669 stations * 13 rows) and 21 variables. All variables are character.
Month (1 to 12) or yearly value. Escaped by stringi::stri_escape_unicode().
Mean air pressure on land (hPa)
Mean air pressure on sea level (hPa)
Mean precipitation (mm)
Mean temperature (degree Celsius)
Mean of daily maximum temperature (degree Celsius)
Mean of daily minimum temperature (degree Celsius)
Mean steam pressure (hPa)
Mean relative humidity (percent)
Mean wind speed (m/s)
Most frequent wind direction. Escaped by stringi::stri_escape_unicode().
Mean sunshine duration (hour)
Mean global solar radiation (MJ/m^2)
Mean snow fall (cm)
Mean of daily maximum snow fall (cm)
Mean of deepest snow (cm)
Mean cloud cover
Mean days of snow
Mean days of fog
Mean days of thunderstorm
Station no. Can be joined to station_no of data(station_jp_full). Escaped by stringi::stri_escape_unicode().
library(magrittr) library(stringi) library(dplyr) data(climate_jp_full) climate_jp_full %>% head_3() %>% dplyr::mutate_if(is.character, stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(climate_jp_full) climate_jp_full %>% head_3() %>% dplyr::mutate_if(is.character, stringi::stri_unescape_unicode)
Climate data downloaded from Japan Meteorological Agency web pages. URLs of each station are listed in data(station_links). https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/
climate_world world_climateclimate_world world_climate
A data frame with 41328 (3444 stations * 12 months) rows and 12 variable:
Station no
Continent. Escaped by stringi::stri_escape_unicode().
Country name. Escaped by stringi::stri_escape_unicode().
Station name. To avoid duplication, including country name after station name. Can split by "_". Escaped by stringi::stri_escape_unicode().
Month
Mean temperature
Mean precipitation
Latitude. (degree)
North or South.
Longitude. (degree)
West or East.
Altitude (m)
An object of class tbl_df (inherits from tbl, data.frame) with 41328 rows and 12 columns.
library(magrittr) library(stringi) library(dplyr) data(world_climate) world_climate %>% dplyr::mutate_all(stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(world_climate) world_climate %>% dplyr::mutate_all(stringi::stri_unescape_unicode)
detail_colnames() joins header rows into column names, clean_detail_value() removes marks of 'JMA' from values, and to_numeric_when_all() converts a vector into numeric only when all values can be converted.
detail_colnames(header) clean_detail_value(x) to_numeric_when_all(x)detail_colnames(header) clean_detail_value(x) to_numeric_when_all(x)
header |
A dataframe of header rows. |
x |
A string vector of values. |
detail_colnames(): a string vector of column names. clean_detail_value(): a string vector. to_numeric_when_all(): a numeric vector or a string vector.
header <- data.frame(x = c("temperature", "temperature"), y = c("temperature", "mean")) detail_colnames(header) clean_detail_value(c("21.6", "0.0 )", "--", "///")) to_numeric_when_all(c("21.6", "0.0", NA)) to_numeric_when_all(c("21.6", "north"))header <- data.frame(x = c("temperature", "temperature"), y = c("temperature", "mean")) detail_colnames(header) clean_detail_value(c("21.6", "0.0 )", "--", "///")) to_numeric_when_all(c("21.6", "0.0", NA)) to_numeric_when_all(c("21.6", "north"))
Japan Meteorological Agency ('JMA') provides detail climate data (yearly, monthly, daily, hourly and 10 minutes values) for each station. detail_url() builds urls of the data, and download_detail() downloads the data of the url. You can see web page as below. https://www.data.jma.go.jp/obd/stats/etrn/index.php
detail_url( prec_no, block_no, item = "daily", year = NULL, month = NULL, day = NULL ) detail_item(item, block_no)detail_url( prec_no, block_no, item = "daily", year = NULL, month = NULL, day = NULL ) detail_item(item, block_no)
prec_no |
A string or numeric of area no. |
block_no |
A string or numeric of station no. Keep leading zero of 'AMeDAS' as a string (ex. "0588"). |
item |
A string to specify data type. See details. |
year |
A numeric or string of year. |
month |
A numeric or string of month. |
day |
A numeric or string of day. |
A station is specified by a pair of prec_no (area) and block_no (station). Use download_prec_no() and download_block_no() to get the numbers. Note that block_no of "kansho" (weather station) has 5 digits (ex. "47759") and that of "AMeDAS" has 4 digits (ex. "0588"). Suffix of item ("_s" or "_a") is set to match the type of block_no.
item can be a short name in the following list, or a name of php file without extension (ex. "daily_s1").
Yearly values.
Monthly values in a year (year is required).
Monthly values of all years ("kansho" only).
Daily values in a month (year and month are required).
Hourly values in a day (year, month and day are required).
10 minutes values in a day (year, month and day are required).
Normals of each month.
Normals of each day in a month (month is required).
Normals of first and last day of frost, snow and ice ("kansho" only).
A string vector of urls.
detail_url(61, 47759, "daily", 2023, 6) detail_url(61, "0588", "hourly", 2023, 6, 20) detail_url(61, 47759, "daily", 2023, 1:3)detail_url(61, 47759, "daily", 2023, 6) detail_url(61, "0588", "hourly", 2023, 6, 20) detail_url(61, 47759, "daily", 2023, 1:3)
For polite scraping, 5 sec interval is set in download_climate(), it takes over 5 hours to get climate data of all stations. Please use existing links by "data(climate_world)", if you do not need to renew climate data. You can see web page as below. https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/
download_climate(url)download_climate(url)
url |
A String to specify target html. |
A tibble including climate and station information, or NULL when failed.
# If you want all climate data, remove head(). # The codes take > 5 sec because of poliste scraping. library(magrittr) library(stringi) library(dplyr) data(station_links) station_links <- station_links %>% dplyr::mutate_all(stringi::stri_unescape_unicode) %>% head(3) %T>% { continent <<- `$`(., "continent") no <<- `$`(., "no") } %>% `$`("url") climate <- list() for(i in seq_along(station_links)){ print(stringr::str_c(i, " / ", length(station_links))) climate[[i]] <- download_climate(station_links[i]) } # run only when download_climate() successed if(sum(is.null(climate[[1]]), is.null(climate[[2]]), is.null(climate[[3]])) == 0){ month_per_year <- 12 climate_world <- dplyr::bind_rows(climate) %>% dplyr::bind_cols( tibble::tibble(continent = rep(continent, month_per_year))) %>% dplyr::bind_cols( tibble::tibble(no = rep(no, month_per_year))) %>% dplyr::relocate(no, continent, country, station) climate_world }# If you want all climate data, remove head(). # The codes take > 5 sec because of poliste scraping. library(magrittr) library(stringi) library(dplyr) data(station_links) station_links <- station_links %>% dplyr::mutate_all(stringi::stri_unescape_unicode) %>% head(3) %T>% { continent <<- `$`(., "continent") no <<- `$`(., "no") } %>% `$`("url") climate <- list() for(i in seq_along(station_links)){ print(stringr::str_c(i, " / ", length(station_links))) climate[[i]] <- download_climate(station_links[i]) } # run only when download_climate() successed if(sum(is.null(climate[[1]]), is.null(climate[[2]]), is.null(climate[[3]])) == 0){ month_per_year <- 12 climate_world <- dplyr::bind_rows(climate) %>% dplyr::bind_cols( tibble::tibble(continent = rep(continent, month_per_year))) %>% dplyr::bind_cols( tibble::tibble(no = rep(no, month_per_year))) %>% dplyr::relocate(no, continent, country, station) climate_world }
For polite scraping, 5 sec interval is set in download_detail(). Please do not download too many data at once. Use detail_url() to build an url of the data.
download_detail(url, as_numeric = TRUE)download_detail(url, as_numeric = TRUE)
url |
A string to specify target html. |
as_numeric |
A logical. If TRUE, columns of numbers are converted into numeric. Marks of 'JMA' for missing values are treated as NA. |
A tibble including detail climate data, or NULL when failed.
url <- detail_url(61, 47759, "daily", 2023, 6) download_detail(url)url <- detail_url(61, 47759, "daily", 2023, 6) download_detail(url)
For polite scraping, 5 sec interval is set in download_links(), it takes about 15 minutes to get all station links. Please use existing links by "data(station_links)", if you do not need to renew links. You can see web page as below. https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/
download_area_links( url = "https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/" ) download_links(url)download_area_links( url = "https://www.data.jma.go.jp/gmd/cpd/monitor/nrmlist/" ) download_links(url)
url |
A String to specify target html. |
A string vector of url links, or NULL when failed.
# If you want links for all countries and all sations, remove head(). # The codes take over 5 sec because of poliste scraping. library(magrittr) library(stringi) library(dplyr) library(tibble) area_links <- download_area_links() station_links <- NULL continent <- NULL continents <- c("\\u30a2\\u30d5\\u30ea\\u30ab", "\\u30a2\\u30b8\\u30a2", "\\u5357\\u30a2\\u30e1\\u30ea\\u30ab", "\\u5317\\u4e2d\\u30a2\\u30e1\\u30ea\\u30ab", "\\u30aa\\u30bb\\u30a2\\u30cb\\u30a2", "\\u30e8\\u30fc\\u30ed\\u30c3\\u30d1") area_links <- head(area_links, 1) # for test for(i in seq_along(area_links)){ print(stringr::str_c("area: ", i, " / ", length(area_links))) country_links <- download_links(area_links[i]) country_links <- head(country_links, 1) # for test for(j in seq_along(country_links)){ print(stringr::str_c(" country: ", j, " / ", length(country_links))) links <- download_links(country_links[j]) station_links <- c(station_links, links) continent <- c(continent, rep(continents[i], length(links))) } } station_links <- tibble::tibble(url = station_links, continent = continent) station_links# If you want links for all countries and all sations, remove head(). # The codes take over 5 sec because of poliste scraping. library(magrittr) library(stringi) library(dplyr) library(tibble) area_links <- download_area_links() station_links <- NULL continent <- NULL continents <- c("\\u30a2\\u30d5\\u30ea\\u30ab", "\\u30a2\\u30b8\\u30a2", "\\u5357\\u30a2\\u30e1\\u30ea\\u30ab", "\\u5317\\u4e2d\\u30a2\\u30e1\\u30ea\\u30ab", "\\u30aa\\u30bb\\u30a2\\u30cb\\u30a2", "\\u30e8\\u30fc\\u30ed\\u30c3\\u30d1") area_links <- head(area_links, 1) # for test for(i in seq_along(area_links)){ print(stringr::str_c("area: ", i, " / ", length(area_links))) country_links <- download_links(area_links[i]) country_links <- head(country_links, 1) # for test for(j in seq_along(country_links)){ print(stringr::str_c(" country: ", j, " / ", length(country_links))) links <- download_links(country_links[j]) station_links <- c(station_links, links) continent <- c(continent, rep(continents[i], length(links))) } } station_links <- tibble::tibble(url = station_links, continent = continent) station_links
A station of detail climate data is specified by a pair of prec_no (area) and block_no (station). download_prec_no() downloads all prec_no, and download_block_no() downloads block_no of the area. Please use existing data by "data(station_jp_full)", which includes prec_no and station_no of all stations, if you do not need to renew them. For polite scraping, 5 sec interval is set in both functions. You can see web page as below. https://www.data.jma.go.jp/obd/stats/etrn/index.php
download_prec_no() download_block_no(prec_no)download_prec_no() download_block_no(prec_no)
prec_no |
A string or numeric of area no. |
download_prec_no(): a tibble of prec_no and area. download_block_no(): a tibble of prec_no, block_no, station and type. NULL when failed.
prec_no <- download_prec_no() prec_no download_block_no(61)prec_no <- download_prec_no() prec_no download_block_no(61)
Graceful fail
gracefully_fail(remote_file)gracefully_fail(remote_file)
remote_file |
A string of remote file. |
An XML document when successed, or invisible NULL when failed.
https://gist.github.com/kvasilopoulos/47f24348ed75cdb6365312b17f4b914c
Wrapper function to head 3 items
head_3(x)head_3(x)
x |
An object. |
An object like x with length 3.
Raw tables downloaded from Japan Meteorological Agency web pages, before cleaning into data(climate_jp_full). Kept for reproducibility of data-raw/climate_jp_full.R. Each element is a table of a station of data(station_jp_full), and the order of the elements is the same as the rows of the data. Column names are not set, and the first rows are headers of the web page. https://www.data.jma.go.jp/obd/stats/etrn/index.php
mean_climean_cli
A list of 1673 tibbles.
157 tables of "kansho" (weather station).
1512 tables of "AMeDAS".
4 stations, which have no normals.
data(mean_cli) length(mean_cli) dim(mean_cli[[1]])data(mean_cli) length(mean_cli) dim(mean_cli[[1]])
Wrapper function to sleep
sleep(sec = 5)sleep(sec = 5)
sec |
A numeric to sleep (sec). |
No return value, called for side effects.
Climate stations in Japan
station_jpstation_jp
A data frame with 157 rows and 11 variable:
Region. Escaped by stringi::stri_escape_unicode().
Prefecture. Escaped by stringi::stri_escape_unicode()
Station no.
Station name. To avoid duplication, including country name after station name. Can split by "_". Escaped by stringi::stri_escape_unicode().
Altitude. (m)
Latitude. (degree)
Longitude. (degree)
North or South.
West or East.
Pronunciation in Japanese. Escaped by stringi::stri_escape_unicode()
City name. Escaped by stringi::stri_escape_unicode().
library(magrittr) library(stringi) library(dplyr) data(station_jp) station_jp %>% dplyr::mutate_all(stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(station_jp) station_jp %>% dplyr::mutate_all(stringi::stri_unescape_unicode)
Station information downloaded from Japan Meteorological Agency web pages. data(station_jp) includes only 157 stations of climate data, while this data includes all 1673 stations. A pair of prec_no and station_no specifies a station in detail_url() and download_detail(). Note that station_no of "kansho" (weather station) has 5 digits (159 stations) and that of "AMeDAS" has 4 digits (1514 stations). https://www.data.jma.go.jp/obd/stats/etrn/index.php
station_jp_fullstation_jp_full
A data frame with 1673 rows and 16 variables:
Station name. Escaped by stringi::stri_escape_unicode().
Area no of 'JMA'. Escaped by stringi::stri_escape_unicode().
Station no of 'JMA', which is used as block_no in detail_url(). Escaped by stringi::stri_escape_unicode().
Pronunciation in Japanese. Escaped by stringi::stri_escape_unicode().
Altitude (m)
Flag of observation (1: observed, 0: not observed). Character, not numeric.
Flag of observation (1: observed, 0: not observed)
Flag of observation (1: observed, 0: not observed)
Flag of observation (1: observed, 0: not observed)
Flag of observation (1: observed, 0: not observed)
Flag of observation (1: observed, 0: not observed)
Area name of prec_no. Escaped by stringi::stri_escape_unicode().
Latitude (degree)
Longitude (degree)
Date of the end of observation. "9999-99-99" shows that the station is in operation (1287 stations). Escaped by stringi::stri_escape_unicode().
Note of 'JMA'. Escaped by stringi::stri_escape_unicode().
library(magrittr) library(stringi) library(dplyr) data(station_jp_full) station_jp_full %>% head_3() %>% dplyr::mutate_if(is.character, stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(station_jp_full) station_jp_full %>% head_3() %>% dplyr::mutate_if(is.character, stringi::stri_unescape_unicode)
Station name and its URL
station_linksstation_links
A data frame with 3444 rows and 4 variable:
Station no
Station information including no, month, temperature, precipitation, station, country, latitude, NS, longitude, WE, altitude. The information is NOT cleaned Row information downloaded from each URL. Escaped by stringi::stri_escape_unicode().
URL of station.
Continent. Escaped by stringi::stri_escape_unicode().
library(magrittr) library(stringi) library(dplyr) data(station_links) station_links %>% dplyr::mutate_all(stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(station_links) station_links %>% dplyr::mutate_all(stringi::stri_unescape_unicode)
Climate stations of the world
station_worldstation_world
A data frame with 3444 rows and 9 variable:
Station no
Station name. To avoid duplication, including country name after station name. Can split by "_". Escaped by stringi::stri_escape_unicode().
Continent. Escaped by stringi::stri_escape_unicode().
Country name. Escaped by stringi::stri_escape_unicode().
Altitude (m)
Latitude (degree)
North or South.
Longitude (degree)
West or East
library(magrittr) library(stringi) library(dplyr) data(station_world) station_world %>% dplyr::mutate_all(stringi::stri_unescape_unicode)library(magrittr) library(stringi) library(dplyr) data(station_world) station_world %>% dplyr::mutate_all(stringi::stri_unescape_unicode)
Calculate warm index and cold index
wi(x) ci(x)wi(x) ci(x)
x |
A numeric vector |
A string vector of url links.
Kira, T. 1945. A new classification of climate in eastern Asia as the basis for agricultural geography, Hort. Inst. Kyoto Univ., Kyoto. (in Japanese) Warmth Index (WI) and Cold Index (CI) was proposed by Kira (1945), which is known closely related to the distribution of vegetation. Indices can are calculated by following equations. wi = sum (Ti - 5), where wi is Warm index, Ti (Celsius) is mean temperature of each month in a year when Ti > 5. Indices can are calculated by following equations. wi = -sum (Ti - 5), where wi is Cold index, when Ti < 5.
temp <- c(-7.8, -7.2, -2.4, 5.2, 11.7, 16.5, 20.5, 21.1, 15.6, 8.8, 2.0, -4.1) wi(temp) ci(temp) wi <- sum(c(0, 0, 0, 0.2, 6.7, 11.5, 15.5, 16.1, 10.6, 3.8, 0, 0)) ci <- sum(c(12.8, 12.2, 7.4, 0, 0, 0, 0, 0, 0, 0, 3.0, 9.1))temp <- c(-7.8, -7.2, -2.4, 5.2, 11.7, 16.5, 20.5, 21.1, 15.6, 8.8, 2.0, -4.1) wi(temp) ci(temp) wi <- sum(c(0, 0, 0, 0.2, 6.7, 11.5, 15.5, 16.1, 10.6, 3.8, 0, 0)) ci <- sum(c(12.8, 12.2, 7.4, 0, 0, 0, 0, 0, 0, 0, 3.0, 9.1))