Systematic literature review

1 Systematic Literature Review using R

Display code

library("cowsay")
library("multicolor")
say(what = "    Hello, we aim to show how to do a systematic literature review using Scopus
    directly through the API using R. We then analyze the documents and the different data
    to have a better understanding of the scholar landscape for a specific concern.
    Here, we have a specific interest in NLP methods and we filter by the reviews 
    that contain the words \"consumer\" or \"marketing\".",
    by = "cat",
    what_color = "white",
    by_color = "rainbow")

 -------------- 
    Hello, we aim to show how to do a systematic literature review using Scopus
    directly through the API using R. We then analyze the documents and the different data
    to have a better understanding of the scholar landscape for a specific concern.
    Here, we have a specific interest in NLP methods and we filter by the reviews 
    that contain the words "consumer" or "marketing". 
 --------------
    \
      \
        \
            |\___/|
          ==) ^Y^ (==
            \  ^  /
             )=*=(
            /     \
            |     |
           /| | | |\
           \| | |_|/\
      jgs  //_// ___/
               \_)
  

2 Libraries

Display code

package_list <- c(
  "tidyverse",
  "ggplot2",
  "bib2df", # for cleaning .bib data
  "janitor", # useful functions for cleaning imported data
  "rscopus", # using Scopus API
  "biblionetwork", # creating edges
  "tidygraph", # for creating networks
  "ggraph", # plotting networks
  "bibliometrix", #using shiny to do analysis of literature review
  "rscopus",#for using scorpus
  "readr", #manipulate csv, excel files
  "quanteda", #textual manipulation
  "tidyr", #to use separate_wider_delim function
  "splitstackshape", #to use cSplit function
  "rcrossref", #to use API for crossref
  "rstudioapi", #to secure the tokens
  "igraph", #to construct wonderful graphs
  "stringr", #to count number of characters in a string
  "plotly", #use plotly to have interactive graphs
  "networkD3", #have fun manipulating interactive networks
  "htmlwidgets", #to save widget 3d graphs
  "cowsay", #funny animals talking
  #"multicolor", #to have a multicolor font 
  "hrbrthemes", #nice theme for ggplot2
  "ggstatsplot", #nice visualizations with stats included in graphs
  "ggridges", #nice graphs with density
  "see", #Model Visualisation Toolbox for 'easystats' and 'ggplot2'
  "countrycode", #gives continent based on country name
  "tidygeocoder", #to get latitude and longitude to place points on worldmap, ggmap can apparently do the same but needs an API
  "lmtest", #to do linear regression
  "flextable", #to present results in nice tables
  "ggthemes", #have nice themes for ggplot
  "gganimate", #to animate ggplot
  "transformr",# to animate ggplot
  "gifski", #to animate ggplot
  "text2vec", #word embeddings
  "gglorenz",#lorenz curve and gini coefficient
  "clipr", #to copy paste dataframe with write_clip(data)
  "reticulate", #execute python code from R
  #"citecorp" #
  "lubridate", #manipulate dates, soon to be by default in the tidyverse package
  "textrank",
  "reactable"
)
for (p in package_list) 
{
  if (p %in% installed.packages() == FALSE) 
  {
    install.packages(p, dependencies = TRUE)
  }
  library(p, character.only = TRUE)
}
rm(p,package_list)

We can also do it from the Scopus website. We can type our search and even get the query by enabling the “advance query” button. We filter by NLP methods in marketing journals with the keywords marketing or consumer in the source (title, abtract, author). The pro of using directly the rscopus package is that we can execute it easily without having to connect on the website, download the file and place it in the right folder every time we want to update our code. Also, the references are easy to manipulate this way.

3 Getting data from Scopus through API

The Scopus data that we get from the rscopus package is a bit different from the file we get when we download the metadata on the website. We here get raw data that we transform with rscopus into exploitable data. From there, three data frames are useful:

Name of data frame	Description
df	A data frame with all the papers and the information attached
affiliation	A data frame with the affiliations
author	A data frame with the authors’ information. (the largest data frame since there are often multiple authors per paper per affiliation).

All these data can be merged in one data frame only providing we use the authors’ data frame, containing one row for every author present in the articles.

3.1 Articles data extraction

Display code

#hiding api keys and Elsevier's institutional token in the code

t1 <- Sys.time()
#api_key <- rstudioapi::askForPassword(prompt = "Please enter the API key")
api_key <- "xxxx"
set_api_key(api_key)
#token <- rstudioapi::askForPassword(prompt = "Please enter the Institutional Token")
token <- "yyyy"
hdr = inst_token_header(token)


nlp_query <- rscopus::scopus_search("TITLE-ABS-KEY(\"natural language processing\" OR \"nlp\" OR \"text mining\" OR \"text-mining\" OR \"text analysis\" OR \"text-analysis\") AND SRCTITLE(\"marketing\" OR \"consumer\")", 
                                     view = "COMPLETE",
                                     headers = hdr)

list(query = "TITLE-ABS-KEY(\"natural language processing\" OR \"nlp\" OR \"text mining\" OR \"text-mining\" OR \"text analysis\" OR \"text-analysis\") AND SRCTITLE(\"marketing\" OR \"consumer\")", 
    count = 25, start = 0, view = "COMPLETE")
$query
[1] "TITLE-ABS-KEY(\"natural language processing\" OR \"nlp\" OR \"text mining\" OR \"text-mining\" OR \"text analysis\" OR \"text-analysis\") AND SRCTITLE(\"marketing\" OR \"consumer\")"

$count
[1] 25

$start
[1] 0

$view
[1] "COMPLETE"

Response [https://api.elsevier.com/content/search/scopus?query=TITLE-ABS-KEY%28%22natural%20language%20processing%22%20OR%20%22nlp%22%20OR%20%22text%20mining%22%20OR%20%22text-mining%22%20OR%20%22text%20analysis%22%20OR%20%22text-analysis%22%29%20AND%20SRCTITLE%28%22marketing%22%20OR%20%22consumer%22%29&count=25&start=0&view=COMPLETE]
  Date: 2023-05-11 17:01
  Status: 200
  Content-Type: application/json;charset=UTF-8
  Size: 115 kB


  |                                                                            
  |                                                                      |   0%
  |                                                                            
  |====                                                                  |   6%
  |                                                                            
  |=========                                                             |  12%
  |                                                                            
  |=============                                                         |  19%
  |                                                                            
  |==================                                                    |  25%
  |                                                                            
  |======================                                                |  31%
  |                                                                            
  |==========================                                            |  38%
  |                                                                            
  |===============================                                       |  44%
  |                                                                            
  |===================================                                   |  50%
  |                                                                            
  |=======================================                               |  56%
  |                                                                            
  |============================================                          |  62%
  |                                                                            
  |================================================                      |  69%
  |                                                                            
  |====================================================                  |  75%
  |                                                                            
  |=========================================================             |  81%
  |                                                                            
  |=============================================================         |  88%
  |                                                                            
  |==================================================================    |  94%
  |                                                                            
  |======================================================================| 100%

Display code

#we get a list with 4 different dataframes : df, affiliations, author, prism:isbn
#nlp_data_raw <- readRDS("nlp_data_raw.rds")
nlp_data_raw <- gen_entries_to_df(nlp_query$entries)
#saveRDS(nlp_data_raw,"nlp_data_raw.rds")

#res = author_df(last_name = "Benavent", first_name = "Christophe", verbose = FALSE, general = FALSE, headers = hdr)
#head(res)
#dataframes ready to use/transform
nlp_papers       <- nlp_data_raw$df
nlp_affiliations <- nlp_data_raw$affiliation #details of universities
nlp_authors      <- nlp_data_raw$author

#we replace "-" by "_" in the columnns' names for easier manipulation
names(nlp_authors)      <- str_replace_all(names(nlp_authors), "-", "_")
names(nlp_affiliations) <- str_replace_all(names(nlp_affiliations), "-", "_")
names(nlp_papers)       <- str_replace_all(names(nlp_papers), "-", "_")

nlp_authors <- nlp_authors %>%
  dplyr::rename("afid" = `afid.$`)

#we get the continent for each country
nlp_authors      <- nlp_authors %>% select(entry_number,everything())
nlp_affiliations <- nlp_affiliations %>% select(entry_number,everything()) %>%
  mutate(continent = countrycode(sourcevar   = affiliation_country,
                                 origin      = "country.name",
                                 destination = "continent"))

nlp_papers <- nlp_papers %>% select(entry_number,everything())

authors_affiliations <-left_join(nlp_authors,nlp_affiliations)
nlp_papers <- nlp_papers %>%
  mutate(entry_number = as.numeric(entry_number))

nlp_full <- left_join(authors_affiliations,nlp_papers)

nlp_full <- nlp_full %>%
  dplyr::rename("author_count" = `author_count.$`) %>%
  mutate(author_count   = as.numeric(author_count)) %>%
  mutate("year"         = substr(`prism:coverDate`,1,4)) %>%
  mutate(citedby_count  = as.numeric(citedby_count))


#csv2 uses comma as separator while csv uses decimal point .
readr::write_excel_csv2(nlp_full,"nlp_full_data.csv")
#nlp_references <- read.csv("nlp_references.csv", sep=";", encoding="UTF-8-BOM")
#nlp_full <- read.csv("nlp_full_data.csv", sep=";", encoding ="UTF-8-BOM")
print(Sys.time()-t1)

Time difference of 17.8854 secs

3.2 Collecting the references of the articles

The scopus_search function doesn’t allow us to get the references of the articles. We have to use the abstract_retrieval function.

Display code

t1 <- Sys.time()
citing_articles <- nlp_papers$`dc:identifier` # extracting the SCOPUS IDs of our articles
citation_list <- list()
subjects_list <- list()
number_ref <- data.frame()
number_ref <- number_ref %>% #dataframe with scopus_ID and the number of references attached
  mutate(`dc:identifier` = "", 
         nref = "")

#extraction of references is somewhat complicated. Scopus doesn't allow to get all the references
#of all the articles in one function. We then need to loop through each article we got from
#the previous query and get the citation list of each article.
for(i in 1:length(citing_articles))
{
  #list of citations
  cat(paste0("Article n°",i,"\n"))
  citations_query <- abstract_retrieval(citing_articles[i],
                                        identifier = "scopus_id",
                                        view       = "REF",
                                        headers    = hdr)
  #list of subjects associated with the article
  subjects_query <- abstract_retrieval(citing_articles[i],
                                        identifier = "scopus_id",
                                        view = "FULL",
                                        headers = hdr)
  
  #we put the list of references in the citations variables.
  #gen_entries_to_df transforms the complicated data structure sent by Scopus into an easily manipulable dataframe
  citations <- gen_entries_to_df(citations_query$content$`abstracts-retrieval-response`$references$reference)
  
  #same with subject areas
  subjects <- gen_entries_to_df(subjects_query$content$`abstracts-retrieval-response`$`subject-areas`$`subject-area`)
  
  if(length(citations$df) > 0) #if request is not null
  {
    nb_total_refs <- as.numeric(citations_query$content$`abstracts-retrieval-response`$references$`@total-references`)
    cat("Total number of references :", nb_total_refs,"\n")
    
    nb_ref_article <- nb_total_refs-40 #number of references we still need to get by querying
    
    if(nb_ref_article > 0)
    {
      cat("Number of references left to retrieve : ", nb_ref_article, "\n")
    } else
    {
      cat("All references are already retrieved.\n")
    }
    
    message(paste0(citing_articles[i], " ref is not empty."))
    citations <- citations$df %>% 
    as_tibble(.name_repair = "unique")
    
    #we put the number of references of the article in the number_ref dataframe created above
    number_ref[i,1] <- citing_articles[i]
    number_ref[i,2] <- citations_query$content$`abstracts-retrieval-response`$references$`@total-references`
    
    citation_list[[citing_articles[i]]] <- citations
        
    if (nb_total_refs > 40) #we can only get 40 references at a time, so we need to loop if there are more than 40
    {
        nb_query_left_to_do <- ceiling((nb_ref_article) / 40) #the number of query we need to do
        cat("Number of requests left to do  :", nb_query_left_to_do, "\n")
        
      for (j in 1:nb_query_left_to_do)
      {
            cat("Request n°", j , "\n")
            citations_query <- abstract_retrieval(citing_articles[i],
                                                identifier = "scopus_id",
                                                view       = "REF",
                                                startref   = 40*j+1,
                                                headers    = hdr)
            citations <- gen_entries_to_df(citations_query$content$`abstracts-retrieval-response`$references$reference)
            
             if(length(citations$df) > 0) #if request is not null
          {
                message(paste0(citing_articles[i], " ref is not empty."))
                citations <- citations$df %>% 
                as_tibble(.name_repair = "unique")
                #citation_list is a list of lists which contain every references for a specific scopus ID
                #
                citation_list[[citing_articles[i]]] <- bind_rows(citations,citation_list[[citing_articles[i]]])
          }
       }
    }
      
    if(length(subjects$df) > 0)
    {
      message(paste0(citing_articles[i], " subject is not empty."))
      subjects <- subjects$df %>% 
      as_tibble(.name_repair = "unique")
      
      subjects_list[[citing_articles[i]]] <- bind_rows(subjects,subjects_list[[citing_articles[i]]])
    }
  }
}

Article n°1
Total number of references : 119 
Number of references left to retrieve :  79 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°2
Total number of references : 172 
Number of references left to retrieve :  132 
Number of requests left to do  : 4 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Article n°3
Total number of references : 136 
Number of references left to retrieve :  96 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°4
Total number of references : 76 
Number of references left to retrieve :  36 
Number of requests left to do  : 1 
Request n° 1 
Article n°5
Total number of references : 66 
Number of references left to retrieve :  26 
Number of requests left to do  : 1 
Request n° 1 
Article n°6
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°7
Total number of references : 73 
Number of references left to retrieve :  33 
Number of requests left to do  : 1 
Request n° 1 
Article n°8
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°9
Total number of references : 70 
Number of references left to retrieve :  30 
Number of requests left to do  : 1 
Request n° 1 
Article n°10
Total number of references : 91 
Number of references left to retrieve :  51 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°11
Total number of references : 48 
Number of references left to retrieve :  8 
Number of requests left to do  : 1 
Request n° 1 
Article n°12
Total number of references : 72 
Number of references left to retrieve :  32 
Number of requests left to do  : 1 
Request n° 1 
Article n°13
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°14
Total number of references : 69 
Number of references left to retrieve :  29 
Number of requests left to do  : 1 
Request n° 1 
Article n°15
Total number of references : 48 
Number of references left to retrieve :  8 
Number of requests left to do  : 1 
Request n° 1 
Article n°16
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°17
Total number of references : 34 
All references are already retrieved.
Article n°18
Total number of references : 131 
Number of references left to retrieve :  91 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°19
Total number of references : 79 
Number of references left to retrieve :  39 
Number of requests left to do  : 1 
Request n° 1 
Article n°20
Total number of references : 95 
Number of references left to retrieve :  55 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°21
Total number of references : 48 
Number of references left to retrieve :  8 
Number of requests left to do  : 1 
Request n° 1 
Article n°22
Total number of references : 96 
Number of references left to retrieve :  56 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°23
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°24
Total number of references : 176 
Number of references left to retrieve :  136 
Number of requests left to do  : 4 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Article n°25
Article n°26
Article n°27
Article n°28
Article n°29
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°30
Article n°31
Total number of references : 87 
Number of references left to retrieve :  47 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°32
Total number of references : 66 
Number of references left to retrieve :  26 
Number of requests left to do  : 1 
Request n° 1 
Article n°33
Total number of references : 105 
Number of references left to retrieve :  65 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°34
Total number of references : 47 
Number of references left to retrieve :  7 
Number of requests left to do  : 1 
Request n° 1 
Article n°35
Total number of references : 135 
Number of references left to retrieve :  95 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°36
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°37
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°38
Total number of references : 12 
All references are already retrieved.
Article n°39
Total number of references : 111 
Number of references left to retrieve :  71 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°40
Total number of references : 79 
Number of references left to retrieve :  39 
Number of requests left to do  : 1 
Request n° 1 
Article n°41
Total number of references : 51 
Number of references left to retrieve :  11 
Number of requests left to do  : 1 
Request n° 1 
Article n°42
Article n°43
Total number of references : 79 
Number of references left to retrieve :  39 
Number of requests left to do  : 1 
Request n° 1 
Article n°44
Total number of references : 44 
Number of references left to retrieve :  4 
Number of requests left to do  : 1 
Request n° 1 
Article n°45
Total number of references : 76 
Number of references left to retrieve :  36 
Number of requests left to do  : 1 
Request n° 1 
Article n°46
Total number of references : 105 
Number of references left to retrieve :  65 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°47
Total number of references : 45 
Number of references left to retrieve :  5 
Number of requests left to do  : 1 
Request n° 1 
Article n°48
Total number of references : 98 
Number of references left to retrieve :  58 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°49
Total number of references : 89 
Number of references left to retrieve :  49 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°50
Total number of references : 70 
Number of references left to retrieve :  30 
Number of requests left to do  : 1 
Request n° 1 
Article n°51
Total number of references : 92 
Number of references left to retrieve :  52 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°52
Total number of references : 114 
Number of references left to retrieve :  74 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°53
Total number of references : 80 
Number of references left to retrieve :  40 
Number of requests left to do  : 1 
Request n° 1 
Article n°54
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°55
Total number of references : 79 
Number of references left to retrieve :  39 
Number of requests left to do  : 1 
Request n° 1 
Article n°56
Total number of references : 91 
Number of references left to retrieve :  51 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°57
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°58
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°59
Total number of references : 94 
Number of references left to retrieve :  54 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°60
Total number of references : 107 
Number of references left to retrieve :  67 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°61
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°62
Total number of references : 93 
Number of references left to retrieve :  53 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°63
Total number of references : 88 
Number of references left to retrieve :  48 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°64
Total number of references : 104 
Number of references left to retrieve :  64 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°65
Total number of references : 55 
Number of references left to retrieve :  15 
Number of requests left to do  : 1 
Request n° 1 
Article n°66
Total number of references : 77 
Number of references left to retrieve :  37 
Number of requests left to do  : 1 
Request n° 1 
Article n°67
Total number of references : 108 
Number of references left to retrieve :  68 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°68
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°69
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°70
Total number of references : 12 
All references are already retrieved.
Article n°71
Article n°72
Total number of references : 290 
Number of references left to retrieve :  250 
Number of requests left to do  : 7 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Request n° 7 
Article n°73
Total number of references : 189 
Number of references left to retrieve :  149 
Number of requests left to do  : 4 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Article n°74
Total number of references : 80 
Number of references left to retrieve :  40 
Number of requests left to do  : 1 
Request n° 1 
Article n°75
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°76
Total number of references : 70 
Number of references left to retrieve :  30 
Number of requests left to do  : 1 
Request n° 1 
Article n°77
Total number of references : 129 
Number of references left to retrieve :  89 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°78
Total number of references : 115 
Number of references left to retrieve :  75 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°79
Total number of references : 51 
Number of references left to retrieve :  11 
Number of requests left to do  : 1 
Request n° 1 
Article n°80
Total number of references : 107 
Number of references left to retrieve :  67 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°81
Total number of references : 82 
Number of references left to retrieve :  42 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°82
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°83
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°84
Total number of references : 119 
Number of references left to retrieve :  79 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°85
Total number of references : 65 
Number of references left to retrieve :  25 
Number of requests left to do  : 1 
Request n° 1 
Article n°86
Total number of references : 40 
All references are already retrieved.
Article n°87
Total number of references : 30 
All references are already retrieved.
Article n°88
Total number of references : 49 
Number of references left to retrieve :  9 
Number of requests left to do  : 1 
Request n° 1 
Article n°89
Total number of references : 96 
Number of references left to retrieve :  56 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°90
Total number of references : 90 
Number of references left to retrieve :  50 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°91
Total number of references : 86 
Number of references left to retrieve :  46 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°92
Total number of references : 49 
Number of references left to retrieve :  9 
Number of requests left to do  : 1 
Request n° 1 
Article n°93
Total number of references : 213 
Number of references left to retrieve :  173 
Number of requests left to do  : 5 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Article n°94
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°95
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°96
Total number of references : 50 
Number of references left to retrieve :  10 
Number of requests left to do  : 1 
Request n° 1 
Article n°97
Total number of references : 40 
All references are already retrieved.
Article n°98
Total number of references : 94 
Number of references left to retrieve :  54 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°99
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°100
Total number of references : 52 
Number of references left to retrieve :  12 
Number of requests left to do  : 1 
Request n° 1 
Article n°101
Total number of references : 99 
Number of references left to retrieve :  59 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°102
Total number of references : 66 
Number of references left to retrieve :  26 
Number of requests left to do  : 1 
Request n° 1 
Article n°103
Total number of references : 53 
Number of references left to retrieve :  13 
Number of requests left to do  : 1 
Request n° 1 
Article n°104
Total number of references : 131 
Number of references left to retrieve :  91 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°105
Total number of references : 21 
All references are already retrieved.
Article n°106
Total number of references : 7 
All references are already retrieved.
Article n°107
Total number of references : 58 
Number of references left to retrieve :  18 
Number of requests left to do  : 1 
Request n° 1 
Article n°108
Total number of references : 60 
Number of references left to retrieve :  20 
Number of requests left to do  : 1 
Request n° 1 
Article n°109
Total number of references : 54 
Number of references left to retrieve :  14 
Number of requests left to do  : 1 
Request n° 1 
Article n°110
Total number of references : 53 
Number of references left to retrieve :  13 
Number of requests left to do  : 1 
Request n° 1 
Article n°111
Total number of references : 23 
All references are already retrieved.
Article n°112
Total number of references : 250 
Number of references left to retrieve :  210 
Number of requests left to do  : 6 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Article n°113
Total number of references : 10 
All references are already retrieved.
Article n°114
Total number of references : 4 
All references are already retrieved.
Article n°115
Total number of references : 54 
Number of references left to retrieve :  14 
Number of requests left to do  : 1 
Request n° 1 
Article n°116
Total number of references : 130 
Number of references left to retrieve :  90 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°117
Total number of references : 67 
Number of references left to retrieve :  27 
Number of requests left to do  : 1 
Request n° 1 
Article n°118
Total number of references : 135 
Number of references left to retrieve :  95 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°119
Total number of references : 19 
All references are already retrieved.
Article n°120
Total number of references : 12 
All references are already retrieved.
Article n°121
Total number of references : 34 
All references are already retrieved.
Article n°122
Article n°123
Article n°124
Article n°125
Article n°126
Article n°127
Article n°128
Total number of references : 85 
Number of references left to retrieve :  45 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°129
Total number of references : 69 
Number of references left to retrieve :  29 
Number of requests left to do  : 1 
Request n° 1 
Article n°130
Total number of references : 25 
All references are already retrieved.
Article n°131
Article n°132
Article n°133
Article n°134
Article n°135
Total number of references : 47 
Number of references left to retrieve :  7 
Number of requests left to do  : 1 
Request n° 1 
Article n°136
Article n°137
Total number of references : 12 
All references are already retrieved.
Article n°138
Total number of references : 5 
All references are already retrieved.
Article n°139
Total number of references : 10 
All references are already retrieved.
Article n°140
Total number of references : 7 
All references are already retrieved.
Article n°141
Total number of references : 22 
All references are already retrieved.
Article n°142
Total number of references : 10 
All references are already retrieved.
Article n°143
Total number of references : 72 
Number of references left to retrieve :  32 
Number of requests left to do  : 1 
Request n° 1 
Article n°144
Total number of references : 112 
Number of references left to retrieve :  72 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°145
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°146
Total number of references : 78 
Number of references left to retrieve :  38 
Number of requests left to do  : 1 
Request n° 1 
Article n°147
Total number of references : 51 
Number of references left to retrieve :  11 
Number of requests left to do  : 1 
Request n° 1 
Article n°148
Total number of references : 94 
Number of references left to retrieve :  54 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°149
Total number of references : 86 
Number of references left to retrieve :  46 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°150
Total number of references : 77 
Number of references left to retrieve :  37 
Number of requests left to do  : 1 
Request n° 1 
Article n°151
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°152
Total number of references : 44 
Number of references left to retrieve :  4 
Number of requests left to do  : 1 
Request n° 1 
Article n°153
Total number of references : 26 
All references are already retrieved.
Article n°154
Total number of references : 66 
Number of references left to retrieve :  26 
Number of requests left to do  : 1 
Request n° 1 
Article n°155
Total number of references : 104 
Number of references left to retrieve :  64 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°156
Total number of references : 54 
Number of references left to retrieve :  14 
Number of requests left to do  : 1 
Request n° 1 
Article n°157
Total number of references : 39 
All references are already retrieved.
Article n°158
Total number of references : 52 
Number of references left to retrieve :  12 
Number of requests left to do  : 1 
Request n° 1 
Article n°159
Total number of references : 9 
All references are already retrieved.
Article n°160
Total number of references : 45 
Number of references left to retrieve :  5 
Number of requests left to do  : 1 
Request n° 1 
Article n°161
Total number of references : 124 
Number of references left to retrieve :  84 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°162
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°163
Total number of references : 65 
Number of references left to retrieve :  25 
Number of requests left to do  : 1 
Request n° 1 
Article n°164
Total number of references : 73 
Number of references left to retrieve :  33 
Number of requests left to do  : 1 
Request n° 1 
Article n°165
Total number of references : 14 
All references are already retrieved.
Article n°166
Total number of references : 235 
Number of references left to retrieve :  195 
Number of requests left to do  : 5 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Article n°167
Total number of references : 55 
Number of references left to retrieve :  15 
Number of requests left to do  : 1 
Request n° 1 
Article n°168
Total number of references : 120 
Number of references left to retrieve :  80 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°169
Total number of references : 76 
Number of references left to retrieve :  36 
Number of requests left to do  : 1 
Request n° 1 
Article n°170
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°171
Total number of references : 51 
Number of references left to retrieve :  11 
Number of requests left to do  : 1 
Request n° 1 
Article n°172
Total number of references : 90 
Number of references left to retrieve :  50 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°173
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°174
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°175
Total number of references : 127 
Number of references left to retrieve :  87 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°176
Total number of references : 75 
Number of references left to retrieve :  35 
Number of requests left to do  : 1 
Request n° 1 
Article n°177
Total number of references : 55 
Number of references left to retrieve :  15 
Number of requests left to do  : 1 
Request n° 1 
Article n°178
Total number of references : 62 
Number of references left to retrieve :  22 
Number of requests left to do  : 1 
Request n° 1 
Article n°179
Total number of references : 30 
All references are already retrieved.
Article n°180
Total number of references : 251 
Number of references left to retrieve :  211 
Number of requests left to do  : 6 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Article n°181
Total number of references : 85 
Number of references left to retrieve :  45 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°182
Total number of references : 77 
Number of references left to retrieve :  37 
Number of requests left to do  : 1 
Request n° 1 
Article n°183
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°184
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°185
Total number of references : 67 
Number of references left to retrieve :  27 
Number of requests left to do  : 1 
Request n° 1 
Article n°186
Total number of references : 98 
Number of references left to retrieve :  58 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°187
Total number of references : 58 
Number of references left to retrieve :  18 
Number of requests left to do  : 1 
Request n° 1 
Article n°188
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°189
Total number of references : 41 
Number of references left to retrieve :  1 
Number of requests left to do  : 1 
Request n° 1 
Article n°190
Total number of references : 35 
All references are already retrieved.
Article n°191
Total number of references : 106 
Number of references left to retrieve :  66 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°192
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°193
Total number of references : 73 
Number of references left to retrieve :  33 
Number of requests left to do  : 1 
Request n° 1 
Article n°194
Total number of references : 78 
Number of references left to retrieve :  38 
Number of requests left to do  : 1 
Request n° 1 
Article n°195
Total number of references : 25 
All references are already retrieved.
Article n°196
Total number of references : 8 
All references are already retrieved.
Article n°197
Total number of references : 12 
All references are already retrieved.
Article n°198
Total number of references : 12 
All references are already retrieved.
Article n°199
Total number of references : 15 
All references are already retrieved.
Article n°200
Total number of references : 6 
All references are already retrieved.
Article n°201
Total number of references : 5 
All references are already retrieved.
Article n°202
Total number of references : 90 
Number of references left to retrieve :  50 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°203
Total number of references : 166 
Number of references left to retrieve :  126 
Number of requests left to do  : 4 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Article n°204
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°205
Total number of references : 89 
Number of references left to retrieve :  49 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°206
Total number of references : 121 
Number of references left to retrieve :  81 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°207
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°208
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°209
Total number of references : 109 
Number of references left to retrieve :  69 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°210
Total number of references : 47 
Number of references left to retrieve :  7 
Number of requests left to do  : 1 
Request n° 1 
Article n°211
Total number of references : 110 
Number of references left to retrieve :  70 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°212
Total number of references : 93 
Number of references left to retrieve :  53 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°213
Total number of references : 153 
Number of references left to retrieve :  113 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°214
Total number of references : 70 
Number of references left to retrieve :  30 
Number of requests left to do  : 1 
Request n° 1 
Article n°215
Total number of references : 99 
Number of references left to retrieve :  59 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°216
Total number of references : 17 
All references are already retrieved.
Article n°217
Total number of references : 44 
Number of references left to retrieve :  4 
Number of requests left to do  : 1 
Request n° 1 
Article n°218
Total number of references : 123 
Number of references left to retrieve :  83 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°219
Total number of references : 28 
All references are already retrieved.
Article n°220
Total number of references : 15 
All references are already retrieved.
Article n°221
Total number of references : 54 
Number of references left to retrieve :  14 
Number of requests left to do  : 1 
Request n° 1 
Article n°222
Total number of references : 7 
All references are already retrieved.
Article n°223
Total number of references : 50 
Number of references left to retrieve :  10 
Number of requests left to do  : 1 
Request n° 1 
Article n°224
Total number of references : 5 
All references are already retrieved.
Article n°225
Total number of references : 4 
All references are already retrieved.
Article n°226
Total number of references : 89 
Number of references left to retrieve :  49 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°227
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°228
Total number of references : 65 
Number of references left to retrieve :  25 
Number of requests left to do  : 1 
Request n° 1 
Article n°229
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°230
Total number of references : 52 
Number of references left to retrieve :  12 
Number of requests left to do  : 1 
Request n° 1 
Article n°231
Total number of references : 24 
All references are already retrieved.
Article n°232
Total number of references : 105 
Number of references left to retrieve :  65 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°233
Total number of references : 63 
Number of references left to retrieve :  23 
Number of requests left to do  : 1 
Request n° 1 
Article n°234
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°235
Total number of references : 67 
Number of references left to retrieve :  27 
Number of requests left to do  : 1 
Request n° 1 
Article n°236
Total number of references : 66 
Number of references left to retrieve :  26 
Number of requests left to do  : 1 
Request n° 1 
Article n°237
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°238
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°239
Total number of references : 62 
Number of references left to retrieve :  22 
Number of requests left to do  : 1 
Request n° 1 
Article n°240
Total number of references : 17 
All references are already retrieved.
Article n°241
Total number of references : 17 
All references are already retrieved.
Article n°242
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°243
Total number of references : 57 
Number of references left to retrieve :  17 
Number of requests left to do  : 1 
Request n° 1 
Article n°244
Total number of references : 53 
Number of references left to retrieve :  13 
Number of requests left to do  : 1 
Request n° 1 
Article n°245
Total number of references : 23 
All references are already retrieved.
Article n°246
Total number of references : 68 
Number of references left to retrieve :  28 
Number of requests left to do  : 1 
Request n° 1 
Article n°247
Total number of references : 105 
Number of references left to retrieve :  65 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°248
Total number of references : 63 
Number of references left to retrieve :  23 
Number of requests left to do  : 1 
Request n° 1 
Article n°249
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°250
Total number of references : 39 
All references are already retrieved.
Article n°251
Total number of references : 60 
Number of references left to retrieve :  20 
Number of requests left to do  : 1 
Request n° 1 
Article n°252
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°253
Article n°254
Article n°255
Article n°256
Article n°257
Article n°258
Total number of references : 14 
All references are already retrieved.
Article n°259
Article n°260
Total number of references : 68 
Number of references left to retrieve :  28 
Number of requests left to do  : 1 
Request n° 1 
Article n°261
Article n°262
Total number of references : 32 
All references are already retrieved.
Article n°263
Total number of references : 57 
Number of references left to retrieve :  17 
Number of requests left to do  : 1 
Request n° 1 
Article n°264
Total number of references : 19 
All references are already retrieved.
Article n°265
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°266
Total number of references : 111 
Number of references left to retrieve :  71 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°267
Total number of references : 60 
Number of references left to retrieve :  20 
Number of requests left to do  : 1 
Request n° 1 
Article n°268
Total number of references : 159 
Number of references left to retrieve :  119 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°269
Total number of references : 58 
Number of references left to retrieve :  18 
Number of requests left to do  : 1 
Request n° 1 
Article n°270
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°271
Total number of references : 49 
Number of references left to retrieve :  9 
Number of requests left to do  : 1 
Request n° 1 
Article n°272
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°273
Total number of references : 109 
Number of references left to retrieve :  69 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°274
Total number of references : 86 
Number of references left to retrieve :  46 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°275
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°276
Total number of references : 91 
Number of references left to retrieve :  51 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°277
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°278
Total number of references : 15 
All references are already retrieved.
Article n°279
Total number of references : 6 
All references are already retrieved.
Article n°280
Total number of references : 4 
All references are already retrieved.
Article n°281
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°282
Total number of references : 40 
All references are already retrieved.
Article n°283
Total number of references : 12 
All references are already retrieved.
Article n°284
Total number of references : 75 
Number of references left to retrieve :  35 
Number of requests left to do  : 1 
Request n° 1 
Article n°285
Total number of references : 49 
Number of references left to retrieve :  9 
Number of requests left to do  : 1 
Request n° 1 
Article n°286
Total number of references : 28 
All references are already retrieved.
Article n°287
Total number of references : 32 
All references are already retrieved.
Article n°288
Total number of references : 91 
Number of references left to retrieve :  51 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°289
Total number of references : 54 
Number of references left to retrieve :  14 
Number of requests left to do  : 1 
Request n° 1 
Article n°290
Total number of references : 43 
Number of references left to retrieve :  3 
Number of requests left to do  : 1 
Request n° 1 
Article n°291
Total number of references : 95 
Number of references left to retrieve :  55 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°292
Total number of references : 3 
All references are already retrieved.
Article n°293
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°294
Total number of references : 92 
Number of references left to retrieve :  52 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°295
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°296
Total number of references : 47 
Number of references left to retrieve :  7 
Number of requests left to do  : 1 
Request n° 1 
Article n°297
Total number of references : 9 
All references are already retrieved.
Article n°298
Total number of references : 14 
All references are already retrieved.
Article n°299
Total number of references : 5 
All references are already retrieved.
Article n°300
Total number of references : 10 
All references are already retrieved.
Article n°301
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°302
Total number of references : 269 
Number of references left to retrieve :  229 
Number of requests left to do  : 6 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Article n°303
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°304
Total number of references : 32 
All references are already retrieved.
Article n°305
Total number of references : 9 
All references are already retrieved.
Article n°306
Total number of references : 124 
Number of references left to retrieve :  84 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°307
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°308
Total number of references : 89 
Number of references left to retrieve :  49 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°309
Total number of references : 35 
All references are already retrieved.
Article n°310
Article n°311
Article n°312
Total number of references : 25 
All references are already retrieved.
Article n°313
Total number of references : 89 
Number of references left to retrieve :  49 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°314
Total number of references : 62 
Number of references left to retrieve :  22 
Number of requests left to do  : 1 
Request n° 1 
Article n°315
Total number of references : 9 
All references are already retrieved.
Article n°316
Total number of references : 58 
Number of references left to retrieve :  18 
Number of requests left to do  : 1 
Request n° 1 
Article n°317
Total number of references : 19 
All references are already retrieved.
Article n°318
Total number of references : 14 
All references are already retrieved.
Article n°319
Total number of references : 5 
All references are already retrieved.
Article n°320
Total number of references : 12 
All references are already retrieved.
Article n°321
Total number of references : 72 
Number of references left to retrieve :  32 
Number of requests left to do  : 1 
Request n° 1 
Article n°322
Total number of references : 206 
Number of references left to retrieve :  166 
Number of requests left to do  : 5 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Article n°323
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°324
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°325
Total number of references : 33 
All references are already retrieved.
Article n°326
Total number of references : 74 
Number of references left to retrieve :  34 
Number of requests left to do  : 1 
Request n° 1 
Article n°327
Total number of references : 277 
Number of references left to retrieve :  237 
Number of requests left to do  : 6 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Article n°328
Total number of references : 15 
All references are already retrieved.
Article n°329
Total number of references : 18 
All references are already retrieved.
Article n°330
Total number of references : 37 
All references are already retrieved.
Article n°331
Total number of references : 43 
Number of references left to retrieve :  3 
Number of requests left to do  : 1 
Request n° 1 
Article n°332
Article n°333
Article n°334
Article n°335
Article n°336
Article n°337
Total number of references : 311 
Number of references left to retrieve :  271 
Number of requests left to do  : 7 
Request n° 1 
Request n° 2 
Request n° 3 
Request n° 4 
Request n° 5 
Request n° 6 
Request n° 7 
Article n°338
Total number of references : 64 
Number of references left to retrieve :  24 
Number of requests left to do  : 1 
Request n° 1 
Article n°339
Total number of references : 11 
All references are already retrieved.
Article n°340
Total number of references : 34 
All references are already retrieved.
Article n°341
Total number of references : 25 
All references are already retrieved.
Article n°342
Total number of references : 36 
All references are already retrieved.
Article n°343
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°344
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°345
Total number of references : 98 
Number of references left to retrieve :  58 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°346
Total number of references : 91 
Number of references left to retrieve :  51 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°347
Total number of references : 35 
All references are already retrieved.
Article n°348
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°349
Article n°350
Article n°351
Total number of references : 46 
Number of references left to retrieve :  6 
Number of requests left to do  : 1 
Request n° 1 
Article n°352
Total number of references : 49 
Number of references left to retrieve :  9 
Number of requests left to do  : 1 
Request n° 1 
Article n°353
Total number of references : 14 
All references are already retrieved.
Article n°354
Total number of references : 34 
All references are already retrieved.
Article n°355
Total number of references : 50 
Number of references left to retrieve :  10 
Number of requests left to do  : 1 
Request n° 1 
Article n°356
Total number of references : 34 
All references are already retrieved.
Article n°357
Total number of references : 52 
Number of references left to retrieve :  12 
Number of requests left to do  : 1 
Request n° 1 
Article n°358
Total number of references : 53 
Number of references left to retrieve :  13 
Number of requests left to do  : 1 
Request n° 1 
Article n°359
Article n°360
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°361
Total number of references : 12 
All references are already retrieved.
Article n°362
Total number of references : 6 
All references are already retrieved.
Article n°363
Total number of references : 10 
All references are already retrieved.
Article n°364
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°365
Article n°366
Total number of references : 69 
Number of references left to retrieve :  29 
Number of requests left to do  : 1 
Request n° 1 
Article n°367
Total number of references : 23 
All references are already retrieved.
Article n°368
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°369
Total number of references : 11 
All references are already retrieved.
Article n°370
Total number of references : 30 
All references are already retrieved.
Article n°371
Total number of references : 35 
All references are already retrieved.
Article n°372
Total number of references : 27 
All references are already retrieved.
Article n°373
Article n°374
Total number of references : 32 
All references are already retrieved.
Article n°375
Total number of references : 23 
All references are already retrieved.
Article n°376
Total number of references : 126 
Number of references left to retrieve :  86 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°377
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°378
Total number of references : 46 
Number of references left to retrieve :  6 
Number of requests left to do  : 1 
Request n° 1 
Article n°379
Total number of references : 17 
All references are already retrieved.
Article n°380
Total number of references : 2 
All references are already retrieved.
Article n°381
Total number of references : 28 
All references are already retrieved.
Article n°382
Total number of references : 56 
Number of references left to retrieve :  16 
Number of requests left to do  : 1 
Request n° 1 
Article n°383
Total number of references : 50 
Number of references left to retrieve :  10 
Number of requests left to do  : 1 
Request n° 1 
Article n°384
Total number of references : 61 
Number of references left to retrieve :  21 
Number of requests left to do  : 1 
Request n° 1 
Article n°385
Total number of references : 8 
All references are already retrieved.
Article n°386
Total number of references : 52 
Number of references left to retrieve :  12 
Number of requests left to do  : 1 
Request n° 1 
Article n°387
Total number of references : 2 
All references are already retrieved.
Article n°388
Total number of references : 21 
All references are already retrieved.
Article n°389
Total number of references : 44 
Number of references left to retrieve :  4 
Number of requests left to do  : 1 
Request n° 1 
Article n°390
Total number of references : 17 
All references are already retrieved.
Article n°391
Article n°392
Total number of references : 37 
All references are already retrieved.
Article n°393
Total number of references : 41 
Number of references left to retrieve :  1 
Number of requests left to do  : 1 
Request n° 1 
Article n°394
Total number of references : 121 
Number of references left to retrieve :  81 
Number of requests left to do  : 3 
Request n° 1 
Request n° 2 
Request n° 3 
Article n°395
Total number of references : 79 
Number of references left to retrieve :  39 
Number of requests left to do  : 1 
Request n° 1 
Article n°396
Total number of references : 11 
All references are already retrieved.
Article n°397
Total number of references : 40 
All references are already retrieved.
Article n°398
Total number of references : 8 
All references are already retrieved.
Article n°399
Total number of references : 71 
Number of references left to retrieve :  31 
Number of requests left to do  : 1 
Request n° 1 
Article n°400
Total number of references : 50 
Number of references left to retrieve :  10 
Number of requests left to do  : 1 
Request n° 1 
Article n°401
Total number of references : 15 
All references are already retrieved.
Article n°402
Total number of references : 29 
All references are already retrieved.
Article n°403
Total number of references : 41 
Number of references left to retrieve :  1 
Number of requests left to do  : 1 
Request n° 1 
Article n°404
Total number of references : 11 
All references are already retrieved.
Article n°405
Total number of references : 14 
All references are already retrieved.
Article n°406
Total number of references : 28 
All references are already retrieved.
Article n°407
Total number of references : 59 
Number of references left to retrieve :  19 
Number of requests left to do  : 1 
Request n° 1 
Article n°408
Total number of references : 8 
All references are already retrieved.
Article n°409
Total number of references : 81 
Number of references left to retrieve :  41 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°410
Total number of references : 16 
All references are already retrieved.
Article n°411
Total number of references : 10 
All references are already retrieved.
Article n°412
Total number of references : 26 
All references are already retrieved.
Article n°413
Total number of references : 20 
All references are already retrieved.
Article n°414
Total number of references : 37 
All references are already retrieved.
Article n°415
Total number of references : 60 
Number of references left to retrieve :  20 
Number of requests left to do  : 1 
Request n° 1 
Article n°416
Total number of references : 83 
Number of references left to retrieve :  43 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°417
Total number of references : 44 
Number of references left to retrieve :  4 
Number of requests left to do  : 1 
Request n° 1 
Article n°418
Total number of references : 11 
All references are already retrieved.
Article n°419
Total number of references : 3 
All references are already retrieved.
Article n°420
Total number of references : 32 
All references are already retrieved.
Article n°421
Total number of references : 42 
Number of references left to retrieve :  2 
Number of requests left to do  : 1 
Request n° 1 
Article n°422
Total number of references : 28 
All references are already retrieved.
Article n°423
Total number of references : 8 
All references are already retrieved.
Article n°424
Total number of references : 17 
All references are already retrieved.
Article n°425
Total number of references : 84 
Number of references left to retrieve :  44 
Number of requests left to do  : 2 
Request n° 1 
Request n° 2 
Article n°426
Article n°427
Total number of references : 17 
All references are already retrieved.
Article n°428
Total number of references : 30 
All references are already retrieved.
Article n°429
Article n°430
Total number of references : 26 
All references are already retrieved.
Article n°431
Total number of references : 27 
All references are already retrieved.
Article n°432
Article n°433
Article n°434
Total number of references : 14 
All references are already retrieved.

Display code

nlp_references <- bind_rows(citation_list, .id = "citing_art")
nlp_subjects   <- bind_rows(subjects_list, .id = "citing_art")

nlp_subjects <- nlp_subjects %>%
  group_by(citing_art) %>%
  summarise(subjects_area = paste(`$`,collapse = " | ")) %>%
  rename("dc:identifier" = citing_art)


nlp_full <- left_join(nlp_full, nlp_subjects)
nlp_full <- left_join(nlp_full, number_ref) %>%
  mutate(nref = as.numeric(nref))

set_flextable_defaults(
  font.size    = 12, font.family = "Open Sans",
  font.color   = "#333333",
  table.layout = "fixed",
  border.color = "gray",
  padding.top  = 3, padding.bottom = 3,
  padding.left = 4, padding.right = 4)



#nlp_full <- read_csv2("nlp_full_data.csv")
#nlp_references <- read_csv2("nlp_references.csv")
print(Sys.time()-t1)

Time difference of 12.84557 mins

We now have collected all the data we could. A summary:

Display code

nb_articles <- n_distinct(nlp_full$entry_number)
nb_references_articles <- n_distinct(nlp_references$citing_art)
count_ref <- nrow(nlp_references)
count_ref_unique <- n_distinct(nlp_references$`scopus-eid`)
dat <- data.frame(
  Type = c("Articles", "Articles with refs", "Missing articles with refs", "References", "Unique Reference"),
  n = c(nb_articles, nb_references_articles, nb_articles-nb_references_articles, count_ref, count_ref_unique ), 
  Percentage = c(100, round(nb_references_articles/nb_articles*100,2), round(100-nb_references_articles/nb_articles*100,2), "/", "/")
)

flextable(dat)

Type	n	Percentage
Articles	434	100
Articles with refs	392	90.32
Missing articles with refs	42	9.68
References	24,576	/
Unique Reference	18,928	/

Display code

readr::write_excel_csv2(nlp_full,"nlp_full_data.csv")
readr::write_excel_csv2(nlp_references,"nlp_references.csv")
write.csv(nlp_full,"nlp_full_data_utf8.csv", fileEncoding = "UTF-8") #sometimes we have problem with the encoding
write.csv(nlp_references,"nlp_references_utf8.csv", fileEncoding = "UTF-8")

3.3 Checking the number of references retrieved

Let’s check if we have retrieved all the references we could get:

Display code

verifreferences <- nlp_references %>%
  group_by(citing_art) %>%
  rename("dc:identifier" = citing_art) %>%
  summarise(number_refs_retrieved = n())

nlp_full <- left_join(nlp_full,verifreferences)

screnforelsevier <- nlp_full %>%
  distinct(`dc:identifier`, .keep_all = TRUE) %>%
  select(`dc:identifier`, nref, number_refs_retrieved, `dc:title`, `dc:creator`, citedby_count)

  
reactable(screnforelsevier, searchable = TRUE, minRows = 5, outlined = TRUE, highlight = TRUE, paginationType = "jump")